[{"content":"It’s been just over two years since the publication of The Essence of Software (EOS). In that time, engaging with readers, consulting and teaching to students and practitioners has given me a new perspective on concept design.\nIn a series of notes (of which this is the first), I hope to share some of the respects in which my understanding has evolved. In addition to having a better sense of what matters most, I think I have a greater appreciation of the obstacles that make concept design seem (at the same time!) trivial to some and obscure to others. I also hope to point to some exciting new directions for concept design\u0026mdash;in particular how concepts make it easier to exploit LLMs for generating code.\n","permalink":"https://essenceofsoftware.com/drafts/concepts-redux/new-take/","summary":"\u003cp\u003eIt’s been just over two years since the publication of \u003cem\u003eThe Essence of Software\u003c/em\u003e (EOS). In that time, engaging with readers, consulting and teaching to students and practitioners has given me a new perspective on concept design.\u003c/p\u003e\n\u003cp\u003eIn a series of notes (of which this is the first), I hope to share some of the respects in which my understanding has evolved. In addition to having a better sense of what matters most, I think I have a greater appreciation of the obstacles that make concept design seem (at the same time!) trivial to some and obscure to others. I also hope to point to some exciting new directions for concept design\u0026mdash;in particular how concepts make it easier to exploit LLMs for generating code.\u003c/p\u003e","title":"A new take on concept design"},{"content":"As I explain in EOS, I’ve coopted the term “software design” to mean something different from what most people in the software world expect. To me, it means designing function: how the software will behave and interact with its users and the world.\nSo the design of the software is what determines whether or not it fulfills the user’s needs, because if the function is wrong (or confusing), the user won’t be happy. The user interface matters too, but that’s a separate issue (of how the design is presented).\nWhat I particularly do not mean by software design is how the code is structured. That’s important, but more for the programmer than the user. It will turn out that concept design influences code design, but that’s not my focus here.\nWhy is it important to reclaim the term “software design”? Why not just call this “product design” or “UX design”? The problem with those terms is that they reinforce the way in which software design is typically siloed into different roles in a company, even though it usually involves people across all of them\u0026mdash;and in other roles too (such as software engineering and architecture). I want to advocate for the centrality of software design in shaping the experience of users, just as Mitchell Kapor did in his manifesto 25 years ago.\nThe term \u0026ldquo;product design\u0026rdquo; also encourages the misconception that software is no different from any other kind of product, and that it doesn\u0026rsquo;t have its own design criteria, challenges and principles. My work is in part aimed at dispelling this misconception by showing how rich the field of software design is, and how much has been missed by thinking of software in generic product terms, and drawing ideas from other fields (industrial design, visual design, behavioral economics, psychology, etc) that have limited leverage.\n","permalink":"https://essenceofsoftware.com/drafts/concepts-redux/what-is-design/","summary":"\u003cp\u003eAs I explain in EOS, I’ve coopted the term “software design” to mean something different from what most people in the software world expect. To me, it means designing \u003cem\u003efunction\u003c/em\u003e: how the software will behave and interact with its users and the world.\u003c/p\u003e\n\u003cp\u003eSo the design of the software is what determines whether or not it fulfills the user’s needs, because if the function is wrong (or confusing), the user won’t be happy. The user interface matters too, but that’s a separate issue (of how the design is presented).\u003c/p\u003e","title":"What is software design?"},{"content":"Concept design builds on many familiar ideas. To mention a few:\nFrom use cases and user stories, concept design takes the idea that software interacts with the world through scenarios that deliver value. From user-centered design (as taught by Don Norman in The Design of Everyday Things), concept design takes the idea that the user interface is a kind of apparition (a “system image” in Norman’s phrasing) that stands between the user and the system, revealing only indirectly the underlying reality. When the user’s understanding (their “mental model”) is not aligned with this reality (the “conceptual model”), the system becomes unusable. From domain-driven design (and its predecessors, such as OMT and JSD), concept design takes the idea that the most stable part of a system’s function may be grounded in structures that already exist in the problem domain. In each of these cases, concept design goes in new directions. Unlike use cases, the scenarios of concept design are not used as specifications. Whereas user-centered design focuses on building the system image that connects the conceptual model to the mental model, concept design focuses on the conceptual model itself. And while domain-driven design looks for existing structures in data, concept design looks for them in function.\nMore fundamentally, concept design is about making the function of software modular: that is, dividing it into separable chunks that can be understood independently. Modularity is the key to many good things: separation of concerns, division of labor, reuse, decoupled implementation, etc.\nTo achieve modularity, concept design draws on well-established ideas from computer science (in particular the idea of defining actions over states, and composing processes by synchronizing actions).\nWhat about objects and classes? Given objects’ success in programming, and their use in simulations for modeling domains, you might have expected them to be the right modularity mechanism for concepts too. But as we’ll see, objects aren’t a good fit for structuring function.\n","permalink":"https://essenceofsoftware.com/drafts/concepts-redux/building-on-ideas/","summary":"\u003cp\u003eConcept design builds on many familiar ideas. To mention a few:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFrom \u003cstrong\u003euse cases and user stories\u003c/strong\u003e, concept design takes the idea that software interacts with the world through scenarios that deliver value.\u003c/li\u003e\n\u003cli\u003eFrom \u003cstrong\u003euser-centered design\u003c/strong\u003e (as taught by Don Norman in \u003cem\u003eThe Design of Everyday Things\u003c/em\u003e), concept design takes the idea that the user interface is a kind of apparition (a “system image” in Norman’s phrasing) that stands between the user and the system, revealing only indirectly the underlying reality. When the user’s understanding (their “mental model”) is not aligned with this reality (the “conceptual model”), the system becomes unusable.\u003c/li\u003e\n\u003cli\u003eFrom \u003cstrong\u003edomain-driven design\u003c/strong\u003e (and its predecessors, such as OMT and JSD), concept design takes the idea that the most stable part of a system’s function may be grounded in structures that already exist in the problem domain.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn each of these cases, concept design goes in new directions. Unlike use cases, the scenarios of concept design are not used as specifications. Whereas user-centered design focuses on building the system image that connects the conceptual model to the mental model, concept design focuses on the conceptual model itself. And while domain-driven design looks for existing structures in data, concept design looks for them in function.\u003c/p\u003e","title":"Building on good ideas"},{"content":"For a video version of this post, see my recent TEDx talk.\nNow translated into Japanese by Takeo Imai!\nDiagnosing software success If you want to make software that’s successful, it helps to start by looking at examples of success. In this post, I look at one well-known example and try and identify what made it succeed\u0026mdash;its secret sauce\u0026mdash;and then show how the same strategy can be applied to similar successes.\nWhen you try to figure what made some app super successful, and you ask users why they like it, they’ll often say “because it just works.” Uncovering what this is about\u0026mdash;what “just working” really means\u0026mdash;is not easy, but it’s possible to identify the key elements of “just working” and I’ll do that later.\nHow Zoom killed Skype In 2019, Skype had four billion registered users, and about a third of the video-call market. Then the pandemic hit. Zoom, a product few had even heard of, took off and “zooming” became a household word. By 2021, Zoom had half the market and almost a billion dollars in revenue, and Skype had receded out of view.\nZoom is one of those apps that people say “just works.” To explore this, let’s think about how Zoom works, from a user’s perspective. You create a meeting, which generates a link you share with your friends. Then, you start the meeting and your friends can join in their own time, using the link you gave them. When you’re done, you end the meeting. We can depict this scenario in a diagram showing the actions taken, with forking arrows in the joining step to suggest that multiple people can join in whatever order:\nNow compare this to using Skype. To set up a group video call in Skype, you add each of your friends to the group, then you call, some of them answer, and then you hangup:\nAt first blush, this doesn’t look any worse. It’s not more complicated in terms of the number of steps or the structure of the scenario. The problem is that some of the steps, which I’ve outlined with red boxes, are painful. The add step means going through your contact list and finding them one at a time. If one of your friends isn’t in your contact list, you’ll have to get their contact info and include it. Worse, if one of your friends isn’t a Skype user, you won’t be able to invite them at all. The answer step isn’t so easy either. Your friend has to open the app and either be there at the moment that your call comes in, or has to find a notification that they missed.\nNow, maybe you’re thinking this makes a mountain out of a molehill. If you just want to talk to two friends and they’re already on Skype, it will work just fine. And you’d be right. But what happened in the pandemic was that things changed, and suddenly people wanted to talk to all kinds of different groups of people\u0026mdash;friends, colleagues, neighbors\u0026mdash;and in larger numbers than before. So the small annoyances of Skype became huge barriers.\nIn that context, Zoom’s solution was what my colleague Merrick Furst calls a “not not.” It was not OK to not have the Zoom scenario, which allowed you to organize a meeting merely by sharing a link, and to join a meeting just by clicking on it, whether or not you had a Zoom account.\nNot the whole story That’s not the whole story, of course. The switch from Skype’s video call scenario to Zoom’s meeting scenario wasn’t so sudden or clear cut.\nThe Zoom idea (having a meeting link created asynchronously in advance) might have originated with LogMeIn in 2010, a company that built a video conferencing solution that was aiming to be much more lightweight than existing offerings from companies like Citrix. Zoom used this idea from its founding in 2012, but by 2020, it wasn’t the only company to have recognized its value. Skype added meeting links in 2015, and Google Hangouts had them a year later.\nBut in both those cases, the idea was integrated half-heartedly. It wasn’t the default way to start a meeting. In Skype you couldn’t link to a video call directly (but only to a chat, from which a video call could be launched). When Skype realized this (belatedly, in April 2020), they added a new feature called Meet Now that allowed direct links to video calls. And in Hangouts, although the host could admit users without accounts, they had to be approved (as in Zoom’s “waiting room”), a nuisance for a large (and not private) meeting.\nThere are other complications too. Zoom’s success undoubtedly benefited from the ease of installing the app, which relied in part on disreputable techniques (learned these from malware developers) for working around operating system protections that would have required additional approval steps from the user. Zoom also lied about end-to-end encryption.\nNone of these complications, however, obscures the fact that Zoom offered a scenario\u0026mdash;a pattern of interaction\u0026mdash;that was a better match to what users needed than its competitors.\nWhat “just works” means With all this in mind, we can now address the question: what did people mean when they said that they switched to Zoom because it “just works”?\nA compelling scenario, so that it’s obvious what steps you have to take, and in what order. Zoom makes this easy, because there are so few steps, and at any time, you have very few options. Having created meeting, you can’t do much except start it; and having started or joined it, there’s nothing to do except leave. (Of course, there are other actions you can take, like muting your microphone, but it’s pretty obvious that those are orthogonal to the main meeting scenario). No pain points when you execute the scenario, like having to find contacts to add to a group call in Skype, or having to open an account to respond to a call invitation. A simple purpose, against which an app can be readily judged. For Zoom, that purpose was opening a video call with multiple participants. Not all apps have such a unitary purpose; Photoshop, for example, fulfills so many different and intricate purposes that I doubt anyone expects it to “just work.” Just So stories Merrick Furst, whom I mentioned earlier for his idea of “not nots” is skeptical of what he calls “just so stories” that present supposedly compelling explanations of why some innovation succeeded or failed. It’s a great term, harking back to Rudyard Kipling’s children’s stories that according to the biologist Lewis Held “offered fabulous tales about how the leopard got its spots, how the elephant got its trunk, and so forth” and that, despite being wonderfully entertaining, were “poor substitutes for real understanding.”\nIt’s possible that my theory of Zoom’s rise is a “just so story.” Perhaps that Zoom simply reached a tipping point and became dominant due to the network effect, for example.\nBut here’s the nub of it. If my goal were to give you a way to determine reliably whether a particular company’s innovation would fail or succeed in the marketplace, this might be fatal.\nMy goal instead is to find ways to improve the design of software, and from that perspective the Zoom vs Skype story seems to me on solid ground. Whether or not Zoom succeeded because of its meeting scenario is not so crucial. What matters is that it succeeded with this scenario. This design, in which a link is sent asynchronously, can be recognized\u0026mdash;in the context of the emergence of larger, ad hoc video calls\u0026mdash;as far superior to the design in which participants are invited directly as existing users of a platform.\nMore examples Two more examples of this idea. Apple’s iPod was sleeker than existing MP3 players when it came out in 2001, but it didn’t explode until a few years later. One crucial innovation might explain this: the opening of the iTunes store (in 2003). The scenario upgrade was from having to rip CDs (or worse find pirated songs online) to a being able to buy single songs, sync them to your device and play them immediately:\nWalter Isaacson credits Steve Jobs with the crucial insight here, and marvels at how nobody else saw this opportunity (especially Sony, which not only lead in consumer electronics but also had its own music studios).\nWhatsApp was founded in 2009, but its meteoric rise began in 2011. End-to-end encryption was still five years away, and free texting had been around since 2006. So what happened in 2011? In short, group chat, so that rather than having to add recipients one at a time to each thread, you could invite your friends to join a group, and then all your messages would be shared with whoever had joined:\nOther companies were trying to capitalize on group chat at the same time, but WhatsApp was the clear winner, perhaps because it already had a strong user base.\nLessons In some sense, the key point here is unsurprising: people adopt new tools that change their pattern of behavior, by making it simpler and by eliminating pain points.\nBut despite its simplicity, this idea has some major ramifications for software:\nScenarios explain software. Software apps, like other innovations, can be explained by usage scenarios. These scenarios aren’t the mundane use cases or user stories that we’re familiar with; they’re more like Michael Polanyi’s operational principles that capture the very essence of a machine’s design. Innovation = scenario change. Bruno Latour explains the emergence of technology in a similar way. Without doors, you’d have to break a hole in a wall every time you leave a building, and patch it on your return: hardly convenient. When a software scenario looks simple or trivial, appreciating it may require comparison to what came before. Simplicity matters. Don Norman taught us that even knowing how to operate a door can be hard. For software, many pitfalls lie between the basic scenario and its realization in the context of a user interface. If the scenario isn’t conveyed clearly and easy to execute, it might as well not exist. The lesson for any start-up, in short, is: know your scenario. And make sure you don’t get distracted and that you focus on delivering it as directly and compellingly as you can.\nBut what about more complex apps that seem to embody multiple scenarios? That’ll be the topic of my next post.\n","permalink":"https://essenceofsoftware.com/tutorials/gentle-intro/how-software-succeeds/","summary":"\u003cp\u003e\u003cem\u003eFor a video version of this post, see my recent \u003ca href=\"https://www.youtube.com/watch?v=LtvkRJuMa6A\"\u003eTEDx talk\u003c/a\u003e.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eNow \u003ca href=\"https://bonotake.hatenablog.com/entry/2024/04/05/150812\"\u003etranslated into Japanese\u003c/a\u003e by Takeo Imai!\u003c/em\u003e\u003c/p\u003e\n\u003ch1 id=\"diagnosing-software-success\"\u003eDiagnosing software success\u003c/h1\u003e\n\u003cp\u003eIf you want to make software that’s successful, it helps to start by looking at examples of success. In this post, I look at one well-known example and try and identify what made it succeed\u0026mdash;its secret sauce\u0026mdash;and then show how the same strategy can be applied to similar successes.\u003c/p\u003e\n\u003cp\u003eWhen you try to figure what made some app super successful, and you ask users why they like it, they’ll often say “because it just works.” Uncovering what this is about\u0026mdash;what “just working” really means\u0026mdash;is not easy, but it’s possible to identify the key elements of “just working” and I’ll do that later.\u003c/p\u003e","title":"How software succeeds"},{"content":"Defining design What\u0026rsquo;s a good working definition of the word design and how it differs from engineering? For those of us who care about design, this is an important question, because it helps explain what we do, and why it matters.\nHerb Simon said “To design is to devise courses of action aimed at changing existing situations into preferred ones.” This definition nicely highlights the design aspect of Simon’s own focus on decision making in organizations. But it’s too broad to help us understand what “software design” might be.\nKapor’s definition of design For that, I turn to Mitchell Kapor who defined design like this:\nIt’s where you stand with a foot in two worlds—the world of technology and the world of people and human purposes—and you try to bring the two together.\nKapor gives building design as an example. It’s the architect who designs the building and shapes the experience of its occupants. The engineer makes sure the building won’t corrode or sway in the wind, concerns that are critical but don’t require an understanding of people.\nIn just the same way, software development involves two very different kinds of activity: designing the functionality that shapes the user’s experience and engineering the system to support it.\nDesign is not just visual Just because software design involves people doesn’t mean that it’s only about the user interface (or more narrowly its visual appearance), or that it’s mostly about psychology and how people feel.\nThe focus of software design is devising and structuring behavior. Doing this well requires deep technical understanding. Just as the architect of a building has to understand not only how people experience spaces, but also what kinds of spaces can be built, so a software designer has to understand how people experience functionality and what kinds of functionality software can support.\nThe design/engineering spectrum You can sometimes do without an engineer (creating a poster or a teacup, for example) and sometimes without a designer (building a memory allocator or an integrated circuit).\nBut for most artifacts, including software, design and engineering are needed in equal measure.\nAnd sometimes one or other is needed when you might not expect it. The “screwpull” wine opener is not just a design product; it relies on the engineering of a Teflon coating that is slippery enough for the spiral not to get stuck in the cork.\nYou might think home routers and wireless access points would use designers only for making pretty boxes. But setting up a wireless network turned out to be more complicated and error-prone than most engineers imagined, and devices like Amazon’s Eero and Google’s Nest are distinguished from their competitors not by their bandwidth but by their ease of setup.\nWhether design or engineering matters more may depend on whether a technology already exists, so it changes over time. Marconi’s first radio was an engineering achievement, but Dieter Rams’s radio was a design product.\nWhere design differs Because design, unlike engineering, focuses on the meeting point between people and products, some big differences in practice emerge:\nMisfits. In engineering, the context of usage is more clearly understood. In design, the context invariably brings surprises, and failures are usually not because specifications aren’t met but because the specifications themselves were wrong. See Misfits. Role of specs. In engineering, the problem statement is a specification. In design, the problem is harder to characterize and finding specs of desirable behavior is the output, not the input of the design process. Qualitative vs. quantitative. In engineering, criteria can often be expressed numerically: a storage device has capacity, latency, bandwidth etc. In design, the criteria are usually more nebulous, if known at all. Nevertheless, design and engineering activities have much in common. Perhaps most importantly (and often neglected) is the role of patterns and prior knowledge. New creations never come out of thin air, but are always adaptations and extensions of earlier ones.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/design-vs-engineering/","summary":"\u003ch2 id=\"defining-design\"\u003eDefining design\u003c/h2\u003e\n\u003cp\u003eWhat\u0026rsquo;s a good working definition of the word \u003cem\u003edesign\u003c/em\u003e and how it differs from \u003cem\u003eengineering\u003c/em\u003e? For those of us who care about design, this is an important question, because it helps explain what we do, and why it matters.\u003c/p\u003e\n\u003cp\u003eHerb Simon said “To design is to devise courses of action aimed at changing existing situations into preferred ones.” This definition nicely highlights the design aspect of Simon’s own focus on decision making in organizations. But it’s too broad to help us understand what “software design” might be.\u003c/p\u003e","title":"Design vs. engineering"},{"content":"Think about your favorite product, whether it’s a photo-editing app or a toaster, and ask yourself: how did it get to be so good?\nWith all the hype in the design world about creativity and brainstorming, it’s easy to imagine that great designs simply appear, out of the blue. Someone conjures up a new idea, and it just needs to be translated from sketch to product. Maybe many details need to be added, but the quality of the design was there at the start—like a tiny seed that grows spontaneously into a magnificent tree.\nA much better model of design is described by George Saunders in his book A Swim in the Pond in the Rain. Saunders is talking about writing, but his ideas apply equally well to designing.\nSaunders starts by writing down a bunch of sentences. Then he prints out the draft, and painstakingly marks it up, imagining, as he reads along, a meter stuck on his forehead with a needle that swings from N (negative) to P (positive) as he reacts to it intuitively. Then he fixes up the draft with his corrections, and starts the whole cycle over again.\nIn this way, the tone and quality of the writing—and the voice of the author—emerge gradually over time. By the end of the process, the writer has something in front of her that she could never have imagined at the start. Even the writer’s persona, as reflected in the piece, grows and adapts over time, until it reaches a state that may even be an improvement on the reality!\nMost radically, this process makes the starting point less critical. As Saunders explains:\nThe beauty of this method is that it doesn’t really matter what you start with or how the initial idea gets generated. What makes you you, as a writer, is what you do to any old text, by way of this iterative method. This method overturn the tyranny of the first draft. Who cares if the first draft is good? It doesn’t need to be good, it just needs to be, so you can revise it. You don’t need an idea to start a story. You just need a sentence.\nApplying this model to design has profound and encouraging implications:\nWhen creativity happens. The creative work of design isn’t just at the start during some kind of “ideation” phase. On the contrary, most of it resides in the iterative adjustment of the design, in which the design is critiqued and adjusted in response to a stream of new ideas. Where quality resides. What makes a design great is almost never a single novelty arising from a momentary insight, but rather an elegant alignment and uniformity that results from an accumulation of many small steps. Perhaps this is Christopher Alexander’s “quality without a name.” The journey. You might worry that a process of design revision could leave you stuck in a local minimum in the design space, like someone digging a hole for treasure in a field who just digs deeper and deeper when they can’t find it. But if you take the revision process seriously, and view it as a creative journey, then every step offers an opportunity for a small change of direction, and you are likely to end up far from where you started. What about the practical consequences?\nEmbrace iteration. If you want great design, you’ll need to embrace iteration. Sometimes this will mean postponing implementation as much as possible, so that you have the time to evolve your design before committing to code. Or if you must implement early, consider implementing a simpler design with known flaws rather than a complicated design whose features may turn out to be unnecessary in practice. You might even place a doorbell in the jungle. Don’t obsess about ideation. Don’t invest too much in ideation activities. Save most of your effort (and your best designers) for revising your design. And don’t reject initial ideas because they’re not novel enough. On the contrary, it’s often best to start with a familiar concept that has been tried and tested, and whose limitations are well understood, and then morph it if it fails to solve the problem. Be obsessive. Every small flaw you detect when reviewing a design, or that gets exposed from the experience of users, is not only an opportunity for a small improvement. If you think about it deeply, it can be a window into a new way of viewing your design, and a chance to take a step in a new direction. ","permalink":"https://essenceofsoftware.com/tutorials/design-general/great-design/","summary":"\u003cp\u003eThink about your favorite product, whether it’s a photo-editing app or a toaster, and ask yourself: how did it get to be so good?\u003c/p\u003e\n\u003cp\u003eWith all the hype in the design world about creativity and brainstorming, it’s easy to imagine that great designs simply appear, out of the blue. Someone conjures up a new idea, and it just needs to be translated from sketch to product. Maybe many details need to be added, but the quality of the design was there at the start—like a tiny seed that grows spontaneously into a magnificent tree.\u003c/p\u003e","title":"How great design happens"},{"content":"What is a design problem? What exactly is a design problem? And why are design problems often hard to solve?\nThe most compelling answer to this question, in my view, comes from Christopher Alexander in his influential book Notes on the Synthesis of Form (1964).\nMost of the book is about a way to structure goals into a tree, by noting alignments and conflicts between them, and grouping them to minimize conflicts across groups. This idea is very similar to Herb Simon’s notion of nearly decomposable systems, which was described in his “Architecture of Complexity” paper (1962) two years before.\nForm, context, ensemble and fit For me, however, the most interesting and valuable part of the book is the framework laid out at the very beginning on the essence of the design problem.\nDesign, Alexander explains, is the creation of a form within a context. Together these make an ensemble. The designer’s task is to ensure that the form fits the context within the bounds of the ensemble as a whole. A cork, for example, is a form that has to fit in the context of a wine bottle; together the cork and bottle make an ensemble.\nA rich framing This framing is richer and more subtle than it first appears to be:\nChoice of ensemble. Factors beyond the ensemble are not considered, so the choice of the ensemble places implicit bounds on the fitness criterion. If the ensemble for the cork design is expanded to include the wine cellar, the designer might need to consider the propensity of the cork to rot or dry out in different conditions. If expanded even further to include the trees from which the material of the cork is harvested, ecological considerations may arise, and suggest a different material. Choice of boundary between context and form. Even if the ensemble is fixed, the boundary between form and context is not. Suppose the ensemble for the cork design comprises the wine and its containers, the producers and the consumers, and fitness includes not only maintaining the freshness of the wine but also cost. In this case, a designer might propose switching from glass bottles to aluminum cans, identifying the context not as the glass bottle into which the cork is inserted but as the conditions of production, transportation and usage of the container. Simple fitness criteria. In simple cases, fitness can be reduced to simple criteria. Alexander gives an example of a metal face fitting against the surface of a standard steel block, which is (for all practical purposes) perfectly smooth. In this case, fitness amounts simply to whether or not the surface of the face is level enough; it can be tested by inking it and noting marks on the high spots; and it can be judged without testing in context by specifying the permitted variance and measuring it. This kind of situation is more common in engineering than in the design of artifacts for human use. Unknowable fitness criteria. In more complicated cases, the criteria for fitness are so numerous that they cannot even be articulated. Worse, some criteria cannot even be predicted. The ring/silent switch on the iPhone X turned out for many users (including me) to be a hair puller. For some reason, the gap between the switch and the bezel traps hairs which then get yanked out when the phone is moved away from your head. Needless to say, users were not delighted by this feature and Apple reportedly was willing to replace their phones. If one were writing a list of requirements for a ring/silent switch, it seems unlikely that “don’t pull hair out” would be included. Misfits The implications for software design, and design more generally, are far reaching:\nCompleteness in requirements. The idea that requirements should be “complete” is rendered absurd, since the list of fitness criteria is infinite and unknowable. Misfits. The best we can do, Alexander asserts, is identify the most likely misfits: ways in which form and context and likely to not fit together. In designing a file system, for example, we might identify files getting accidentally deleted as a critical misfit: this is what motivates the Trash concept (and indicts the design of the Unix command line*). Being concrete, negative scenarios, misfits are explicit and tangible, so they’re easy to explain and understand. Experiment. The only completely reliable way to identify misfits is to try the form in the context and see what happens. In practice, however, misfits may take time to emerge, and may have already caused unacceptable damage before they are detected. Prototyping lets you test your design with lower risk and lower cost, but the context of the prototype may not match the real context well enough to catch even serious misfits (especially those with rare but disastrous consequences). Experience. Most misfits are identified not through experiments but from experience. If you’re designing an app for selling event tickets, for example, you would mitigate the risk of scalping; if you’re designing a backup utility, you’d want to defend against ransomware attacks. Neither of these misfits is predictable in the abstract, but failing to consider them, given all the experience we have building such systems, would be unforgivable. Patterns. Where does such knowledge come from if the designer lacks the right experience? This is what patterns offer: a way to record our accumulated experience about particular design problems and their solutions. Patterns not only codify designs that have worked well in practice. They can also highlight common misfits and their mitigations. In a catalog of concept patterns, the entry for Upvote would mention the problem of duplicate votes, and ways to prevent them (such as using cookies or IP addresses). * Just consider what happens if instead of typing rm *.o you fail to hold down the shift key and type rm *\u0026gt;o instead.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/misfits/","summary":"\u003ch2 id=\"what-is-a-design-problem\"\u003eWhat is a design problem?\u003c/h2\u003e\n\u003cp\u003eWhat exactly is a design problem? And why are design problems often hard to solve?\u003c/p\u003e\n\u003cp\u003eThe most compelling answer to this question, in my view, comes from Christopher Alexander in his influential book \u003cem\u003eNotes on the Synthesis of Form\u003c/em\u003e (1964).\u003c/p\u003e\n\u003cp\u003eMost of the book is about a way to structure goals into a tree, by noting alignments and conflicts between them, and grouping them to minimize conflicts across groups. This idea is very similar to Herb Simon’s notion of nearly decomposable systems, which was described in his “Architecture of Complexity” paper (1962) two years before.\u003c/p\u003e","title":"Form, context \u0026 misfits"},{"content":"Seven criteria What makes a design fit for purpose depends—surprise!—on the purpose. But generic criteria are helpful for two reasons. First, they give you some basis for evaluation, however crude. Second, they can be used as a checklist to suggest more specific ones.\nA software product should be:\nEfficient: letting users accomplish their goals with no more steps or effort than needed; Flexible: supporting not just one fixed and narrow goal, but a range of goals that users are likely to have; Responsive: responding to requests from the user in a timely way; Understandable: behaving predictably, offering options and producing outcomes that make sense to users; Robust: tolerant of errors made by the user and other common perturbations of context, such as failures of other systems or agents to perform their expected roles; Secure: resistant to malicious attacks, including social engineering attacks that induce the user to perform unintended actions. Just: allocating benefits amongst different groups of users and stakeholders in a fair way. Non-criteria Some criteria I have intentionally omitted from this list:\nSimple: Simplicity is about the design itself, rather than the user’s experience of it, and it impacts all of the criteria (most notably understandability). Learnable: How easy it is to learn to use a product is part of how understandable it is. A well designed product not only conveys its states and actions for the user’s immediate understanding, but also teaches the user, helping build a mental model over time. The “product,” of course, must be broadly construed to include help features and training materials. Consistent: Some qualities are not ends in themselves, but means to achieving fitness, so are heuristics rather than criteria. Maintainable: Many qualities matter for developers but not for users (or at least, only indirectly); these are engineering rather than design criteria. Minimalist: Some writers advocate minimalism as a universal requirement of good design. It’s an attractive and compelling position, but it reflects a preference that is cultural and personal rather than universal. Many designers in the past (such as John Ruskin and Louis Sullivan) have argued that ornament and decoration are not surface treatments that detract from a design but are integral to it. Perhaps the pendulum will swing back again, and our affection for grey interfaces with text in Helvetica Light will wane. The design space If we imagine each criterion has having a value between zero and one, then each design sits at a point in a 7-dimensional space, and we can compare designs with respect to their positions.\nComparisons are most instructive for designs that address the same problem: two email clients, for example, or two word processors.\nComparisons between designs that address different problems often reflect more on the problems than on the designs. Adobe Photoshop, for example, is less understandable than Adobe Lightroom, in that the cognitive burden of becoming familiar with Photoshop is greater. This is as expected, since Lightroom (roughly speaking) only lets you adjust the values of the pixels in a single photo, whereas Photoshop lets you combine and rearrange pixels from different photos.\nTradeoffs The designs that solve a given problem are not usually distributed uniformly over the design space. Instead, they sit on a plane so that increasing the value of one criterion means decreasing another.\nA design that is more flexible and efficient will usually be less understandable and less robust. Take, for example, the two file managers that are available in MacOS: the Macintosh Finder (comprising the windows and icons, and manipulated mainly by clicking and dragging), and the Unix command line (accessible through the Terminal application, which takes textual commands).\nThe Unix design is more flexible (you can put a file in two folders, for example) and also more efficient (you can more easily move many files at once, for example). But it is at the same time less understandable. Even the most basic command, mv, performs two distinct actions (moving and renaming files and folders) and can behave differently when a trailing slash is added, contrary to the assumptions of many users. It is also less robust: if you inadvertently mv one file onto another (by typing mv a b, intending to rename a to b, unaware that b already exists), the second will be irretrievably lost.\nSometimes, a higher value in one criterion can mitigate a lower value in another. For example, greater understandability can make up for some loss of robustness. In the Backblaze backup utility, the message “Files backed up as of 1:00pm” gives the user the impression that file contents generated prior to 1:00pm will be safe. In fact, the backup is not truly continuous, but instead periodically creates a list of modified files and then uploads them. Files changed after the creation of the list are not uploaded. This evident lack of robustness would be mitigated by a clearer message that helped users understand the “blind spot” in the backup process.\nNotes The generalization of error tolerance to robustness is inspired by Eunsuk Kang’s recent work. Justness is a big topic, which I address in my teaching through Batya Friedman’s Value Sensitive Design. These criteria expand on a list initially drawn up by Rob Miller for the MIT user interface design class 6.831. ","permalink":"https://essenceofsoftware.com/tutorials/design-general/fitness-criteria/","summary":"\u003ch2 id=\"seven-criteria\"\u003eSeven criteria\u003c/h2\u003e\n\u003cp\u003eWhat makes a design fit for purpose depends—surprise!—on the purpose. But generic criteria are helpful for two reasons. First, they give you some basis for evaluation, however crude. Second, they can be used as a checklist to suggest more specific ones.\u003c/p\u003e\n\u003cp\u003eA software product should be:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eEfficient\u003c/strong\u003e: letting users accomplish their goals with no more steps or effort than needed;\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFlexible\u003c/strong\u003e: supporting not just one fixed and narrow goal, but a range of goals that users are likely to have;\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResponsive\u003c/strong\u003e: responding to requests from the user in a timely way;\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eUnderstandable\u003c/strong\u003e: behaving predictably, offering options and producing outcomes that make sense to users;\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRobust\u003c/strong\u003e: tolerant of errors made by the user and other common perturbations of context, such as failures of other systems or agents to perform their expected roles;\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSecure\u003c/strong\u003e: resistant to malicious attacks, including social engineering attacks that induce the user to perform unintended actions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eJust\u003c/strong\u003e: allocating benefits amongst different groups of users and stakeholders in a fair way.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"non-criteria\"\u003eNon-criteria\u003c/h2\u003e\n\u003cp\u003eSome criteria I have intentionally omitted from this list:\u003c/p\u003e","title":"Generic fitness criteria"},{"content":"Most of us* believe that simplicity is the key to design. I’m a sucker for the best quotes about it. Here are some of my favorites. First, Tony Hoare in his Turing Award lecture lamenting the complexity of Algol-68:\nI conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.\nand on PL-1:\nAt first I hoped that such a technically unsound project would collapse but I soon realized it was doomed to success. Almost anything in software can be implemented, sold, and even used given enough determination. There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way—and that is reliability. The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay.\nNot just reliability—usability too. Here’s Edsger Dijsktra in a similar vein:\nThe opportunity for simplification is very encouraging, because in all examples that come to mind the simple and elegant systems tend to be easier and faster to design and get right, more efficient in execution, and much more reliable than the more contrived contraptions that have to be debugged into some degree of acceptability.\nBut perhaps best of all, and most relevant to software design, is the principle known as Gall’s Law:\nA complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.\n(I learned about Gall’s Law from Gordon Brander’s pattern collection.)\nGall’s formulation is particularly good because it doesn’t deny that complexity is inevitable in all real systems. It just reminds us that having a complicated problem is no excuse for designing a complicated solution.\n*Well, not all. I once visited Charles Simonyi at Microsoft and asked him, rather cheekily, if the length of the Windows manuals on his shelf, which extended all the way from one side to another, might be a symptom of a design that wasn’t simple enough. His response? “Simplicity is crap.”\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/simplicity/","summary":"\u003cp\u003eMost of us* believe that simplicity is the key to design. I’m a sucker for the best quotes about it. Here are some of my favorites. First, Tony Hoare in his Turing Award lecture lamenting the complexity of Algol-68:\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eI conclude that there are two ways of constructing a software design: One way is to make it so simple that there are \u003cstrong\u003eobviously\u003c/strong\u003e no deficiencies and the other way is to make it so complicated that there are no \u003cstrong\u003eobvious\u003c/strong\u003e deficiencies.\u003c/em\u003e\u003c/p\u003e","title":"Simplicity"},{"content":"When you’re diagnosing a usability snag, or coming up with a new design idea, it’s helpful to know what level you’re working at.\nOver the years, designers and researchers have defined various levels, reflecting their view of design and what they emphasize.\nI find the following scheme most helpful:\nPhysical. If you want to think of levels as low and high, this one is at the bottom. It’s the level of design in which you’re concerned about the physical (and physiological) qualities of human users. So it might involve physical things (like picking colors), but it includes less tangible aspects too (such as how long an action can take before you need to show a progress bar). To do this kind of design, you need to know a bit about human anatomy and physiology. For example, Fitts’s Law tells you how long it takes to move a pointing device to its target; perceptual fusion tells you that a delay of more than 10ms will be perceptible; and the fact that 1 in 12 males is red-green colorblind says you’d better not rely on those colors for important distinctions. Linguistic. This level involves design around language, and how you can use icons and words to convey information to users. Unlike the physical level, this level is culturally dependent: a white circle with a red border means no entry to Europeans, but not to Americans. One of the most important design heuristics is to use linguistic cues consistently within your app and across apps. Conceptual. This level is about the semantics of an app: its behavior, given by the actions you can perform and the state the actions read and write, and the meaning and implications of those actions and state. Here’s a picture illustrating these levels. The illustration for the linguistic level is a British road sign affectionately known as “man having trouble opening umbrella” (illustrating the subjectivity of linguistic signals).\nLevels of impact? An orthogonal dimension You may wonder if these levels correspond to the amount of impact a design decision can have. There’s some truth in that: in particular, because concept design questions address the question of what an app does and what purposes it fulfills, they tends to be more impactful. But the scope of impact—on a scale that might include personal, group, society, planet—is not so neatly tied to the levels.\nAt the conceptual level, a design decision can have impact at any scale. We’ve come to understand, for example, that the Upvote concept used in social media apps has its immediate impact in the quality of the ranking, and how well it serves the interests of an individual user; a deeper impact in terms of the user’s psychology (for example, encouraging addiction to social media); a more widespread impact on a social group (for example teenagers competing for attention and approval); and even impact on an entire society (for example, with the way the concept enables the spread of disinformation).\nAnd wider impacts can occur even at the “lowest,” physical level. A design that requires a particular physical ability can disadvantage large numbers of users. Poor use of color, for example, might make an app unusable to the almost 8% of men who experience color-blindness.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/levels-of-design/","summary":"\u003cp\u003eWhen you’re diagnosing a usability snag, or coming up with a new design idea, it’s helpful to know what \u003cem\u003elevel\u003c/em\u003e you’re working at.\u003c/p\u003e\n\u003cp\u003eOver the years, designers and researchers have defined various levels, reflecting their view of design and what they emphasize.\u003c/p\u003e\n\u003cp\u003eI find the following scheme most helpful:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003ePhysical\u003c/strong\u003e. If you want to think of levels as low and high, this one is at the bottom. It’s the level of design in which you’re concerned about the physical (and physiological) qualities of human users. So it might involve physical things (like picking colors), but it includes less tangible aspects too (such as how long an action can take before you need to show a progress bar).  To do this kind of design, you need to know a bit about human anatomy and physiology. For example, Fitts’s Law tells you how long it takes to move a pointing device to its target; perceptual fusion tells you that a delay of more than 10ms will be perceptible; and the fact that 1 in 12 males is red-green colorblind says you’d better not rely on those colors for important distinctions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLinguistic\u003c/strong\u003e. This level involves design around language, and how you can use icons and words to convey information to users. Unlike the physical level, this level is culturally dependent: a \u003ca href=\"https://en.wikipedia.org/wiki/Prohibitory_traffic_sign\"\u003ewhite circle with a red border\u003c/a\u003e means no entry to Europeans, but not to Americans. One of the most important design heuristics is to use linguistic cues consistently within your app and across apps.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eConceptual\u003c/strong\u003e. This level is about the semantics of an app: its behavior, given by the actions you can perform and the state the actions read and write, and the meaning and implications of those actions and state.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eHere’s a picture illustrating these levels. The illustration for the linguistic level is a British road sign affectionately known as “man having trouble opening umbrella” (illustrating the subjectivity of linguistic signals).\u003c/p\u003e","title":"Levels of design"},{"content":"In the 1980s, there was a revolution in software. The Apple Macintosh came out (building on the invention of the WIMP interface at PARC), Don Norman wrote The Design of Everyday Things, and the first CHI conference was held.\nNow everyone knows how important user interface design is, and there’s no excuse for messing it up. You can find several collections of UI design heuristics (such as Nielsen’s) online, and many companies publish standards for how to use UI widgets consistently (here’s Google’s, for example).\nThe downside of all this success is that people sometimes forget that UI design is only part of software design. A good user interface is necessary for a good user experience but it’s not sufficient. Just as important—maybe more important—is the underlying functionality, defined by the concepts that the software provides.\nIn my book, I open with a story about Dropbox. Its UI design is state of the art, but people still get confused and end up deleting other people’s files. The reason, I explain, is that the underlying conceptual model is confusing.\nTo see how UX is more than the UI, consider three widgets that let the user do a thumbs-up on something:\nFrom a UI perspective, these look pretty similar. In each case, you click on some kind of “up” button to signal that you like something.\nBut behind this apparent similarity in the UI, there are actually three completely different concepts, which a user must grasp to make sense of these buttons.\nIn the first, on the left, the arrow-up action is part of an Upvote concept in StackOverflow: clicking it says you approve of the answer (or the question). The accumulation of clicks gives a crowd-sourced measure of the credibility of items, affecting the order in which they appear.\nIn the second, the thumbs-up action is part of the Reaction concept of Slack, and is just a kind of minimal message sent back to the author of a post. It has no effect on the order in which posts appear.\nIn the third, the thumbs-up action is part of the Recommendation concept in Netflix, and is used when you liked a movie that you saw. Your approval isn’t shared with other users, and it doesn’t affect the rank of this particular movie. What it does is indicate that you want to see more movies similar to this one.\nThe design considerations in these three examples are all very different, and each has subtleties and problems of its own. One example: a friend told me that she saw a movie that she thought was really sexist, so she didn’t want to give it a thumbs up, but she liked the genre so she wondered if she should anyway.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/beyond-ui/","summary":"\u003cp\u003eIn the 1980s, there was a revolution in software. The Apple Macintosh came out (building on the invention  of the WIMP interface at \u003ca href=\"https://en.wikipedia.org/wiki/Xerox_Alto\"\u003ePARC\u003c/a\u003e), Don Norman wrote \u003ca href=\"https://en.wikipedia.org/wiki/The_Design_of_Everyday_Things\"\u003e\u003cem\u003eThe Design of Everyday Things\u003c/em\u003e\u003c/a\u003e, and the first \u003ca href=\"https://sigchi.org/conferences/conference-history/\"\u003eCHI conference\u003c/a\u003e was held.\u003c/p\u003e\n\u003cp\u003eNow everyone knows how important user interface design is, and there’s no excuse for messing it up.  You can find several collections of UI design heuristics (such as \u003ca href=\"https://www.nngroup.com/articles/ten-usability-heuristics/\"\u003eNielsen’s\u003c/a\u003e) online, and many companies publish standards for how to use UI widgets consistently (here’s \u003ca href=\"https://material.io/design\"\u003eGoogle’s\u003c/a\u003e, for example).\u003c/p\u003e","title":"Beyond the user interface"},{"content":"Two modes of design thinking Design, whatever the domain, includes two different modes of thinking. In one, the designer generates ideas freely, often responding only loosely to any given need or problem. In the other, the designer takes some previously articulated design ideas, and attempts to improve them.\nThe first mode is expansive, and most successful when critical judgment is suspended; the second is reductive, and calls for focus and analysis. The first tends to complicate a design and make it less coherent as new ideas are introduced that have yet to find their place; the second brings order and clarity.\nAn experienced design oscillates between these modes at all times. Finding a flaw during a careful and focused analysis of a design, she might decide to take a step back and brainstorm some ideas for how to overcome it.\nTwo distinct phases of design In the early stages of a design, a standard approach is to separate these modes of thinking into distinct design phases. In the first, divergent phase, the goal is to come up with as many ideas as possible, postponing judgment and critical analysis; then, in the second, convergent phase, the goal is to coalesce a ragtag collection of ideas into a coherent design. One rationale for this is that the expansive thinking of the divergent phase is fragile and easily inhibited when criticisms are raised.\nSome misconceptions The rising popularity of design thinking, and enthusiasm for divergent design in particular, has led to some common misconceptions.\nDesign is primarily divergent Perhaps the most pervasive misconception is one that is rarely articulated explicitly, but is often implied. It is that the fundamental nature of design is divergent, and that designers spend most of their time “ideating,” playfully generating wild and whacky ideas. In practice, however, most design work is convergent, and the essential qualities of a great design often come from working through fine details. Just watch Jony Ive talk about the design of the Macintosh laptop, and the emphasis he places on getting exactly the right curvature on the corners.\nWhen creativity happens Related to this misconception is the assumption that creativity happens mostly during the divergent phase: some people even seem to equate creativity with ideation. But as Barbara Boden explains in a widely cited article, there are different kinds of creativity. She calls one of the kinds of creativity combinational creativity, which involves producing unfamiliar combinations of existing ideas and making associations between ideas that were previously only indirectly linked. This kind of creativity arises more often in convergent design, when existing ideas have been articulated and the designer is exploring how they might be coalesced.\nWhere simplicity comes from Because convergent design involves smoothing off rough edges and eliminating needless complications, it’s easy to imagine that it’s the only source of simplicity in design work. But even if it’s the primary source of simplicity, it’s not the only one.\nDivergent design may produce a collection of ideas that are all too complicated to be refillable into a simple solution, in a case of “you can’t get there from here.” And during convergent design itself, a complexity problem may be resolved only by stepping beyond the immediate constraints of the current design and engaging in some expansive design thinking to generate a new idea and get out of a rut. This is like what happens in a gradient-descent search when the algorithm gets stuck in a local minimum and one option for finding a better solution is to take a random jump to a different location.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/diverge-converge/","summary":"\u003ch2 id=\"two-modes-of-design-thinking\"\u003eTwo modes of design thinking\u003c/h2\u003e\n\u003cp\u003eDesign, whatever the domain, includes two different modes of thinking. In one, the designer generates ideas freely, often responding only loosely to any given need or problem. In the other, the designer takes some previously articulated design ideas, and attempts to improve them.\u003c/p\u003e\n\u003cp\u003eThe first mode is expansive, and most successful when critical judgment is suspended; the second is reductive, and calls for focus and analysis. The first tends to complicate a design and make it less coherent as new ideas are introduced that have yet to find their place; the second brings order and clarity.\u003c/p\u003e","title":"Divergent and convergent design"},{"content":"Tactics for divergence Let’s consider some tactics that you might find useful as a software designer to make divergent design more effective. To make these concrete, I’ll suppose that we’re developing an app to support architectural walking tours.\nInterviewing stakeholders Talking to potential users and other stakeholders is good not only for identifying needs but also for getting design ideas. You can ask what tools and techniques a user currently finds helpful, and what changes might make them more effective. Just resist the temptation to ask whether a particular design idea would be attractive; most interviews (even if just out of consideration for the interviewer) are likely to express unwarranted enthusiasm.\nExample: I asked a potential user who often walks around cities with her family what snags she experienced. She told me that family members would often get separated, and that it could be hard to figure out where everyone was and get back together again. This suggested several potential features to me (for example, showing avatars for all members of the group on a shared map) . Imagining a situation Imagine yourself in a situation that calls for the app that you are designing. Think systematically through what you might be doing, and simulate your application ideas in your mind’s eye.\nExample: When I’m taking a walking tour, I’m usually on vacation, and (especially if it’s a long one) I like to have coffee breaks along the way. So I’d like to be able to plan a route around cafes with great coffee. Gap analysis Consider existing apps and the gaps between the features they offer and needs you have identified.\nExample: I’ve looked at several apps for architectural tours, and (surprisingly) none of them integrate a live map, so it’s hard to match the descriptions of buildings to your location. Viability For an app to be successful, fulfilling some users needs may not be enough. Without a financial model, the app may not be sustainable. And many apps require a critical mass of users, or at least benefit from a network effect (in which the value of the app to open user increases with the number of overall users). So think about how an app may be monetized and costs covered (or how costs may be reduced) and how the user base might be expanded.\nExample: How will architectural descriptions be obtained? Perhaps individual owners (whether homeowners or institutions) could provide data about their own buildings. Users could be offered subscriptions to cover content costs. To avoid such costs, information about many buildings could be provided by Wikipedia. Discounts for tourers at local cafes, bookshops and other businesses relevant to walking tours might be offered through the app. To incentivize users to introduce others to the app, users might be able to share favorite buildings or entire routes with friends; they might even be able to generate postcards that combines interesting facts with photos. Analogies to other apps Think about apps that serve similar purposes, or that address some of the same subproblems.\nExample: When considering subscription features, I started wondering about the models offered by blogging platforms such as Medium and Substack, and explicit support models as in Patreon. Individual content providers might be rewarded by a profit share. Feature lists Scan some lists of features of existing apps. Some features might fit your app directly; others might be useful if adapted; and just considering a large number of features might spur new ideas. Places to look for feature lists include: company webpages (such as a page from Apple listing features of MacOS Ventura), Wikipedia articles (such as this list of Facebook features), and blog posts in which users talk about features they wish apps had (such as this list for browsers).\nExample: I looked at a list of essential travel app features. The first listed feature was booking for flights and other things, and it occurred to me that it would be good to know if any buildings on the walking tour allowed visits, but only at certain hours or with booking in advance. Using an LLM Large language models such as GPT are very effective at generating features ideas for an app. The only thing to be wary of is that they can be so fast and seemingly comprehensive that it’s easy to be intimidated and not go to brainstorm your own, more creative, features.\nExample: I gave GPT 3.5 the following prompt: “I’m designing an app to support architectural walking tours. It would help people walking around a city identify buildings and learn about them. Can you help me by suggesting some features that such an app might have?”. It generated a list of about 20 features, including ones I’d already thought of (such as augmented reality integration) and ones I hadn’t (such as showing original blueprints and building plans). Exploring Values during Divergent Design Software designers are increasingly coming to see the social and ethical impacts of their products as central. In short, they want to bring their values to their design work. But this can’t just be tacked on as yet another design phase. Values can’t be incorporated as an afterthought but need to be integrated into your design work.\nThe divergent design phase provides a good context for considering social and ethical questions. By thinking about the value implications of your design ideas, you can make sure that your design is going in a direction that will be net positive to society.\nThis has several benefits. Most obviously, not building something that causes damage to society is a moral imperative. And building something that contributes positively to society can bring meaning and satisfaction to your work.\nThere are benefits for your organization too. Companies that are regarded as more trustworthy and honest by consumers reap a big advantage in market share and popularity.\nPerhaps most relevant to our discussion are the benefits that these considerations bring to the product itself. Social and ethical considerations sometimes suggest eliminating some features. But just as often they can lead to new features, and improvements in the app overall.\nHere are some examples of social and ethical considerations, drawn from the envisioning cards of value-sensitive design, applied to our example app.\nIndirect stakeholders Indirect stakeholders are people who may be affected by a system without directly using it. A good design must them into account.\nExample: What if some buildings became popular in the app for reasons that might upset local inhabitants? An apartment block in London, for example, experienced a famous fire in which dozens of occupants were killed. This suggests that features that use crowdsourcing to focus attention on certain buildings and routes should be deployed carefully, and some kind of flagging of moderation feature might be needed. Differing abilities Different users have different physical abilities, and a software design should accommodate these whenever possible.\nExample: Blind users will not be able to see photos of buildings. This suggests that photos should have captions for a screen reader, and that a separate audio feature would be useful. Like many accommodations for disabled people, including audio would benefit all users, many of whom might prefer to listen to a commentary as they walk without having to check their phones. Considering children An app designer should remember that even apps that are not designed with children in mind can end up being used by children.\nExample: Children often accompany their parents on walking tours, so thinking about how our app might serve them is important. One possibility is that descriptions are tagged by age and expertise level, and a user can set that level at the start and only be shown appropriate content. Or we might introduce some gamification features, such as rewarding children with points for spotting particular architectural features. Again, these features can benefit all users: there are adults who like game features too. Diverse geographies A solution designed for one geographic setting may perform quite differently in another. Example: River tours are popular in many cities (obviously in Venice and Amsterdam, but in many others, such as London and Chicago too). In designing our app, we might want to make sure that it’s still possible to use it if you’re not walking.\n","permalink":"https://essenceofsoftware.com/tutorials/design-general/divergent-tactics/","summary":"\u003ch1 id=\"tactics-for-divergence\"\u003eTactics for divergence\u003c/h1\u003e\n\u003cp\u003eLet’s consider some tactics that you might find useful as a software designer to make divergent design more effective. To make these concrete, I’ll suppose that we’re developing an app to support architectural walking tours.\u003c/p\u003e\n\u003ch2 id=\"interviewing-stakeholders\"\u003eInterviewing stakeholders\u003c/h2\u003e\n\u003cp\u003eTalking to potential users and other stakeholders is good not only for identifying needs but also for getting design ideas. You can ask what tools and techniques a user currently finds helpful, and what changes might make them more effective. Just resist the temptation to ask whether a particular design idea would be attractive; most interviews (even if just out of consideration for the interviewer) are likely to express unwarranted enthusiasm.\u003c/p\u003e","title":"Tactics for divergent design"},{"content":"Any software app, service or system can be viewed as a collection of interacting concepts.\nConcepts are the building blocks of software Imagine explaining Twitter to someone who\u0026rsquo;s never used it. You might tell them about the Tweet and Follower concepts, then maybe Hashtag and Like (aka Upvote), and if they\u0026rsquo;re keen to know even more, you could explain VerifiedAccount or Bookmark.\nEach concept can be explained with a purpose (what it\u0026rsquo;s for) and an operational principle (how you use it):\nFollower. Purpose: let you see content from preferred users. OP: if you follow someone, then their content will appear in your feed. VerifiedAccount. Purpose: prevent impersonators. OP: if someone verifies their identity to Twitter\u0026rsquo;s satisfaction (eg, by showing a link to it on their official website), then their account gets marked with a blue tick that tells people it\u0026rsquo;s real. I picked the word concept because its informal meaning is close to what I want to express. Technical usages of the term “concept” have meant specific and different things. A concept in software is something richer than a concept in concept lattices or conceptual models (where it usually means a class, so that “dog” refers to the class of all dogs, for example). A software concept has a rich behavior that serves a purpose. You can define the full behavior of a concept with a state machine, by declaring a state space (aka a data model) and some actions that read and write states.\nConcepts are software genes Concepts are like a kind of genetic code. Just as the behavior of an organism is determined by its genes, so the behavior of an app is determined by its concepts.\nAnd like genes, most concepts are common across apps. All social media apps have Friend or Follower (or both); any app with content from many users has Upvote; almost all apps have Password or something similar for user authentication; and so on. This is why it\u0026rsquo;s usually easy to learn a new app: you know almost all the concepts already.\nAnd like genes, concepts can mutate. Twitter\u0026rsquo;s Tweet, Facebook\u0026rsquo;s StatusUpdate and WhatsApp\u0026rsquo;s Message are all mutants of a basic Post concept.\nConcepts are life patterns Most concepts in software apps are just software equivalents of real world concepts: patterns of behavior that humans have enacted for centuries.\nConcepts such as Post (sharing content with the public) or Message (conveying content to one or more specified recipients) are communication patterns that have been around for millennia, ever since people carved words into stone tablets. Folder and Label reflect traditional ways to organize things: partitioning into groups, or attaching visible classification marks.\nConcepts are technological inventions But even the most well established life patterns were invented, by some person at some time. Restaurant reservations? Late 1800s. Social security accounts? 1930s. Using passports to identify people when traveling to foreign countries? Apparently Henry V in 1400s England. Publishing content? From short after the advent of writing, I’d guess.\nNot surprisingly, people keep inventing new concepts. Some are life patterns first and only become software concepts when apps or systems are built to support them. Most banking concepts are like this.\nSome concepts don’t mirror a pattern in daily life, or do it only weakly (and so we think of these as “metaphors”). Photoshop’s Layer concept is a tiny bit like an artist’s acetate layer, but it’s so much richer and more powerful that it counts as an invention in its own right.\nI suspect there are also concepts that appear for the first time in software and then make their way into daily life. Let me know if you can think of one!\nWhat’s new about this? Perhaps you’re wondering: aren’t these concepts you’re talking about just what people call “features”?\nThe fundamental difference is that concepts, unlike features, can be defined independently. A concept is free standing, and can be reused between one app and another. That’s what makes concepts understandable: when you learning how to use HackerNews and you discover that is has an Upvote concept, you know what it’s for and how to use it, because you’ve seen the same concept in other apps (for upvoting comments in your newspaper, for example).\nFeatures are generally inseparable from the app they belong to, and their effect is often defined as an extension of modification of some existing behavior. This means that features are much more general than concepts, and almost any step in the development of a program can be viewed as adding or removing a feature. But this generality means that features provide less leverage for thinking about design, because they can’t be designed and analyzed independently.\nWhat are concepts most like? Some people see a connection between concepts and object-oriented classes, but that seems confusing to me. Classes are a very code-oriented notion, and a class (traditionally) defines a collection of objects. A concept typically involves multiple collections of objects and relationships between them: for Follower, there are the users who follow each other and the posts they publish.\nA better analogy is that a concept is like a tiny service—a microservice, but even smaller. You could call it a nanoservice. Like a service, and unlike a class, a concept provides value by itself, and interacts directly with the outside world.\nHow do concepts fit together? Concepts are independent only in the sense that they can be understood independently. They can’t execute fully independently of each other or they wouldn’t need to be part of the same app.\nIn a social media app, the Friend concept will ensure that users can view content published by their friends; and the Post and Comment concepts will govern how users create posts and add comments. But these can’t execute independently, because the comments will have to be added (by Comment) to posts (from Post), and both comments and posts will have to be treated as published content (by Friend).\nThis is what concept synchronization is for: a way to compose concepts so that they interact together without requiring one concept to be defined in terms of another.\nTips Ways to use these ideas:\nInventory the concepts of an app to understand it and identify its strengths and weaknesses. Define a product family by the concepts that its members share. Point to “differentiator” concepts (like Photoshop’s Layer) that give a product an advantage over its competitors. When designing an app, steal existing (and familiar) concepts before inventing new ones. Whenever possible, align concepts with life patterns. ","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/sw-as-concepts/","summary":"\u003cp\u003eAny software app, service or  system can be viewed as a \u003cstrong\u003ecollection of interacting concepts\u003c/strong\u003e.\u003c/p\u003e\n\u003ch2 id=\"concepts-are-the-building-blocks-of-software\"\u003eConcepts are the building blocks of software\u003c/h2\u003e\n\u003cp\u003eImagine explaining Twitter to someone who\u0026rsquo;s never used it. You might tell them about the \u003cem\u003eTweet\u003c/em\u003e and \u003cem\u003eFollower\u003c/em\u003e concepts, then maybe \u003cem\u003eHashtag\u003c/em\u003e and \u003cem\u003eLike\u003c/em\u003e (aka Upvote), and if they\u0026rsquo;re keen to know even more, you could explain \u003cem\u003eVerifiedAccount\u003c/em\u003e or \u003cem\u003eBookmark\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eEach concept can be explained with a purpose (what it\u0026rsquo;s for) and an operational principle (how you use it):\u003c/p\u003e","title":"Software = concepts"},{"content":"A compelling way to explain how something works is to tell a story. Not any story, but a kind of defining story that shows, through a typical scenario, why the thing is useful and fulfills its purpose.\nThe Minuteman Library Network, for example, offers a wonderful service. If I request a book, then when it becomes available at my local library, I get an email notifying me that it’s ready to be picked up.\nNote the form this scenario takes: if you perform some actions, then some result occurs that fulfills a useful purpose. Many kinds of mechanism can be described in this way:\nIf you make social security payments every month while you work, then you will receive a basic income from the government after you retire. If you insert a slice of bread into the toaster and press down the lever, then a few minutes later the lever will pop up and your bread will be toast. If you become someone’s friend and they then publish an item, you will be able to view it. One reason that scenarios are so compelling is that we observe them all around us. Using an elevator is actually quite complicated, but we learned how to do it by watching other people. Many people haven’t yet learned how to use Schindler’s PORT elevator, so the story needs to be told explicitly:\nEnter the floor you want to go to on the keypad and note the elevator number that is subsequently displayed; now wait for the elevator with that number, and it will stop at your floor. Each concept has one or more such stories that explain how to use the concept, and show that the concept fulfills its purpose. I call such a story an operational principle (OP), a term introduced by the chemist philosopher Michael Polanyi and made known to the software world by my father Michael Jackson.\nUse cases, OPs and snooze alarms Because an OP is a scenario, you might confuse it with a use case. But these are very different things. Use cases are for specifying the requirements of a software system, so even a simple system will have a large number of use cases, covering not only ones in which the user succeeds but also those in which the user fails.\nIn contrast, the OPs of a concept aren’t intended to specify it fully, so they can be few and brief. The OPs characterize a concept’s essential behavior, explaining how it works and fulfills its intended purpose. All details that are not fundamental to the design are left unspecified.\nConsider a digital snooze alarm clock. The first OP is this:\nIf you set the alarm for a given time, and turn the alarm on, then when that time comes around, the alarm will ring (for some fixed time or until you turn it off). We can add a second OP to explain the snooze function:\nWhen the alarm starts ringing, you can press the snooze button, and it will stop ringing and start ringing again after some fixed time, and you can then turn it off. These OPs tell you enough to use the alarm clock, and they justify its design. But they barely begin to answer all the questions you might have about the details of its behavior. What happens if you press the snooze button again? If you turn the alarm off after pressing the snooze button but before the alarm rings again?\nOf course the designer will have to answer all these questions in a reasonable way, but they’re not the essence of the design. That doesn’t mean that we won’t come across additional scenarios that matter enough to be recorded as OPs. This one, for example:\nIf you set the alarm for a given time, do any sequence of actions except for setting the time, and then turn the alarm on, then it will ring when the given time comes around. This ensures that when you set the alarm time it “sticks” from day to day, and snoozing (in particular) won’t change it. It’s not the only possible design. You could imagine a clock that resets the alarm time to 8:05am if it was set previously to 8:00am and then snoozed.\nSpecifying full behavior So if the OP doesn’t fully specify the behavior of a concept, how do you do that? I’ll cover that in detail in another post, but here’s the idea. In short, you define the state of the concept and then define the effect of each action on the state (just like writing high-level code):\nstate alarmTime, ringTime, now: Time on: Bool ringing: Bool = on \u0026amp;\u0026amp; 0 \u0026lt; (ringTime - now) \u0026lt; RINGT actions\tsetAlarmTime (t: Time) alarmTime := t tick (t: Time) now := t alarmOn () on := true; ringTime := alarmTime alarmOff () on := false; snooze () if ringing ringTime := ringTime + SNOOZET Notes. The alarm is defined to ring for a time RINGT after it goes off and to snooze for a time SNOOZET. The passage of time is modeled explicitly by an action that updates the current time.\nExercises for the reader. (1) How must SNOOZET be related to RINGT, and why? (2) What happens if you set the alarm time to just before the current time?\nIf what you’re trying to do is fully specify a concept’s behavior, this kind of state machine model is the way to go. It’s much more succinct than scenarios (and so better than use cases, in my view, in this role).\nBut OPs offer something else. In a technical sense, you can infer all scenarios from the state machine, but that doesn’t mean you can make sense of them. To understand the snooze function you need to know how it’s typically used: the alarm rings, you press snooze and then it rings again a bit later.\nSo the OP helps by distinguishing the archetypal scenario from other scenarios that are equally plausible (in terms of the state machine) but play no role in explaining the concept’s purpose. For example, you can let the alarm ring until just before the moment at which it switches off and press snooze then. This scenario is as valid (to the state machine) as our first OP (in which you press snooze when it starts ringing) but it’s not helpful in understanding why the snooze function was invented.\nHow many OPs? A simple concept may have just one OP. Here’s an example of a concept whose OP is almost too trivial to explain:\nComment. If you add a comment to an item, then the comment will appear in the list of comments displayed with it. Some concepts, though, need two or more OPs to explain their essential functionality:\nTrash. (1) If you delete a file, you can then restore it by moving it out of the trash. (2) If you delete a file and then empty the trash, the file is gone forever and its storage space can be reused. Password. (1) If you register with a user name and password, and then you login with that same user name and password, you will be authenticated as the user who registered (that is, able to act in their name). (2) If you register with a username and password, and then you login with that same username but a different password, you will not be authenticated. Personal access token (PAT). (1) If you create an access token for a resource and pass it to another user, then that user can enter the token string and obtain access. (2) If you create an access token, pass it to another user, and then revoke it, the other user will not be able to access the resource with it. It’s not unusual for a more elaborate OP to be the one that actually motivates the concept design. For the PAT concept, it’s the revocation that’s the essence: without it, a simple password would do.\nGeneric and context-specific OPs Most concepts are generic, which means that they can be instantiated in different contexts, applying to different kinds of objects. The Trash concept, for example, can be applied to files and folders in a file system; or to messages in an email client; or to photos in a catalog.\nWhen writing an OP as part of a concept design, you’ll have to decide whether to write it in a generic form, or instantiated for the context at hand. My recommendation is that you write it initially in its instantiated, more concrete form, as this will let you assess more easily whether the concept will provide the value that’s expected. Then, once you’re happy that the concept is the right one, you can reformulate it in a more generic way to gain the benefit of a simpler and more flexible concept, and to ensure that you haven’t specialized the concept in some way that will make it unfamiliar.\nHere’s an example. Suppose you’re designing an email client, and realize that it would be convenient if the app were to automatically complete email addresses when they are typed in to the to or cc fields. You might start with an OP like this:\nPreviousRecipientCompletion. After you have sent emails to a variety of addresses, each time you send an email and start to type an address, the app automatically suggests a completion based on the addresses previously used. Having written this, you realize there is nothing email-specific about this; this is just a simplified form of a generic concept:\nAutocompletion. When you start typing a string, the system suggests suffixes that would extend the string to one that appears in a predefined dictionary, or to one that you previously typed. Purposes, not user goals The design of a concept is driven by some purpose that the concept fulfills. In some cases, this purpose is aligned with a simple user goal. The purpose of the TextMessage concept is to convey messages between users; a user following the OP (if you send a message to another user, then they will receive it) has the simple goal of conveying a single message.\nIn general, however, user goals are not so easily aligned with purposes. A user always has some motivation to perform some action or sequence of actions, but the motivation is often inchoate and not worth dwelling on in the design process.\nThe purpose of the Password concept, for example, is to authenticate users. Neither the user who successfully logs in nor the user who fails has a goal that fully matches this purpose, whose essence is to distinguish the two cases.\nOne of the risks of focusing on user goals is that it becomes tempting to articulate shorter scenarios that have no relationship to the purpose at all. There’s no point writing a scenario for registering a password, for example. It provides no value in itself to the user, and matters only in the context of the OP in which it’s followed by a successful (or unsuccessful) login.\nOPs with multiple users Another reason an OP might have no user goal is that the purpose involves collaboration of multiple users. The purpose of Upvote is to let users rank items by their popularity. When you as a user give a thumbs-up to an item, you’re not fulfilling a goal worth articulating, but simply contributing to a larger communal purpose. The OP for Upvote must include upvotes by multiple users on multiple items, to illustrate the key idea that the ranking of items depends on the number of upvotes they receive.\nThe danger of “secondary” goals One final problem with goals: they encourage you to think that some goals are primary and others secondary. But it may well be the supposedly secondary goal that motivates the design of the concept.\nTake the ShoppingCart concept, for example. You might be tempted to label as primary the OP in which a user adds an item and checks out immediately (thus buying the item in the most efficient way), and label as secondary the OPs in which the user adds multiple items, or adds and then removes items.\nBut just as the purpose of the Trash concept is not deletion but undeletion, which is illustrated by the OP in which an item is deleted and then restored, so the purpose of ShoppingCart is to allow items to be selected for purchase without commitment. If it were merely to make it easy to buy items, a different, simpler concept would suffice (such as Amazon’s one-click BuyNow).\nWriting good OPs OPs offer a lightweight and provocative way to explore a new design or record an existing one. Here are some tips on writing good OPs.\nPurpose. Have the concept purpose in mind: that the purpose of Trash is undeletion (and not deletion), or that the purpose of Password is authentication (and not creating accounts or anything like that). Full history. Remember to include a long enough history of actions to demonstrate the purpose. Registering a password is useless by itself; it only becomes interesting when you log in later. One concept. If your OP looks complicated, you may be mixing the OPs of distinct concepts. A scenario in which a user logs in, views a friend’s post and then comments on it involves at least three concepts (Password, Friend, Comment). Actions not buttons. Write the OP in terms of the actions and states that form the concept (and the user’s mental model), not the realization of the concept in the user interface. So keep out details of UI buttons and views, and avoid especially breaking a single action (such as a login attempt) into multiple steps (enter name, enter password, press submit). ","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/principle/","summary":"\u003cp\u003eA compelling way to explain how something works is to tell a story. Not any story, but a kind of defining story that shows, through a typical scenario, why the thing is useful and fulfills its purpose.\u003c/p\u003e\n\u003cp\u003eThe Minuteman Library Network, for example, offers a wonderful service. If I request a book, then when it becomes available at my local library, I get an email notifying me that it’s ready to be picked up.\u003c/p\u003e","title":"Operational principles"},{"content":"The role of purposes From what to why. The beginning of wisdom for a designer is to stop asking “what?” and start asking “why?”.\nThis applies at every level. You can ask “what application shall I design?”. But a better question is “why would I design that?”. And at a finer grain, the question “what features should my app have?” gives way to “why should my app have these particular features?”.\nGranular justifications. Software designs are always taught to think about the purpose of the product as a whole: who it serves, and what value it delivers. This is an essential part of needfinding (as it’s called in the HCI community) or requirements analysis (as it’s called by software engineers).\nBut they’re rarely taught to justify the elements of their designs. Instead, a designer will often assemble an elaborate array of functions and will ask only whether the totality of the design meets the users’ needs. No case is made for each individual element; it’s included just because it’s part of the overall design. The result of this is a lot of functionality that is poorly targeted, or worse, unnecessary.\nThe alternative is that each element of functionality gets its own justification. But what’s an element? Do we have to explicitly justify every line of code?\nConcept purposes. In concept design, each concept is given a justification called a purpose. You might sometimes find it helpful to record justifications for smaller elements (such as the actions of a concept), but in practice that’s too much work, and justifying entire concepts is good enough.\nA rationale for existing. A concept purpose is an explanation of why the concept exists; it’s the rationale for going to the trouble of designing it, or when the concept already exists, the rationale for including it. This is critical input to the designer’s decision of whether including the concept is worth the cost (in design, implementation, user interface complexity, etc).\nKnowing why. The concept purpose is useful to the user too. Knowing why a piece of functionality exists is usually the first thing a user needs to understand, before they even consider how to use it.\nExample: Melania Trump’s mistake. In my book I tell the story of how Melania Trump “favorited” a tweet that made fun of her husband. She presumably thought that this would save the tweet for her privately; unfortunately, her action made public that she had “liked” the tweet. Her mistake was understandable, because Twitter’s explanation of the Favorite concept didn’t explain what it was for, so she confused what was essentially an Upvote concept for a Bookmark concept (which in fact was missing, and added to Twitter later). Some examples of purposes Articulating the purpose of a concept can be hard, but the effort pays off in deeper understanding and making the design work that follows more effective (because you know what you’re really trying to do).\nHere are some examples of purposes for familiar concepts:\nTrash. The purpose of the Trash concept, introduced originally in the Apple Macintosh in 1984, is not to make deletion easy. On the contrary, its purpose is to allow undeletion. Style. The purpose of the Style concept, widely used in all desktop publishing tools and word processors, is not to let you change the typographic settings of a piece of text; the Format concept is sufficient for that. Its purpose is to help you keep a document consistent by changing the formatting of a whole collection of paragraphs (eg, all the headings) in one go. Notification. The purpose of the Notification concept in social media apps is to draw your attention to activity that may be of interest. At least, that was the original purpose, and that’s the purpose that most users would like it to have. Unfortunately, in many cases its purpose seems to be to increase user engagement by drawing users back to the app even when there is nothing of value for them there. AuthenticationToken. This purpose of this concept, used for example by OAuth and many other schemes, is to allow users to grant access to some resource in some service (for example, the contacts in your Gmail account) to different parties. The idea is that each party gets a different token, so the user can revoke access for one party without revoking it for another. Criteria for good purposes What makes a good purpose definition for a concept? Here are some criteria:\nNeed-focused. The purpose should be stated in terms of the needs of the user. For example, the purpose of the Upvote concept should not be to “express your approval of an item” since that has no tangible benefit; a better purpose might be to “use crowd-sourced approval to rank items.” Specific. The purpose should be specific to the design of the concept at hand. For example, even though an Autocomplete concept in Gmail may make the app more attractive to consumers, it wouldn’t be useful to say that its purpose is to “increase the Gmail user base” since presumably all concepts have that goal. A better purpose might be “to save the user typing effort.” Evaluable. A purpose should be a yardstick against which to measure a concept design. For example, a good purpose for the Trash concept is “to allow undeletion.” The apparently similar purpose “to prevent accidental deletion” is not good because evaluating the concept design against that goal would require all kinds of assumptions about user behavior. Concept specificity Obviously, a concept should have at least one purpose. If it has no purpose, there’s no reason for it to exist.\nMore controversially, a concept should have at most one purpose. This might seem strange, especially since some people think that a good design decision solves multiple problems at once (or feeds many birds with one scone, as the politically correct version of the old adage goes).\nIn fact, this is a misconception, and rigorously separating purposes so that each concept has only one purpose is the path to more flexible software. The chapter on specificity in my book has many examples of design flaws that arose because a concept was given more than one purpose.\nExample: Facebook’s Reaction concept. In Facebook, the Reaction concept is used both to register approval (for deciding which posts to include in users’ feeds) and to send an emotional reaction back to a post’s author. Users are especially confused by the fact that clicking on the angry emoticon is a positive upvote for a post. The entangling of purposes means that it’s not possible for a user to do one with the other: you can’t approve a post, for example, without conveying a reaction back to the author, or vice versa. Example: Facebook’s Friend concept. In Facebook, the Friend concept serves two purposes. One is to provide access control so you can limit who sees your posts. The other is to filter your own incoming content, so you can choose whose posts you want to see. (Arguably the design of the Feed concept has undermined this second purpose, but that’s another matter.) These two purposes are not always in alignment; you might want to see someone’s posts but not share yours with them, or vice versa. Avoiding having more than one purpose for a design component is an idea that appears in many different domains. Programming advice often recommends that each function should do only one thing (and David Parnas identifies combining purposes in a single function as one of the main causes of a program not being amenable to extension or contraction). In mechanical engineering, ensuring that each design parameter corresponds to at most one functional requirement is one of the “axioms” of axiomatic design.\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/purpose/","summary":"\u003ch2 id=\"the-role-of-purposes\"\u003eThe role of purposes\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eFrom what to why\u003c/strong\u003e. The beginning of wisdom for a designer   is to stop asking “what?” and start asking “why?”.\u003c/p\u003e\n\u003cp\u003eThis applies at every level. You can ask “what application shall I design?”. But a better question is “why would I design that?”. And at a finer grain, the question “what features should my app have?” gives way to “why should my app have these particular features?”.\u003c/p\u003e","title":"Concept purposes"},{"content":"If user experience runs deeper than the user interface, we need a way to talk about an app that isn’t just visual, and that captures behavior in a more fundamental way.\nProgrammers think in terms of complicated things like objects, callbacks, streams, functionals, and so on. They’re essential for structuring code, but for behavior a much simpler model is effective.\nState machines That model is the state machine. You’ve probably come across state machines, either in a class about the theory of computation, or seeing diagrams like this:\nThis diagram describes the state machine of a shopping cart. You start with an empty cart; adding an item produces a non-empty cart; then you can add and remove more items; checkout; and then return to the initial state.\nDiagrams like this are useful for capturing basic control flow. But they’re not rich enough for modeling behavior, because the number of states must be finite (and small enough to draw!). This diagram doesn’t say that the order you submit when you checkout contains the items you added and didn’t remove. And because individual items aren’t tracked, it needs some non-determinism to model the fact that when you remove an item from a non-empty cart you might end up back with an empty cart (if it was the last item) or you might not (if it wasn’t).\nWhat exactly is a state machine? To understand exactly what a state machine is, let’s look at an example that’s so simple we can draw it in full. Here’s a rather basic egg timer:\nIt works like this. You start in the state s0, and then press the up button to increment the timer to s1 corresponding to one minute, s2 for two minutes, or s3 for three minutes; you can press the down button to decrement the timer; and then when you’re ready, you press start and a countdown begins, with r3 meaning running for three minutes, etc; and then when no time is remaining (r0) the timer rings.\nWe could write down this state machine as a collection of explicit transitions, but to be more succinct we could introduce two separate variables:\ntime: 0..3 running: bool and then define an action like this:\nup () when not running and time \u0026lt; 3 time += 1 start () when not running and time \u0026gt; 0 running := true This looks more like a program but it’s still a state machine. The states are all the possible environments (that is, bindings of values to variables), and the actions are sets of transitions like (e, a, e’) where e and e’ are the environments before and after and a is the action name. To be pedantic, we might use the term action for the name of an action (and its transitions) and action occurrence for a particular transition.\nFor example, the action up has three occurrences, of which the first (corresponding to the leftmost arrow in the diagram) is\n({time: 0, running: false}, up, {time: 1, running: false}) States In a more realistic app, the variables won’t have primitive values. Instead, each variable will be a data structure itself. The easiest way to represent such data structures uniformly is to think of them as relations.\nThe variables representing the state of an online store, for example, might include one holding the number of each item in stock:\nstock: Item -\u0026gt; one Number sets of pending and fulfilled orders for each user:\npending, fulfilled: User -\u0026gt; set Order a shopping cart for each user:\ncart: User -\u0026gt; one Cart and, for each cart or order, the items it contains:\nitems: (Cart + Order) -\u0026gt; set Item The value of each of these variables is a binary relation. The fulfilled variable, for example, is a set of pairs of the form (u, o) where u is a user and o is one of u’s fulfilled orders. Viewed as a table, there’s a separate row for each user and order. So if there are two fulfilled orders for Alice and one for Bob, the table might look like this:\nUser Order Alice Order_1 Alice Order_3 Bob Order_2 Actions Just as the states of a real app are richer, so the actions are a bit richer too. The label of the action includes not only the action name, but often some parameters too.\nFor example, here’s an action for adding an item to a cart:\nadd (c: Cart, i: Item) when i.stock \u0026gt; 0 c.items += i i.stock -= 1 An example occurrence might be add (c0, i1), meaning an addition of item i1 to cart c0.\nThe definition of actions is just like it was with the simple egg timer, except now we have to read and write relating values. Here I’m using an Alloy-like notation, in which\nc.items += i means that item i is added to the set associated with the cart c by the items relation. In traditional mathematical notation, it’d be written like this\nitems' = items U {(c, i)} where items’ is the value of the items relation after. So this action definition says that when the requested item is in stock, the item is added to the given cart and its inventory count is reduced by one. The when condition is a guard; if it doesn’t hold the action can’t happen. So you can’t add an item to your cart if the item is out of stock.\nRemoving an item from a cart looks like this:\nremove (c: Cart, i: Item) when i in c.items c.items -= i i.stock += 1 When you checkout, a new order is created and made pending, the cart is deleted from the user’s carts, and the order is given the set of items that were in the cart:\ncheckout (u: User, c: Cart, o: Order) new o u.pending += o c in u.carts u.carts -= c o.items := c.items Finally, when an order gets fulfilled, it’s moved from the set of pending orders to the set of fulfilled orders:\nfulfill (u: User, o: Order) when o in u.pending u.pending -= o u.fulfilled += o ER diagrams An entity-relationship diagram offers a nice way to show the state variables:\nThere are a few differences from the textual declarations. I’ve used symbols rather than words for multiplicity (! in place of one, for example). I’ve shown the multiplicity on both ends of a relation: the ! on the source end of the cart arrow says that each cart belongs to one user. You can do both of these textually too. The diagram doesn’t naturally accommodate unions of sets, so for the items relation I introduced a superset called Bag, with each Bag being either a Cart or an Order.\nThese diagrams are very lightweight and contain lots of information. Working out the details of the state as you the draw the diagram always reveals interesting questions. Can a user have more than one cart, for example? That’s certainly possible for Amazon, because a client-side cart is created before you even log in.\nWhat’s the point? UI-independent view. Viewing an app in terms of its state and actions lets you think about its behavior concretely, but without having to consider the details of the user interface. And when you come to design the UI, you can ask yourself how to map the states and actions to UI views and widgets. More succinct than use cases. A use case can be useful for describing the details of a workflow, but often a single action will suffice. The checkout action, for example, could encompass all the steps in checking out a shopping cart, and the flow would be better described using wireframes than text. For capturing a fuller journey (that a user adds and removes items from a cart, then checks it out, then the order is fulfilled), I recommend using an operational principle written in terms of actions. Path to implementation. It’s easy to extract a relational database schema from the ER diagram. OMT showed how to do this systematically. You can also generate a class hierarchy for an object-oriented approach. Small details expose big issues. The biggest advantage of writing down a state machine is that as you work out small details you inevitably encounter questions that expose serious design issues. For example, when I wrote the add action, I had to decide whether it should check the stock of the item, and if so, whether it should decrement it. I decided to do both, which means that (a) you can only add an item that is in stock; and (b) you’re assured that when you checkout, the item you wanted won’t have been taken by someone else. But this convenience for the user comes at a cost for the company, because it means that items will be shown as unavailable when they haven’t yet been paid for. And there will need to be a way to return an item to the inventory if too much time elapses with the cart not being checked out. Describing a whole app as a single state machine rarely makes sense (unless it’s a very small app). Concepts to the rescue! Just as you can describe an app as a state machine, we’ll see that you can do the same thing for an individual concept. Then an app is just a composition of concepts, and you can define in detail only the concepts that seem tricky or interesting.\nSome technical notes History. This kind of state machine model was introduced by the Z Notation in the late 1970s, and became universal in all formal modeling languages (VDM, B, TLA and my own language Alloy).\nGlobal state. The state represents the entire global state of the app, frontend and backend. A state machine can describe a distributed system too. Butler Lampson shows how to do this in his course notes on operating system design.\nNotational nitpicks. The notation I used in the example is mostly from Alloy, with a shorthand for updating a relation. The statement c.items -= i would be written c.items’ = c.items + i in Alloy. There’s a crafty overloading going on in the treatment of the inventory counting: since the stock variable maps each item to exactly one number, I’ve written i.stock += 1 for i.stock’ = plus(i.stock, 1), with the + meaning integer additional and not the set union it usually means in Alloy.\nVisible state. In many formal methods, the state is assumed to be hidden, and the user’s view of the state is represented by observer actions that produce outputs but don’t update the state. It’s easier to just assume that the state is by default visible to the user, and to describe how and when the user can view the state as part of the user interface mapping.\nRelations and RDBs. Relations don’t need to be binary. But relations of higher-arity should only be used when there’s a genuine need to relate more than two things: in RDB terms, the relations should be fully normalized. For example, to allow more than one of a given item in a cart or order, we could declare a 3-place relation containing the tuple (c, i, n) when cart c has n copies of item i:\nitems: (Cart + Order) -\u0026gt; Item -\u0026gt; one Number Output actions. Actions can be inputs (initiated by a user) or outputs (initiated by the system). An input action can have output parameters as well as input parameters.\nDeterminism. I prefer all actions to be deterministic. This doesn’t actually prevent the system from making choices that aren’t controlled by the user; it just means that those choices must be exposed in action parameters. For example, an airline reservation system might assign a seat to a passenger with an action whose signature is\nassignSeat (p: Passenger, out s: Seat) Thanks to Gordon Cassie for comments on this post which led to several improvements.\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/app-as-machine/","summary":"\u003cp\u003eIf user experience runs \u003ca href=\"../../design-general/beyond-ui\"\u003edeeper\u003c/a\u003e than the user interface, we need a way to talk about an app that isn’t just visual, and that captures behavior in a more fundamental way.\u003c/p\u003e\n\u003cp\u003eProgrammers think in terms of complicated things like objects, callbacks, streams, functionals, and so on. They’re essential for structuring code, but for behavior a much simpler model is effective.\u003c/p\u003e\n\u003ch2 id=\"state-machines\"\u003eState machines\u003c/h2\u003e\n\u003cp\u003eThat model is the \u003cstrong\u003estate machine\u003c/strong\u003e. You’ve probably come across state machines, either in a class about the theory of computation, or seeing diagrams like this:\u003c/p\u003e","title":"Apps are state machines"},{"content":"As I explained in another tutorial, the easiest and most effective way to define the behavior of an app precisely is to model it as a state machine. The app has a set of possible states, and actions that update and query the states. An execution of the app is just a sequence of actions (or more correctly, action instances that include particular action arguments), and the behavior as a whole is the set of all possible executions.\nDescribing concepts themselves as state machines brings the same advantages. The state machine of the app as a whole is just a combination of the machines of the individual concepts. This is how concepts bring modularity to thinking about the functionality of an app.\nAn example of a concept as a state machine Let’s model the core concept of Yellkey, a URL shortening service that many teachers use. The novelty of Yellkey is that the short URLs it generates use common English words, so you get short URLs like “https://yellkey.com/hello” rather than ones that contains strange sequences of letters that are harder to transcribe.\nHere’s the Yellkey concept as a state machine:\nconcept Yellkey purpose shorten URLs to common words principle after registering a URL u for t seconds and obtaining a shortening s, looking up s will yield u until the shortening expires t seconds from now: register (u, t, s); lookup (u, s') {s' = s} state used: set String shortFor: used -\u0026gt; one URL expiry: used -\u0026gt; one Date const shorthands: set String actions // register URL u for t seconds // resulting in shortening s register (u: URL, t: int, out s: String) s in shorthands - used s.shortFor := u s.expiry := // t secs after now used += s // lookup shortening s and get u back lookup (s: String, out u: URL) s in used u := s.shortFor // system action: shorthand s expires system expire (out s: String) s.expiry is before now used -= s s.shortFor := none s.expiry := none Actions are a good place to start because they capture what the user does to use the concept. There are two actions that are performed by the user: register, which takes a URL and some duration in seconds, and returns a shortening; and lookup, which takes a shortening and expands it by returning a URL. In Yellkey, the user doesn’t get to choose an arbitrary duration, but selects one from a dropdown (which has a few options between 1 and 24 hours).\nThe fact that a shortening can expire is represented by an action that is performed by the system: expire occurs for a given shortening when it has passed its registered duration.\nDefining state To design the state, you consider what must be remembered by the concept in order to support the actions. In this case, what’s needed is: a mapping from shortenings to URLs, and a mapping from shortenings to their expiry times. Since the shortenings must be drawn from a set of common words, that set is stored as part of the state also.\nFor convenience, I’ve also included a component that remembers which shortenings are currently being used. Strictly, this isn’t necessary because the shortenings in use are just the ones mapped by the two mappings. But this makes it a bit easier to define the actions, and to make explicit (in the declarations of the mappings) that every shortening being used maps to exactly one URL and exactly one expiry time.\nDefining Actions Each action has a header that names the action and lists some arguments, and a body that defines the action’s behavior:\nregister (u: URL, t: int, out s: String) s in shorthands - used s.shortFor := u s.expiry := // time now plus secs used += s The arguments that are marked with the keyword out are outputs that are returned by the action; the others are all inputs. (Using a keyword rather than having a special return value turns out to be more convenient, because it allows more than one output, and introduces names for them).\nThe body includes two kinds of statements: preconditions that limit when the action can occur, and postconditions that say what effect the action has.\nLet’s look at each line in turn. The first line\ns in shorthands - used says that any value can be picked for s that is in the set shorthands - used, that is the set of words that are shorthands but not used.\nThe second line\ns.shortFor := u says that the mapping between shortenings and their URLs is updated so that the new shortening s maps to the given URL u.\nThe third line\ns.expiry := // t secs after now says (informally) that the mapping between shortenings and their expiry times is updated so that the new shortening s maps to the time corresponding to t seconds after the current time.\nAnd finally\nused += s says that the set of used shortenings has s added to it.\nA note on notation. I’ve used a simple notation for the action formulas based on Alloy, extended with some C-style operators. Given a relation r: A -\u0026gt; B, and variables a and b holding values drawn from the sets A and B, the statement a.r := b makes a map to b in r—that is, adds the pair (a, b) to r.\nNon-determinism The register action doesn’t say how the shortening is chosen. It says any shortening is acceptable so long as it’s in the set of shorthands that are not currently being used.\nThis is called “non-determinism”. It doesn’t mean that the selection has to be random, or that if you could somehow undo the action and do it again that you might get a different result.\nAll it means is that the specification doesn’t say how the shortening is chosen, and whoever implements the concept can decide how it’s done (so some prefer the term “under-determined” to “non-deterministic”).\nFor example, the set of shorthands could be represented as a ring with a pointer to the last shorthand issued. When register is called, it returns the next shorthand in the ring, or skips it if it’s being used, carrying on round the ring until it finds an unused one.\nDesign is in the details The details are not the details. They make the design.—Charles Eames\nBeing precise about the states and actions helps in several ways. The first is communication: it helps you convey the concept behavior unambiguously to others.\nThe second is that it counters wishful thinking. As anyone who’s written software knows, it’s easy to convince yourself that some function is obvious until you actually try and code it, and then you discover you don’t even understand what it should do. Writing a model is like writing code in this respect, except that it’s less work and lets you know sooner that you’re confused.\nThe third is that different options come up as you formalize the behavior, prompting you to address design questions you might not even have noticed.\nFor example, an alternative specification of the register action might not just add a new shortening for a URL but might replace any existing ones.\nOn the one hand, one could argue that it’s not in the spirit of Yellkey to support multiple shortenings for a single URL, and it would also save resources to do this.\nOn the other hand, this would open up a potential malicious attack. Suppose a lecturer writes a Yellkey short URL on the board for a class quiz. A naughty student in the class could register a new one for the same URL, causing the old one to stop working.\nConcepts aren’t classes People are sometimes confused about the difference between concepts and classes in object-oriented languages. With our concept example in hand, we can now see how different a concept is from a class.\nUser-facing, end-to-end functionality. A concept provides user-facing functionality, in an end-to-end way. A class, in contrast, is usually hidden from users, and provides only a piece of functionality that is combined with functionality in other classes to deliver some value to the user. For example, an implementation of Yellkey might have a Shorthand class each of whose instances holds a shorthand and its associated URL. But this class doesn’t offer lookups; that would have to be in another class. No intrinsic dependencies. Concepts don’t have intrinsic dependencies on other concepts, and can be understood in isolation. Classes almost always use other classes, and can’t operate without making calls to them. System actions. Concepts can have both user and system actions; the methods of a class are called from outside (and if they were called internally, the call would not be visible to the outside). Richly associative state. Concepts hold state that associates different objects and values in rich ways, and allows flexible navigation. Classes, in classic OOP, are more constrained. A class is represented by a set of objects, and each object can contain references to other objects or collections of objects. But this structure does not support querying for particular objects within a class. For example, as I just noted, given a Shorthand class whose objects represent registrations of shorthand/URLs, you can’t ask for the objects that match a particular shorthand. The OOP workaround for this is to introduce another class whose instances are shorthand-to-URL mappings. None of this means that concepts can’t be implemented in an object-oriented language. In fact, a reasonable approach is to implement each concept as a class, supported by whatever other classes are needed to fulfill its functionality (but avoiding any references to the classes of other concepts).\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/concept-as-machine/","summary":"\u003cp\u003eAs I explained in another \u003ca href=\"../apps-are-state-machines\"\u003etutorial\u003c/a\u003e, the easiest and most effective way to define the behavior of an app precisely is to model it as a state machine. The app has a set of possible states, and actions that update and query the states. An execution of the app is just a sequence of actions (or more correctly, action instances that include particular action arguments), and the behavior as a whole is the set of all possible executions.\u003c/p\u003e","title":"Concepts are state machines"},{"content":"Motivation: Defining Behavior The operational principle of a concept explains its archetypal behavior: how it’s typically used, and how it fulfills its purpose. But for a concept to be flexible and powerful, it should work in many different scenarios, so we need a way to pin down the behavior in detail.\nUsing the notion of a state machine, we can define how a concept will behave in every possible situation. Then we can analyze the design to make sure that the behavior is always acceptable. And as we transition from design to code, the state machine provides an ideal stepping stone. The state of the state machine will become the state of the program and the actions of the state machine will become the functions of its API.\nA relational formulation of state is the best, in my view, because it is abstract enough to avoid making any premature implementation commitments, but at the same time is easily translated into code. For traditional or object-oriented data structures (in languages like JavaScript and Python) or collection databases (like Mongo.db), the translation is usually straightforward; for a relational database (where the state is defined by a schema in SQL) the translation is even easier.\nThe Role of State Novices are sometimes confused about what the actual role of the state of a concept is. Part of the reason for the confusion is that state declarations are very similar to ontologies or knowledge graphs, and this encourages novices to think that the state should somehow embody everything that is known about the relevant objects in the domain and their relationships.\nBut that’s a mistake. The state of a concept plays a very straightforward role, and that’s to determine what the concept remembers at runtime. The state is the concept’s memory. If some piece of information is needed in an action, then it will have to be in the memory; if it will never be needed, it shouldn’t be there, even if it’s interesting.\nFor example, suppose we’re designing a concept called Group for messaging within a group of users. The state will certainly need to include which users belong to which groups, because there will be actions that depend on this (the action post for posting a message will likely check that the user is a member of the group they’re posting in). The state will also need to distinguish regular members of a group from the owner of the group, since some actions (such as approving new members) will only work for owners. Should the state also include the date at which a user joined a group? That would depend on the details of the behavior. If a user who joins a group can only see messages posted after they joined, the date will be necessary, since otherwise this check can’t be performed. But if a user can see all old messages, the date wouldn’t play a useful role, and can be omitted.\nThis criterion of only including in a concept’s state what the concept behavior requires helps divide state up between concepts too. Suppose we’re designing an app for a lending library. We might have a Catalog concept for finding books, and a Hold concept for putting a hold on a book. Since the Hold concept seems to be about books, you might be tempted to include in its state things like the name of a book, or the email address of a user issuing a hold. But this would be a mistake: the Hold concept should support just the actions of creating and tracking holds on books, and for those functions it doesn’t need to know anything about a book except for its identity. The Catalog concept will contain information about a book, including its title. Likewise, the Hold concept will only need to know the identity of a user, and the user’s contact details will be stored in another concept.\nDefining State We’ll define the state of a concept as a series of variable declarations. Each variable will correspond to a set, or to a relation. A relation may map one set to another (in which case it’s called a binary relation), or may associate elements of more than two sets (in which case I call it a multirelation). If you think of a relation as a table, then a binary relation has two columns, and a multirelation has three or more. A set is just a table with one column, so it’s a relation too.\nThis way of thinking about state is taken from the Alloy language, and we’ll use some basic Alloy-like syntax to define the relations.\nFor example, suppose we are designing a concept called Invitation that captures the kind of functionality that appears in many apps in which one user can issue an invitation to another user, to be a friend, to access some files, or whatever. The state of the concept might include\ninvited: User -\u0026gt; User which introduces a relation called invited between users. You can think of a relation like this in several ways: as a mapping from users to users which can be access by a lookup (where u.invited might represent the set of users invited by the user u), as a tuple set or predicate that contains the tuple (u1, u2) when user u1 has invited user u2; or as a table with two columns, with the understanding that the first column represents the inviting user and the second the invited user, and each row represents an invitation. These are all equivalent ways to view a relation.\nIf we want to store the date on which invitation was issued, we could use a three-way relation:\ninvited: User -\u0026gt; User -\u0026gt; one Date where (u1, u2, d) would be included when user u1 invited user u2 on date d. (The multiplicity keyword one says that each pair of users maps to a single date.)\nAnother way to model this would be to introduce a set of invitation objects, and then we can define the date of an invitation and who it’s from and to as additional projections:\nfrom, to: Invitation -\u0026gt; one Person date: Invitation -\u0026gt; one Date As an example of a set declaration, we might have a set of pending invitations that have yet to be accepted:\nPending: set Invitation Novices sometimes wonder why it’s preferable to use a set rather than a mapping to a boolean value. Why not, for example, represent the pending invitations as a relation like this?\nisPending : Invitation -\u0026gt; one Bool The answer is that it’s much easier to write relational expressions about the state using the set rather than the boolean function. For example, the set of users who have issued invitations that are pending can be written as Pending.from; to write this with the boolean function you’d need to construct a set comprehension with a formula.\nAlso, introducing sets lets you include certain invariants in the declaration of the state. For example, suppose the Invitation concept includes a remind action which reminds a user who has received an invitation to accept or reject it, and this action is triggered some fixed time after the invitation is issued. We could represent the time for the reminder like this:\nremindOn: Pending -\u0026gt; one Date which says that each pending invitation has a reminder date. Without the Pending set, we’d need to associate the reminder date with all invitations:\nremindOn: Invitation -\u0026gt; lone Date which says that each invitation is associated with zero or one dates at which the reminder should occur. Because the declaration maps all invitations, not just pending invitations, we have to weaken the multiplicity from one to lone (zero or one), and the invariant that says that every pending invitation has a reminder date is no longer expressed.\nDiagram notation The relation declarations of the state can also be represented in a diagrammatic form. Here, for example, is a diagram for the Invitation concept:\nThis kind of diagram is called an extended entity-relationship diagram. It’s “extended” because it lets you show that one set is a subset of another, in this case that the Pending invitations are a subset of all invitations.\nNote how the diagram expresses the constraint that only pending invitations have reminder dates. This kind of constraint requires subsets and can’t be expressed in standard entity-relationship diagrams.\nPrinciples Here are some principles to help you design the state of a concept:\nSufficiency. The state must be rich enough to support the actions. Put another way, the concept must remember enough about actions in the past to be able to perform actions in the future correctly. Example: an Upvote concept, which tracks how many times users have approved some item, must remember which user issued each upvote in order to prevent double voting, but the date of the upvote would likely not be needed.\nNecessity. Generally, the state should not include components that are not necessary to support the actions. Example: a Reservation concept need not store reservations that have been canceled; a Trash concept need not store the original location of an item if there is no restore action; a Voting concept need not store the time at which a vote occurred, if all votes received before some deadline are treated equally.\nThere is a subtle qualification to this principle. Sometimes it is useful to enrich the state in ways that anticipate future functionality. Example: in a Moderation concept, there might be no actions that require storage of posts that have been rejected by a moderator, but it might be sensible to store them anyway if you anticipate adding, for example, functionality that recommends acceptance or rejection based on treatment of prior posts from the same user.\nRelation direction. The direction in which you declare a relation usually corresponds to the most common direction of lookup. Example: in the Folder concept, the relation between folders and the objects they contain would be declared as\ncontents: Folder lone -\u0026gt; set Object and not\nparent: Object -\u0026gt; lone Folder since you expect to navigate from a folder to its objects, not vice versa. But like tables in a relational database (and unlike mappings in a programming language), the direction of a relation doesn’t fundamentally matter, and you can always do it either way, so you long as you’re consistent in how you refer to it. Example: if an Invitation concept included the relation\ninv: Person -\u0026gt; Person you would have to make sure that you interpreted it consistently, so that p.inv means either the persons invited by p or the persons who have invited p, but not both!\nUnordered relationships. Mathematically, a relation is always directed, and corresponds to a set of ordered pairs or arrows between objects. To represent a symmetric, unordered relationship, you can define a symmetric relation that includes pairs in both directions. Example: the Friend concept may have a relation\nfriends: User -\u0026gt; User that contains both (u1, u2) and (u2, u1) when users u1 and u2 are friends.\nMost abstract structures. The state should not include structural decisions that needlessly complicate the state and limit the freedom of the implementer. In particular, don’t use a sequence (ordering some elements) when a set suffices. Example: you might think that the messages in a Group concept should be stored as a sequence, but since none of the actions involve sequence manipulations, and each message can appear only once, a set of messages is better. If you want the concept to contain enough information for messages to be displayed in order, you can associate a date with each message and assume the mapping of the concept to the user interface sorts messages by date.\nAbstract types. Don’t use a primitive type that has properties that aren’t necessary for the concept behavior. Example: a Ticket concept may generate identifiers for tickets which, in the implementation, will happen to be integers that are generated in order. In the concept design, however, all that matters is that the ticket identifiers are distinct, so you should introduce an abstract type TicketId to represent them rather than using Int.\nRicher abstract types. Another reason to introduce an abstract type is to record in the design that a type will have more properties than a primitive type. Example: rather than representing an email address as String, represent it as an abstract EmailAddress type, to capture the fact that email addresses have a particular form that can be validated.\nPolymorphism. In an app that you’re designing, a concept will often be motivated by a need that is related to a particular kind of object, but the concept itself will not rely on any properties of that object. Example: a Hold concept in a library app that governs how borrowers place holds on books need only associated borrowers with books, and doesn’t need the borrowers or the books to have any particular properties.\nTypes of objects that are not specific to the concept (and are generated by it) should be represented by type parameters, making the concept polymorphic or generic. Example: the Hold concept might be declared as Hold\u0026lt;User, Item\u0026gt;, where User and Item are type variables (that is placeholders for concrete types that will be provided when the concept is used).\nCarrier types. A concept will often introduce a type to represent a set of objects that is central to the concept. In this case, you can overload the name and use it for both the concept and the type, disambiguating them by context. Example: the Group concept has a set of groups of type Group; the Post concept has a set of posts of type Post; the Invitation concept has a set of invitations of type Invitation. When instantiating a polymorphic concept, you can refer to these types in a qualified name. Example: an Upvote concept is defined polymorphically over a generic type of Item (the items that are upvoted). The concept would be declared as Upvote\u0026lt;Item\u0026gt;, and can then be instantiated as Upvote\u0026lt;Post.Post\u0026gt; to indicate that, in this app, the target items are the posts of the Post concept.\nExample: the Hold\u0026lt;User, Item\u0026gt; from the library app might be instantiated as Hold\u0026lt;User.User, LibraryCatalog.Book\u0026gt; with a User type from a User concept and a Book type from a LibraryCatalog concept.\nNo external types. Concepts should be fully independent of one another. This is what makes them understandable to users and reusable across applications and even domains. So a concept’s state should never refer to types that are introduced in another concept. Example: an Upvote concept in a social media app should not refer to the type of posts of a Post concept through a state component such as\nconcept Upvote state upvotes: Post -\u0026gt; set Vote Instead, any type whose objects will (at runtime) come from another concept should be declared as a type parameter so that the concept is fully generic (see polymorphism above):\nconcept Upvote \u0026lt;Item\u0026gt; state upvotes: Item -\u0026gt; set Vote This principle will actually force you to introduce polymorphism whenever objects are to be shared between concepts, because there’s no other way to say that a type in one concept matches a type in another (except for the primitive types such as String).\nPrimitive types. The standard primitive types are assumed to be defined globally, so a concept can refer to types such as String, Integer, Bool, etc., and use their standard operations (such as concatening strings, adding integers) in defining actions.\nSequences. When an ordered collection of elements is needed, a sequence can be used. Example: in a Chatbot concept, the transcript is a sequence of queries and responses:\nQuery, Response: set Entry transcript: seq (Query + Response) (The plus is a union operator, so Query + Response is the set that includes both queries and responses.)\nConcepts aren’t classes. A common misunderstanding is to think of a concept as a class that has instances, with the concept’s state variables being the instance variables of the class. Example: you might write\nconcept User state friends: set User imagining a collection of user objects each of which points to a set of friends. This is wrong, and assumes a much more complicated set up in which types occurring in state declarations can themselves be concepts. The correct view is that a concept is a state machine whose non-primitive types denote objects only in the sense that they have identity, not in the object-oriented sense of carrying methods and so on. The correct way to write this concept is\nconcept Friend \u0026lt;User\u0026gt; state friends: User -\u0026gt; User which introduces a Friend concept that stores a friend graph.\nBackground Implementation bias. Explain origin in VDM. Response to critique of algebraicists. Example of statistics bag from Carroll Morgan.\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/state/","summary":"\u003ch2 id=\"motivation-defining-behavior\"\u003eMotivation: Defining Behavior\u003c/h2\u003e\n\u003cp\u003eThe operational principle of a concept explains its archetypal behavior: how it’s typically used, and how it fulfills its purpose. But for a concept to be flexible and powerful, it should work in many different scenarios, so we need a way to pin down the behavior in detail.\u003c/p\u003e\n\u003cp\u003eUsing the notion of a state machine, we can define how a concept will behave in every possible situation. Then we can analyze the design to make sure that the behavior is always acceptable. And as we transition from design to code, the state machine provides an ideal stepping stone. The state of the state machine will become the state of the program and the actions of the state machine will become the functions of its API.\u003c/p\u003e","title":"Concept state"},{"content":"Two sides of a coin The key to concept design is, perhaps not surprisingly, the idea of concepts. A concept is two things at once. On the one hand, a “concept” means you’d expect it to mean: a mental construct that you need to understand to be able to use an app effectively. So you might that to use Photoshop you need to understand the “concept of a layer”; to use Facebook you need to understand the “concept of friending”; and so on. But if concepts were just vague mental constructs, there wouldn’t be much you could do with them.\nOn the other hand, a concept is a coherent unit of functionality. In this sense, concepts belong to the long line of modularity mechanisms in software that include procedures, classes, abstract data types, processes, and so on.\nThe novelty of concepts comes in part from the way in which these two views are aligned. The concept as a mental construct is the same concept as the functional unit. The alignment of these two makes concepts a little unusual when viewed through these two perspectives.\nA new kind of mental construct Because the concept as a mental construct corresponds to a unit of function, it is inherently dynamic. Many people think of concepts as static, often as a form of classification in which a concept (“pond”, say) is defined as a set of objects (bodies of water, say) that have certain properties (small, still, land-based, etc). In concept design, however, concepts are defined by their dynamic behavior. A restaurant reservation isn’t a category of things in the world that you’d identify by classifying all possible things (dogs, car dealerships, headaches, etc) and then rejecting the ones that don’t match. Imagine a Martian appearing on Earth and asking you to explain what a restaurant reservation is. You wouldn’t explain how to distinguish one from other things (a social security number, a tax return). You’d explain it by saying what it’s for (to help diners so they can be confident a table will be available, and to help restaurant owners fill their tables) and how to use (call the restaurant, pick an available time, and then turn up at that time).\nThree parts This explanation has demonstrated three essential parts of a concept: the name (“restaurant reservation”), the purpose (ensuring a table will be available) and the operational principle (picking a time in advance and turning up at that time).\nA new kind of functional unit Just as the requirement to correspond to functionality makes a concept an unusual kind of mental construct, so the requirement to a mental construct makes a concept an unusual kind of functional unit. Most importantly, a mental construct has to be understandable without reference to other mental constructs. This property of independence is what makes mental constructs understandable.\nFor example, the concept of a “comment” in a social media app is easy to understand because you don’t need to know anything else. The purpose is for people to share their reactions to particular artifacts, and the operational principle is simply that you pick your artifact, write your thoughts, and they then appear associated with the artifact.\nIn practice of course that artifact that the comment is about will be a social media post, or perhaps a reply, or maybe even another comment. But you don’t need to know that to understand what the concept of a “comment” means; the concept is independent of whatever it’s attached to. In computer science lingo, the concept is “polymorphic.”\nWhy go to the trouble? The reason for defining concepts this way isn’t just to align the mental model and functional units views. It’s that the properties concepts end up having make them a particularly way to modularized function. In particular, the independence of concepts makes them more likely to be reusable in different contexts, not only at the design level but at the implementation level too.\nConcept Criteria Uses of criteria A more comprehensive list of criteria can be given for distinguishing concepts from other things. These criteria are useful for a variety of purposes:\nWhen learning concept design, they help you understand the difference between concepts and other kinds of software notions (such as entities, classes, features, microservices, etc). When analyzing an application, they help you identify coherent units of functionality that shape the app and characterize it. When designing an app, they help you recognize the larger units of functionality whose design can be pursued independently. The criteria Here is a list of criteria that every concept must satisfy:\nUser facing. Concepts are always experienced by the user of an application. So the structures that are only internal to an application and are hidden from the user are not concepts. Of course an implementation structure may support a user-facing concept. And programmers themselves should be viewed as users when considering the design of an API. Semantic. Concepts are made of static and dynamic structures that are abstract and semantic in nature. A user interface widget is not a concept, nor is a color scheme or a UI skin. Independent. A concept can be understood by itself without reference to other concepts. In contrast to classes in OOP, there are no “dependencies” that link one concept to another and prevent the former from being used without the latter. Behavioral. Every concept has a behavior associated with it. Usually the behavior is quite simple. For example, in the upvoting concept, when people upvote items, the items with the most upvotes rise to the top of the ranking. Applications can have much more complex behaviors, of course, but these are often due to the interactions of many small concepts. Purposive. A concept serves a particular purpose that is intelligible to its users and that brings real value by itself. Something pretending to be a purpose may just be a role played in a larger context. For example, the concept of a social security number has the purpose of allowing the government to associate pensions with individuals. The action of obtaining a social security number may seem at first to have a purpose (“getting a social security number”?) but on reflection you’ll realize that’s just a role it plays in the larger concept, and by itself it delivers no real value. End-to-end. A criterion related to being purposive and independent is that a concept’s functionality must be end-to-end. A user authentication concept, for example, couldn’t include actions for registering an account but not also include the authentication action itself. (To check that a concept is end-to-end, try writing an operational principle, and if it takes the form “when this action happens, the state is updated in this way” you’ll know something is missing.) Familiar. Most concepts are familiar to users. This is what makes it possible for users to start using a new app without reading a manual. Of course an app can have new concepts, but novelty can come even more a new combination of old concepts, or subtle adjustments to them. The key concepts used in social media (post, friend/follow, upvote, reply, comment, hashtag, etc) appear in almost an identical form in every app (Facebook, Twitter/X, Instagram), albeit with some small variations that may have significant impacts on usage (such as Instagram’s better support for images, or Twitter/X’s restricted post length). Reusable. As a consequence of many of these criteria (being purposive, embodying end-to-end functionality, independence, etc), concepts are often reusable in different contexts. For example, Zoom introduced the concept of a meeting identifier (whose purpose was to allow parties to join a meeting without requiring each participant to be called), but it was eventually copied by Google Meet and Microsoft Teams. Distinctions We can apply these criteria to distinguish concepts from some other software constructs:\nClasses. Classes in OOP (and similarly abstract data types) are not generally user-facing, and are rarely independent. Features. Application features are indeed user facing, but they are usually not independent. Typically a feature is defined in terms of some base functionality; without that, the feature may be hard to understand. For example, one feature of Instagram is that “accounts can be made private”; from a concept design point of view, this is an augmentation of the follower concept with an approval mechanism that some users can elect to use. User stories. In agile development, user stories are a popular way to organize requirements. A single user story, however, may not be end-to-end (or purposive). For example, the story “As a teacher, I would like to record which students in my class are present today” does not, by itself, bring any value. From a concept design point of view, the concept might be attendance, and it would cover not only daily entry of attendance data but also end-of-term summaries, and perhaps also repeated absence warnings, etc. Because user stories are not end-to-end, I am concerned that they cannot be properly evaluated and so are not appropriate increments of functionality for implementation. Microservices. Microservices are a popular way to structure large applications. But because they aggregate many distinct pieces of functionality with distinct purposes, and because they are not generally independent of one another, they are not usually reusable either. ","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/criteria/","summary":"\u003ch2 id=\"two-sides-of-a-coin\"\u003eTwo sides of a coin\u003c/h2\u003e\n\u003cp\u003eThe key to concept design is, perhaps not surprisingly, the idea of \u003cem\u003econcepts\u003c/em\u003e.\nA concept is two things at once. On the one hand, a “concept” means you’d expect it to mean: a mental construct that you need to understand to be able to use an app effectively. So you might that to use Photoshop you need to understand the “concept of a layer”; to use Facebook you need to understand the “concept of  friending”; and so on. But if concepts were just vague mental constructs, there wouldn’t be much you could do with them.\u003c/p\u003e","title":"Concept criteria: what's a concept?"},{"content":"In another tutorial, I showed a concept definition for Yellkey, a popular URL shortening service:\nconcept Yellkey purpose shorten URLs to common words principle if you register a URL u for t seconds and obtaining a shortening s, looking up s will yield u until the shortening expires t seconds from now state used: set String shortFor: used -\u0026gt; one URL expiry: used -\u0026gt; one Date const shorthands: set String actions // register URL u for t seconds // resulting in shortening s register (u: URL, t: int, out s: String) s in shorthands - used s.shortFor := u s.expiry := // t secs after now used += s // lookup shortening s and get u back lookup (s: String, out u: URL) s in used u := s.shortFor // shorthand s expires system expire (out s: String) s.expiry is before now used -= s s.shortFor := none s.expiry := none Although it’s fairly simple and straightforward, there’s something unsatisfying about this concept.\nThe criteria for concepts include being familiar and reusable. The essential functionality that this concept offers—shortening URLs—is common to many different shortening services, some freestanding (such as tinyurl.com) and others embedded in larger applications (eg, Google short URL option in Google Forms). But this concept would not be a good match for those other settings because it includes the expiration functionality.\nWhat’s going on here is that we seem to have two concepts: one about URL shortening and the other about expiry. Can we disentangle these?\nFactoring concepts Here’s how we might factor the Yellkey concept into two more basic and reusable concepts. First, we define a concept that provides shorthands:\nconcept Shorthand [Target] purpose provide access via shorthand strings principle after registering a target t and obtaining a shorthand s, looking up s will yield t: register (t, s); lookup (s, t') {t' = t} state used: set String shortFor: String -\u0026gt; opt Target const shorthands: set String actions register (t: Target, out s: String) s in shorthands - used s.shortFor := t used += s unregister (s: String) s in used used -= s s.shortFor := none lookup (s: String, out t: Target) s in used t := s.shortFor This is just like our Yellkey concept, but the expiry-tracking functionality has been removed. Note also that I’ve made the concept polymorphic: it no longer translates shorthands into URLs, but into objects of some unspecified type Target. This would allow the Shorthand concept to be used in many more contexts; you could have shorthands for file paths in a file system, for example, or for commands in a user interface.\nNow we turn the expiry-tracking functionality into its own concept:\nconcept ExpiringResource [Resource] purpose handle expiration of short-lived resources principle after allocating a resource r for t seconds, after t seconds the resource expires: allocate (r, t); expire (r) state active: set Resource expiry: Resource -\u0026gt; one Date actions allocate (r: Resource, t: int) r not in active active += r r.expiry := // t secs after now deallocate (r: Resource) r in active active -= r r.expiry := none renew (r: Resource, t: int) r in active r.expiry := // t secs after now system expire (out r: Resource) r in active r.expiry is before now active -= r r.expiry := none Again, I’ve made this polymorphic, so that any kind of resource can have its expiry tracked. This concept could be used to implement (the rather mean) control that turns off wifi access after the time paid for has passed.\nAnd because I’m anticipating using this concept in a more general setting, I’ve included two additional actions that let you deallocate a resource before it expires, or renew so that its life is extended.\nComposing concepts with syncs Now we need to put our two concepts together to recover the functionality of our original Yellkey concept. We do this by first instantiating the concepts with the appropriate types:\napp YellKey include HTTP include Shorthand [HTTP.URL] include ExpiringResource [HTTP.URL] I’ve included a concept called HTTP that I haven’t specified that defines the URL type. In a fuller description of Yellkey, this concept would allow us to augment the lookup with a redirect.\nWith the concepts in hand, we now associate their actions through synchronizations:\nsync register (url: URL, short: String, life: int) when Shorthand.register (url, short) ExpiringResource.allocate (short, life) sync expire (out short: String) when ExpiringResource.expire (short) Shorthand.unregister (short) sync lookup (short: String, url: URL) Shorthand.lookup (short, url) A synchronization (or “sync”) constrains the executions of the concepts so that whenever one particular action occurs in one concept, some other actions occur in other concepts. So the first one says that when a shorthand is registered (in the Shorthand concept) that same shorthand is allocated as a resource (in the ExpiringResource concept). The second one says that when a shorthand expires (in ExpiringResource) it is unregistered (in Shorthand). The third sync includes only one action; its effect is just to make the lookup action (in Shorthand) available as an application action.\nConcept actions that aren’t mentioned in syncs don’t occur in the application. Actions like Shorthand.unregister are excluded because Yellkey only lets you register shorthands and doesn’t let you unregister them. Similarly, you can’t renew a shorthand (even though the ExpiringResource includes an action to describe that functionality).\nConcept synergy In any design, composing concepts brings to whole (the overall app) the benefits of each of the parts (the constituent concepts). In some designs, something magical happens, and the benefit to the whole is more than the sum of the benefits of the parts. This is compositional synergy.\nSuch a synergy occurs here. You might think that composing with ExpiringResource would add only a limitation from the user’s perspective: that a resource that would otherwise be free is now limited. But in this case, there is a real benefit that comes with this limitation. Because the shorthands have brief lifetimes, it’s possible to serve the same size user base with a much smaller dictionary of possible shorthands (the set-valued shorthands component in Shorthand), and that means that the shorthands can all be familiar words.\nAnother example: user sessions Factoring Yellkey into more basic concepts may not seem very surprising since it’s not (yet at least) a design pattern that is used elsewhere. But sometimes even a concept that seems to be a widely used pattern can profitably be broken down into more elemental concepts.\nThe standard functionality that governs website sessions is such a case. Although we could easily describe it as a single concept, we can expose more structure (along with more opportunities for reuse) by treating it as a composition.\nFirst, let’s define the most basic form of user authentication in which a user registers with a username and password and can later authenticate by entering the same username and password:\nconcept User purpose authenticate users principle after a user registers with a username and password, they can authenticate as that user by providing a matching username and password: register (n, p, u); authenticate (n, p, u') {u' = u} state registered: set User username, password: registered -\u0026gt; one String actions register (n, p: String, out u: User) u not in registered registered += u u.username := n u.password := p authenticate (n, p: String, out u: User) u in registered u.username = n and u.password = p The state holds the set of registered users, and for each of them, a username and password. The register action allocates a user identity (which is an output of the action) and associates the provided name and password with it. The authenticate action has no effect on the state; its specification consists only of a precondition that the provided name and password match those of some registered user, whose identity is returned.\nNow we define a separate concept to represent session management:\nconcept Session [User] purpose authenticate user for extended period principle after a session starts (and before it ends), the getUser action returns the user identified at the start: start (u, s); getUser (s, u') {u' = u} state active: set Session user: active -\u0026gt; one User actions start (u: User, out s: Session) s not in active active += s s.user := u getUser (s: Session, out u: User) s in active u := s.user end (s: Session) active -= s The state here holds the set of active sessions, and for each one, the user associated with it. There’s an action to start a session, which takes a user and returns a fresh session identifier; an action to end a session; and an action (performed mid-session that obtains the associated user).\nNow we can assemble the conventional user session by synchronizing these two concepts:\nconcept UserSession include User include Session [User.User] sync register (username, password: String, out user: User) User.register (username, password, user) sync login (username, password: String, out user: User, out s: Session) when User.authenticate (username, password, user) Session.start (user, session) sync authenticate (s: Session, u: User) Session.getUser (s, u) sync logout (s: Session) Session.end (session) The actions of the composition are:\nregister: the user registers with a username and password; this is just the register action of the User concept; login: the user authenticates with a username and password (in the User concept) and a fresh session is allocated (in the Session concept); authenticate: the user is spontaneously authenticated mid-session by the execution of the getUser action of Session; logout: the user closes the session with the end action of Session. Concepts are abstract It’s important to understand that a concept describes a pattern of behaviors that users may observe. It may be implemented in a single computing device, but that is not necessary. The state may be distributed, with even a single component spread across nodes, and the actions likewise can be performed at different locations.\nIn a standard web stack, the User and Session states are both stored on the server. To enable the client to present a session identifier to the server, there is usually an additional piece of distributed state, which could be modeled like this:\ncookies: Client -\u0026gt; opt Session Each client is optionally associated with a session identifier; this information is typically stored in a cookie that the server sends to the client browser that is then set aside and sent back to the server with every request to its domain.\nActions likewise are abstractions, and a single action typically corresponds to a (potentially elaborate) sequence of steps. For example, when the user executes login, the following steps typically occur:\nThe user enters a username and a password in a web form and clicks the submit button; The client program running in the user’s browser makes an HTTP request to the server with this data; A controller method executes in the server, and requests are made to a database, often running on another machine; The session identifier is returned, encrypted, in a cookie to the client, which saves the cookie locally. Why more basic concepts are better To see why factoring this functionality into two concepts makes sense, consider some variant designs:\nBiometric authentication. Suppose the user logs in not with a username and password, but rather using some biometric property. To accommodate this, we can just swap out the password based authentication for a biometric authentication concept. The Session concept remains unchanged. Mid-session authentication. Some apps that have sessions still require an explicit user authentication in the middle of a session for extra security. For example, if you try to execute a large financial transaction on a banking app, you will typically be required to enter your username and password again even if you’re already logged in. This is easily accommodated with the existing authenticate action in User. If we hadn’t had a separate concept, we would have needed to add a new action. Non-session authentication. Some situations use authentication without sessions at all. For example, some apps let you unsubscribe from a service by clicking on a link that opens a browser but then requires you to enter your username and password to perform that one action. And operating systems often require you to authenticate for certain actions: MacOS requires authentication for opening apps for the first time, for example, and for any action that requires superuser status. This functionality would require the creation of the User concept if it hadn’t already been defined. Making sessions expire It’s regarded as good security practice to terminate sessions automatically after some time has passed. This is especially important when the app is running on a public machine, and if a user forgot to check out, the next user would be able to use their existing session.\nHow can we achieve this? You saw this coming: we can use our friend the ExpiringResource concept. All we need to do is treat sessions as resources, allocating them when sessions start, and ending sessions when the resources expire:\nconcept ExpiringUserSession include User include Session [User.User] include ExpiringResource [Session.Session] sync register (username, password: String, out user: User) User.register (username, password, user) sync login (username, password: String, out user: User, out s: Session) when User.authenticate (username, password, user) Session.start (user, session) ExpiringResource.allocate (session, 300) // set expiration to 5 mins sync logout (s: Session) when Session.end (session) ExpiringResource.deallocate (session) sync authenticate (s: Session, u: User) Session.getUser (s, u) sync terminate (s: Session) when ExpiringResource.expire (s) Session.end (s) ","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/sync/","summary":"\u003cp\u003eIn another \u003ca href=\"../concept-as-machine\"\u003etutorial\u003c/a\u003e, I showed a concept definition for \u003ca href=\"https://yellkey.com\"\u003eYellkey\u003c/a\u003e, a popular URL shortening service:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003econcept Yellkey\npurpose shorten URLs to common words\nprinciple\n  if you register a URL u for t seconds\n  and obtaining a shortening s, looking up s\n  will yield u until the shortening expires\n  t seconds from now\nstate\n  used: set String\n  shortFor: used -\u0026gt; one URL\n  expiry: used -\u0026gt; one Date    \n  const shorthands: set String\nactions\n  // register URL u for t seconds\n  // resulting in shortening s\n  register (u: URL, t: int, out s: String)\n    s in shorthands - used\n    s.shortFor := u\n    s.expiry := // t secs after now\n    used += s\n\n  // lookup shortening s and get u back\n  lookup (s: String, out u: URL)\n    s in used\n    u := s.shortFor\n\n  // shorthand s expires\n  system expire (out s: String)\n    s.expiry is before now\n    used -= s\n    s.shortFor := none\n    s.expiry := none\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAlthough it’s fairly simple and straightforward, there’s something unsatisfying about this concept.\u003c/p\u003e","title":"Concept composition and sync"},{"content":"An idea you need to know One of the most important and useful ideas in software also happens to be one of the least well known. It was introduced in a paper that David Parnas wrote in 1979, entitled Designing Software for Ease of Extension and Contraction.\nProgram families. Following on from an earlier paper that introduced the idea of “program families”\u0026mdash;that when you design software you should realize that you’re designing a whole family of programs, not just one\u0026mdash;this paper addresses the question of how to do this. How do you come up with one design that can be easily extended (by adding new functionality) or contracted (by removing functionality)?\nFour flaws. Parnas starts with four flaws that make programs hard to extend or contract. Two are particularly relevant to concept design. One is having a component that serves two different functions. Concept design avoids this by demanding specificity: that each concept has only one purpose. The other is the presence of excessive dependencies. Concept design avoids this by requiring that there are no dependencies between concepts.\nThe uses relation. Parnas then introduces his big idea as a way to address these problems. He calls it the “uses relation.” Today we’d call it a dependency diagram.\nA snarky remark. Even back in 1979, Parnas was aware of the fact that this idea was getting less attention than it deserved. He wrote rather snarkily:\nAfter studying a number of such systems, I have identified some simple concepts that can help programmers to design software so that subsets and extensions are more easily obtained. These concepts are simple if you think about software in the way suggested by this paper. Programmers do not commonly do so.\nIn writing this post, I hope to do my part in putting this right, albeit 40 years later!\nDefining subsets The dependency diagram. Here’s the idea. Suppose our program has components A, B, C, D, etc. We draw a graph with components as nodes and an edge from one component x to another component y if any member of the program family that includes x must also include y.\nFor example, in this diagram the edges from A to B and C mean that any program that includes A also includes B and C; and the edge from B to D means that any program including B includes D.\nThis diagram now implicitly defines a collection of subsets corresponding to each of the programs that might be built. So one subset, for example, is the program that contains only D; another contains B and D. On the other hand, B and C do not form a legitimate program, because any program containing B must also contain D.\nListing all the possible subsets, we get:\n{}, {C}, {D}, {B, D}, {C, D}, {B, C, D}, {A, B, C, D} (I’ve rather pedantically included the empty subset, since it obeys the rules, but admittedly it’s not a very useful program.)\nConcept subsets Let’s apply this idea to concepts. Suppose we have a social media app that has the following concepts:\nPost: share content with others Comment: share reactions to content Friend: limit access to your content User: authenticate a participant Now let’s construct the diagram by taking each of these concepts in turn and asking which other concepts it would require.\nThis question is more subtle than it first appears to be, because it forces you to think about variants of a program that might be unfamiliar.\nSo we start with Post. Now you might assume that a post needs a user to author it. But that’s not true. It would be perfectly possible to have an app in which someone can create a post (and even write in their name as the author) without any kind of user authentication. Admittedly, this might not work too well for a large-scale social media app, but it might be just fine for a noticeboard inside an organization. (A puzzle for curious readers: what would the consequences be of not having a User concept if the Post concept included an edit action?)\nThe Comment concept is simpler. It clearly requires the Post concept, because if there are no posts, there’s nothing to comment on!\nHow about Friend? You might imagine that Friend could be present without Post, in an app that lets you navigate a graph of friendships independently of any content. And indeed, that’s how friending worked in the earliest social media apps (notably Friendster).\nBut the Friend concept that we’re familiar with (from Facebook) is a very different beast. Its purpose is to not support navigation through the graph of connections but to control access to content (and also to filter what content you see, although that’s arguably a different purpose that should be separated out). Moreover, if the Friend concept were to play the role it played in apps like Friendster, we’d need an additional concept to hold information about a user (such as Profile or HomePage), but no such concept is present in our design.\nConsequently, Friend must require Post, because without content there can be no access to control. Friend also requires User, because it can’t work without authors of posts being authenticated.\nFinally, User itself requires Friend, because without Friend there is no reason to include User.\nThe resulting diagram looks like this:\nUsing the dependency diagram So now you have a dependency diagram, what can you do with it?\nClarifying the role of concepts. First, let’s recognize that the construction of the diagram alone forced us to think hard about the role the concepts play in our design. If we’d just thrown the concepts together and not considered their dependencies, we would surely have realized that Comment builds on Post, but we might not have understood how Friend and Post are related.\nSubsets: a family of programs. One dependence diagram defines a collection of possible subsets. For our social media app, we have the following subsets\n1. {Post} 2. {Comment, Post} 3. {User, Friend, Post} 4. {User, Friend, Post, Comment} each corresponding to a different app: (1) a noticeboard app in which people can post anonymous or unauthenticated messages; (2) the same app with comments; (3) a rudimentary social media app with user authentication and a friend network; (4) the same app with comments.\nExplanation order. An advantage of using familiar concepts is that they don’t need to be explained to most users. But there will always be a need for training material that explains how to use an app, especially when the app is complex or contains novel concepts. Because concepts are freestanding and can be understood independently of one another, they can be presented separately.\nBut if you’re explaining concepts in some order, or encouraging users to learn them in some order, the diagram can suggest orders that make the most sense. The rule is that if concept A requires concept B, you want to explain B before A. So that means that an order such as\n\u0026lt;Post, Comment, Friend\u0026gt; is good, but\n\u0026lt;Friend, Comment, Post\u0026gt; is less good, because the Friend concept isn’t motivated until you see Post (when you can explain that the idea is to share posts with your friends), and likewise Comment doesn’t make sense until you’ve seen Post.\nNote, by the way, that the mutual relationship between Friend and User means that there’s no order of one concept at a time that satisfies the rule: those two have to be explained together.\nIntrinsic dependencies Concrete descriptions. Let’s move to thinking of software components not in terms of abstract functionality but in terms of their description, whether some text describing a component at the design level, or some code that implements a component.\nIntrinsic dependencies. Now the possibility arises that a component contains an explicit reference to another component. At the abstract level, its design mentions the other component; or at the code level, it makes a call to it (for example). I call such a dependency an “intrinsic dependency” because it’s part of the component itself rather than a property of the overall design. When is such a dependency legitimate?\nParnas’s rule. In his paper, Parnas gives the following rule: if a component A has an intrinsic dependency on a component B, then there should be no plausible subset that contains A but not B.\nThat sounds complicated, but it’s really very simple. Putting it more simply:\nIf you can’t use A without B, you should never want to use A without B. The uses relation. True confessions: Parnas doesn’t start with the interpretation of the diagram that I started with. In fact, he follows exactly the reverse path. He suggests that you construct a “uses relation” with an edge from A to B whenever A uses B (that is, has an intrinsic dependency on it). The diagram you obtain from this then defines the subsets that are actually possible; you then check it to see if they are desirable (or more accurately that some important subsets are not possible).\nExample of bad code. For example, suppose we were building our social media app in an object-oriented style, and we defined two classes Post and Comment that implemented those concepts. Now suppose that each post contained a list of the associated comments:\nclass Post { List\u0026lt;Comment\u0026gt; comments; ... } This is, in fact, the way many inexperienced programmers might write this code. Notice that this introduces an intrinsic dependency of Post on Comment. According to Parnas’s rule, that means there should be no variant of the app in which there are posts but no comments.\nClearly that’s absurd! The intrinsic dependency is exactly the wrong way round. If there were to be a dependency, it should be of Comment on Post, and not Post on Comment.\nUnfortunately, achieving that is not so easy in OOP. The problem is that dependencies tend to follow the direction of data navigation, and since the programmer will likely want to go from a post to its associated comments (so they can displayed along with the post), the dependency goes in that direction too.\nThere are various ways around this in OOP, but that’s another subject. So I’ll just point out here that the problem doesn’t arise in a relational setting, in which the Post and Comment concepts are implemented as modules each with their own database tables. The Comment concept can then hold a table mapping posts to comments, and navigating from a post to its comments just involves performing a relational join.\nApplying this to concepts. Back to concepts. Unlike many other kinds of software modules, concepts are always independent, so there are no intrinsic dependencies. That’s why I started with subset interpretation of the dependency diagram: it’s the only one that matters.\nIndependent? What?. Aren’t I contradicting myself? I’ve just said that concepts are always mutually independent, but doesn’t the dependency diagram show dependencies? The resolution of this apparent contradiction is that we’re talking about two different kinds of dependencies. Concepts have no intrinsic dependencies, but they have dependencies in the context of the application design.\nSo the Comment concept has no intrinsic dependence on the Post. To understand comments, you don’t need to know about posts. This is achieved by describing the Comment concept generically in terms of some arbitrary targets that comments can point to; they could be posts, but they could also be newspaper columns or answers in a Q\u0026amp;A forum. In computer science lingo, Comment is polymorphic.\nBut in the context of our social media app, including the Comment concept makes no sense without the Post concept, because there aren’t any other suitable targets for the comments. So there is an extrinsic dependency of Comment on Post, but not an intrinsic one.\nA parting thought. Lest you imagine that intrinsic dependencies are simple, let me make your life a bit harder. When you try to pin down exactly what it means for one component to “use” another, most simple attempts fail. You might start by saying that A uses B if A makes a call to B. But then a round-robin scheduler that calls each of its tasks in turn “uses” those tasks (even though it’s actually the tasks that depend on the scheduler). In a paper I wrote with Jimmy Koppel, we present a collection of such puzzles, and suggest some ideas for a more robust notion of intrinsic dependency.\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/dependency/","summary":"\u003ch2 id=\"an-idea-you-need-to-know\"\u003eAn idea you need to know\u003c/h2\u003e\n\u003cp\u003eOne of the most important and useful ideas in software also happens to be one of the least well known. It was introduced in a \u003ca href=\"https://ieeexplore.ieee.org/document/1702607\"\u003epaper\u003c/a\u003e that \u003ca href=\"https://en.wikipedia.org/wiki/David_Parnas\"\u003eDavid Parnas\u003c/a\u003e wrote in 1979, entitled \u003cem\u003eDesigning Software for Ease of Extension and Contraction\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProgram families\u003c/strong\u003e. Following on from an earlier paper that introduced the idea of “program families”\u0026mdash;that when you design software you should realize that you’re designing a whole family of programs, not just one\u0026mdash;this paper addresses the question of how to do this. How do you come up with one design that can be easily extended (by adding new functionality) or contracted (by removing functionality)?\u003c/p\u003e","title":"Concept dependencies and subsets"},{"content":"Why modularity matters To see why modularity is so critical in software design, consider what goes wrong when it’s missing.\nFor the programmer, life becomes miserable. Whenever you need to understand or modify some piece of functionality, there’s no single place in the code to go to, and you may have to visit many widely separated locations (and just finding them can be a major challenge). Even removing (let alone adding) functionality is hard, because the parts of the codebase are all interconnected in subtle ways, and taking anything away risks breaking the parts that remain.\nFor the user, things are bad too. It’s possible that the code has no modularity but that, viewed from the user’s perspective, the app is perfectly modular. But this is unlikely, and poor modularity in the code is usually reflects a lack of structure in the app’s functionality. Consequently, users have trouble understanding how the software works, because the individual functions aren’t separable. Everything is interconnected, and to understand one part you have to understand another, and then another\u0026hellip;\nModularity: the holy grail of software Not surprisingly, then, achieving modularity has been a central goal of programming practice, and a primary motivation in software engineering and programming language research for decades.\nWe’ve invented a host of new abstractions (functions, closures, abstract types, objects/classes, streams, etc), new programming paradigms (functional, object-oriented, aspect-oriented, subject-oriented, role-oriented, process-oriented, reactive, prototype-based, actor-based, \u0026hellip;), principles (information hiding, representation independence, the Liskov substitution principle, the Law of Demeter), and patterns (for object-oriented code, enterprise architectures and more)\u0026mdash;all of which are motivated by improvements in modularity.\nAnd it’s not just in software; modularity may be the most valuable asset in design.\nWhere concept design comes in The new idea in concept design is that modularity is born not in the code, but in the function of the software. If you can structure an app’s functionality in a modular way, then you can carry over that modularity to the code.\nAnd because the function is modular, users will be able to understand it in a modular way. This is essential for usability, because it lets users learn how to use an app in pieces incrementally, and reuse their knowledge across systems.\nA concept is a bundling of functionality that meets three modularity criteria:\nSpecificity: The concept fulfills a specific purpose that brings real value to the user, and does not mix, multiple separable purposes. Completeness: The concept’s functionality meets its purpose fully. Independence: The concept stands by itself, without reference to other concepts. Let’s now look at each of these criteria in turn, see why they’re needed, how they might be violated, and what you have to do make them hold.\nSpecificity Having a specific purpose includes having enough function to bring real value, but not so much that separable purposes are being mixed.\nFor example, consider a User concept whose purpose is to provide authentication of users. Almost all web-based apps include this concept. Suppose the concept just included registration, in which a user creates an identity by providing a username and a password.\nThis might be regarded as a reasonable increment of functionality in some approaches. As a user story, it might be expressed like this: “As a customer, I can create register as a user so that I can go shopping.”\nBut in fact registering brings no real value to the user, because without the functionality associated with the subsequent authentication check, registration does nothing useful. The value that the user receives (which is more about other users bring prevented from acting on their behalf) is provided by the authentication mechanism in its entirety. So the purpose must be user authentication, and not user registration alone.\nThis would be apparent if you tried to write an operational principle:\nconcept User purpose help users register for services principle after a user registers... ??? actions register (name, password: String, out u: User) There’s no way to complete the scenario because there aren’t any other actions to include! And when you start trying to figure out what action you might add, you’ll see that the one you need is the authentication checking action.\nYou might be tempted to expand the purpose to include all user-specific settings. For example, perhaps each user has a display name, and an avatar. These are not relevant to the purpose of authentication, however; they serve a different purpose (letting users choose how to present their identities to others), and should be part of a different concept.\nEnsuring that a concept has a single, specific purpose makes it more understandable. If User included the avatar feature, you might start wondering whether that plays a role in authentication somehow. And it makes it more reusable by unbundling features: you might want authentication without avatars, for example.\nIn practice, formulating a suitably narrow purpose can be tricky, and often the easiest way to do it is to consider which particular features would be included in the concept’s functionality. Then, as you realize that certain features should be included (both registering and checking) and that others should be excluded (display names and avatars), you can refine your purpose accordingly (in this case, that the purpose is simply authentication and nothing more).\nCompleteness Completeness just means that the concept includes all the functionality needed to fulfill the purpose. So if the purpose of User is authentication, then offering registering without checking is obviously insufficient.\nA concept may fail to be complete for more subtle reasons. Suppose our User concept supported checking by offering an action that, given a user id, username and password, returns an indication of whether or not the username and password are correct for that user. This design fails to fulfill the purpose completely, because when the check is applied, the identity of the purported user isn\u0026rsquo;t known! Instead, the concept must include a lookup, for example by having the action take the username and password and return the user identity (if a matching one exists).\nThis example might seem silly and pedantic, but it illustrates a key difference between concepts and classes in object-oriented languages. In an object-oriented implementation of user authentication, you might have a User class with each instance holding a username and password. The functionality I just mentioned would be exactly what such a class provides, and it wouldn’t support the lookup. For that, you’d need to add a static component to the class that maps usernames to User objects. Using static state isn’t really in the spirit of object-oriented programming through, so you might instead create a Registry class each instance of which holds such a mapping. Thus a single concept becomes several classes that are tightly coupled together.\nFinally, it’s worth noting that completeness only means that the purpose is minimally satisfied. There are always additional features that might be desirable; for example, it would be useful for users to be able to change their passwords (and even their usernames).\nBeing complete ensures that all the functionality associated with the concept’s purpose is in one place, encapsulated within the concept.\nIndependence A concept not only has to meet its purpose completely, but it has to do it without the help of other concepts.\nRecognizing that storing and checking passwords involves some non-trivial functionality (notably encrypting and salting), you might think to put it in its own concept.\nThat would be mistake, however. First, encrypting and salting passwords is not user-facing functionality, so it doesn’t make a good concept. Second, suppose you did factor it out, say into a Password concept, assuming the User concept could delegate some functionality to it. Now your User concept would fail the independence test, since it would require the Password concept to operate.\nConcepts have no dependencies of this sort, and have to self-contained. As a result a concept can’t fail because of a bug in another concept, and functionality is truly localized.\nYou may wonder whether this rule undermines some modularity in the code. It doesn’t, because nothing prevents you from implementing password salting and encryption in a separate module, which can be one of several modules realizing the concept.\nConcept implementations will also inevitably depend on libraries for general services (such as string manipulation, arithmetic, file management, etc). So concept structuring won’t spare you entirely from the problem that fixing a bug in a function may require looking in multiple places. For example, your password encryption might fail because of a bug in a mathematical library.\nBut, in contrast to conventional design approaches, there won’t be pieces of connected functionality at the same level in scattered locations (for example, passwords being encrypted in one place and decrypted in another, far away).\nGenericity A concept may be functionally independent of other concepts while still mentioning them in its design.\nHere, for example is a very rudimentary Email concept:\nconcept Email purpose exchange of messages principle after a user sends a message to another user, they can receive that message state from, to: Msg -\u0026gt; one User body: Msg -\u0026gt; one String actions send (from, to: User, body: String, out m: Msg) recv (u: User, m: Msg) This is obviously very simplified; the state is a global set of messages each of which is associated with the users it’s from and to, and some body. A more realistic description might include state components for holding sent and received messages, and an explicit system action for transferring messages from one to the other.\nHere’s another concept, commonly used in mail clients:\nconcept Label purpose organize and filter messages principle after adding a label to a message m, if you get messages with that label, the message m will be included in the results state labels: Msg -\u0026gt; set String actions addLabel (msg: Msg, l: String) // return messages that have all labels in ls getMessages (ls: set String, out msgs: set Msg) Again, this is obviously simplified: the concept would usually support rich queries over label combinations.\nNotice that the Label concept appears to depend on the Email concept, because it mentions the Msg type that it generates. But if you think about the behavior of Label, since it involves only the identity of the messages (and not their content), the fact that the labeled items are messages is completely irrelevant. It matters only in understanding the role that the concept plays in the application as a whole.\nSo to make this clear and improve the concept, we can eliminate the reference to messages, and make it generic over some unspecified Item type:\nconcept Label [Item] purpose organize and filter items principle after adding a label to an item i, if you get items with that label, the item i will be included in the results state labels: Item -\u0026gt; set String actions addLabel (i: Item, l: String) // return items that have all labels in ls getItems (ls: set String, out items: set Item) This is much better because it makes it clear that Label is a more general concept that allows the organization and filtering of any kind of item.\nMore subtly, our Email concept has the same problem, because it refers to the type User, which presumably is the type of user objects generated by our User concept. But again, the concept does not depend on the user objects having any particular properties, so we can make the User type a parameter:\nconcept Email [User] In this case, we could keep the references to “users” in the description of the concept, but if we wanted to be really pedantic we could replace it by a name (such as Principal or Account) that makes it clearer that a human user might not be involved.\nWith all these generic concepts in hand, to describe our application we need to instantiate the type parameters:\napp EmailClient includes User Email [User.User] Label [Email.Msg] This says that the type of users in the Email concept will be the User type from the User concept, and the type of the targets in the Label concept will be the Msg type from the Email concept.\nPermutation invariant: for experts We can make precise this idea that a type is generic with the notion of permutation invariance. Suppose there is some execution of the concept comprising a sequence of actions interleaved with the resulting states. Now imagine taking the elements of the set in a given type, for example all the user objects in the set User, and permuting them, swapping u0 and u1, swapping u2 and u5, and so on. If you applied this permutation to the execution, would the result also be a valid execution? This is just a fancy way of saying that all that matters is the identity of the objects. If the answer is yes, then this permuted type is being treated generically.\nThis reasoning will tell you that the Msg type in Label and the User type in Email are generic. But it will also tell you that the String type in both concepts is generic! That’s because the concepts don’t do any string-specific operations: they just store the strings. So the Email concept, for example, could be written even more generically like this\nconcept Email [User, Content] purpose exchange of messages principle after a user sends a message to another user, they can receive that message state from, to: Msg -\u0026gt; one User body: Msg -\u0026gt; one Content actions send (from, to: User, body: Content, out m: Msg) recv (u: User, m: Msg) making it clear that the concept works with any kind of content. For email this might seem a bit artificial, since email messages are always textual, But for a social media post concept the genericity is more important (since a post’s body might be text, image or a combination), and factoring out the content into specialized concepts (such as a concept for creating and formatting rich text) would be helpful.\nWow, this is hard! This may seem pretty challenging to you. And you’d be right. Designing modular concepts is not at all easy. But that’s because design is hard, and it’s even harder when you’re trying to be modular.\nWith practice, you’ll develop this skill. You’ll see immediately when a concept isn’t modular and what you need to do to fix it. And you’ll begin to think naturally in terms of modular concepts.\nAnd it’s worth the effort! Your designs will be more understandable, more focused and more \u0026hellip; well, modular. When you come to implement them, you’ll be able to translate them very directly into code and you’ll be unlikely to hit structural obstacles. And you’ll be spared all the refactoring that is often needed when programmers build an app before they’ve really understood what they’re trying to do.\n","permalink":"https://essenceofsoftware.com/tutorials/concept-basics/modularity/","summary":"\u003ch2 id=\"why-modularity-matters\"\u003eWhy modularity matters\u003c/h2\u003e\n\u003cp\u003eTo see why modularity is so critical in software design, consider what goes wrong when it’s missing.\u003c/p\u003e\n\u003cp\u003eFor the programmer, life becomes miserable. Whenever you need to understand or modify some piece of functionality, there’s no single place in the code to go to, and you may have to visit many widely separated locations (and just finding them can be a major challenge). Even removing (let alone adding) functionality is hard, because the parts of the codebase are all interconnected in subtle ways, and taking anything away risks breaking the parts that remain.\u003c/p\u003e","title":"Concept modularity"},{"content":"Workshop Schedule Time Activity Topics 10:00 Welcome Introductions, goals, schedule, book 10:10 A new modularity AI challenges, concept modularity 10:40 What software design is about Function, patterns, eliminating friction 11:15 Exercise 1 Functional design of sidecar app 11:45 Designing with concepts Action-state models, concerns, syncs 12:45 Exercise 2 Concept design of sidecar app 1:15 Demo Code generation and context 1:35 Conclusion Slido links What is your company’s posture on AI-based coding?\nWhat in your view is the major issue with AI-based coding?\nLLM background document on concept design Click to download\nSlides Exercise\nSidechat exercise Topics\nWelcome Challenges Modularity Designing Function Patterns Chunking Friction Behavior State Actions Concerns Syncs Conclusions Tiny survey Thanks so much for your feedback here\nNewsletter signup Sign up for my occasional newsletter here\n","permalink":"https://essenceofsoftware.com/workshops/alliances-2026/","summary":"CSAIL Alliances, April 2026","title":"CSAIL Alliances Concepts \u0026 AI Workshop"},{"content":" Designing function (4 mins)\nEliminating friction (7 mins)\nChunking behavior (5 mins)\nPatterns in software design (6 mins)\nThinking about behavior (6 mins)\nDefining actions (8 mins)\nModeling state (9 mins)\nSeparating concerns (8 mins)\n","permalink":"https://essenceofsoftware.com/workshops/autodesk-videos/","summary":"Short videos about concept design","title":"Autodesk Concept Design Tutorials"},{"content":"Summary. Concepts should not be confused with objects in object-oriented design. While objects model individual entities with identity (a universally applicable idea), object-oriented thinking goes further, and forces behavior, relationships, and multi-object actions into single objects, creating complexity and needless coupling. Many real-world actions operate over sets of entities or relationships between them, which objects model awkwardly. Concept design instead “chunks” behavior around a purpose, encapsulating related actions and relationships across multiple individuals. This separation improves clarity, modularity, and reuse, avoiding the pitfalls of object-oriented domain modeling.\nGetting confused by object orientation I\u0026rsquo;ve been surprised at how many people think concepts are objects (in the object-oriented sense). And I suspect that I\u0026rsquo;m largely to blame, for giving names to concepts that sound like objects—Post rather than Posting for example—and for not being forceful enough in explaining what\u0026rsquo;s wrong with objects.\nWhen I wrote my book, I underestimated how much object orientation has shaped how people think about software. I mistakenly thought I could mostly ignore objects and that it was better just to make the case for a new idea. And I didn\u0026rsquo;t want to pick fights. So I relegated my explanations of why concepts aren\u0026rsquo;t objects to two page-long end notes (on p.246 and p.275) and left it at that. Perhaps I should have guessed that almost nobody would read the end notes, even though many of the key ideas of the book are there (including my favorite pasta recipe!).\nAs I\u0026rsquo;ve seen people use concept design over the last few years, in industry and in my classes at MIT, I\u0026rsquo;ve come to see how object orientation leads them astray. I came to realize it was time to be more blunt about why concept design isn\u0026rsquo;t object oriented, to explain how concepts address some of the flaws of objects, and to show what goes wrong if you don\u0026rsquo;t understand these things.\nI need to explain first what I mean by \u0026lsquo;object oriented.\u0026rsquo; It\u0026rsquo;s a term with a rich history that has come to mean different things to different people. This piece has turned out to be much longer than I\u0026rsquo;d planned, in part because we\u0026rsquo;ll need to disentangle several distinct ideas.\nObjects as Individuals First is the idea that the world can be modeled as a set of \u0026lsquo;objects\u0026rsquo; that have persistent identity. In this view, I\u0026rsquo;m an object, you\u0026rsquo;re an object, this post is an object, the machine you\u0026rsquo;re reading it on is an object, and so on. Because these objects have identities distinct from their properties, it makes sense to talk about how they change over time. You get older as time passes; the post may be edited; your computer breaks and is repaired. This notion of object is important in software because we need to track these identities (by generating explicit identifiers and ensuring that they point to objects unambiguously) and to have a firm grasp on how representations of objects in code are related to objects in the real world. We need to be clear, for example, about the difference between the object that represents my book Essence of Software and the object that represents the particular copy that I just looked at to find those end notes.\nThis notion of object was known to philosophers long before computing, but it has been popularized by software developers and computer scientists, and we\u0026rsquo;ve contributed to explaining and clarifying it. Jean-Raymond Abrial\u0026rsquo;s 1974 paper entitled Data Semantics, which is often cited as a key influence on object orientation, talks explicitly about the problem of mapping identities between the world and software. Peter Chen\u0026rsquo;s invention of entity relationship diagrams in 1976 laid the foundations of modern data modeling, with the idea that the state of the world could be represented as a collection of object sets (called entities) and relationships between them.\nThis notion is just as essential to concept design as to any other software development approach. It doesn\u0026rsquo;t require thinking of objects as composite structures that have \u0026lsquo;values\u0026rsquo; distinct from their \u0026lsquo;identities.\u0026rsquo; Better to take the relational view that an object is represented by nothing more than its identity, and can be associated with other objects and with non-object values by relationships (or mathematically, relations). Thus the fact that Alice and Bob are friends can be represented by a relation called friend that includes the pair (Alice, Bob); and the fact that Alice is 25 years old can be represented by a relation age that includes the pair (Alice, 25), and there\u0026rsquo;s no fundamental difference between these two cases.\nObjects as Machines The second idea is the one embodied by early object-oriented programming languages, first Simula and later Smalltalk. In these languages, an object is a little machine that has its own state and interacts with the outside through \u0026lsquo;methods.\u0026rsquo; A counter, for example, might have a state that is just a number, initialized at zero, and methods increment, reset, and display. By including so-called \u0026lsquo;observer methods\u0026rsquo; you can keep the state secret to the object; to know the current count, you just call display and don\u0026rsquo;t need to see the counter variable.\nImplicit in this idea is the assumption that objects are formed into a system by calling each others\u0026rsquo; methods, or \u0026lsquo;sending messages\u0026rsquo; in Smalltalk lingo. In practice, this means that objects tend to be intertwined with one another, with a method call to one setting of a cascade of method calls through a chain of objects. This introduces undesirable dependencies between objects, because it means that if one method fails (or even just returns an unexpected result) the method that calls it might fail too.\nMuch of object-oriented programming therefore has been concerned with mitigating these dependencies. The influential Gang of Four book catalogued dozens of design patterns, mostly concerned with reducing dependencies, often by introducing elaborate structures. These structures—factories, proxies, observers, etc—are often ingenious and elegant, but they undoubtedly contribute to the complexity and lack of legibility of modern software (more on that in an upcoming post).\nThis notion of object, like the first, is compatible with concept design, so long as it\u0026rsquo;s used strictly as an implementation technique rather than a domain-modeling principle. Concept design uses action synchronization to eliminate dependencies, which can be seen as a particular implementation pattern (related to the patterns used in many event-driven architectures) that is readily coded in object-oriented languages.\nObjects as Datatypes A third notion of object orientation is that sets of objects grouped into classes represent abstract data types. Data abstraction emerged in the 1970s and 80s, and is one of the most powerful ideas in programming. The easiest way to understand it is through Barbara Liskov\u0026rsquo;s observation that early programming languages provided a collection of built-in datatypes—booleans, numbers, strings, and so on—each with their own operations. Except in low-level languages like C, you couldn\u0026rsquo;t see how the values of these types were represented: an integer was an integer, not a sequence of bits, and if you wanted to know something about its value, you had to use the standard integer operations (for example, equals and greater than). That\u0026rsquo;s good.\nLess good is that you couldn\u0026rsquo;t extend the collection of datatypes with a new datatype. For example, you could build dates using records with year, month and day fields, but your dates would look different from built-in types like integers. Liskov argued that the programming language should let you define new types, so you could define a type Date that had its own operations and, like the built-in types, could be used without knowledge of its representation.\nThis idea is orthogonal to the idea of objects. But Liskov aligned the two ideas in her language CLU (1977), resulting in a very elegant programming model. There was no distinction between values and objects; everything was an object, including the integer 3. (In this respect, she was following Smalltalk. Java unfortunately followed C++ in treating primitives and objects differently, creating a mess that despite attempts to patch things up later marred the language for ever.)\nOther languages, such as SML, provided data abstraction without objects. Whereas object-oriented languages (like Java) achieve representation hiding by the simple rule that only the methods of an object can access it, these other languages needed a different trick (notably assigning one type to a value inside a module and a different type outside).\nThe object-oriented scheme for data abstraction, despite its elegance, has its flaws. Chief amongst these is that methods privilege one object (often called \u0026lsquo;self\u0026rsquo; or \u0026rsquo;this\u0026rsquo;) that is the target of the method. This makes it hard to handle operations that involve more than one object. If you define a method for adding two complex numbers, for example, the implementation has to treat them differently, and will let you access the representation only of the first of the two. In a language like SML, you could treat them uniformly. (In fact, you could do that in CLU too, since it adopted a hybrid approach that included both object-oriented structuring and an explicit type discipline to distinguish internal and external views of a data type. As Tony Hoare said about Algol-60, CLU was not only an improvement on its predecessors, but also on nearly all its successors.)\nAnother flaw, or at least a complication, is that this approach seemed to require that some datatypes be \u0026lsquo;mutable.\u0026rsquo; A set, for example, might have an operation for adding an element. Now things get more complicated. The meaning of equality becomes subtle, since we have to distinguish between two sets that happen to have the same elements right now but are actually different sets. And the hiding of state becomes a liability, because you need to track when two variables (or object fields) are pointing to the same object, and how a call to a method on one object might result in a change in the value of another. Decades later, separation logic and languages like Rust were developed to cope with these challenges.\nIn short, data abstraction and object orientation are different ideas, and just happen to have been conflated in the history of programming languages. Whether there\u0026rsquo;s some deep synergy here, as Liskov might have argued, or merely a convenient overloading (in the concept design sense) is beside the point. To understand this topic more deeply, read William Cook\u0026rsquo;s thoughtful analysis in his 1990 paper Object-Oriented Programming Versus Abstract Data Types.\nObject-oriented Design Finally we come to the notion of object orientation that is most relevant at the design level, and which I\u0026rsquo;ll argue is misguided and the cause of much trouble.\nThis notion has its roots in a very compelling idea: that a software system can be constructed in two parts, one a model that tracks the state of the world outside, and the other a collection of queries on this model.\nMichael Jackson\u0026rsquo;s JSD method (1986) is the most systematic and elegant formulation of this idea. A banking system, for example, has a model that has an entity for each bank account; as customers perform deposits and withdrawals, the state of the entity is updated; a bank statement is then just a query on the state of the bank account object. Jackson modeled each entity as a long-lived process; this made it possible to handle real-time functionality using the same idea.\nThe Object Management Technique (OMT, 1991) applied this idea in the more conventional context of data modeling and databases, and was one of the sources of UML, a modeling language that has been largely abandoned but lives on in spirit.\nEric Evans\u0026rsquo;s Domain-driven Design (2003) augmented this idea with a collection of contemporary object-oriented patterns (and introduced more controversially the idea of bounded context, that the teams building services that work together should develop their own domain models independently).\nSo where\u0026rsquo;s the snag? It\u0026rsquo;s in the assumption, which often followed, that the required functionality of the system can be described purely (or even primarily) in terms of familiar domain objects. Let\u0026rsquo;s see what goes wrong when you try to do this.\nFunctions on more than object Some functions can be assigned to a single object: an increment of a counter, a deposit in a bank account, or a change of a user\u0026rsquo;s name. In each case, some action that happens in the world corresponds to a method that can be called on a single object, and results in a change to the state of that object (and no other). But many functions aren\u0026rsquo;t like this. A bank transfer, for example, updates the state of two accounts, so it can\u0026rsquo;t be represented by a method on just one.\nWhen you look closely, it turns out that many actions that seem to concern a single object actually involve the set of all objects of its class. Changing a user\u0026rsquo;s name might seem to be about just the one user object, but what if names are to be unique? Then the change can only be allowed if no other user has the new name. Or consider password authentication. At first sight, you might think it\u0026rsquo;s about individual users. But think about what happens when users are authenticated. The first step isn\u0026rsquo;t to check the password but to find the user associated with the given user name. After that, we can check whether the password matches the recorded password for that user. Finding which user is associated with a name isn\u0026rsquo;t a method of any user object, but involves querying the entire set of users to see which, if any, has that name.\nNow of course these functions that act on sets can be represented as methods of objects too, but to do so will require new kinds of collection objects. For user authentication, we\u0026rsquo;ll need an object that represents all the users of the system, for example.\nRelational functions Things get worse when we consider functions that are about relationships between objects. Most social media functions fall in this category: friending and following relate users to each other; upvoting, bookmarking and favoriting relate users to posts; commenting relates comments to posts; and so on. In any of these cases, trying to assign an action to a method of one object is a fool\u0026rsquo;s errand. Who should follow be assigned to, the following or followed user? Is upvoting a method of a user or a post?\nOne workaround for this problem is to introduce objects to model the relational links themselves: friend objects, upvote objects, bookmark objects, and so on. In data modeling this makes sense and can work well; for one thing, making relational tuples explicit as objects lets you attach other properties to them (for example, the date on which an upvote occurred).\nBut this doesn\u0026rsquo;t solve the modeling problem if we continue to insist that actions need to belong to objects. Sure you can model friending as the creation of a friend object, but how will you determine whether two users are friends? You\u0026rsquo;ll still need a collection object. And what are you going to call it, by the way? It\u0026rsquo;s not easy to find a compelling name, and that alone should make you suspicious. And if you\u0026rsquo;re tempted to think that you might just augment objects with some built-in collection features, consider how you\u0026rsquo;d deal with LinkedIn\u0026rsquo;s \u0026lsquo;degrees of connection\u0026rsquo; query that tells you how many friend steps you are away from someone.\nAggregates Even if you can assign an action to a single object, it may rely on other objects for its fulfillment. Suppose your online store has a cart object, with an action to add an item to a cart. So far so good. But if you model your cart as containing separate line item objects, now you have a problem: add is no longer a method on a single object. Either it involves creating a new line item, which is then added to the cart (say as cart.add (lineItem), or the action is invoked just by calling a cart method (say as cart.add (itemSku)). Either way, the two objects (the cart and line item) have become intertwined, and neither can exist without the other.\nDomain driven design deals with this problem by defining some objects as \u0026lsquo;aggregates\u0026rsquo; but as we\u0026rsquo;ll see this complication isn\u0026rsquo;t necessary, and arises only because of the insistence that the domain should be modeled with objects whose methods correspond to real-world actions.\nDirectionality and dependencies Two major limitations of object orientation have drawn the attention of researchers, and led to many proposals for new paradigms. One is that relationships between objects must be represented in the instance variables of the objects themselves, and these instance variables prescribe a direction of navigation.\nTake the relationship between posts and comments in a social media app. A novice would likely endow the post object with an instance variable that holds an array of comments on that post; this makes it easy to display comments along with a post. But as I note in my book, this is actually exactly the wrong way round. Since an app is more likely to have posts without comments than to have comments without posts, the reference should go the other way, with each comment holding the post that it targets. This improves the dependency situation, but now navigating from a post to its comments is no longer possible. In situations like this in many object-oriented codebases, you end up with links in both directions, which makes navigation easy but now introduces an invariant that will need to be maintained.\nIn short, relationships between objects result in a mass of complications. There have been many proposals to fix this (for example, with links that can be traversed in both directions) but none have been introduced into mainstream languages.\nSeparating Views I\u0026rsquo;ve left until last a problem with object orientation that is arguably the most serious, and which has been extensively explored by researchers. It\u0026rsquo;s that object orientation brings all the functionality associated with an object together in one place.\nConsider a user object in a typical web app, for example. The conventional way to construct such an object is to ask yourself: what do I know about users? what properties do users have? what actions can be performed on them? This leads to a conflation of all the different aspects of functionality that users are involved in. The very same user object will have instance variables for the user name, password, display name, bio—and maybe even karma points, lists of friends, upvoted posts, and so on.\nThese all belong to very different functions and should be separated. Friending and upvoting and karma clearly don\u0026rsquo;t belong together, and one can easily imagine apps that have subsets of these, so representing them all together makes an object that is fundamentally not reusable.\nEven if we look at properties that seem to be more basic to users—user name, password, display name, bio—we see conflation. The user name and password are for authentication; the display and bio are for profiles. By separating these, we can describe and even implement these functions entirely separately. My student Eagon Meng points out that even the user name and password can profitably be teased apart, with the user name being relevant only to naming and the password to authenticating that a person presenting themselves is in fact the user that has been named.\nAttempts to address this problem go back decades, and include mixins, subject-oriented programming, aspect-oriented programming and roles. In gaming, where separating out performance critical functions (such as moving things around) from other functions (such as trading belongings) is vital, a pattern called entity-component is widely used. It separates different views of objects and is similar in some ways to our standard implementation of concepts.\nChunking with Concepts Here\u0026rsquo;s a way to understand what\u0026rsquo;s going on. When we build software, we model three primary kinds of thing:\nindividuals, which have persistent identity; relationships between individuals; and actions that result in changes in relationships. For example, in a social media app, the individuals might be users, posts, comments and so on; the relationships are that some users are friends of others, that posts belong to users, that comments are about posts, and so on; and the actions are become a friend, create a post, add a comment, etc. If it helps, you can think of individuals as nouns, relationships as adjectives, and actions as verbs. These provide a vocabulary of behavior, but they don\u0026rsquo;t suggest any structuring principle for how to put behaviors together.\nIt\u0026rsquo;s well known that in any domain of expertise, what distinguishes experts from novices is their ability to think in terms of larger structures. Psychologists call this chunking. A musician doesn\u0026rsquo;t think of a composition in terms of individual notes, but in terms of bars, or motifs, or themes; harmony is understood in terms of chords or progressions, and large structures such as canons or fugues.\nIn design, patterns (introduced by Christopher Alexander) can be seen as a way to chunk design elements to give designers a language for talking about design at a higher level.\nHow should software behavior be chunked? What are the appropriate patterns that will allow software designers to think at a higher level? Concept design proposes a particular kind of pattern, namely the concept, that brings together all the behaviors associated with a particular purpose. So, for example, PasswordAuthentication is a concept that serves the purpose of authenticating users by storing secret passwords which are set when a user registers and provided again later to establish that the user is the same user who registered before.\nIn object-oriented design, the chunks are equated with the individuals themselves. The economy of this idea is very appealing: why introduce a new notion when an existing one will suffice? (Concept design gurus will feel a twinge of worry here, at the prospect of overloading of purposes.) And indeed the idea works very nicely in some cases. To model a social media group chat (as in WhatsApp, for example) it seems natural to have a GroupChat object that corresponds to each group, and whose methods support all the actions that users of the group chat perform: joining and leaving, posting and replying to messages, and so on.\nAs we\u0026rsquo;ve seen, however, this is actually the less common case, and more often the behaviors associated with an object can\u0026rsquo;t be modeled by the object itself. Take password authentication. It would be tempting to define a Password object in the hope that it might encapsulate password-related behaviors. But this won\u0026rsquo;t work. Suppose our object contains a user name and a hashed password string. This would allow a method to change a user\u0026rsquo;s password (just update the password string), and a method to check that a presented user name and password match the stored user name and hashed password. But that\u0026rsquo;s only part of the functionality. Given a user name, how will we find the appropriate password object? And what about updating the user name? That can\u0026rsquo;t be allowed if another password object contains the new user name, but that can\u0026rsquo;t be implemented in the scope of a single object. So we\u0026rsquo;ll need some other object, PasswordTable, say, that represents a set of user names and passwords. Nothing new here; these are the same issues I outlined above.\nThis is not an isolated example; it happens everywhere, and it leads to an elaborate collection of interrelated objects where it\u0026rsquo;s no longer clear which object embodies the behavior. With both PasswordTable and Password objects, we now have password-related behaviors split across multiple objects.\nIn contrast, in concept design the story is much simpler. All password-related behaviors will be encapsulated in the PasswordAuthentication concept, expressed as a collection of actions that take different combinations of arguments (users, user names, passwords, etc). The state of the concept will capture relationships between individuals (objects, if you like) and simple values. Users, for example, will be individuals. Passwords actually won\u0026rsquo;t be, because passwords are just strings, and what makes a particular string a password is that it\u0026rsquo;s the password of some user.\nBad smells: spotting an object-oriented concept Emerging concept designers are often confused by object orientation, and they write concepts as if they were class declarations. For example, you might see a concept that looks like this:\nconcept User\npurpose manage users\nstate\nuserid\nusername\npassword\nemail\ndisplayname\nactions\nregister (username, password)\neffects creates a new user\nauthenticate (username, password)\neffects returns true if the username and password match\nThe state suspiciously has no sets, and seems instead to list the properties of a single user. But the actions can\u0026rsquo;t be defined on this single user, and their specs sweep this difficulty under the rug. The authenticate action makes no sense: which user is to be compared with the given username and password? The presence of the email and displayname fields conflates user authentication with user profiles; this is why it\u0026rsquo;s not possible to write a coherent purpose for the concept.\nInstead the concept should have been written more like this:\nconcept PasswordAuthentication\npurpose authenticate users\nstate\na set of Users with\na username String\na password String\nactions\nregister (name: String, pass: String): (user: User)\neffects creates a new user with username name and password pass and returns it\nauthenticate (name: String, pass: String): (user: User)\nrequires some user with username name and password pass\neffects returns the matching user\nauthenticate (name: String, pass: String): (error: Error)\nrequires no user with username name and password pass\neffects returns an appropriate error message\nThe scope of the actions is now explicitly a set of users, each with a user name and password. There is no need for a userid field; each User individual implicitly has an identity. The profile fields have been removed, so the concept is solely about authentication. Profiling can be described in a separate concept.\nConcepts are expressive enough to describe an object-oriented structure when needed. Here\u0026rsquo;s a version that lets you group users into different scopes, corresponding to the example mentioned above. Note that there\u0026rsquo;s an action corresponding to a constructor, and all the other actions take the object (the table) as their first argument:\nconcept PasswordTable\npurpose authenticate users in different scopes\nstate\na set of Tables with\na set of Users\na set of Users with\na username String\na password String\nactions\nnew (): (table: Table)\neffects creates a fresh empty table\nregister (table: Table, name: String, pass: String): (user: User)\neffects creates a new user in the given table\nwith username name and password pass and returns it\nauthenticate (table: Table, name: String, pass: String): (user: User)\nrequires some user in table with username name and password pass\neffects returns the matching user\nauthenticate (table: Table, name: String, pass: String): (error: Error)\nrequires no user in table with username name and password pass\neffects returns an appropriate error message\nConclusions Object orientation is a powerful programming mechanism. But the apparent correspondence between objects and individuals in the world is misleading, and modeling the world as a collection of objects turns out to be far from straightforward. Like object orientation, concept design recognizes the identity of distinct individuals, but chunks behavior not around the individuals themselves but around more flexible abstractions that typically contain multiple individuals of different types.\n","permalink":"https://essenceofsoftware.com/posts/concepts-and-oop/","summary":"Object oriented modeling and its pitfalls.","title":"Why concepts aren't objects"},{"content":"Full paper\nDividing labor with LLMs. As LLMs get better at writing code, it seems inevitable that there will be less work for human programmers. Thomas Dohmke is right that low-level coding skills will matter less and that \u0026ldquo;the future belongs to developers who can model systems, anticipate edge cases, and translate ambiguity into structure—skills that AI can’t automate.\u0026rdquo;\nDohmke says \u0026ldquo;We need to teach abstraction, decomposition, and specification not just as pre-coding steps, but as the new coding.\u0026rdquo; As someone who has been teaching these things as the core of coding for a few decades now, influenced by colleagues and my own teachers, I can\u0026rsquo;t disagree with that.\nBut this \u0026ldquo;division of labor\u0026rdquo; viewpoint—the human makes the big, high-level decisions and the LLM fills in the details—may not be the best way to think about AI in coding. First, it ignores the fact that LLM capabilities will undoubtedly improve, so the line between what LLMs can do and what must be left to a human coder is constantly shifting. Second, it assumes that all tasks at a particular level are alike, when in fact some high level design might be amenable to LLMs, and some low level coding may require human skills.\nBack to basics. Instead, we might ask: how should we build software so that humans and LLMs can work more effectively together? Most observers realize that the advent of LLMs is going to produce radical changes in software development. What they may not have recognized is how, paradoxically, LLMs will force us to re-adopt classic software engineering principles that we\u0026rsquo;ve only paid lip service to in recent years.\nThe end of software design? When ChatGPT came out, I worried that AI would suck all the air out of the room and it\u0026rsquo;d be ages before anyone would care about software design again. But then it began to dawn on me that the success of AI will be more dependent on good software design than most people realize, and that the advent of AI is more likely to revive interest in software fundamentals. A new era for software design?\nLLM coders are amazing. Coding has turned out to be one of the major successes of LLMs. Their capabilities are spectacular. Now that LLMs can one-shot entire applications, it\u0026rsquo;s hard to remember that just a few years ago researchers were struggling to synthesize ten-line functions.\nHype and reality. But, as anyone who has used LLM coders knows, the reality is not as pretty as the hype would suggest. LLMs quickly hit a complexity barrier when asked to build a full application, and when they operate as programming partners in a real codebase they tend to fumble around making changes in arbitrary places, breaking existing functionality, introducing security vulnerabilities and corroding the quality of the code—making coding take more time, not less. In short, they behave just like novice programmers.\nToday\u0026rsquo;s playbook. Most companies in this sector are following the same playbook. They hope to overcome the problems by stacking up one agent on another with more and more elaborate guidance and iterative strategies, and by giving these agents more control and wider access. It\u0026rsquo;s possible that this will work, but I\u0026rsquo;m skeptical. Evidence so far suggests that granting more control merely increases the risk of catastrophic errors. Widening access makes it harder to localize changes (and maintain structure), and means more tokens read and higher costs to the user. It\u0026rsquo;s common for agentic tools to ingest the entire codebase or reprocess a file many times even to make a tiny change.\nA new approach. A different approach is possible, however. Rather than seeking to adapt LLMs to the messy reality of software development, we might recognize that LLMs—and their failure to work effectively in complex codebases—are exposing flaws in how we develop software. They are the canary in the mine, sending us a signal that all is not happy in the way we write code.\nThis raises exciting prospects. If we can improve how we develop software, we might make LLMs more effective, so they scale to large and more complex systems and become more reliable even as they get better themselves. If software design can give a factor of 10 improvement for today\u0026rsquo;s LLMs, we might reasonably assume it will do the same for tomorrow\u0026rsquo;s, so that however effective LLMs become, good software design will continue to amplify their capabilities. And the gains improved design brings to LLMs, it should bring for human developers too.\nThe key idea. So what are the design ideas in this new approach? Well, the key idea is one of the oldest (and best). If we could structure our software in a modular fashion, with each module independent of the others, we could generate code one module at a time. That means the LLM\u0026rsquo;s context can be just the one module, and not the whole codebase. And since important behavioral properties would be encapsulated within individual modules, those properties would, by definition, continue to hold when other modules are modified.\nNot so easy. In practice, this is easier said than done. Codebases that purport to be modular turn out to be less modular than you might have hoped. This happens partly because modern software engineering practice encourages complex interfaces, and partly because even if programmers start out with a modular design, they\u0026rsquo;re not inclined to maintain it. We\u0026rsquo;ve coined a term—\u0026ldquo;technical debt\u0026rdquo;—to dignify the practice of creating a mess with the naive hope of fixing it later.\nOOP to blame. Lack of modularity happens for a more fundamental and interesting reason too. Even though most codebases for large systems are not strictly object-oriented, they tend to follow the gestalt of OOP, with functionality grouped according to the key entities of the problem domain. A typical structure for a blogging app, for example, breaks functionality into \u0026ldquo;stacks\u0026rdquo; of modules corresponding to entities such as User, Article, Comment, etc (and then breaks each stack into layers for routing, business logic and data persistence). The implicit rationale is that all functionality about users (authentication, eg) goes in the User stack; all functionality about articles (creating, deleting, editing, etc) goes in the Article stack; and so on. Unfortunately, though, a lot of functionality can\u0026rsquo;t be neatly assigned to an entity. Take favoriting, for example, in which users mark some articles as favorites to make them easy to visit later. Does this functionality belong to the User stack or the Article stack? There\u0026rsquo;s no good answer to this, and in one codebase we looked at it was split across both!\nA concept-based approach. Concept design makes modularity much easier to achieve. Concepts are defined not by entities but by user-facing purposes. So different functions associated with users (authentication, naming, profiles, notifications, preferences, etc) would be factored into distinct concepts. And favoriting would have a natural place—in its own concept.\nConcepts are independent. Concepts are, by definition, completely independent of one another. They have no knowledge of the interfaces (let alone the internals) of other concepts. All references to shared objects are fully polymorphic, and mutable state is never shared across concepts. This means that each concept can be designed and implemented by itself.\nSynchronizations for composition. So if concepts are independent, how are they coordinated at runtime? That\u0026rsquo;s what synchronizations (or syncs) are for. A sync is a kind of mediator that ties together actions across concepts, and manages dataflow between them. Syncs are very granular, and they factor out application-specific behavioral details that might otherwise compromise the reusability of concepts. For example, in our blogging example, syncs might include interactions such as:\nOnly allow an article to be editing if the authenticated user is the author When an article is deleted, delete its comments too When a comment is deleted, notify the author of the comment Syncs do need to know about concepts, but they only need their specifications. What we\u0026rsquo;ve done. We started experimenting with this idea two years ago. Barish Namazov, at the time an undergraduate in my research group, built a prototype tool called Kodless that generated concept implementations from minimal concept specification prompts (short talk here). (In Kodless, syncs were implemented as routes (HTTP endpoints), so they weren\u0026rsquo;t so granular.) Barish demonstrated that this worked by generating a backend for a HackerNews clone.\nOver the last year, my PhD student Eagon Meng has been working on improving this scheme. He developed a patterns for expressing granular syncs, and implemented it first using SPARQL and a graph database, and them more recently as a lightweight Typescript library. He also define a simple concept specification language. To demonstrate this new approach, we applied it to RealWorld, a benchmark collection of implementations of a Medium-like blogging app called Conduit. Standard LLMs have little trouble generating the concept specs from minimal prompts, and then generating concept implementations from them. Syncs are a bit trickier and needed some iteration, but LLMs seem to have no trouble with this pattern despite it being a bit unconventional.\nWYSIWID. The key quality of this approach is that code becomes \u0026ldquo;legible\u0026rdquo;, with a more straightforward correspondence between the behaviors that the user cares about the modular structure of the code. Concepts are a new kind of module which, unlike traditional modules, are user-facing (always having a purpose that maps directly to some user needs), fully decoupled from one another, and that separate concerns more cleanly. Syncs allow application-specific details to be factored out and represented in a granular way. In short, what you see (in the code) is what it does (for the user). Together, concepts and syncs offer a new structural pattern that allows LLMs to be applied in a more incremental and focused way, reducing costs while increasing scalability and integrity.\nProgress so far. We\u0026rsquo;ve written a paper explaining all of this which will appear at SPLASH Onward! this fall. We\u0026rsquo;re teaching this approach to undergraduates in the Software Design class that I\u0026rsquo;m running with my colleague Mitchell Gordon at MIT this fall. We\u0026rsquo;ve been using concept design and implementation in the class for several years, so what\u0026rsquo;s new is not the structure and modularity but leveraging LLMs to generate code. As a dry run, we did a hackathon with the wonderful folks from the Sundai Club a few weeks ago. About 40 people came and they spent the day using a concept-infused version of Cursor to generate some really impressive little apps.\nProspects. We\u0026rsquo;re excited to see where this goes. One direction, in the spirit of low-code apps, would be a SAAS tool for non-experts that offers a library of reusable concepts, and synthesizes apps by selecting concepts and compiling application-specific syncs to glue them together. Another, aimed at developers, would be an agentic workflow that not only compiles concept specs but also helps write them. We\u0026rsquo;re also thinking hard about front ends, and how they might also be structured with concepts and syncs, eliminating the aggregation of cross-concept data that currently makes endpoints needlessly complicated. Whatever happens, it seems that software design is here to stay\u0026hellip;\n","permalink":"https://essenceofsoftware.com/posts/wysiwid/","summary":"How concepts enable a new approach to LLM coding.","title":"WYSIWID: What you see is what it does"},{"content":"Workshop Schedule Time Monday Tuesday 9:00 Welcome (slides, goals poll, challenge poll) Presentations (continued) 9:30 Introducing concepts (slides) Designing systems (slides) 11:00 Break Break 11:10 Exercises (slides, comparisons) Exercises (slides, reading, writing) 12:00 Lunch Lunch 1:00 Presentations Presentations 1:30 Designing concepts (slides) Autodesk exercises (slides, catalog, rubric) 3:15 Break Break 3:30 Exercises (slides, reading, writing) Autodesk exercises 4:30 Presentations Presentations 4:45 End of day reflections End of workshop reflections (survey) 5:00 Adjourn Adjourn Goals To give participants basic proficiency in concept design To train leaders for infusing concept design into Autodesk To help align participants in their design work, across roles \u0026amp; teams To get some experience applying concept design in a work project To encourage participants to reflect on their strategies when designing General structure The workshop will be on August 25 and 26 Each day will run from 9:00am to 5:00pm Each day will break for lunch from 12:00-1:00 Shorter breaks mid-morning and mid-afternoon Sessions will include presentations, discussions \u0026amp; exercises Agenda Monday am: Conceptual models \u0026amp; the idea of concepts Monday pm: Defining user-facing behavior in a UI-independent way Tuesday am: Composing concepts \u0026amp; modularity Tuesday pm: Project work (eg, on Unified Design concepts) Pre-reading In advance of the workshop, I’d like you to read some short pieces for some background:\nAn overview of concept design: Three Stages of Enlightenment (13 minute read) A brief explanation of how concept design fits in with other kinds of design: Levels of Design (3 minute read) A short piece about the spirit of design, and how great design happens: How Great Design Happens (4 minute read) I hope you enjoy these and find them helpful. There are several other pieces on the concept design website that you might find interesting, for example one about how LLMs will upend the principles of agile development. Homework In advance of the workshop, I\u0026rsquo;d like you to think a little about your own design work, and answer the following questions by submitting this form by August 10:\nWhat are some key strategies you use in design? What is a recent user-facing design problem that you\u0026rsquo;ve worked on? What was the problem and how did you solve it (or not)? What questions or concerns about concept design are you hoping the workshop will answer? ","permalink":"https://essenceofsoftware.com/workshops/autodesk-oslo/","summary":"Oslo, August 2025","title":"Autodesk Concept Design Workshop"},{"content":"One way to improve your understanding of concept design is to compare related concepts, and try to determine (and then articulate) how and why they differ.\nConfusion between related concepts can be a design flaw in itself, when the designer fails to convey (through the user interface and support materials) what concept is being offered in a particular situation.\nFor example, X/Twitter used to have a feature called \u0026ldquo;favorites,\u0026rdquo; but it wasn\u0026rsquo;t clear if this was an instance of an Upvoting concept (which collected votes from users for popularity ranking) or a Favoriting (or Bookmarking) concept (which let users save favorite tweets). In 2015, Twitter renamed \u0026ldquo;favorites\u0026rdquo; to \u0026ldquo;likes\u0026rdquo; and replaced the yellow star icon with a red heart, explaining in a press release \u0026ldquo;We are changing our star icon for favorites to a heart and we’ll be calling them likes… We know that at times the star could be confusing, especially to newcomers. You might like a lot of things, but not everything can be your favorite.\u0026rdquo; Needless to say, this didn\u0026rsquo;t help explain what concept was actually at play, and in 2017, Melania Trump famously \u0026ldquo;liked\u0026rdquo; a tweet that taunted her husband, presumably believing that she was bookmarking it, when in fact her \u0026ldquo;like\u0026rdquo; was shown publicly as a gesture of approval. In 2018, Twitter finally added a Bookmark concept.\nIn this case, it seems possible that the confusion was intentional, since public likes, unlike private bookmarks, serve the company\u0026rsquo;s business goals of increasing visible user engagement (and indeed, even after they were added, the bookmarking actions were buried in the user interface and far harder to access than the upvoting actions). In other cases, confusion between concepts is simply a consequence of poor design, and even with good design it is not possible to ensure complete clarity for all users given their diversity of background and assumptions.\nExercise: Style Concepts Most tools that offer formatting of text or graphical objects include concepts for styling them, which allow a user to apply formatting rules that were previously defined. There is a fundamental difference between a concept that merely stores formatting rules as \u0026ldquo;styles\u0026rdquo; and one that maintains associations between objects and their styles.\nTo explore this difference, start by experimenting with some of the following concepts, using whichever applications you have convenient access to or are familiar with, selecting at least one example from each category:\nCategory A\nParagraph styles in Microsoft Word, Apple Pages and Google Docs Color swatches in Adobe Indesign Site styles in Squarespace Selection colors in Figma Color themes in Visual Studio Code Shape styles in Apple Keynote Theme colors in Microsoft Powerpoint Category B\nColor palettes in the Apple color picker Text color selector in Google Docs Cell styles in Microsoft Excel Layer styles in Adobe Photoshop Shape styles in Microsoft Powerpoint Text styles in Apple TextEdit Now articulate the difference between these categories by formulating two concepts, one for each category. Make your concepts simple and generic; they do not need to capture all the details of a particular example. Point to the essential difference in behavior between the two concepts (in operational principle and actions). Do the two concepts share a purpose, or are their purposes different? Is one easier to implement than the other (hint: look at the concept state)?\nExercise: Passwords and Personal Access Tokens Github offers a form of authentication called personal access tokens. This is described as \u0026ldquo;an alternative to using passwords.\u0026rdquo; Creating a token sounds very much like creating a password with a strong password helper: you click a button to create a token, and get an obscure string which you can then save. Using a token appears to be the same as using a password: you enter a username and the token string, and will then be authenticated if the string matches the one that was generated when the token was created. Moreover, at the very start of the article we are told \u0026ldquo;Treat your access tokens like passwords.\u0026rdquo;\nSo what exactly is the difference between the standard Password concept and the PersonalAccessToken concept? Read the Github page carefully*, and document the two concepts, paying particular attention to their purposes and operational principles. Now consider how they differ, and explain how their behavior is motivated by different purposes. You will also want to consider whether one of the concepts has actions that are not provided by the other concept.\n* Note: consider only \u0026ldquo;personal access tokens (classic)\u0026rdquo; and not \u0026ldquo;fine-grained personal access tokens.\u0026rdquo;\n","permalink":"https://essenceofsoftware.com/exercises/concept-comparison/","summary":"Concept comparison exercise","title":"Concept comparison exercise"},{"content":"To help you get going with designing individual concepts, this exercise asks you to read a mostly complete concept and answer some questions about it.\nconcept GiftRegistry [User, Item]\npurpose track purchases of requested gifts\nprinciple\na recipient creates a registry, and adds items to it indicating the number of each requested; opens the registry so it becomes publicly visible; then givers can view which items are still available and purchase them; and finally the recipient closes the registry, after which it is no longer publicly visible but the recipient can see which items were purchased and by whom.\nstate\na set of Registrys with\nan owner User an active Flag a set of Requests a set of Requests with\nan Item a count Number a set of Purchases a set of Purchases with\na purchaser User an Item a count Number actions\ncreate (owner: User): (registry: Registry)\ncreate a new registry with this owner, active set to false and no requests addItem (registry: Registry, item: Item, count: Number)\nrequires registry exists effects if a request for this item exists, add the count\notherwise create a new request for the item with this count and add to registry removeItem (registry: Registry, item: Item)\nrequires a request for this item exists in the registry effects remove the request from the registry open (registry: Registry)\nrequires registry exists and it is not active effects make registry active close (registry: Registry)\nrequires registry exists and it is active effects make registry not active purchase (purchaser: User, registry: Registry, item: Item, count: Number)\nrequires registry exists, is active and has a request for this item with at least count effects create a new purchase for this purchaser, item and count and decrement\nthe count in the matching request Questions Invariants. What are two invariants (aka integrity constraints) of the state? What are their purposes? Is one more important than the other? Which actions are primarily responsible for preserving each of the invariants, and how do they do it? Fixing an action. Can you identify an action that potentially breaks one of these invariants, and explain how this might happen? Can you suggest two different ways in which it might be fixed, and give some reasons in favor and against each (pointing to how they might affect users)? Inferring behavior. The operational principle describes the typical scenario in which the registry is opened and eventually closed. But a concept specification often allows other scenarios. By reading the specs of the concept actions, determine whether a registry can be opened and closed repeatedly. What might be some reasons to allow this? Registry deletion. There is no action to delete a registry. Why in practice is this probably a good design decision? Queries. What are some common queries that you would expect to be executed against the concept state? Hiding purchases. A common feature of gift registries is to allow the recipient to choose not to see purchases so that an element of surprise is retained. How would you augment the concept specification to support this? Generic types. The User and Item types are specified as generic parameters. The Item type might be populated by SKU codes, for example. Explain why this is preferable to representing items with their names, descriptions, prices, etc. Assigning functionality. For each of these possible extensions, say whether supporting it should involve changes to this concept, synchronization with other concepts (which you should name) or both: Showing most recently purchased items Including gift messages from givers Grouping requests by category (kitchen, bath, etc) Notifying recipients when a purchase is made Incorporating requests for money rather than items Allowing givers to join together in purchasing an item ","permalink":"https://essenceofsoftware.com/exercises/concept-reading/","summary":"Concept reading exercise","title":"Concept reading exercise"},{"content":" criterion name criterion description evidence example failing Independence Concepts are fully independent of each other, and can therefore be understood and used independently of one another. Concept description limits any references to context of use to notes. Concept purpose mentions the way in which the concept is intended to be used (eg, a payment concept whose purpose says “enables payment for magazine subscription”). Concept does not refer to another concept by name. Concept mentions working in concert with another (eg, session concept says “works with authentication concept to provide authenticated sessions”). Concept does not rely on any properties of other concepts. Concept action “calls” an action of another concept or queries the state of another concept. All external datatypes are either generic parameters or built-in types (such as String). Concept treats arguments as objects that have been constructed elsewhere (eg, takes in a user object that is assumed to have a name field). Completeness Each concept provides a complete and coherent unit of functionality that delivers the value described in the purpose without the help of other concepts. Concept functionality covers entire lifecycle of the purpose. Concept doesn’t include actions for set up (eg, defining available slots for reservations), or for closing down (eg, no deletion for an account). Concept embodies real functionality that fulfills a compelling purpose. Concept is a data structure with CRUD actions when purpose calls for richer behavior (eg, concept holds contact info for a user but doesn’t include any notification behaviors). Concept state is rich enough to support all the concept actions. Concept state is expressed as the instance variables of a single object (eg, password auth concept that declares state as username and password, failing to support lookup by username needed to check password). Concept actions are sufficient to provide essential functionality to users. No action to allow users to undo the effects of prior actions (eg, to cancel a reservation). Separation of concerns Concept does not conflate two concerns that could be broken into separate concepts that could be reused independently of one another. All components of the state work together for a single purpose. The state admits a factoring into two or more independent parts (eg, a user concept mixes preferences and profile fields). No state component can be dropped without compromising essential functionality. The concept gratuitously includes state that is not needed to support actions (eg, a password authentication concept that stores, in addition to username and password, the date on which the user first joined). The concept does not include state components that could be easily expanded into much richer, self-contained structures. The concept contains references to external objects and stores properties of them that are not needed for this concept (eg, references to users along with their names, which would better be stored in a separate profile concept). Concept represents at most one reusable and ideally familiar units of functionality. The concept does not include a subpart that could easily stand by itself, and may even be familiar in its own right (eg, user concept includes karma points). The concept is balanced in the attention to behavioral detail. The concept does not include a fragment of functionality that would in practice grow into a full and complex concept of its own (eg, a restaurant reservation concept including some details of table sizes, which would in practice belong to a concept that managed table layouts). Purpose Define the purpose of a concept that motivates its inclusion or invention Purpose is a succinct and compelling description of a need or problem that the concept solves. The purported purpose is instead a partial description of behavior (eg, purpose of Authentication concept is defined as being able to register and login, rather than as a means of identifying users). Purpose expresses a need and not a means by which the need is fulfilled. Purposes hints at mechanism of concept (eg, purpose for a Reservation concept saying that it enables users to obtain commitments of allocation of a resource in advance). Purpose is focused on the concept at hand and not a larger need. Purpose cannot be fulfilled by the concept itself, but would require other concepts too (eg, purpose of Friend concept is to allow users to connect to and share posts with each other, which cannot be fulfilled without a Post concept in addition). Purpose is expressed in an intelligible way that is easy to understand. Purpose uses technical terms or the name of the concept itself without explanation (eg, saying that the purpose of a Following concept is to allow users to follow each other). Purpose captures an end-to-end need that brings real value and does not focus on an aspect of behavior that brings no value in itself. Purpose specifies ability to enter data without indicating why the data is useful (eg, saying that the purpose of a school Attendance concept is to record whether students are present or absent, without saying how the concept makes use of this information, say for generating end-of-term reports). Purpose is expressed in an application-independent way that would make sense for any context of use. Purpose includes application-specific details that are not relevant to the design of the concept (eg, saying that the purpose of an Authentication concept in a concert ticketing app is to ensure that tickets are used by the people who bought them, which goes beyond authentication and includes irrelevant details of who is authenticated and why). Operational principle Summarize archetypal concept behavior as a scenario OP is a scenario that involves a sequence of steps. OP is instead a restatement or elaboration of the purpose (eg, an OP for Authentication that says that users are identified so that decisions can be made about what actions they may perform, rather than enumerating the key steps of registration, login, etc). OP covers the full lifecycle of the concept. OP covers only a user interaction that requires a prior setup (eg, an OP for Authentication that starts with login and neglects registration). OP includes actions by all stakeholders that modify the state. OP neglects the setting up the conditions by administrators or company employees that make consumer interactions possible (eg, an OP for a ProductCatalog concept that includes a customer searching the catalog but does not include the actions that create the catalog). OP only includes actions of the concept at hand, and not the actions of other concepts. OP describes a user journey that involves multiple concepts (eg, an OP that says that a user registers for an account, logs in and then posts a message, mixing Authentication and Post). State Design the abstract state of a concept, aka the data model State clearly defines distinct components. Not clear exactly what is being stored in the state (eg, state for a UserProfile concept mentions only “information about users”, rather than saying that it stores a display name and bio for each user, for example). State covers all the objects needed to support the actions. State lists the properties of a single object, as if the concept were an object-oriented class (eg, state for an Authentication concept listing “username and password” and not recognizing that a username and password is required for each user, so that the login action can match against all possible users). State indexes components appropriately by object. State includes some components that are either not indexed, or are indexed incorrectly (eg, state for a Friend concept defines friends as a set of users, failing to say that there is a set of users for each user; or says that each user has a set of friends and a date the friendship started, when instead there should be a set of friendships per user, each with the friend and the date started). State includes components that belong to other concepts and are not needed for the actions. State includes properties of an object when just the identity would be sufficient (eg, the state of an Article concept for a blogging app represents the author of an article as a username and thumbnail image, when only the identity of the author is required and these additional fields should appear only in a UserProfile concept). State references external objects by properties when identities would suffice. State references objects by user-friendly names (eg, in an Article concept, the author is represented by their username rather than by a user type that represents the identity of a user). State is sufficiently rich to support all actions. A precondition or postcondition cannot be expressed fully because of a missing state component (eg, the state of a Token concept fails to include an expiry date needed to determine whether a validation action succeeds). State is abstract and not tainted by implementation concerns that are irrelevant to the behavior. A collection of objects is declared as a list (or worse a tree) when a simple set would suffice (eg, in a Friend concept, the friends of a user are given as a list rather than a set). State does not include needless redundancies (except those introduced to enable easier querying). State includes a set of objects that are implicitly ordered by some property, and needlessly declares the set as an ordered list (eg, in a GroupChat concept, declaring the collection of messages in a chat as an ordered list when the messages are already implicitly ordered by send time). Actions Design the actions of a concept that update the state Actions required to set up the state are included. State includes components that have to be set up in advance of typical user interaction but actions are not provided to do so (eg, a ProductCatalog concept that does not include actions for populating a catalog in the first place). Set of actions is sufficient to reach all states. The state includes a special case that is not handled by an action (eg, a Style concept that allows styles to be organized into a hierarchy, and includes an “as is” value for a format associated with a style, but offers no way to specify this through an action). Set of actions is sufficient to update state components as needed. A state component is assumed to be mutable, but no action allows its mutation (eg, in a concept for handling user authentication by passwords, no action is provided to allow a user to change their password). For objects managed by the state, actions are provided to create, update and delete the objects as needed. Objects can only be created but never deleted (eg, a user account concept permits the creation of user accounts but no way to delete them). Undo or compensating actions are present when needed. A user can submit a request for a resource but has no action to cancel the request (eg, in a restaurant reservation concept, there is no action for canceling a reservation). Actions should not include getter methods. Actions are included that correspond to queries that a user would not typically be aware of and that might be performed repeatedly internally (eg, for a Post concept, an action that gets the author of a post). Actions should specify all necessary preconditions. (For a code-level specification, a valid alternative is to specify an error return.) An action depends on a resource being allocated, and will fail if the resource is not available (eg, for an action to reserve a table in a restaurant reservation concept, a slot at the given time must be available). Actions should only refer to state components of this concept. A common mistake is to refer to a component associated with an object that belongs to another concept (eg, an action in a Post concept takes a user object as an argument, and its specification refers to the name of the user, which belongs to a separate UserProfile concept). Set of actions should be minimal and not include actions that are easily expressed in terms of other actions. An action is included that performs another existing action over all the elements of some set, wrongly included because the designer imagines this might be useful (eg, a room reservation concept has an action for reserving a room at multiple times, but which does not create any kind of repeat reservation which might justify the action). Synchronizations for security Sufficient synchronizations are included for authentication and access control Synchronizations are included to identify user appropriately. An action that takes a user as an argument is invoked by a synchronization that binds that user to the currently active user (eg, when the user creates a post, the author of the post is assigned to the user creating it). Synchronizations are included to prevent unauthorized access An action that requires a user to have a particular relationship to an object is appropriately synchronized with authentication actions and state access (eg, in a group chat concept, an attempt to post a message requires the creator of the post to be a member of the group; in a Article concept, the editor of an article is required to be its author). Synchronizations to maintain coherent state Synchronizations are included so that the states of different concepts make sense globally Synchronizations are included to maintain invariants in which the presence of one object requires the presence of another. Actions that perform deletions are synchronized with cascading deletion actions for associated objects (eg, when a post is deleted in a Post concept, the comments of that post are deleted in a Comment concept). edit of post and associated upvotes and comments Synchronizations to support concepts that require it notifications, logging, auditing Use state appropriately to check conditions to find appropriate object karma ","permalink":"https://essenceofsoftware.com/exercises/concept-rubric/","summary":"Criteria for reviewing concept definitions","title":"Concept rubric"},{"content":"Concepts can be designed and understood entirely independently of one another, even if eventually they will be composed with other concepts in a larger application context. Designing and evaluating individual concepts is thus an important skill.\nIn this exercise, you\u0026rsquo;ll define concepts for some standard behaviors that you are likely to be familiar with. For each one, you should provide all the standard elements of a concept (name, purpose, operational principle, state and actions), along with some succinct notes describing any common variations, or pointing to subtleties in the concept. Also, for each concept you should list also some other concepts that are commonly found with it (giving each a name and a few words describing it, but not a full concept definition).\nURL Shortener Define a concept for the essential function of a URL shortening service such as tinyurl.com or bit.ly. Your concept should support both user-defined and autogenerated URL suffixes.\nBillable Hours Tracking Many companies that bill clients for work by the hour use tracking software to help automate record keeping. Define a concept that handles the core functionality. Assume that, in the application that uses this concept, an employee marks the beginning of a session by selecting a project and entering a string describing the work to be done, and then marks the end of the session with another interaction. Think about how to handle a case in which someone forgets to end a session.\nConference Room Booking Define a concept for the essential function of a service for booking conference rooms in a company or university department, like CSAIL\u0026rsquo;s room booking system. Note: you do not need to include recurring bookings.\nElectronic Boarding Pass Define a concept for the essential function of the kind of electronic boarding pass that airlines typically provide, which can be inserted into a digital wallet on a phone, and which is updated in realtime to reflect new gate assignments and modified departure times. Here is an article about creating boarding passes using Apple\u0026rsquo;s Passkit API which might be helpful, but note that it does not cover the functionality in full and includes many user interface details that are not conceptual.\nAddress Verification A variety of applications use address verification, in which the identity of a user is authenticated by asking them to provide some or all of their mailing address. For example, online credit card transactions typically require that the user enter a full address; gas station purchases with credit cards often ask users to enter their zipcode; the NextDoor social network obtains the billing address record associated with a user\u0026rsquo;s phone number to ensure that they are within the catchment area of a particular forum.\nDefine address verification as a concept, bearing in mind that the concept is inherently distributed: it involves actions performed at different locations by different stakeholders, and its state may not be stored at the location at which the action is performed.\nTime-Based One-Time Password (TOTP) Multifactor authentication schemes improve on simple password-only schemes. One common scheme uses an app that runs on the user\u0026rsquo;s phone (or computer) that generates time-based tokens that are then entered by the user as part of the authentication process. Define a concept for this that captures the essential behaviors. Note that you should not dwell on the actual cryptographic functions that are used to generate tokens, but should just assume them in your concept action definitions. Pay particular attention to the concept\u0026rsquo;s purpose, and include some additional explanation of the respects in which the concept actually improves security (and what kinds of attacks it still leaves open).\n","permalink":"https://essenceofsoftware.com/exercises/concept-writing/","summary":"Concept writing exercise","title":"Concept writing exercise"},{"content":"To help you get going with designing systems of concepts, this exercise asks you to read a a few concepts and syncs and answer some questions about them.\nConcepts Consider the task of designing a URL shortening service. We might define three concepts: URLShortening, which maintains mappings between the short URLs and their expansions; ExpiringResource, which manages the relinquishing of resources that have finite lifetimes; and NonceGeneration, which generates the unique strings that the shortener uses. Here are some abbreviated specifications of the concepts:\nA UrlShortening Concept concept UrlShortening\npurpose shorter or more memorable way to link\nprinciple after create generates a short url, lookup will return the original url\nstate\na set of Shortenings with\na targetUrl String a shortUrl String actions register (shortUrlSuffix, shortUrlBase, targetUrl: String): (shortUrl: String)\nforms and saves a new short URL by appending the suffix to the base and associates it with the targetUrl lookup (shortUrl: String): (targetUrl: String)\nreturns the target URL associated with a short URL delete (shortUrl: String)\ndeletes a short URL and its association with a target URL A NonceGenerator Concept concept NonceGenerator [Context]\npurpose generate unique strings within a context\nprinciple each generate returns a string not returned before for that context\nstate\na set of Contexts with\na used set of Strings actions\ngenerate (context: Context) : (nonce: String)\nreturn a nonce that is not already used by this context An ExpiringResource Concept concept ExpiringResource [Resource]\npurpose expire resources automatically to manage costs\nprinciple after setting expiry for a resource, the system will expire it after the given time\nstate\na set of Resources with\nan expiry DateTime actions\nsetExpiry (resource: Resource, seconds: Int)\nsystem expireResource () : (resource: Resource)\nSome explanatory notes The expireResource action of ExpiringResource is a system action, performed spontaneously when its precondition is true (namely when the expiry time for some resource, which it returns, has arrived). The register action of UrlShortening takes a base URL; this ensures that the concept is not hardwired to a particular domain (such as tinyurl.com), and would allow the service to support user-specific domain names. Concept Questions Read the concept specifications carefully and answer these questions:\nContexts. The NonceGenerator concept ensures that the short strings it generates will be unique and not result in conflicts. What are the contexts for, and what might a context end up being in the URL shortening app? Storing used strings. Why must the NonceGenerator store sets of used strings? One simple way to implement the NonceGenerator is to maintain a counter for each context and increment it every time the generate action is called. In this case, how is the set of used strings in the specification related to the counter in the implementation? (In abstract data type lingo, this is asking you to describe an abstraction function.) Words as nonces. One attractive option for nonce generation is to use common dictionary words (in the style of yellkey.com, for example) resulting in more easily remembered shortenings. What are some disadvantages of this scheme? How would a variant of the NonceGenerator concept defined for this purpose differ from NonceGenerator? (Hint: consider in particular the state, and the generate action.) Synchronizations Here are three of the synchronizations that might be included in a full design of the URL shortening application:\nsync generate\nwhen Web.request (method: \u0026ldquo;shortenUrl\u0026rdquo;, shortUrlBase)\nthen NonceGenerator.generate (context: shortUrlBase)\nsync register\nwhen Web.request (method: \u0026ldquo;shortenUrl\u0026rdquo;, targetUrl, shortUrlBase) and NonceGenerator.generate (): (nonce)\nthen UrlShortening.register (shortUrlSuffix: nonce, shortUrlBase, targetUrl)\nsync setExpiry\nwhen UrlShortening.register (): (shortUrl)\nthen ExpiringResource.setExpiry (resource: shortUrl, seconds: 3600)\nSome explanatory notes Completions and invocations. In the synchronization language, actions referred to in the when clause always correspond to action completions, and actions in the then clause to action invocations. Named arguments and results. The arguments and return values of actions are named (they\u0026rsquo;re represented in the code simply as dictionaries), so, for example, the first sync invokes in its then clause the generate action of the NonceGenerator concept with the argument named context bound to shortUrlBase. Bound variables. The when clause can bind variables used in the other clauses; for example, the return of expireResource in the last sync says that the return value given by the return name resource is bound to the variable shortUrl. This bound variable is then passed as an argument to the delete action in the then clause. Omitted names. By convention, it is not necessary to include both the argument name and the bound variable name when they are the same (writing \u0026ldquo;x: x\u0026rdquo;), and mentioning the name once is short for this. So the mention of targetUrl in the when clause of the first sync is short for targetUrl: targetUrl, and binds the variable targetUrl to the argument of the request action called targetUrl. Synchronization Questions Read the synchronizations carefully and answer these questions:\nPartial matching. In the first sync (called generate), the Web.request action in the when clause includes the shortUrlBase argument but not the targetUrl argument. In the second sync (called register) both appear. Why is this? Omitting names. The convention that allows names to be omitted when argument or result names are the same as their variable names is convenient and allows for a more succinct specification. Why isn\u0026rsquo;t this convention used in every case? Inclusion of request. Why is the request action included in the first two syncs but not the third one? Fixed domain. Suppose the application did not support alternative domain names, and always used a fixed one such as \u0026ldquo;bit.ly.\u0026rdquo; How would you change the synchronizations to implement this? Adding a sync. These synchronizations are not complete; in particular, they don\u0026rsquo;t do anything when a resource expires. Write a sync for this case, using appropriate actions from the ExpiringResource and URLShortening concepts. ","permalink":"https://essenceofsoftware.com/exercises/system-reading/","summary":"System design reading exercise","title":"System design reading exercise"},{"content":"In this exercise, you\u0026rsquo;ll design a few concepts and some synchronizations to fulfill the features of a simple app. Then you\u0026rsquo;ll modify your concepts (if necessary) to make them sufficiently modular to be used for some variants of the app. You should include the synchronizations that are needed for the primary web requests that define the app, and should give some examples of how they might be modified to account for the variants in functionality.\nApplication Description The app is a simplified audience polling tool like Slido, which allows presenters to create simple questions that are then presented to an audience. Members of the audience answer the questions and the presenter can see (and share) aggregate results. A question may have a simple short textual answer, or it may ask for selections from some predefined options; it\u0026rsquo;s up to do you to choose, but don\u0026rsquo;t make it complicated by offering both.\nApplication Variants Here are the application variants that your design should support without changing the essential concepts:\nDisplaying the polling link: Variant A: the link is displayed as a QR code Variant B: the link is displayed as a short URL User accounts: Variant A: creators of polls must have accounts and can view all the polls they created in one view Variant B: creators of polls do not have accounts, but received a link whenever they create a poll through which it can be viewed and edited later Expiry: Variant A: polls do not expire. Variant B: polls expire some fixed period of time after they have been created. ","permalink":"https://essenceofsoftware.com/exercises/system-writing/","summary":"System design writing exercise","title":"System design writing exercise"},{"content":"Workshop Schedule Date Title Topics Links Recording June 10 Concepts how this project began; conceptual models and beyond; the need for granularity in conceptual models; concept structure; concepts as atoms, apps as molecules slides recording June 12 Behavior details matter; state machines; concepts and objects; group chats in WhatsApp; directories in Unix; file synchronization; heuristics for states and actions slides recording June 16 Behavior (redux) describing behavior in a UI-independent way; a sample concept design in full; modal vs noun/object designs; integrity constraints \u0026amp; how to preserve them; how data models are related to states; folder concepts, Unix and implications for shared file systems slides, cheatsheet recording June 24 Modularity de/composition; modularity criteria; concept synchronization; independence constraints; decomposing data models; restaurant modularity example; purposes and conflation; Zoom reaction example slides, modularity-rubric recording (coming soon), feedback Pre-read In advance of the workshop, I\u0026rsquo;d like you to read some short pieces for some background. The first is an overview of concept design:\nThree Stages of Enlightenment (13 minute read)\nThe second is a very brief explanation of how concept design fits in with other kinds of design:\nLevels of Design (3 minute read)\nAnd the third is a short piece I wrote about the spirit of design, and how great designs don\u0026rsquo;t spring out of the air but emerge from hard work polishing and refining ideas:\nHow Great Design Happens (4 minute read)\nI hope you enjoy these and find them helpful. There are several other pieces on the concept design website that you might find interesting, for example one about how LLMs will upend the principles of agile development, and one about how concept design is being used at Palantir.\n","permalink":"https://essenceofsoftware.com/workshops/autodesk-online/","summary":"Online, June 2025","title":"Autodesk Concept Design Workshop"},{"content":"Workshop Schedule Time Monday Tuesday 9:00 Welcome (slides, goals poll, sw poll) Questions \u0026amp; review 9:30 Introducing concepts (slides) Design moves (slides) 10:45 Break Break 11:00 Identifying concepts (slides) Team vision exercise (slides, yours) 12:15 Lunch Lunch 1:15 Designing concepts (slides) Team integration exercise (slides, yours) 2:45 Break Break 3:00 Design exercise (slides) Presentations (slides) 4:30 Reflections (slides, survey) Reflections (slides, survey) 5:00 Adjourn Adjourn Welcome and Pre-read Dear Autodesk Friends,\nI’m very excited about our upcoming workshop next Monday and Tuesday (March 17-18), and looking forward to meeting you all!\nThe schedule for our workshop is above. There might be some small adjustments as we see how things go, but the general contours won\u0026rsquo;t change.\nOur sessions will start promptly at 9:00am in the morning, with an hour for lunch at 12:15pm, and ending for the day at 5:00pm. There are two short breaks during the day. We\u0026rsquo;ll have some informal chat over lunch, which will be served in the room, so don\u0026rsquo;t plan to go somewhere else!\nIn advance of the workshop, I\u0026rsquo;d like you to read some short pieces for some background. The first is an overview of concept design:\nThree Stages of Enlightenment (13 minute read)\nThe second is a very brief explanation of how concept design fits in with other kinds of design:\nLevels of Design (3 minute read)\nAnd the third is a short piece I wrote about the spirit of design, and how great designs don\u0026rsquo;t spring out of the air but emerge from hard work polishing and refining ideas:\nHow Great Design Happens (4 minute read)\nI hope you enjoy these and find them helpful. There are several other pieces on the concept design website that you might find interesting, for example one about how LLMs will upend the principles of agile development, and one about how concept design is being used at Palantir.\nDon\u0026rsquo;t hesitate to contact me if you have any questions. I can be reached by email at dnj@conceptualstrategy.com.\nBest wishes,\nDaniel\n","permalink":"https://essenceofsoftware.com/workshops/autodesk-boston/","summary":"Boston, March 2025","title":"Autodesk Concept Design Workshop"},{"content":" Time Tuesday Wednesday Thursday 9:00 Welcome \u0026amp; intros (slides) Questions from yesterday Questions from yesterday 9:30 Why software is hard (slides) Disentangling concepts (slides) Team activity 10:45 Break Break Break 11:00 Innovation scenarios (slides) Quintessential design move (slides) Team activity 12:15 Discussion Discussion Discussion 12:45 Lunch Lunch Lunch at 1pm; adjourn 13:45 Designing concepts (slides) Product families \u0026amp; catalogs (slides) Debrief \u0026amp; discussion 15:00 Break Break 15:15 Designing syncs (slides) Team activity 16:30 Review of day Review of day 17:00 Adjourn (survey, responses) Adjourn (survey,responses) All the slides linked to from this page, with the exception of the Autodesk-specific exercises, are background IP and may be used freely within Autodesk (but not beyond).\n","permalink":"https://essenceofsoftware.com/workshops/autodesk-woodinville/","summary":"Woodinville, December 2024","title":"Autodesk Concept Design Workshop"},{"content":"Background Our Kodless prototype seems to be very promising and allows effective LLM-based code generation of both concepts and syncs, as demonstrated by the HackerNews example.\nSome limitations and opportunities:\nCatalog. Currently all concepts are generated afresh, aside from some basic ones like Session. It would be easy to extend Kodless to allow concepts to be selected from a catalog. This would allow (a) reuse; (b) greater confidence in the security of critical concepts; (c) concepts that might be hard to build with an LLM because they involve tricky functionality or use complex APIs (such as maps); (d) faster app composition. Cross-concept queries. In the current design, when a query must access the databases of multiple concepts, the sync code makes action calls to the concepts and essentially aggregates results. This is not ideal, both because it complicates the sync code and because it requires complex view actions in concepts (such as getByIds). Not clear how to fix this. Request structure in syncs. Some routes require fairly elaborate structure, in having multiple params and in packaging results into structured objects. This is currently handled by requiring the user to write prompts that contain this information. It could be mitigated by some metaprompts that provide general guidance on how request results should be packaged. It could also be absorbed inside standard BFF components (see below). Frontend code. Barish designed a nice DSL for frontend code which makes it easy to build a reactive tree that interleaves components from different concepts. He has also had some success using GPT to generate the DSL code. I am convinced, however, that the base functionality of Kodless should offer reusable frontend components. New Architecture The proposed architecture offers reusable frontend components so that apps can be built without any frontend code generation.\nSome elements of the new architecture might be:\nPer-concept BFF. Each concept comes with not only its backend component encapsulating a data store and actions, but also a default frontend component, perhaps implemented in the Backend for Frontend pattern. This new component is a server-side component that sits between the frontend and the backend like a proxy, and offers actions that respond with frontend widgets (in which calls to other component actions are embedded). Global CSS. The frontend components use a standardized set of CSS styles to make it easy to achieve a uniform appearance, and to change it. Default syncs. By default, each action of a backend concept has an associated sync that simply wraps the action in a route. The frontend components don’t call the concept actions directly, but instead call the syncs. [Should these syncs be plain function calls instead of routes, so that the BFF components encapsulate all HTTP-related functionality?] Overriding syncs. When more application-specific behavior is needed, the default syncs can be replaced by specialized syncs, for example, adding authentication actions. Predefined schema objects. For standard things (like books, movies, mailing addresses), we can use schemas/formats as defined in schema.org. These will often be passed as parameters to polymorphic concepts. For example, Catalog concept that lets you find items based on several metadata fields could be instantiated with the book schema. Examples Adam’s programming language fan table ","permalink":"https://essenceofsoftware.com/drafts/concept-architecture/","summary":"A Proposal for a Concept-based App Architecture","title":"Concept Architecture"},{"content":"In concept design, a running app is viewed as a configuration of concept instances. I’ve been thinking about how these configurations should be designed, and had planned to write a short blurb about it. But as I wrote it, I realized there was more to say than I’d expected, and also more to figure out.\nSo I thought I’d share it with some friends for feedback. I’d welcome help and comments from all you concept designers out there, but I’m thinking particularly of the following people whose recent work is connected to these ideas:\nKevin Sullivan, who’d argued, in our collaborative work on concept design for moral distress, that surveys and temperature readings could be linked externally in the application state. I’d resisted this but now I think Kevin was right all along. Abutalib (Barish) Namazov, whose concept-driven, LLM-based code generator Kodless might exploit configurations as a construction strategy. Geoffrey Litt, whose Patchwork system corrects a flaw of Github’s and binds chats to branches rather than pull requests. A discussion with Geoffrey prompted me to write this piece. Alcino Cunha, whose work on software configurations separates regular state from configuration state. The ideas here might help make that distinction more precise, and the configuration-enhanced version of Alloy might help analyze configurations in concept designs. Concept Instances \u0026amp; Extents Concepts manage sets. A concept instance handles sets of things. For example, the Label concept, which is used in Gmail for organizing messages, contains the functionality for creating labels, associating them with messages, and finding messages with particular labels. Only one instance of the concept is needed to represent these sets and relationships.\nOOP is different. In contrast, the instances (or objects) of a Label class in an object-oriented program would each represent a single label. You could have each label object hold links to the associated messages, and this would make it easy to find the messages with a given label.\nOOP still needs concept-like obejcts. In an object-oriented program, you’d still need some way to navigate from label names entered by the user to the actual labels. For this, you’d probably define another class, say LabelRegistry, whose objects contain mappings from strings to label objects. Or you might add a static mapping to the Label class, although this is a bad idea because, as we’ll see below, you’re likely to want to have more than one such mapping. Either way, this simple example shows how the relationship between domain concepts and objects or classes in object-oriented programming isn’t as simpler as people sometimes claim, and more often than not a single domain concept is implemented as a collection of classes.\nHow many instances per concept? In the simplest cases, having a single instance of each concept will suffice. But often multiple instances will be needed, to correspond to different, disjoint contexts. In the Label case, different Gmail users will have different instances, so that their labels aren’t confused.\nConcept extent. Each instance of a concept has an extent that determines its lifetime and span, and is defined by some other concept (or object) instances it’s associated with. In Gmail, for example, the extent of each Label instance is a single User. Two distinct users will have different Label instances so that their labelings will never interact (in particular, labels aren’t included in sent messages), and the same Label instance will be associated with a given user for that user’s lifetime.\nDeciding on a concept’s extent is often a critical design question. Apple’s macOS file system, for example, has a Label concept, with labels called “tags,” but there seems to be just one instance across multiple users, with some surprising consequences.\nZoom: Design Snags with Chat Extent Zoom’s Chat concept shows how concept extents can have serious impacts on usability. There are of course many instances of the Chat concept in Zoom, since otherwise all users would have to be within a single chat. But what exactly is the extent of a Chat instance?\nNot just a meeting. At first glance, it seems that the extent of a chat is a single meeting. But this isn’t quite right. A Zoom meeting can be restarted multiple times, even one that wasn’t defined as “recurring” when it was created. When you join the restarted meeting, you won’t see any messages from the chat of the previous session. So each restart associates a fresh chat instance with the meeting. In other words, the extent of each chat instance is a meeting session.\nPrivate chats. When you post a message in Zoom, you can optionally send it privately to a single participant. One could view this as a special feature of Zoom’s Chat concept. But a simpler (and more robust) interpretation is that there are separate chat instances for each pair of participants. In this case, the extent of a private chat is defined by a tuple (meeting-session, participant, participant).\nBreakout rooms. Finally, a chat extent issue that reflects a serious design flaw in Zoom. When you open breakout rooms in a Zoom session, each room gets its own chat instance. If this were an additional instance, it might be justified\u0026mdash;as a means of allowing the members of the breakout room to converse privately. But unfortunately it’s the only instance that is accessible.\nThe consequences of this design decision are disastrous. Since the chat instance of the main meeting session is no longer accessible, instructions or background information for a breakout room cannot be posted in the session chat. Moreover, the host of the meeting, not being a member of the breakout room chats, cannot post a chat message to signal the upcoming end of the breakout, for example. One guide to using breakout rooms in teaching recommends a tedious workaround:\nStudents will no longer have access to anything in the main room once the breakout session has begun (including the chat and the whiteboard). Consider posting the discussion objectives/prompts on Canvas and ensuring each student has Canvas up in another window before the breakout session begins.\nFixing breakout rooms. To mitigate this obvious flaw, Zoom added the concept of Broadcast to allow a host to send a message to breakout rooms. This concept needlessly replicates some of the features of the Chat concept, while missing other important features (such as persistence, and the ability to copy and paste from messages, or to click on links).\nMore Extent Examples In Github, a chat instance is associated with a pull request. This means the conversation must end if a pull request is rejected or withdrawn, and cannot start until the request has been issued. [Thanks to Geoffrey Litt for this example.] In an online store, the extent of the Order concept is a user, but the concept can be lifted over users, since the orders of different users are independent of one another. In contrast, the Auction concept associates bids with users in its state, and cannot be lifted over users because the outcome of an auction depends on the behavior of multiple users. A helpful way to figure out when a concept can be lifted is to consider the operational principle (OP). The OP that explains the Order concept, for example, need never mention a user: “you place an order for an item, then the item is delivered”. But the OP for the Auction concept must mention users: “if one user issues a bid for an item, and then another user issues a higher bid\u0026hellip;”.\nAnother example: in a traditional elevator system, each elevator operates independently and so the assemblage could be modeled as a set of instances of an Elevator concept. In modern systems, there may be some load balancing amongst the elevators, but this might not affect user’s experience. But in Schindler’s PORT system, in which you select a destination and the system directs you to an elevator bank, the concept must clearly include the entire set of elevators.\nLifting: A Design Move Enriching concept state. An alternative to having multiple instances of a concept is for the state of the concept itself to be enriched with mappings.\nFor example, a single instance of the Meeting concept could allocate multiple meeting objects, and manage their association with chats. This would be a bad idea for two reasons.\nFirst, it would make the Meeting concept dependent on the Chat concept: if we wanted to build a system that had meetings but not chats, such as meeting concept would not be suitable (and making an associated chat optional would not solve this).\nSecond, the meeting-chat association is not an intrinsic part of either concept, but is a property of how they are configured together. Including it in Meeting corrupts and needlessly complicates the concept.\nContexts: a bad idea. In an earlier attempt to model Zoom, I augmented the Chat concept with a “context” representing the extent of the chat. Because this context is polymorphic, and because it’s plausible that every chat has some context, it doesn’t seem to introduce a dependence (which augmenting Meeting would). But it nevertheless feels like a kludge that complicates the concept by adding functionality that is not concept-specific, and now seems like a mistake.\nLifting as a design move. We can capture this idea as a design move that I call lifting. Whenever a concept state includes relational mappings, we can consider moving a mapping outside the concept into application configuration state.\nFor the meeting-chat example, if we started with this:\nconcept Chat [Context] state chat: Context -\u0026gt; lone Chat posts: Chat -\u0026gt; set Post app Zoom includes Meeting Chat [Meeting] we can lift to this:\nconcept Chat state posts: set Post app Zoom includes Meeting ch: Meeting -\u0026gt; lone Chat In abstract terms, what’s going on is that we have a state relation of the form\nMeeting -\u0026gt; Chat that can either be expressed like this\n[Meeting -\u0026gt; Chat] with square brackets denoting state stored inside a concept, or, in lifted form, like this\n[Meeting] -\u0026gt; [Chat] How Lifting Simplifies Concepts The importance of lifting is that it simplifies a concept by reducing its scope.\nSimple lifting example. For a more basic example, and setting aside the extent question for now, suppose we were modeling the meeting concept and were considering something like this\nconcept Meeting state meetings: set MeetingObject created: meetings -\u0026gt; one Date ... in which the concept maintains a set of meetings (represented by members of the allocated type MeetingObject), and mappings to associate each meeting with its creation date, etc.\nThis is a poor design, because all the functionality of the meeting concept can be described within the span of a single meeting. There are no actions that merge meetings, for example, or any invariants that would limit the properties of a meeting due to the presence of other meetings. Lifting fixes this, and results in a simpler concept state:\nconcept Meeting state created: Date ... Invariants prevent lifting. Now suppose we’d included meeting links in the meeting concept, like this:\nconcept Meeting state meetings: set MeetingObject link: meetings lone -\u0026gt; one Link created: meetings -\u0026gt; one Date ... This introduces an invariant (indicated by the lone keyword in the link declaration) that links are unique, with no two meetings sharing a link.\nTo maintain this invariant, the concept needs access to the set of all meetings and their links. Applying lifting here would mean maintaining this invariant in the synchronizations, violating the concept design principle that all significant functionality is held inside concepts.\nHaving your cake and eating it. But lifting still seems desirable for the other aspects of meeting behavior, which are all scoped within a single meeting. A solution to this dilemma is to separate out the link assignment functionality in a separate concept, and then lift the meeting concept. This would be especially compelling if the link assignment functionality has uses in other situations and thus has a claim to being a concept in its own right. And indeed, this is the case, and the concept, called Capability, is a widely used scheme for granting access to resources through unique keys.\nWhen Non-Lifting Reveals Complexity Sometimes considering whether a concept can be lifted reveals an interesting design complexity.\nCan reservation be lifted? Take the Reservation concept, for example. Reservations associate users with resources (for example, restaurant patrons with tables), but we also need to represent the fact that resources have owners (the restaurants that own the tables).\nMust the Reservation concept include ownership of resources in its state? Or can we lift this? In the lifted version, we would have one instance of the Reservation concept for each business that takes reservations.\nThis seems promising. The tables of one restaurant or the rooms of one hotel are not coupled in any way to the tables of another restaurant or the rooms of another hotel. And indeed many restaurants and hotels do run their own reservation systems, so it seems that separate instances for each business should make sense even for a service that manages multiple businesses.\nConflict detection. But there’s a feature of reservation services that does actually couple the businesses together. In order to prevent consumers who want to preserve their choices from gaming the system by making multiple reservations and then dropping all but one, these services prevent such reservations from being made.\nApps like OpenTable and Resy, for example, prevent you from making two reservations on the same night. This successfully rules out foul play, but it frustrates legitimate demands too (such as wanting to book a bar for a cocktail and then a restaurant for a meal, as this aggrieved user of Resy explains).\nSo, in its current form, the Reservation concept cannot be lifted if it is to support this kind of multiple booking.\nChallenge to readers. Is there a neat way to factor this behavior out into a separate concept (whose purpose is to detect and prevent abuse), in the same way that the Capability concept was used to factor linking out of the Meeting concept?\nConcept code To make concrete the explanations above, here’s the Zoom meeting-chat design in the formal concept notation.\nconcept Chat [User] state posts: set Post date: Post -\u0026gt; one Date by: Post -\u0026gt; one User msg: Post -\u0026gt; one Text members: set User joined: User -\u0026gt; one Date actions new (out c: Chat) join (u: User) leave(u: User) post (u: User, t: Text, out p: Post) delete (u: User, p: Post) concept Meeting [User] state host: User participants: set User + Endpoint active: bool actions new (h: User, out m: Meeting) start (u: User) end (u: User) join (u: User + Endpoint) leave (u: User + Endpoint) concept Capability [Resource] principle after allocate, can get (and can't access without get) state used: set Key private resource: used -\u0026gt; lone Resource actions allocate (r: Resource, out k: Key) k not in used used += k k.resource := r get (k: Key, out r: Resource) r := k.resource concept Auth state username: User -\u0026gt; one Text password: User -\u0026gt; one Text actions register (n, p: Text, out u: User) authenticate (n, p: Text, out u: User) changePassword (u: User, p: Text) delete (u: User) app Zoom includes let U = [Auth.User], M = Meeting [U], C = Chat [U], K = Capability [M].Key au: Auth ca: Capability [M] ch: M -\u0026gt; lone C syncs createMeeting (host: U, out k: K, out m: M) M.new (host, m) ca.allocate (m, k) startMeeting (host: U, k: K, out m: M) ca.get (k, m) m.start (host) C.new (c) m.ch := c // replaces on restart c.join (host) postInChat (m: M, u: U, t: Text, out p: C.Post) m.ch.post (u, t, p) Notes\nI haven’t introduced an explicit session type corresponding to Zoom meeting sessions. Instead, chats are bound to meetings, and the binding is reassigned on start. I’ve used uppercase names (eg, C, M, U) as shorthands for types, and lowercase names (eg, ca) for concept instances. The new action allocates a fresh instance of the concept. Previously, I’ve often overloaded names, using the same name for a concept and for a type used in that concept. For example, in the unlisted version of the Meeting concept, I might have used the name Meeting both for the concept and for meeting objects within the concept. I now think this was a bad idea. So the name Meeting within the new action of the Meeting concept refers only to the concept type itself (and is the type of the instance returned by the action). Here’s the variant in which the chat is associated with the meeting and persists across sessions of that meeting:\ncreateMeeting (host: U, out k: K, out m: M) M.new (host, m) Ca.allocate (m, k) C.new (c) m.ch := c c.join (host) startMeeting (host: U, k: K, out m: M) ca.get (k, m) m.start (host) Note that the declaration of the application-level binding is unchanged, but the chat is now allocated and bound to the meeting when the meeting is created rather than when the meeting is started.\nAnd here’s the variant for private chats:\napp Zoom includes let U = [Auth.User], M = Meeting [U], C = Chat [U], K = Capability [M].Key au: Auth ca: Capability [M] ch: M -\u0026gt; lone C privateCh : M -\u0026gt; U -\u0026gt; U -\u0026gt; lone C syncs privatePost (m: M, from, to: U, t: Text, out p: C.Post) if no privateCh [m, from, to] C.new (c) privateCh[m, from, to] := c privateCh[m, to, from] := c privateCh[m, from, to].post (u, t, p) ","permalink":"https://essenceofsoftware.com/drafts/concept-extents/","summary":"Notes on the design of concept configurations","title":"Concept Configuration"},{"content":"Feeling like Cinderella To celebrate my lab’s 60th birthday, we put on a two-day bash: one day of talks from alumni and one day or talks from the faculty. I started my talk by noting that the title of the event was “AI Frontiers \u0026amp; Implications”, and that I felt like Cinderella, being a plain old computer scientist gatecrashing an AI ball.\nOur lab’s name (CSAIL) stands for “Computer Science and Artificial Intelligence Laboratory,” but of course nowadays it’s all about AI. Like many of us, I’ve sometimes wondered if AI will make everything I’ve worked on irrelevant. After all, won’t GPT-N (for some sufficiently large N) be able to design and code all our software without human intervention?\nWhy AI will never make a great programmer For now, I’ve come to the conclusion that there’s no risk of this in the near future. I noted in another recent talk that there are three predictions you should never make: (3) that the stock market will crash this year; (2) that a certain politician is too stupid and corrupt to win election; and (1) that GPT won’t ever be able to do X (for any X). Nevertheless, despite all of GPT’s amazing capabilities, there’s good reason for skepticism:\nLLMs like GPT aren’t reliable. They make stuff up, and they make mistakes. There’s no evidence that this will ever change. GPT is really good at writing code that’s like all the code that’s been written before. Nobody’s shown any examples of it inventing new and better ways to write code, so if we just leave it to GPT-4, our code will be stuck with the knowledge that StackOverflow had in 2021, when GPT-4’s training ended. And if we stop innovating, a future version of GPT won’t know any more. I think of this as a kind of vigilance decrement writ large: just as a human driver can’t take over when a self-driving car gets in a pickle, so we human coders won’t be able to fix a massive software mess that LLMs create if we don’t continue to hone our skills. Sometimes it’s OK if software mostly works. But this is less common than you might imagine, and software has a habit of becoming critical the more useful it is. We just won’t tolerate software that has occasional but devastating failures. Are LLMs like compilers? Some people say LLMs will be like compilers. Initially, there were skeptical programmers who believed they could write better machine code themselves. But eventually people realized that compilers saved so much trouble that it was rarely worth the effort of writing low level code yourself.\nAre the LLM naysayers like those early compiler skeptics? LLMs are already successfully generating complex code fragments, and many programmers rely on them daily.\nBut this analogy misses a crucial point. The reason that high-level languages took over is that compilers are completely dependable. Nothing is ambiguous in the instructions a compiler is given. So we can define correctness for a compiler and, given enough resources, even verify it.\nWhat’s so great about LLMs as coders, in contrast, is that they take a vague specification and fill in the implicit details from the context. This is what makes using an LLM to generate code so compelling: you just nudge a bit, offer some hints, and if you’re lucky, you get what you wanted. The downside is that there’s no guarantee that the ambiguities will be resolved the way you expected.\nAn impressive example of LLM coding Here’s an example of a coding task that I gave ChatGPT last week that illustrates its amazing ability to fill in the gaps in a vague specification.\nMy dad has recorded our family tree in a massive text file. I wanted GPT to generate a program to read this file and draw the tree. I started by telling it what problem I was trying to solve, giving it just one entry from the file and asking it to write a function to parse it.\nHere’s what my entry looks like:\n0001 born 09-01-1964 London 0001 male 0001 marr 01 0014 01-02-1989 NYC USA 0001 name Daniel Nicholas Jackson 0001 fthr 0002 0001 mthr 0003 0001 chld 01 0482 0014 0001 chld 02 0505 0014 0001 chld 03 0515 0014 0001 note 01 Computer Scientist, CMU \u0026amp; MIT 0001 note 02 Oriel College, Oxford, physics 0001 note 03 MIT MS, PhD Computer Science I wasn’t surprised that GPT was able to write the parsing code. What blew me away was that it correctly interpreted every single field in this structure even though I’d told it nothing about the encoding. It inferred without any help that the line\n0001 chld 02 0505 0014 meant that the person with ID 0001 has a second child with ID number 0505 who was born also to the person with ID 0014 (matching the ID of the person 0001 is married to, as shown on an earlier line).\nI then gave GPT a series of additional instructions: to extend this function across the whole file; to handle some new fields it hadn’t seen before; and to generate a graph in DOT format. All this worked fine and the resulting 100 or so lines of code had no bugs, as far as I could tell.\nDomain knowledge is critical My takeaway from the family tree experiment is that domain knowledge is GPT’s secret sauce. It’s indeed impressive that it can write routine code (such as a basic parser) so effectively, but its ability to fill in the gaps in a spec is even more remarkable.\nTo illustrate this, let me show you one more example. I was playing around with having GPT generate code for some basic concepts. I wanted some code for a Friend concept, so I started (mischievously) by giving it a rather abstract specification:\nCan you write a JavaScript class for me? I want it to maintain a graph between nodes, and associate a set of items with each node. It should include (1) a method that connects a node n1 to a node n2, (2) a method for associating an item i with a node n, and (3) a boolean method that takes a node n and an item i and returns true if the node is connected to a node associated with the item.\nMy intent was that the nodes would represent users, and the items would represent posts. So this would embody the key operational principle of Friend: that you can see your friends’ posts.\nThe code GPT generated in response had a basic flaw: it made the node relation asymmetric, so if I friended you, it meant I could read your posts but you couldn’t read mine. This asymmetry resulted from my asking for “a method connects a node n1 to a node n2”; if I said instead “a method for connecting one node to another”, it correctly implemented a symmetric relation.\nSo what if instead I revealed the concept?\nCan you write me a Javascript class to implement the friend concept in a social media app? It should track who is friends with who, and which posts someone has written, and then allow access to a post to a user if it was written by one of the user\u0026rsquo;s friends. There should be (1) a method to make one user a friend of another, (2) a method to record that a user wrote a post, and (3) a method that returns true if a user can access a post.\nThe code GPT produced for this prompt was much better. What was most exciting was that it not only made inferences from my spec (eg, that the friend relation should be symmetric) but it actually violated my spec in a desirable way. Look at this method:\nclass SocialMediaApp { ... canAccessPost(username, postId) { if (this.users.has(username) \u0026amp;\u0026amp; this.posts.has(postId)) { const postAuthor = this.posts.get(postId); if (postAuthor === username) { return true; // User can access their own post } const user = this.users.get(username); return user.friends.has(postAuthor); } return false; } } GPT has correctly inserted a special case: that even though you’re not friends with yourself, you should be able to read your own posts.\nThe future of programming: unremarkable observations What are the implications of all this for the future of programming? First, some unremarkable observations that others have made many times before:\nGPT is amazing at writing routine code. It can save you tons of time and relieve the tedium of writing basic functions. GPT is a great interlocutor for discussing technical questions (what platform or API to use, what algorithmic challenges you might face, etc). GPT writes competent, standard code and can’t be expected to do anything that requires true creativity (like a new design pattern). GPT isn’t like a compiler: it makes mistakes, so code review is more important than ever. These facts alone mean that the way you program (and teach programming) will change a lot. I’m particularly intrigued by the new role that code review will play, and how classic ideas about invariants and verification might become more relevant than ever.\nThe end of agile? I’m most excited by the ways in which GPT differs from all software engineering tools we’ve previously encountered.\nWhen you use GPT as a programmer, you can spend much less time writing code (and tests), because GPT can do that for you. Instead, you spend your time writing prompts (aka specifications), and creating an overall structure for the code that GPT is writing.\nWhat does agile development tell you to do? Roughly, to put all your effort into coding, and to spend almost no time on specification. Thinking hard about overall structure is derided as “big design upfront” (BDUF) and we’re told that we can always “refactor” later.\nLLMs have turned all this on its head. Now the focus of our work is writing thoughtful “documentation”: prompts that we give to GPT that hold our specs and all the background knowledge we’d like it to exploit. Coding matters less than ever before.\nGranular domain knowledge and concepts My two little experiments suggest that GPT’s coding ability is dramatically amplified if it has appropriate domain knowledge. In these cases, the domains (family trees and social media) were presumably well represented in GPT’s training set.\nBut what about less familiar domains? What if GPT hadn’t known about friending? How will GPT exploit domain knowledge in a novel application?\nHere’s where I believe concepts have a vital role to play. Concepts embody domain knowledge in a granular and reusable fashion. If we had a repository of concepts, we could pick the concepts that are relevant to a particular coding task and give them to GPT as prompts.\nMy intuition is that GPT already has a large number of concepts represented implicitly in its training set. I’m excited to explore what would happen if we made concepts explicit. Perhaps GPT could fill in subtle aspects of a specification more effectively if it had a fuller representation of the concept at hand. And perhaps it could replicate the kind of “common sense exception” that we saw (in the Friend example, for reading your own post) in much more novel and unfamiliar domains.\nDesign still matters My family tree story actually had a disappointing ending. Sadly, I wasn’t able to draw a nice tree. It was so big and had so many crossing edges that it was impossible to read. I tried to improve the layout by having GPT write code to assign each person to a generation, and then to layer the output so that each generation occupied one horizontal layer.\nWhen prompted, GPT did helpfully point out some of the challenges in assigning generations (eg, that the notion isn’t even well defined if you have a disconnected subtree). But it became clear that I’d hit a design problem that GPT at least wasn’t able to solve without much more prompting.\nMight a rich set of concepts for family trees have given GPT the knowledge it needed to complete this task? Maybe. But at the very least this example suggests that even apparently simple problems can harbor subtle design challenges, and for now, software designers are still needed.\n","permalink":"https://essenceofsoftware.com/posts/end-of-agile/","summary":"Why agile no longer makes sense in the LLM era.","title":"The End of Agile"},{"content":"Overview About this case study. This case study presents a concept design for a GPT-powered tutor that we built in May 2023 for experimental use in the MIT class 6.1040: Software Studio this coming fall. Geoffrey Litt had the idea for the tutor and implemented it; the concept design was a joint effort. In this video Geoffrey demonstrates the tutor in action.\nPrimary motivations. From a concept design perspective, the case study offered an opportunity to:\nApply concept design in the earliest stages of development of an app; Provide an example of concept design for a real but small problem; Illustrate how concepts can help you address the kinds of design challenges that arise even in simple apps. Modular implementation. Geoffrey used this case study also as a chance to explore the use of concepts as a modularity mechanism in the code. He implemented the tutor in his Riffle framework, and used the concept design to guide the structure of the relational tables and code, so that each Riffle component corresponds to a concept. The results seem to be promising, bringing more clarity and modularity than a traditional design might have produced. I hope that Geoffrey will write something about this at some point.\nDesign Goals and Principles Goals. The goals of building and deploying the tutor are:\nTo get some initial experience using GPT-like tools in education; To help students learn concept design; To gather data both about students’ learning of concept design and about the performance of the AI tutor; To refine the pedagogy around concept design. Pedagogical alignment. Exploiting LLMs in teaching depends not surprisingly on providing good prompts. One of the most exciting aspects of this project has been the realization that the kinds of prompt GPT seems to need correspond closely to (my perceived sense of) what students need, namely short and explicit tutorials that offer actionable advice and concrete examples, both positive and negative. Due to the limitations of window size, we are also encouraged to make our tutorials as granular as possible, so that the prompts for a particular task can be limited to only the advice that is essential for that task. A further pedagogical alignment arises in the need to explicitly identify competences, linking the tasks students are asked to fulfill to the understanding those tasks require (and thus to the advice that will help them develop such understanding).\nDesign principles. In embarking on this project, we adopted a few principles to guide it:\nTo make the Simplest Thing That Works (STTW), as agilistas would say: that is, focusing on the smallest set of concepts and behaviors that would meet the goal of supporting students effectively; While serving the needs of concept design, to make the tutor as general as possible, avoiding any needless specialization, so that it might be applied in other domains; To reduce barrier to usage by students, which suggested not requiring accounts (and keeping interactions anonymous), implementing as a cross-platform web app (without browser extensions), and not being prescriptive about the order in which students attempt exercises; Making maintenance and deployment easy for course staff, and ensuring that adding new questions and advice was straightforward, which suggested making a serverless app (the implementation requiring only a tiny lambda function to proxy API requests to GPT) and not implementing a web-based UI for staff, but instead taking questions and advice from a table-structured file that is easily edited. Choice of tech stack. Initially, one of our principles was to implement the tutor using the tech stack taught in the class (Node/Express/Vue) so that the implementation might serve as a pedagogical example for students, but we decided in the end to use Riffle instead, both to \u0026ldquo;eat our own dog food\u0026rdquo;, exercising our research on that platform, and taking advantage of Riffle’s benefits, both in ease of development and to the user (most notably the full history of the user\u0026rsquo;s interactions being persisted locally in a SQL database inside the browser, giving the appearance of a backend storage without the need to implement it or compromise the user\u0026rsquo;s privacy).\nInitial design Basic conception. We started with the simple idea that the student would be presented with a collection of structured exercises. Attempted solutions to the exercises would be graded by GPT, providing feedback to the student, and allowing repeated submissions until the student was able to submit a solution for which GPT reported no evident flaws.\nPrompt contents. We assumed that we would not fine-tune the LLM, but would use GPT out of the box with appropriate prompts. We planned to include in the GPT prompt for a particular exercise the following elements:\nThe text of the exercise itself; A general prompt instructing GPT to behave as a tutor and to provide constructive advice without revealing the answer to the student, and to return its response as a JSON structure with a simple numeric grade (corresponding to poor, ok and good) and feedback text; One or more tutorials relevant to the competence being tested; A few sample graded solutions to the particular exercise (containing both grades and feedback). First concepts. Our initial brainstorm identified the following concepts:\nExercise: a concept defining the structure of exercises and the pattern of behavior of a student in submitting solutions. Advice: a collection of pieces of advice, each being a text string, perhaps indexed in competencies. We anticipated using this advice both as prompts for the LLM and as part of a help facility for students. Bot: we assumed we\u0026rsquo;d need a concept to represent the pattern of interactions with the GPT service. The need for such a concept became less clear as the design progressed, since GPT\u0026rsquo;s API (unlike its browser interface) is not stateful, and treats each query independently, so that the behavior is little more than a series of queries that produce responses. Editor: initially, I assumed we\u0026rsquo;d want a concept to embody the text editor that the student uses to edit and submit candidate solutions to exercises. Here’s a screenshot of the running app, showing the exercises on the left (with the selected exercise part highlighted); the submission and feedback boxes the middle; and the advice and chat panels on the right (explained in more detail below):\nDesign Challenges We pursued the design of these concepts in the light of several key challenges that we identified.\nSaving student solutions. We intended to include a simple way for students solutions to be saved, both so that students themselves could review their work and so that we could evaluate which exercises were the most useful, what students were able or not able to do, and so on.\nInitially, we thought to include this functionality in the Exercise concept. This would result in a nice operational principle (OP) for the concept, with the student selecting an exercise and submitting a series of solutions. If this functionality were instead factored out into another concept, the Exercise concept would be little more than a data structure containing the exercises, and would lack a compelling OP.\nOn the other hand, we realized that the submitted solutions would need to be indexed on the identities of students. In concept design terms, this means that the state of the Exercise concept would include relations like\nsubmitted: Student -\u0026gt; Solution And yet there is no interaction between students; each student\u0026rsquo;s behavior and subsequent submitted solutions are independent. This suggests what I call lifting the concept, and defining the application using an indexed collection of concepts rather than indexing within the concept state itself. So the application might be declared like this\ninclude User UserExercise: User.User -\u0026gt; one Exercise so that it includes a User concept (which exports a type User.User corresponding to the set of users), and an indexed collection UserExercise of concepts, with UserExercise[u] being the exercise concept for user u.\nThis is bad because it inappropriately replicates not only the student-specific submission behavior but also the exercises themselves, which should be defined independently of any students.\nThis dilemma is easily resolved by factoring out the part of the concept that is indeed student specific. So we created a new concept, which we called Drill, to represent the student behavior; this concept can now be indexed, and the Exercise concept can remain a single, student-independent concept instance. As we\u0026rsquo;ll see when we elaborate Drill, the concept will also include the current state of the student\u0026rsquo;s attempted solutions, obviating the need for an Editor concept.\nWe have yet to implement solution submission, and the app runs independently within each student\u0026rsquo;s browser. So we might have maintained a single Exercise concept that is student-independent because there is only one student in scope (namely the one using this browser). But the separation into Exercise and Drill is straightforward and brings some clarity without much overhead, and it ensures that our app is ready for this likely extension.\nThe one downside of this approach is that the Exercise concept no longer has a compelling OP; all the interaction happens in Drill. This seemed a reasonable price to pay, and made me realize that I should acknowledge a class of “CRUD concepts” that are essentially just containers for data. I’ve been resisting this idea but it seems that most applications are likely to have one or more of such concepts.\nLinking advice to exercises. How should advice be linked to exercises? Our initial idea was to include, in the Exercise concept, a mapping from parts of exercises to the “competences” that they tested, and to index advice texts in the Advice concept on “topics” that would be equated to competences. The two concepts would be polymorphic in these types, with an additional Competence concept providing concrete competences to instantiate the two other concepts:\ninclude Competence Advice \u0026lt;Competence.Competence\u0026gt; Exercise \u0026lt;Competence.Competence\u0026gt; Note that I’ve overloaded Competence here so it refers to both the concept and the type of value it exports (as I did for User.User above).\nThis approach makes sense, but there wasn’t much functionality we wanted to associate with competences aside from the linking of exercises to advise. So we decided to simple have the Exercise concept define (and export) the competences. This gives the following structure\ninclude Exercise CompetenceAdvice: Advice \u0026lt;Exercise.Competence\u0026gt; ExerciseAdvice: Advice \u0026lt;Exercise.Part\u0026gt; in which Advice is instantiated with the Competence type provided by Exercise. Note that I’ve instantiated Advice twice: once for the advice associated with competences, and a second time for the advice associated with individual parts of exercises.\nMaintaining editor state. Our sketched UI included a text box in which the student enters attempted solutions. We wanted to persist the content of this box across submissions, so that when a student gets feedback and wants to adjust their last submission to create the new one, they don’t need to cut and paste the prior submission in (or, worse, have to retype it). We also thought it would be good to allow a student to switch between unfinished exercises, so that when you return to an exercise you can continue editing the solution.\nThis functionality was easily accomplished within the Drill concept, with state components for (a) the task that is selected; (b) the current text associated with each task; and (c) the submitted text for each task.\nconcept Drill \u0026lt;Task\u0026gt; state selected: lone Task current: Task -\u0026gt; lone Text submitted: Task -\u0026gt; lone Text Since the Drill concept should work with text being submitted for any kind of task, a type variable Task is used to represent what will be, when instantiated, the exercises (or actually their parts).\nThese state components are updated by actions in the obvious way: editing the text in the text box corresponds to an edit action that replaces the currentText associated with the selected task; submitting a solution corresponds to a submit action that copies the current text of the selected task to the submitted text of that task.\nBy virtue of this state and actions, this concept eliminates the need for a separate Editor concept.\nIntegrating chat. We implemented the design as described above, and also included a help panel with two tabs. One shows advice associated with the current exercise part, produced by a query that looks in the state of the Advice concept for the advice associated with the selected task in the Drill concept. The other offers a chat interaction with GPT, where the user can ask follow-up questions.\nTwo questions arose for the chat interaction. First, what would be the context for the chat? It seemed clear that there should be a separate chat thread for each exercise part, switched using the same action that switches the (solution editing) text box. The prompts for this chat would then be (a) the same advice provided for the grading, along with (b) prior submissions, and (c) earlier questions asked by the student in that thread.\nIt wasn’t obvious at first how to integrate the prior submissions and grading feedback into the chat thread. One possibility was for the chat per se to show only chat questions asked by the student and the responses to them, with the submitted solutions and their feedback shown in a different way\u0026ndash;either in a thread in a separate panel, or shown inserted into the chat thread as special objects.\nA moment’s reflection made it clear that this kind of solution was baroque and needlessly complicated. It would either require yet another UI panel (and make it harder for the user to see which follow-up questions followed which submissions) or it would make the chat thread UI non-uniform. The solution is simply to fully embrace the familiar Chat concept, in which a thread comprises a sequence of messages from two (or more) parties. Then student submissions and feedback are then just injected as regular messages, from the student and bot respectively. It might also have been possible to treat these messages as originating with two additional users, distinguishing the two roles of the student (as submitter of solutions and asker of follow-up questions) and the two roles of the bot (as grader and question-answerer). Rather than doing this, we chose to stick to two parties in the chat and to prefix the submissions and feedback with “You submitted: \u0026hellip;” and “Feedback: \u0026hellip;”.\nHere’s a screenshot showing the chat panel:\nFinal Concept Design We now outline the concepts in detail. The Exercise concept manages the exercise texts, and associates parts of exercises with competences:\nconcept Exercise purpose construct and present structured exercises principle // just a CRUD concept state parts: Exercise one -\u0026gt; set Part competence: Part -\u0026gt; one Competence description: Exercise + Part -\u0026gt; one Text (Since the purpose of the concept is just to store exercises, it doesn’t have an interesting operational principle, so rather than elaborating the expected behavior of CRUD actions, I’ve just omitted them and left the principle empty too.)\nThe Advice concept indexes advice texts on generic topic identifiers (which will later be instantiated with competences and exercise parts):\nconcept Advice [Topic] purpose manage indexed repository of advice principle // just a CRUD concept state advices: Topic -\u0026gt; set Text name: Topic lone -\u0026gt; one Text The student activity of completing an exercise is handled by the Drill concept:\nconcept Drill \u0026lt;Task\u0026gt; purpose store incremental student work on set of tasks principle // if you select a task, edit a solution // and submit it, the final edit is recorded // as the submission selectTask (t); edit (x1); edit (x2); submit (); {t.submitted = x2} state selected: lone Task current: Task -\u0026gt; lone Text submitted: Task -\u0026gt; lone Text actions selectTask (t: Task) selected := t edit (x: Text) selected.current := x submit () selected.submitted := selected.current The Task type is left generic, but will be instantiated later as the Part type of the Exercise concept.\nThe Chat concept will manage the informal exchanges between user and bot, and will have the exercise submissions and their grading injected as additional messages:\nconcept Chat \u0026lt;Party\u0026gt; purpose multi-party exchange of messages principle // series of posts gives expected msg sequence post (p1, t1, m1); post (p2, t2, m2); {msgs = \u0026lt;m1, m2\u0026gt;, m_i.author = p_i, m_i.content = t_i, etc} state msgs: seq Msg author: Msg -\u0026gt; one Party content: Msg -\u0026gt; one Text actions post (p: Party, t: Text, out m: Msg) Finally, the concept that captures the protocol of interacting with LLMs like GPT:\nconcept Bot purpose handle interaction with AI agent principle // a query is submitted and a response is given ask (q1, r1) {queries = \u0026lt;q1\u0026gt; and responses = \u0026lt;r1\u0026gt;} state queries, responses: seq Text actions ask (query: Text, out response: Text) This is almost the simplest form this concept might take, with the prompts included in the question text. A richer concept would allow prompts to be registered upfront. The API version of GPT is actually stateless, so even the question and answer sequences are not strictly necessary.\nConcept Composition The app as a whole is a composition of concepts:\napp ConceptTutor include User Exercise UserDrill: User.User -\u0026gt; one Drill \u0026lt;Exercise.Part\u0026gt; CompetenceAdvice: Advice \u0026lt;Exercise.Competence\u0026gt; ExerciseAdvice: Advice \u0026lt;Exercise.Part\u0026gt; UserPartChat: User.User, Exercise.Part -\u0026gt; one Chat \u0026lt;BOT + USER\u0026gt; Bot We haven’t specified the User concept, since it could take many forms; all that matters for tracking completed exercises is that it provide some identifiers, given by the type User.User, which indexes the Drill concept, so there is one instance of Drill for each user. In our implementation, the browser naturally provides a distinct identifier for each user and no explicit implementation of such a concept is needed. We currently aren’t recording completed exercises, but could easily do that by labeling them with a browser identifier, for example.\nThere are two instantiations of the Advice concept, one holding advice for each competence and one holding specific advice for each exercise part.\nThe Chat concept is instantiated in two dimensions, over users and exercise parts, so each user has a separate chat for each exercise part they are working on. The Party parameter of the Chat concept is instantiated with a type consisting of two simple constant values: USER representing the user, and BOT representing the LLM bot. Don’t be confused by the User vs. Party types: the users of the app as a whole are not the parties to the chats, since each chat involves only one user (and the bot).\nConcept Synchronizations Almost all of the actions of the concepts are executed independently but there are a few key synchronizations.\nOne occurs when the student submits a solution to an exercise part:\nsync userSubmitsPart (u: User)\twhen UserDrill[u].submit () let p = UserDrill[u].selectedTask c = Exercise.competence[p] q = ExerciseAdvice[p] ^ CompetenceAdvice[c] ^ Exercise.description[p] ^ UserDrill[u].current[p] in Bot.ask (q, r) UserPartChat[u,p].post (USER, q, qm) UserPartChat[u,p].post (BOT, r, rm) When a user clicks on the button to submit a solution, the relevant exercise part p is the one that is selected (in the Exercise concept), and it has an associated competence c. The advice texts for the exercise and competence are found in the Advice concept state, and concatenated with the exercise text and the student solution to form the query q to the LLM. The query and the LLM’s response r are then posted in the chat on behalf of the nominal USER party and BOT party.\nA technical note: For brevity, I’ve assumed that the out parameters of actions are declared implicitly. In a more conventional programming style, I might have written\nMsg rm = UserPartChat[u,p].post (BOT, r) for example. I prefer to treat the out parameters more like other arguments so they can be named (which matters more when an action has more than one of them). Also, I haven’t included the extra prefixes that are added to the chat posts (“You submitted: “, “Feedback: “).\nA similar but simpler sync is needed when the user posts a question to the LLM in the chat.\nThe design as a diagram The entire concept design can be shown as a diagram. Here it is:\nEach concept is shown as a box with a thick grey outline. Inside each box, there is the concept state (shown as a kind of entity relationship diagram), and arrows representing actions.\nLet’s look at the Drill concept, for example. There are two sets represented in the state by the capitalized words \u0026lt;Task\u0026gt; and Text. \u0026lt;Task\u0026gt; is a type parameter (hence the angle brackets); you can see from the dotted line that in the composition it is bound to the Part type of the Exercise concept. Text is a primitive type of text strings. The two relations between \u0026lt;Task\u0026gt; and Text are shown as solid lines, and correspond to the declarations in the textual form of the concept:\ncurrent: Task -\u0026gt; lone Text submitted: Task -\u0026gt; lone Text The “pin” labeled selected represents the selected task:\nselected: lone Task There are three solid arrows representing the three actions of the concept, and their target shows which state component they update. So selectTask sets which task is selected; edit updates the current text for a task; and submit sets the submitted text for a task.\nThe dashed arrows represent the synchronizations. So the arrow from submit to ask says that when submit happens there is a sync that causes ask to happen to; and the arrow from ask to post means that this third action will happen too.\nA few more details about the diagrammatic notation:\nRelations. I’m using lines to represent relations, as in entity-relationship diagrams. I realize that programmers who are unfamiliar with ER or UML diagrams find this confusing, but in my view it’s superior to any other way of showing an abstract state and there are no viable alternatives. No relation arrows. I’ve departed from traditional ER representations (and notation that I’ve advocated myself and taught to students in the past, such as the diagrammatic notation for Alloy) in a few key respects. There are no arrows on the relations, even though a mathematical relation is directed. I removed the arrows both because the direction is not essential to the design (it’s only needed to ensure that updates and queries are specified consistently), and more importantly, I want the design diagram to emphasize causality, so I’m reserving arrows for causal effects. Sets as pins. Another departure from ER is to show dynamic subsets as pins, rather than using a special subset arrow. I like this because it’s easier to draw and allows you to follow paths in the diagram (so selected.current, eg, is the path that gets you the current text of the selected task). Omissions. I’ve omitted some details that are present in the textual form that could be included, most notably relation multiplicities. I’ve also omitted some state components (such as the description relation for exercises), and am only showing some synchronizations. Instantiations. I’ve only shown one instantiation of Advice, and the indexing of Drill and Chat isn’t shown. I plan to write a tutorial soon explaining the notation in more detail.\nState Queries and Concept Mapping A simple action involving one concept can still result in a variety of effects in the user interface. Note in particular that when the action\nUserDrill[u].selectTask(p) is performed, in which a user u selects exercise part p as their current task, the editor buffer will be updated to display\nUserDrill[u].current[UserDrill[u].selected] which holds the current version of the text for the selected part, and the help window will display\nCompetenceAdvice.advices[ Exercise.competence[UserDrill[u].selected] ] which is the advice text for the competence associated with the current exercise part. These updates occur because the view in the user interface is defined in terms of queries over the concept states. You should think of these queries being executed reactively. Their definition is part of what I call the concept mapping and might be elaborated more or less explicitly along with the wireframe design.\nSummary of Concept Design Principles This case study has illustrated the following concept design principles:\nModularity. Because the app has only one page, and its various elements interact with each other, conventional approaches would treat the app as a monolith (eg, in a typical UX approach by specifying wireframes and flows, or in a more formal approach by creating a single data model). Using concepts, we were able to break the app into pieces that can be understood independently, and to describe the interactions as syncs. Familiarity. Concept design encourages you to use standard, familiar concepts when possible: we did this with the Chat concept, avoiding specializing it to incorporate submissions and feedback. Polymorphism. Concepts should be as generic as possible. Making Advice polymorphic in the Topic type, and not tying it to competences, allowed us to instantiate it also to hold advice about particular exercise parts. Lifting. Many concepts can be simplified by removing indexing from the concept’s local state and applying it at the app level, with an indexed collection of concept instances instead. We applied this to the Chat and Drill concepts. In the case of Drill, the opportunity to lift suggested factoring Drill out from Exercise, which also helpfully separates static data (the exercises and their components) from dynamic data (the evolving submissions). Abstraction. Details of a concept are excluded if they are not relevant to the essential design questions. For example, Drill.edit embodies all the complexity of a text editor, but for the purpose of high-level design, it’s best just to assume this. ","permalink":"https://essenceofsoftware.com/studies/larger/tutor/","summary":"Notes on design of a GPT-powered tutor","title":"A GPT-powered tutor"},{"content":"API Design at Google Concept design was developed for shaping the functionality of apps, and because it emphasizes the underlying semantics behind the user interface, it should be straightforward to apply it to the design of services. But can it be applied to APIs?\nRecently, inspired by discussions with Meital Tagor Sbero, a UX research manager at Google, I’ve started looking into whether it might be useful in the design of APIs.\nMeital runs a team working on making APIs easier for programmers to use. They have been applying cognitive dimensions of notations, a collection of heuristic principles for usability (due to Thomas Green and Marian Petre) to the design of Android APIs.\nOne of the heuristics is called closeness of mapping and suggests that a notation (or in this case an API) should correspond closely to the problem world. Concept design seems to offer a way to pursue this goal.\nAs an example of an API that programmers find hard to understand, Meital pointed me to the Android Camera API. I decided to focus for a small case study on the Rotation API, because it’s small and seems to have some complexities that concepts can address. I’ve developed a concept model of this API which I believe resolves some of the problems in understanding it, but the model is not complete and it may well contain errors that reflect my own misunderstandings. I’ve also not done the hard work of mapping the concept actions and states to the programmatic components of the API.\nWhy the Rotation API is tricky Before showing you the concepts I came up with, I need to explain why the Rotation API isn’t so easy to understand. I’ll do this by pointing to explanations within the official API documentation.\nThe documentation starts by introducing some key terms:\nReading this, I found myself confused:\nTarget = display? If the display rotation “represents the degrees by which the device is rotated counter-clockwise from its natural orientation” and the target rotation “represents the number of degrees through which to rotate the device clockwise to reach its natural orientation”, why aren’t the two always equal? And indeed, in the graphical examples that follow, the two are equal.\nHow many orientations? The display orientation is said to have only four values, but the phone can surely be held in between these. Indeed, the code sample that appears later shows an onOrientationChanged listener that gives an orientation in degrees:\nDisplay or device? The definitions seem to use the terms “display” and “device” interchangeably, but later it will transpire that the display orientation may not match the device orientation: Furthermore, a second code sample shows an alternative way to set the target rotation not using the onOrientationChanged listener but using the onDisplayChanged listener instead:\nThe next section introduces two new terms: sensor orientation, “a constant value, which represents the degrees (0, 90, 180, 270) the sensor is rotated from the top of the device when the device is in a natural position” and the image rotation which “describes how the data should be rotated clockwise to appear upright.” A series of diagrams then illustrate sensor orientations for different phones:\nThis puzzled me. How can the sensor be rotated and not be aligned with the phone? If the Pixel 3XL indeed has a sensor that is rotated 90 degrees, that would mean that the sensor would be in landscape mode when the phone is held upright in portrait mode.\nA Conceptual Take Here is my attempt to resolve these issues and explain what I believe is going on here.\nFirst, the phone contains an accelerometer that gives its rotation from the upright position. I’ll model this in a simple concept with just one state component (holding the current rotation) and an action that is called whenever the rotation changes:\nconcept DeviceRotation purpose provide physical rotation info from accelerometer principle when user rotates device R degrees counterclockwise from upright position, rotationUpdated(R) occurs state\u2028rotation: Degrees actions // system action performed whenever // change in rotation is detected // outputs current rotation and sets rotation = r rotationUpdated (out r: Degrees) I’m using the word “rotation” throughout for an angular measurement, and reserving the word “orientation” for portrait, landscape, etc. But note that this rotationUpdated action is the onOrientationChanged listener in the API.\nNote that the notion of physical rotation is non-trivial, and isn’t even defined when the phone is parallel to the ground: you’ll have noticed this when you’ve taken photos of something on the ground and been surprised that the resulting image’s orientation seems wrong. But the notion should be intuitively clear enough that we can proceed to the other concepts.\nOur second concept models just one aspect of a digital camera, namely how a captured image has a rotation associated with it:\nconcept Camera purpose take images with recorded rotations principle after setAdditionalRotation (r); capture (i) {i.rotation = r + defaultRotation} state const defaultRotation: Degrees // defined by hardware additionalRotation: Degrees rotation: Image -\u0026gt; Degrees rows: Image -\u0026gt; seq seq Pixel actions setAdditionalRotation (r: Degrees)\t// make new image i with i.rotation = defaultRotation + additionalRotation capture (out i: Image) Images are represented here naively as rows of pixels. Each image has a rotation associated with it. This rotation is intended to be applied to the image when it is displayed, and is chosen so that the image will have the appropriate orientation.\nFor example, if the camera is positioned to take an image in portrait orientation, but the first row in the recorded image corresponds to the pixels in the rightmost column as seen through the camera viewfinder, the image will need to be rotated by 90 degrees (clockwise) to be displayed correctly.\nThis scenario might still occur if the portrait orientation is the natural orientation of the camera (as it is for most mobile phones), because it’s not necessary to require the sensor always record an image so that rows are horizontal in the natural orientation.\nThis is why there is a hardware-specific defaultRotation that holds the rotation that must be applied when the camera is in its natural orientation. So for a camera that has a natural portrait orientation, but whose sensor records rows as just described, the defaultRotation will be 90, and that will determine the image rotation. Now if the camera is held in landscape orientation (by being rotated 90 degrees counterclockwise), the additionalRotation will be set to -90 degrees. As a result, the rotation of the image will be the sum of these, and will be zero, reflecting the fact that in this position the sensor happens to record the image in the expected orientation.\nThe defaultRotation is what was called “sensor rotation” in the API documentation, and the rotation associated with the captured image is the “image rotation”.\nNow finally the most subtle concept. As the user rotates the camera, we need to infer from the extent of the rotation whether the intended orientation is portrait or landscape. Clearly when the camera is perfectly upright, a portrait orientation is intended; and when it is perfectly on its side, rotated 90 degrees counterclockwise, a landscape orientation is intended.\nBut what about in between? Here’s what you might see as you rotate a camera from portrait through to landscape:\nNear the extremes, the intended orientation is clear, but in the middle (as illustrated by the example shown on the bottom right) it seems arbitrary. Perhaps the intended orientation could be inferred by looking at the sensor image, but the angle of rotation alone won’t answer the question definitively.\nTo model this, we define a concept that infers an orientation from a given angle:\nconcept UserOrientation purpose infer intended camera orientation from device angle principle (1) after updateDeviceAngle (a); getInferredOrientation (o, d) {o is the inferred orientation and d its value in degrees} (2) after toggleLock(); updateDeviceAngle (a); getInferredOrientation (o, d) {o is PORTRAIT and d is 0} state Orientation = {PORTRAIT, LANDSCAPE, PORTRAIT_REV, LANDSCAPE_REV} inferredOrientation: Orientation deviceAngle: Degrees orientationLock: Bool = false action toggleLock () updateDeviceAngle (angle: Degrees) getInferredOrientation (out orientation: Orientation, inDegrees: Degrees) The concept holds in its state:\nthe inferred orientation of the device, which has one of four values (including the “reverse” versions of portrait and landscape, namely when the camera is upside down); the angle at which the device is being held; and an orientation lock field, which when true holds the inferred orientation to be the standard portrait orientation. There are three actions:\ntoggleLock simply toggles the orientation lock; updateDeviceAngle updates the angle; getInferredOrientation returns as outputs the inferred orientation, and its equivalent in degrees. This last action embodies whatever (arbitrary) decision we make on when to switch between orientations.\nSynchronizing the concepts Now we can put the concepts together:\napp Camera include concept DeviceRotation concept Camera concept UserOrientation and connect their behaviors with synchronization. The essential one occurs when a device rotation is detected:\nsync updateRotation (out r, o: 0..359) when DeviceRotation.rotationUpdated (r) UserOrientation.updateDeviceAngle (r) UserOrientation.getInferredOrientation (_, o) Camera.setAdditionalRotation (o) When the accelerometer reports a change in the rotation of the device, a new inferred orientation is obtained from the new angle, and the additional rotation in the camera is updated accordingly.\nA diagram Here is a diagrammatic depiction that shows more directly the causal flow, from the rotation being updated in DeviceRotation to the device angle being updated in UserOrientation and then finally the additional rotation being set in Camera:\nThe diagrammatic notation is explained here (and will be explained more fully in an upcoming tutorial).\nNote that some causal steps are still implicit, in particular how the updating of additionalRotation in Camera affects the rotation of the image produced by capture.\nConnecting back to the API Now looking back at the API code sample we can understand it more readily:\nThe firing of the onOrientationChanged listener corresponds to the DeviceRotation.rotationUpdated action that fires the sync; the when statement that assigns rotation based on orientation is an implementation of the UserOrientation.getInferredOrientation action; and the setting of imageCapture.targetRotation is the Camera.setAdditionalRotation action.\nIn the alternative code sample,\nmy guess is that the following is going on. There is a separate and fuller implementation of the UserOrientation concept that rotates the display for apps in general (and is appropriately sync’d with the other concepts as above). The camera can use the orientation reported directly by this module, obtaining the rotation field of the display object. This presumably is affected by whether the orientation lock is on, and perhaps also by whether the app in question is using this feature.\nSummary In summary, we’ve resolved our initial questions as follows:\nThe word “orientation” seems to be used in two distinct ways: for the device rotation, and for the inferred orientation. The “sensor rotation” is just the default rotation associated with captured images, and the “image rotation” is their total associated rotation. Because the phone already includes a module that infers orientation and rotates the display accordingly, display orientation and device orientation are distinct, and a programmer might be able to obtain the display orientation directly, bypassing the need to make the inference in application code. What are the benefits of the conceptual view here? I’d argue that they are:\nA clean separation of concerns, into (a) obtaining the device rotation from the accelerometer; (b) storing rotations with images; and (c) inferring orientations from rotations. A purpose and simple scenarios (OPs) for the concepts, in particular the UserOrientation concept which the API docs didn’t make clear is heuristic. It remains to be seen how this kind of concept design can best be worked into API documentation. It’s clearly not ideal to have a textual notation for concepts that is different from the typical interface specs of APIs. One possibility is to represent concepts diagrammatically.\nAnother challenge involves mapping the concepts to the sometimes complex object-oriented patterns that are used in APIs, which are often more like frameworks (with application code written as subclassing extensions) than service interfaces.\n","permalink":"https://essenceofsoftware.com/studies/larger/rotation/","summary":"Applying concept design to the camera rotation API","title":"Android Camera API"},{"content":"When I wrote my book about concept design, I hoped that its ideas might nudge practicing software developers to think differently about their work. My grander, long-term dream was that whole organizations might restructure themselves around concepts.\nThrillingly, this is starting to happen. About a year ago, Peter Wilczynski got in touch to tell me about some exciting work that he and Taylor Gregoire-Wright were doing at Palantir to center their development process on concepts. Their idea was to add concepts as a new entity type to Palantir\u0026rsquo;s company-wide knowledge database, augmenting existing types (which ranged from representing individual software modules to entire applications and teams). This would be their Trojan horse, not only encouraging developers to think about their projects in terms of concepts, but most significantly, to factor out the concepts that are common to multiple apps, shining a light on them and providing a basis for efforts to align them more fully.\nPeter and Taylor had noticed an increasing number of issues reported against their products that couldn\u0026rsquo;t be neatly assigned to a module or even to an entire app. These issues weren\u0026rsquo;t simple bugs in which some function does the wrong thing; they were manifestations of deeper conceptual flaws, in which an app\u0026rsquo;s behavior didn\u0026rsquo;t match user expectations, or in which two apps in their product suite presented essentially the same functionality in confusingly different ways.\nOver a period of a year or so, Peter and Taylor insinuated concepts into Palantir\u0026rsquo;s daily work. They organized many of their internal design documents to highlight key concepts, creating new concept entries in the knowledge base and linking them to the relevant documents. They invented a new structure, the concept cluster, to represent a collection of concepts that are typically used together. And, most impressively, they persuaded their colleagues to follow along, identifying their own concepts and concept clusters, and thinking about their design work in a new way.\nThis experiment is in its early days, but the results so far are promising. The collection of concepts in the knowledge base is growing, from only in handful at the start of 2023 to over 150 now, with several hundred Palantir employees accessing the concepts in the knowledge graph regularly.\nMost exciting, in my view, is the way concepts are changing the role of product managers, who typically have few opportunities for career advancement unless they move into management. In Palantir, product managers can now become owners of clusters of concepts that cross application boundaries, becoming the company\u0026rsquo;s experts on key product assets. In this role, they can ensure that valuable design experience (which is often tied to concepts rather than code modules, and is thus easily lost as members rotate in and out of product teams) is maintained and exploited as products mature. And with their larger view of the product family, these product managers are well positioned to align different products, so that their functionality is uniform and consistent, and to prevent needless reinvention.\nPeter, Taylor and I recently wrote an experience report about this effort, which will appear in the SPLASH Onward! conference this fall. We expect to improve the paper in line with the helpful suggestions of the reviewers. In the meantime, a draft of the paper is available on Arxiv.\nI was delighted (and somewhat surprised) that my coauthors, with the approval of the company, were able to talk in detail about what Peter calls the \u0026ldquo;dynamics\u0026rdquo; of concept development. In two brief case studies, the paper explains how two concepts were created and evolved, drawing on prior concepts and lessons from the experiences of users. The paper offers some preliminary advice to others interested in reorienting software design around concepts\u0026ndash;for example, the importance of educating engineers about concepts, assigning staff to curation of the concept inventory, and integrating concepts into existing work planning and task tracking.\nI learned a lot working with Peter and Taylor on the paper. One lesson involved microservices. I sometimes explain concepts as \u0026ldquo;nanoservices\u0026rdquo;\u0026mdash;like microservices but smaller. I knew that microservices hadn\u0026rsquo;t fulfilled all their promises (and had supervised an industrial masters thesis focused on reducing microservice dependences). But I was surprised to learn that, in Peter and Taylor\u0026rsquo;s experience, adopting a microservice architecture often leads to even less modularity, since microservices encourage teams to replicate functionality that has been implemented in other teams\u0026rsquo; microservices. Concepts may act as a corrective, allowing teams working on different microservices to identify shared functionality.\nIn a similar vein, domain-driven design, a deservedly popular approach to development, may also have a tendency towards replication of functionality. Teams are discouraged from discovering commonalities with the designs of other teams, since they lie beyond the \u0026ldquo;bounded context\u0026rdquo; that isolates one team\u0026rsquo;s work from another\u0026rsquo;s. In addition to resulting in wasted development effort, this may make it hard to identify the concepts that comprise a company\u0026rsquo;s key assets, and to ensure consistent experiences across products. Concept design may help redress the balance, and break down some barriers between teams by offering a succinct way to express shared assets.\nOne anecdote I can\u0026rsquo;t resist telling you about: We had explained in the paper how including concepts in the knowledge base was helping employees working in marketing and strategy, because the concepts provided a simpler account of each app\u0026rsquo;s functionality and of the value of product improvements. Since the paper shared a considerable amount of detail about Palantir\u0026rsquo;s software development practices and was not shy about reporting some challenges that teams had faced, it was necessary (of course) to run the paper by Palantir\u0026rsquo;s legal team prior to publication. In their email to us approving publication, they suggested that concepts might help them in their work too, by highlighting the essential IP of their products. I\u0026rsquo;d hinted at this role for concepts in Chapter 3 of my book, but was delighted to see it acknowledged by experts.\nOne of the reasons that improvements in design processes are so rare is that companies are often reluctant to make investments that don\u0026rsquo;t pay off immediately. As Nelson Repenning and John Sterman explain in an insightful paper from 2001, improvements to capability may not only fail to bring benefits in the short term, but cutting back on new capabilities may produce a productivity boost (albeit one that is short-lived). It takes courage to slow down the production line to sharpen the tools.\n","permalink":"https://essenceofsoftware.com/posts/palantir/","summary":"Lessons learned from deploying concepts at scale.","title":"A Concept Experiment at Palantir"},{"content":"Goals and Principles These notes are intended to contribute to the design of a tutor based on GPT that Geoffrey is building for use in Software Studio this fall.\nSome of the goals of building and deploying the tutor are:\nTo get some initial experience using GPT-like tools in education; To help students learn concept design; To gather data both about students’ learning of concept design and about the performance of the AI tutor; To refine the pedagogy around concept design. Some design principles of the tutor are:\nTo make the simplest thing that works; While serving the needs of concept design, to make the tutor as general as possible so that it can be applied in other domains; To reduce barrier to usage by students; To make it easy for class staff to change the materials and exercises; To implement using the class tech stack\u0026ndash;Node/Express, Vue (or React if that’s easier for Geoffrey)\u0026ndash;so the tutor can be used as an exemplar for students and also easily modified by staff; Identifying concepts The central concept captures the protocol of interacting with tools like GPT:\nconcept ChatBot purpose interact with AI agent principle new (p, s); ask (s, t); respond (s, t); ... state prompt: Session -\u0026gt; one Text questions, answers: Session -\u0026gt; seq Text actions new (prompt: Text, out s: Session) // create new session with given prompt ask (s: Session, t: Text) // user enters text respond (s: Session, t: Text) // bot responds close (s: Session) // needed? The operational principle says that a new session is created with a prompt; in our design, this will turn out to be hidden from the user. The user then enters some text (modeled by the ask action) and the bot responds.\nThe prompt will turn out to be a concatenation of distinct components, such as a general tactic for the bot to follow; a pedagogical context for the exercise at hand; and the starting seed for the exercise itself (eg, “ask the user to do X”).\nThe student will work in a mutable buffer, generating a sequence of immutable texts:\nconcept Buffer purpose create immutable text with mutable buffer principle // if you create a buffer and edit it, // saving produces the last content new (b); edit (b, t); save (t') {t' = t} state content: Buffer -\u0026gt; one Text actions new (out b: Buffer) // create new buffer edit (b: Buffer, t: Text) // edit buffer so new content is t save (b: Buffer, t: Text) // save contents of buffer close (b: Buffer) // needed? The save action is intended to be sync’d with another concept; I’ve factored out the means by which the text is saved.\nThe chatbot is fueled by a repository of advice, which I’ll model as a concept that offers advice on various topics:\nconcept Advice [Topic] purpose manage repo of advice principle // create topic, add advice, then getting // advice will return all advice added to that // topic newTopic (n, t); addAdvice (t, a1)... addAdvice (t, aN); get (t, a') {a' = a1 + ... + aN} state advices: Topic -\u0026gt; set Text name: Topic lone -\u0026gt; one Text actions newTopic (name: Text, out t: Topic) addAdvice (t: Topic, advice: Text) getAdvice (t: Topic, out advices: set Text) Side notes. This is a strange concept because it has so little behavior but it seems to represent a very significant part of the tutor, and would likely have its own code. I’ve included an action getAdvice to help reify the operational principle even though the state is visible and observer actions aren’t generally needed (but I think I should rethink this for OP-relevant state observations). At first, I incorporated advice into a concept that managed exercise/part structure, but then when I realized that that concept needed to incorporate the student behavior, it became too complex and I decided this was a better concept to factor out than a student behavior concept. Could this concept be made more similar to one that already exists, such as a Q\u0026amp;A concept of the sort you might use to model StackOverflow etc? In the AI tutor setting, the advice will be given to the chatbot, not the student, but this should be irrelevant in the design of the concept.\nThe pedagogical content is organized as a collection of exercises with multiple parts; the parts are associated with competencies (which will be the topics for advice).\nconcept Exercise [Competency] purpose maintain and offer structured exercises principle // admin creates exercise with parts // when student selects part, they get // the relevant descriptions newExercise(te, e); newPart (e, tp, c, p); selectPart (e, p, t', c') {t' = te^tp, c' = c} state parts: Exercise one -\u0026gt; set Part competence: Part -\u0026gt; one Competency description: Exercise + Part -\u0026gt; one Text actions newExercise (t: Text, out e: Exercise) newPart (e: Exercise, t: Text, c: Competency, out p: Part) selectPart (e: Exercise, p: Part, out t: Text, out c: Competency) // when student selects a part of an exercise, // they're given a description that combines // the description of the exercise with the // description of the part (and the competency // they'll acquire) Notes\nBoth exercises and their parts have descriptions Competency associated only with parts and not whole exercises. This will mean we can’t associate advice with exercises as a whole. Perhaps advice common to all parts is just included for those parts? Parts are not shared between exercises Considered including student behavior in this concept to make it richer but then decided it was more important to factor it out so that the student behavior concept could be lifted over students (that is, each student behavior being independent) Concept for student behavior:\nconcept Drill purpose govern student work blah... principle // if you select a task and one of its subtasks // and then submit a solution, it's recorded // appropriately selectTask (t); selectSubTask (t, t'); submit (s) {t' in t.subtasks and t.solution = s} state subtasks: Task one -\u0026gt; set Task solution: Task -\u0026gt; lone Text current: lone Task actions selectTask (t: Task) // set current to t selectSubTask (super, sub: Task) // set current to sub // ensure sub a subtask of super submit (t: Text) // current.solution := t Notes\nDecided to make this cursor-based; not sure this is a good idea, but seems useful to have this state somewhere in the app. OP is not very satisfying but at least suggests how concept is used. Composing concepts app Tutor [Student, Competency] includes\tBuffer Advice [Competency] Exercise [Competency] Drills: Student -\u0026gt; Drill Notes\ndrills is a set of concept instances indexed by student. How to deal with type that has no allocator in any concept? For now, just making such types parameters of the app itself and will assume that they get allocated in syncs. Example of a sync:\nsync selectPart (s: Student, e: Exercise.exercise, p: Exercise.Part) // get part's text to present to student Exercise.selectPart (e, p, t, c) // get advices associated with competency Advice.getAdvice (c, as) // prompt chatbot with concatenation of advices\tChatBot.new (concat(as), session) // where does the session go? // set the student context for answering Drills[s].selectSubTask (e, p) Buffer.new (b) // where does the buffer go? ","permalink":"https://essenceofsoftware.com/drafts/ai-tutor-v1/","summary":"Notes on design of a GPT-powered tutor","title":"Notes on design of a GPT-powered tutor"},{"content":"Goals and Principles These notes are intended to contribute to the design of a tutor based on GPT that Geoffrey is building for use in Software Studio this fall.\nSome of the goals of building and deploying the tutor are:\nTo get some initial experience using GPT-like tools in education; To help students learn concept design; To gather data both about students’ learning of concept design and about the performance of the AI tutor; To refine the pedagogy around concept design. Some design principles of the tutor are:\nTo make the simplest thing that works; While serving the needs of concept design, to make the tutor as general as possible so that it can be applied in other domains; To reduce barrier to usage by students; To make it easy for class staff to change the materials and exercises; To implement using the class tech stack\u0026ndash;Node/Express, Vue (or React if that’s easier for Geoffrey)\u0026ndash;so the tutor can be used as an exemplar for students and also easily modified by staff; Identifying concepts The central concept captures the protocol of interacting with tools like GPT:\nconcept ChatBot purpose interact with AI agent principle prompt (t0); ask (t1); respond (t2); ... state prompt: lone Text questions, answers: seq Text actions prompt (t: Text) // initialize the chatbot with a prompt ask (t: Text) // user enters text respond (t: Text) // bot responds The prompt will turn out to be a concatenation of distinct components, such as a general tactic for the bot to follow; a pedagogical context for the exercise at hand; and the starting seed for the exercise itself (eg, “ask the user to do X”).\nThe student will work in a mutable buffer, generating a sequence of immutable texts:\nconcept Buffer purpose create immutable text with mutable buffer principle // if you open buffer and edit it, // saving produces last content open (t); edit (t'); save (ts) {ts = t'} state content: Text actions open (t: Text) // reset buffer contents to t edit (t: Text) // edit buffer content to be t save (out t: Text) // save contents of buffer The save action is intended to be sync’d with another concept; this factors out the means by which the text is saved and where it’s saved to.\nConcept design theory aside. This concept might seem so simple that its functionality should be absorbed into another concept (eg, Drill, below). But this would be a mistake. First, it embodies distinct and complex functionality\u0026ndash;a complete text editor. The edit action represents very abstractly and generally all possible edits. Second, by separating the concept out we’re able to lift it (that is, create an indexed collection of instances without having to pollute any concept with a spurious index type).\nThe chatbot is fueled by a repository of advice, which I’ll model as a concept that offers advice on various topics:\nconcept Advice [Topic] purpose manage repo of advice principle // create topic, add advice, then getting // advice will return all advice added to that // topic newTopic (n, t); addAdvice (t, a1)... addAdvice (t, aN); get (t, a') {a' = a1 + ... + aN} state advices: Topic -\u0026gt; set Text name: Topic lone -\u0026gt; one Text actions newTopic (name: Text, out t: Topic) addAdvice (t: Topic, advice: Text) getAdvice (t: Topic, out advices: set Text) Concept design theory aside. Another concept that might seem too simple to represent. In this case, the behavior is indeed very simple, but the concept is factored out because it represents very distinct and domain-dependent functionality, and represents a significant part of the tutor. I’ve included an action getAdvice to help reify the operational principle even though the state is visible and observer actions aren’t generally needed. Could this concept be made more similar to one that already exists, such as a Q\u0026amp;A concept of the sort you might use to model StackOverflow etc? In the AI tutor setting, the advice will be given to the chatbot, not the student, but this should be irrelevant in the design of the concept.\nThe pedagogical content is organized as a collection of exercises with multiple parts; the parts are associated with competencies (which will be the topics for advice).\nconcept Exercise purpose maintain and offer structured exercises principle // admin creates exercise with parts // when student selects part, they get // the relevant descriptions newExercise(te, e); newPart (e, tp, c, p); getPart (e, p, t', c') {t' = te^tp, c' = c} state parts: Exercise one -\u0026gt; set Part competency: Part -\u0026gt; one Competency description: Exercise + Part -\u0026gt; one Text actions newCompetency (out c: Competency) newExercise (t: Text, out e: Exercise) newPart (e: Exercise, t: Text, c: Competency, out p: Part) getExercise (e: Exercise, out t: Text) getPart (e: Exercise, p: Part, out t: Text, out c: Competency) // when student selects a part of an exercise, // they're given a description that combines // the description of the exercise with the // description of the part (and the competency // they'll acquire) Notes:\nBoth exercises and their parts have descriptions Competency associated only with parts and not whole exercises. This will mean we can’t associate advice with exercises as a whole. Perhaps advice common to all parts is just included for those parts? Parts are not shared between exercises Considered including student behavior in this concept to make it richer but then decided it was more important to factor it out so that the student behavior concept could be lifted over students (that is, each student behavior being independent) selectexercise was added so that it could be synchronized with the resetting of the buffer as the student begins a new exercise. Competencies will presumably have textual names, but treating them abstractly in the concept. Concept for student behavior:\nconcept Drill [Task] purpose store incremental student work on structured task principle // if you select a task and one of its subtasks // and then submit a solution, it's recorded // appropriately selectTask (t); selectSubTask (t, t'); submit (s) {t' in t.subtasks and t.solution = s} state subtasks: Task one -\u0026gt; set Task solution: Task -\u0026gt; lone Text current: lone Task actions selectTask (t: Task) // set current to t selectSubTask (super, sub: Task) // set current to sub // ensure sub a subtask of super submit (t: Text) // current.solution := t Notes:\nDecided to make this cursor-based; not sure this is a good idea, but seems useful to have this state somewhere in the app. OP is not very satisfying but at least suggests how concept is used. An important design issue. For simplicity, this design assumes that student work is saved only for subtasks (which will correspond to parts of exercises). There is no separate storage of whole exercise solutions. Instead, the text of an exercise solution is derived from the texts of the solutions of the parts. The student may be shown a coherent block of text representing their solution to the whole exercise but will not be able to edit it: only edits to solutions to parts are allowed. The Drill concept maintains the fragmentary solutions; the app can display a solution to an entire exercise by combining state from Exercise and Drill.\nA concept is needed to manage student sessions and provide access to students to the work they have done:\nconcept Session purpose maintain persistent identities of clients principle // if you open a session on a client and then // get the session at the client before closing, // you'll get the session that was created on open open (c, s); getSession (c, s') {s' = s} state session: Client -\u0026gt; one Session actions open (c: Client, out s: Session) // create new session on client getSession (c: Client, out s: Session) // return session associated with client close (s: Session) // close session and forget client Note. Client abstracts the implementation details of how the server associates session state with the user’s browser, tab etc. Probably easiest to generate a session id on open and install a cookie containing it.\nA concept that contains all the functionality for remember email addresses and sending emails:\nconcept Email [Principal] purpose support sending progress emails to users principle // if you register a principal with an email // and send an email to that principal // then it will be sent to their address register (p, a); send (p, t) {message {t} sent to p.address} state address: Principal -\u0026gt; one Address actions register (p: Principal, a: Address) forget (p: Principal) send (p: Principal, t: Text) // send a message containing t to p Composing concepts All the concepts are assembled into an app, with one instance of the Buffer and Drill concepts for each session, the Drill concept instantiated so that its tasks are exercises and parts from the Exercise concept, and one instance of the Chatbot concept for each combination of session and task:\napp Tutor includes\tSession Email Exercise Advice [Exercise.Competency] Buffers: Session.Session -\u0026gt; one Buffer Drills: Session.Session -\u0026gt; one Drill[Exercise.Exercise + Exercise.Part] Chatbots: Session.Session -\u0026gt; Drills.Task -\u0026gt; Chatbot Note. There will actually only be one chatbot instance per subtask, even though the declaration suggests one per task.\nWhen a user opens a new session, their email address is registered:\nsync open (c: Client, addr: Email.Address, out s: Session.Session) Session.open (c, s) Email.register (s, addr) When a user selects an exercise, a new task is selected associated with that exercise, and the user’s buffer is cleared:\nsync selectExercise (c: Client, out s: Session, ex: Exercise.Exercise, out exerciseText: Text) Session.getSession (c, s) Exercise.getExercise (ex, exerciseText) Drills[s].selectTask (ex) Buffers[s].open (\u0026quot;\u0026quot;) The text associated with the exercise (exerciseText) will be displayed to the user.\nWhen a user selects a part of an exercise, a new subtask is selected associated with that exercise, the user’s buffer is reset with the content of the prior solution to that part (or an empty string if none), the advice stored for the competency associated with that part is obtained, and the associated chatbot is prompted with that advice and the textual description of the exercise part:\nsync selectPart (c: Client, out s: Session, ex: Exercise.Exercise, part: Exercise.Part, out comp: Exercise.Competency, out partText: Text, out advice: Text) Session.getSession (c, s) Exercise.getPart (ex, partText, comp, part) Drills[s].selectSubTask (ex, part) Buffers[s].open (emptyIfNone(part.solution)) Advice.getAdvice (comp, advice) Chatbots[s][part].prompt (advice ^ partText) When a user saves the buffer, the content is submitted as a solution (to the current part), that solution is presented as a query to the relevant chatbot, and the solution is also emailed to the user:\nsync submitSolution (c: Client, out s: Session, out solution: Text) Session.getSession (c, s) Drills[s].submit (solution) Buffers[s].save (solution) Chatbots[s][p].ask (solution) Email.send (s, solution) Notes. In this current formulation, the query to the chatbot includes only the solution to the current part of the exercise, but perhaps it should contain the (derived) current version of the entire exercise solution. The chatbot will respond to the query spontaneously; no sync is needed to describe that.\nSummary of Concept Design Ideas Here are some of the principles that guided the conceptual design:\nFactoring. Reasons for factoring out concepts include: standard reusable functionality (Session); separable/optional feature (Email); separable feature with considerable complexity (Buffer, Chatbot); to exploit lifting so that concept need not include indexing (Drill); functionality that is separable and may be subsequently elaborated (Advice). Lifting. Whenever a concept might have included an indexed state component, but the different indexed parts of the state never interact, the concept should instead be lifted. For example, each buffer has its own behavior, and there are no actions across buffers, so rather than having the Buffer concept include an indexed collection of buffers, the concept itself is instantiated, indexed over sessions. Email should probably be lifted too. Abstraction. Details of a concept are excluded if they are not relevant to the essential design questions. For example, Buffer.edit embodies all the complexity of a text editor, but for the purpose of high level design, it’s best just to assume this. Polymorphism. Concepts are made polymorphic when possible. For example, the tasks of the Drill concept and the competencies of Advice can be bound to any types. Concept vs sync design. In some apps, the design of the concepts themselves will be challenging. In this case, the main problem was how they fit together, so laying out the essential actions and figuring out how they would be synced was primary. Open Design Questions The concept design offers only a draft attempt at resolving the tricky questions about how the various parts of the app should interact. Particular decisions embodied in the concept design that should be revisited include:\nAssociating chatbot sessions with parts of exercises. Deriving exercise solutions from part solutions, and not allowing them to be edited explicitly. Conveying solutions to user by emailing submitted solutions to subparts. Clearly, users may prefer an explicit action for sending or saving a solution, which should probably send the entire exercise solution. There should also be a way to send the chatbot transcript. Associating prompts with competencies tied to parts, and not with whole exercises, and feeding the text of the exercise part as part of the prompt. The chatbot instances are associated with parts of exercises, and their sessions persist, so that if you select a part, then select another part and come back to the first, you’ll be in the same session. \u0026hellip; ","permalink":"https://essenceofsoftware.com/drafts/ai-tutor-v2/","summary":"Notes on design of a GPT-powered tutor","title":"Notes on design of a GPT-powered tutor"},{"content":"Intro This is an attempt to apply the concept design framework of EOS to Gordon Brander’s Noosphere. Because I don’t understand Noosphere well, it will no doubt contain many errors, but it seemed like a fun and useful exercise, both as a test for concept design and perhaps also to offer a different perspective to the design of Noosphere itself.\nIdentifying concepts The first step is to identify a collection of concepts that together can reproduce Noosphere’s basic functionality. Rather than starting with the concepts that Noosphere highlights (notably the central Sphere concept), I’ve followed the strategy of identifying concepts bottom-up, trying to be as granular as possible, and whenever possible reusing existing concepts that are domain independent.\nAfter identifying a concept, I attempt to describe it using the standard EOS structure: a name, a purpose for the concept, an operational principle (an archetypal scenario explaining how it’s used), a state schema, and some actions.\nCrucially, concepts can be generic, defined over some type variable (given after the name) which is bound when the concept is instantiated in a given design context.\nThe most basic concept seems to be the concept of a Notebook that is little more than a collection of notes:\nconcept Notebook [Note] purpose manage collection of owned notes principle // if you create a notebook and add a note // the note belongs to the notebook. create(b); add(b, n) {n in b.notes} state notes: Book one -\u0026gt; set Note actions create (out b: Book) // create new book b delete (b: Book) // delete book b and its notes add (b: Book, n: Note) // add note n to book b remove (b: Book, n: Note) // remove note n from book b Related concepts. Notebook is similar to Folder, although notebooks usually can’t be nested. In contrast to Label (used, eg, to organize messages in Gmail) and Collection (used to organize photos in Lightroom), a note usually cannot belong to more than one notebook.\nDesign question: sharing. Can notes be shared between users? The Notebook concept does not allow a note to be shared between notebooks. A single notebook could be owned by multiple users, however. We might add an Ownership concept later.\nDesign question: mutability. Notebooks are mutable, but the concept does not commit to whether notes themselves are. In most existing apps (such as Notion), notes are indeed mutable, but they won’t be in Noosphere.\nDesign question: genericity. The Notebook concept allows any type of note, but in practice this will be restricted to some standard media type (text, rich text, image, audio, movie, etc).\nDesign question: minimality. The boring operational principle suggests that maybe there may be more to a notebook than this. Am I missing something?\nNow we turn to the notes themselves, and for now consider only a concept for textual notes:\nconcept Textnote purpose provide editable notes with textual content principle // if you open a buffer, edit some text in it, // and save the buffer to a note, then the note // contains the text open(b); edit(b,t); save(b, n) {n.content = t} state content: Note -\u0026gt; one static Text current: Buffer -\u0026gt; one Text actions open (out b: Buffer) // create a new buffer with empty text content open (n: Note, out b: Buffer) // create new buffer with text content of existing note edit (b: Buffer, t: Text) // update content of buffer b to be t save (b: Buffer, out n: Note) // create new note with content matching current content of buffer close (b: Buffer) // discard buffer and association with current text Related concepts. Textnote is similar to the File concept in GitHub’s web interface: when you open and edit a file in a repo, you’re modifying a buffer which you can then commit to create a new version of the file.\nDesign question: mutability. This concept introduces a type Note of immutable notes. The concept attempts to define the basic functionality needed for creating immutable notes incrementally; the idea is to introduce a mutable Buffer type for editing, which produces immutable notes on saving.\nNoosphere uses the conventional concept of content-addressable storage for naming notes, which I’ll attempt to define in a concept:\nconcept ContentAddress [Object] purpose name objects by content principle // if you create an object, and later get an // object with the same address, it will // have the same content create(o, a); get(a, o') {o’.content = o.content} state address: Object -\u0026gt; one static Address content: Object -\u0026gt; one static Text actions create (o: Object, t: Text, out a: Address) // create object o with contents t and // compute a new address and store it get (a: Address, out o: Object) // return any object with address a forget (a: Address) // forget text associated with an address Design question: purpose. I am not sure how to express the purpose, that is what motivates this concept. It seems to be a combination of a naming scheme that works across storage sites (since an object implicitly carries its own name), and a kind of built-in authentication (since modifying of corrupting an object will change its content and thus its name).\nDesign question: copies. The concept makes explicit that there may be several objects that are copies of the same content and that therefore have the same name. The get action is undetermined by design: it doesn’t specify which object is returned when more than one with the right address is available.\nDesign question: immutability. Note that the address and content relations are immutable: the address is computed by a hash function that is assumed not to change, and the content is not permitted to change (since otherwise the address would no longer match the contents).\nDesign question: collisions. The operational principle actually relies on there being no hash collisions. This is a reasonable assumption in practice.\nDesign question: content type. I’m assuming for now that the contents are text, since this is compatible with the Textnote concept introduced before. Obviously, content addressing could apply to a lower level representation and could handle other media types uniformly.\nDesign question: indexing. The concept assumes some kind of index (or cache) that comprises the address component of the state; at the moment, I’m not concerned with how this index might be localized, but will just assume it’s always available. Given this, the role of the forget action isn’t clear.\nIn Noosphere, notebooks are named by public keys. The purpose of this, presumably, is to avoid having a global directory mapping principals or notebooks to public keys: instead they just carry their public keys in their names.\nconcept PublicKeyName [Principal] purpose simplify key distribution principle state public, private: Principal one static -\u0026gt; one static Key actions generate (p: Principal, out pr, pu: Key) // generate new key pair for principal find (k: Key, out p: Principal) // return principal with public key k sign (t: Text, p: Principal, out s: Text) // sign text t with private key of p check (t: Text, p: Principal, s: Text, out valid: Bool) // with public key of p check signature s matches t Technical issue. Because concepts express state so abstractly (as relations), the difference between the key/principal relationship being in a global table and being carried by (a representation of) the principal itself isn’t made. This is a bit weird. For now, I’m not sure how to express the fact that the key is a name, except to provide an action to find a principal given a key, which is the quintessential naming lookup (and would presumably be missing in a key directory).\nDesign question. I’ve declared the key/principal relations to be static in both directions, which means that (a) you can’t assign a new key to an existing principal, and (b) you can’t assign a new principal to an existing key (even its principal no longer exists). If not, Noosphere would break because references to notebooks would change in meaning over time. Notebook owners will have to make sure they do key generation just once and save their private keys carefully!\nContent-addressable names and public-key names are not mnemonic, so Noosphere introduces user-defined shorthands:\nconcept Shorthand [Context, Object] purpose mnemonic, local naming of objects principle // if you bind a short to an object // you can then look it up with that short bind (c, o, s); lookup (c, s, o') {o = o'} state resolve: Context, Short -\u0026gt; one Object actions bind (c: Context, o: Object, out s: Short) // set shorthand for o in context c to be s // remove bindings for existing objects named s in c unbind (c: Context, s: Short) // remove shorthand s from context c lookup (c: Context, s: Short, out o: Object) // return object with shorthand s in context c Design issue: uniqueness. In a given context, a shorthand must name a unique object but the same shorthand can name different objects in different contexts. Shorthands (unlike content addresses) are not canonical: the same object can have multiple shorthand names, even in the same context.\nDesign issue: mutability. The shorthand mapping is not static: a shorthand can name different objects at different times. In fact, this will be essential in Noosphere, since this will allow a shorthand to refer to the latest version of a note.\nTechnical issue: naming names. The target of a shorthand, represented by the type variable Object, may itself be a name in another naming concept.\nTechnical issue: shorthand name type. The Short will usually just be a text string, but since no properties of the name beyond equality are assumed, the type is abstracted here.\nNoosphere has a simple kind of versioning, in which notes can point to their prior versions. I’ll model that with a generic versioning concept:\nconcept Version [Object] purpose access to and restoring of old versions principle // if you register p as previous version of o, // you'll get p if you ask for o's previous version register (o, p); getPrev (o, p') {p' = p} state previous: Object -\u0026gt; lone Object actions register (o, prev: Object) // record that prev is previous version of o getPrev (o: Object, out prev: Object) // return previous version of o (it exists) Design issue: invariants. Each object can have at most one predecessor, but potentially any number of successors. Cycles aren’t permitted; in particular, an object can’t be its own previous version.\nComposing concepts Now we can assemble the concepts by composition. The first step is to instantiate the concepts, binding their type parameters to domain-specific types:\napp Noosphere include Textnote Notebook [Textnote.Note] ContentAddress [Textnote.Note] PublicKeyName [Notebook.Book] Shorthand [Notebook.Book, Textnote.Note] Shorthand [Notebook.Book, Notebook.Book] Version [Textnote.Note] These inclusions say that the text notes of the Textnote concept comprise the notes of the Notebook concept and the objects of the ContentAddress concept. The PublicKeyName concept is applied to the books of the Notebook concept: that is, each notebook has its own public key.\nThe Shorthand concept is instantiated twice, to introduce shorthands for notes and for books. In both cases, the context is a notebook, the idea being that each notebook interprets the names within its notes. If a note in my notebook includes the name @gordon/composability, this will be resolved by finding the public key associated with @gordon, and then obtaining within the notebook with that public-key name the note whose content-address is associated with composability.\nThe second step is to synchronize the actions of the concepts. Let’s focus on a few of the more interesting ones.\nHere’s an attempt at explaining what happens when a note is saved:\nsync save_note ( b: Notebook.Book, x: Textnote.Buffer, s: Short, prev: Textnote.Note, out n: Textnote.Note, out t: Text, out a: ContentAddress.Address) when t = Textnote.current[x] and prev in Notebook.notes[b] Textnote.save (x, n) Notebook.add (b, n) ContentAddress.create (n, t, a) Shorthand.bind (b, a, s) Version.register (n, prev) The arguments of the action include the book to which the note is being saved; the buffer being edited, a shorthand name being assigned to the new note, and the identity of the previous version of the note.\nWhen the note is saved, the following actions occur in each of the concepts:\nIn Textnote: the content of the buffer is saved to a note; In Notebook: the resulting note is added to the book; In ContentAddress: a new address is computed for the note; In Shorthand: the shorthand is bound to the new note (and, if used for previous versions, unbound); In Version, the new note is registered as the new version of the previous note. Design question: latest version naming. Assuming that the user of this notebook reuses the same shorthand name for all versions of this note, an external user requesting a note with a given shorthand name will now acquire a copy of the new note. This seems right.\nDesign question: content addresses not distinct. If the text in the buffer happens to be the same as the text in an existing note in this notebook, the new note that is created will have the same content address as the existing note. Two distinct notes (with different version histories) will have the same content address name. Presumably this isn’t a problem, since a request for a note from an external user will provide only its content, and will not reveal the identity of the note or its version history. That is, content addresses aren’t canonical for the local notes in a notebook, but only for the cache of notes obtained externally.\nDesign question: cross-notebook versioning. The precondition requires the previous version of the note to belong to the same notebook. The alternative seems like trouble, because version tracking requires notebook identities, which are hidden outside a notebook.\nDesign question: unstable shorthands. We haven’t specified the relevant syncs for this, but I’m assuming that a notebook does not hold, in its shorthand context, the bindings for external notes. This means that the meaning of a reference appearing in a note in Alice’s notebook to a note in Bob’s notebook may change when Bob rebinds the shorthand. If Alice wants to avoid this, presumably she could reference the note by its content address (you could imagine an action that lets you specify the current shorthand for a link but then inserts the relevant content address). Another possibility would be that a notebook has a shorthand context not only for external notebooks, but also for notes in those notebooks. Then shorthands would be stable but would not get automatic version updates.\nDesign question: local resolving of shorthands. When a reference in a note obtained from an external notebook is resolved, is it cached locally? Presumably this means that different clients in a P2P network will disagree on the meaning of a shorthand?\nDesign question: notebook as shorthand context. It is not necessary that the notebook be the context for shorthands; there could be multiple contexts per notebook, or multiple notebooks per context. The notion of a “sphere” that combines notebook and shorthand context seems like the right default.\nReflections This exploration suggests some opportunities and issues for concept design (CD):\nDecoupling. CD seems to offer a nice way to decouple the elements of an intricate design like Noosphere, and to identify and incorporate the relevant preexisting concepts. Catalog. Defining the preexisting concepts was pretty hard, reflecting partly my own lack of knowledge, but also the subtlety of some of these concepts. This whole exercise makes it clear to me how valuable a catalog would be, so that a designer could just look up the concepts, learn about them, incorporate the right variants, and not have to figure them out. More and less well-documented concepts. Some of the concepts, such as ContentAddress, correspond to technology ideas that have been extensively described but not formulated as pithy concepts. Others correspond to ideas that must be well known to some but haven’t been widely articulated. Textnote, for example, tries to represent a simple scheme for an editing process that gives the appearance of a mutable document but produces a sequence of immutable instances. (This suggests I should probably have focused the concept purely on that, factored out the textual aspect and made it generic.) Synergy. Some of the compositions are probably resulting in design synergies that it would be good to explore. Using the Sphere for bounding the notebook, providing a context for shorthands, and carrying a public key is presumably synergistic in ways it would be good to figure out. Maybe public key naming should be described as a synergistic composition of a generic naming scheme and a naming-independent PKI concept. Allocation. Deciding when to have actions for allocation of objects is tricky, and I wanted to have concepts that could both allocate their own objects and compose with other concepts that would do the allocation instead. (This issue arose for allocating contexts in Shorthand: originally I had an allocation action in the concept itself, but then wasn’t sure how to treat the notebook, which is allocated elsewhere, as a context.) Perhaps the solution to this is to find a different way to model allocation of objects in concepts, maybe with some simple implicit default, so that it doesn’t need to be considered in most concept definitions. ","permalink":"https://essenceofsoftware.com/drafts/noosphere-v1/","summary":"A Concept Exploration of Noosphere","title":"Noosphere concepts"},{"content":"I originally wrote this post in response to a lovely invitation from Gordon Brander to contribute a post to his blog (which I thoroughly recommend—it’s full of provocative ideas).\nIntroduction I\u0026rsquo;ve spent the last ten years or so looking at hundreds of apps, trying to figure out what software design is all about. The result of my explorations is concept design: a new lens for thinking about software, how it’s designed, constructed and used.\nViewing software design through this lens can improve the quality of the software you build, for both developers and users, and bridge the gap between UX and engineering (which are often siloed into different roles). You can think of it as a computational take on UX design.\nI call it a “new” lens, because adopting it dramatically changes the way most software design happens. But in some sense, it\u0026rsquo;s not new at all. It makes explicit what I suspect the best designers have always done, and it distills and combines some ideas that have been floating around for a long time.\nI’ll explain concept design in three stages, each of which suggests adopting a different perspective on design from the conventional one.\nStage One: Design = Functionality Stage One is the least radical, but it’s important to grasp it before moving on: view the design of software as the shaping of its functionality.\nThis might sound obvious, but it’s not what people actually do. Functionality is what the software does, not how it does it (which is what software architecture is about), or how it’s presented (which is what UI design is about). It’s the most basic starting point, because you can’t think about code until you know what function it should implement, and you can’t think about a UI until you know what functions the UI should support. In practice, the functionality is often implicit and apps are “designed” by starting with code fragments or wireframes. Being explicit about what we’re actually designing is the first stage.\nSo imagine we’re designing a social media app—call it WhatsUp—that lets users post in groups. As we’re evolving the design, we’ll need a way to describe the functionality we have in mind. How do we do that?\nWe could start by defining the set of states: there are users, groups and posts; posts are by users and divided into groups; users can be members of multiple groups. We can explore these states more precisely by considering candidate invariants (“integrity constraints” in database lingo) as design questions: Must a post belong to exactly one group? Must a post be from a member of the group it’s in?\nThe state space of WhatsUp, shown as an ER diagram\nThen we can list the actions that users can perform: create, delete, join and leave a group; add a post to a group; reply to a post; etc.\nThe beauty of this scheme is that just by writing down the state and invariants you’re describing a whole (infinite!) set of possible behaviors. A good designer can see through a seemingly straightforward invariant to the impacts it will have on the behavior (and even the software architecture), so the design of the state becomes a powerful design tool in its own right. Suppose we say that the posts in a group must be by members of that same group. Then what happens if a user leaves a group? Presumably their posts will have to be deleted, and that will have complicated consequences if there are replies.\nThe state also acts as a measure of complexity: sometimes you’ll consider some behavior and realize it needs a richer state. For example, we might want to ensure that users can only see posts in a group that were added after they joined. If so, the state will have to include not only dates for posts but also dates of joining for users.\nDescribing an app in terms of states and actions is pretty challenging, and the work is often worth it. But doing it for the whole app still seems like a daunting task. Is there any way to break the task down and make it simpler?\nStage Two: Concepts Stage Two involves the most fundamental move, in which we bring some modular structure to an app’s functionality.\nWhen I introduced WhatsUp, I described it as an app that “lets users post in groups.” You may wonder: Why isn’t that enough? You understood what I meant because you’re already familiar with the concepts I’m alluding to. The concept of Group, for example, is present in many social media apps, and goes back to the earliest chat apps (such as IRC). I could have called it ChatRoom instead.\nWhat kind of “concept” is Group? To a philosopher, a concept is a kind of category or classification: the concept of Dog corresponds to the set of all things that are dogs. Applied to software, this line of thinking leads to concept lattices and ontologies. It’s useful, but I don’t believe it’s what matters most. To a software design, a concept is about behavior: a Group is something you join and leave and post to.\nIn my approach to concept design, a concept is a small service with its own API defined by its actions. I sometimes use the term “nanoservice” because a concept is like a microservice but smaller and more tightly focused. (And, as we’ll see, unlike a microservice, a concept is designed to be independent of other concepts.)\nAt the same time, a concept is a social protocol describing the rules of the game for the users: that you have to join the group before you can post in it, for example. All software concepts are inventions. Some of them were designed within software, and so they impose protocols on users; the Password concept, for example. May of them were designed first as social protocols in the world, and were then adopted by software; the Reservation concept, for example, (at least as applied to restaurants) was invented in the 19th century, long before OpenTable!\nHow to define the behavior of a concept? States and actions, just as we might have done for the app as a whole. But the difference is that now we have some structure: each concept has its own local state and actions. So the grouping of posts is part of the Group concept state, and the content of posts can be part of Post, a different concept. Joining and leaving are actions of Group; editing a post is an action of Post. Different designers (or different teams) can work on different concepts, allowing modular design of the functionality.\nThe state of WhatsUp factored into concepts: each concept has its own state space.\nNow here’s where concept design diverges from current perspectives on design (such as “design thinking”). My claim is that if you look at most apps, you’ll find that they are composed of conventional concepts that appear in many other apps. That’s how we’re able to switch so easily from one social media app to another, for example.\nThis means that design, for the most part, isn’t about inventing a completely new set of behaviors from whole cloth. Rather, it’s about deciding which standard concepts to use, and then how to fit them together. Novelty comes from putting existing concepts together in new ways, and occasionally from introducing brand new concepts.\nApps are like molecules that have different characteristics but are built from the same atoms. So we can think of WhatsApp as the molecule PoGrUsRe (Post + Group + User + Reaction); Facebook as PoFrUsUpRe (Post + Friend + User + Upvote + Reaction); Twitter as PoFoUsUpHa (Post + Follower + User + Upvote + Hashtag); and so on.\nThe WhatsApp molecule and its structure: the graph defines a family of apps, with an arrow from concept A to concept B meaning that any (subset) app in the family that contains A will contain B too.\nHow does this change the design process? It means that your first step is to figure out if some existing concepts will do the job. You might want to adjust them, or even add some new concepts. You won’t need to elaborate the states and actions of most of the concepts because you know them already.\nYou might wonder if this makes design less creative. I’d argue that it makes it more creative, because you spend less time reinventing the wheel and more time focusing on the aspects that are truly novel. When you put concepts together in creative ways, magical things can happen. Two examples: Yellkey takes two familiar concepts—let’s call them LinkShortener and ExpiringService—and puts them together, and by ensuring that links expire quickly, can offer common words as links, completely changing the user’s experience. And the Arc browser makes tabs more manageable by joining the Tab and Bookmark concepts in a new way.\nConcept design helps you identify what makes apps distinct, and what a company’s true assets are. Photoshop succeeded because of the Layer concept; TBL’s invention of the web rests not on hypertext or markup or HTTP (none of which were novel) but on the URL concept; Microsoft Word introduced Style, the concept that almost defines desktop publishing; Dan Bricklin’s VisiCalc had RelativeReference, the concept that makes spreadsheets work.\nStage Three: Independence The full benefit of concepts comes when they can be designed (and even implemented) fully independently of one another.\nAlthough the concepts within an app work together and interact with each other, users understand them independently. HackerNews combines Post and Upvote so that posts to be upvoted, and the state of the Upvote concept will (at runtime) contain references to posts, which are created within the Post concept. But in the user’s mind, the Upvote concept is the same one the New York Times uses for ranking Comments.\nSo we need to describe a concept like Upvote without any dependence on any other concept. This means first that its types can’t refer to the types of other concepts. It must be polymorphic, so that the items being voted can have any type, whether they’re posts, newspaper articles, comments, or even users.\nSecond, independence requires that the actions of one concept can’t call those of another. Consider Facebook’s (controversial) integration of Upvote and Reaction. We’d like to say that when a user reacts to a post, it gets upvoted too. The conventional way to do something like this would be with a function call, so that the action Reaction.react(u, r, p) (which means that user u adds reaction r to post p) is modeled as some code that call the action Upvote.upvote(u, p).\nBut embedding a call to an action of the Upvote concept inside the Reaction concept would couple the concepts together. So what to do? The standard programming trick would be to use callbacks (or objects), but that’s complicated and takes us away from simple state and actions into the world of higher-order functions.\nConcept design uses a much simpler approach, inspired by Tony Hoare’s CSP. Instead of modifying the Reaction concept itself, we synchronize it with the Upvote concept by saying that whenever one action happens the other one should happen too:\nsync Reaction.react(u, r, p) Upvote.upvote (u, p) Synchronizations are like transactions: either all the actions happen or none of them. So this sync\nsync Group.post (u, p) Karma.permit (u, 20) might enforce the requirement that a user can post to a group only when the permit action of the Karma concept can occur (with 20 karma points).\nIn many apps, especially those composed of standard concepts, much of the design challenge is figuring out how the concepts fit together. Concept design factors this out into separate synchronizations, and lets you focus on this aspect independently. Many usability problems can be traced to problematic syncs. In Google’s calendar app, for example, Event and Invitation are synchronized so that deleting an invitation sends a declining message to the inviter, which may be unwanted. (In fact, problems with this particular sync go back a long way, with users of Apple’s iCal complaining that spam events could not be deleted without notifying the spammer that the account was valid!).\nAnd sometimes even clever syncs can’t make a concept composition work. Gmail’s composition of Label and Conversation is ingenious but troubled: you’ll understand this if you know why searching for messages with the label sent includes messages that were instead received.\nFinally, in some cases, a synchronization is so powerful that it enables all kinds of new workflows. Photoshop, for example, syncs Channel, Selection and Image in a way that only experts fully understand but which brings amazing opportunities (you’ll know what I mean if you’ve ever extracted the edges from an image and applied sharpening just to them).\nSo what can you do with all this? As an individual designer, a concept design lens can change how you work:\nApp = concepts. Viewing apps as compositions of concepts brings a new clarity: now you can explain how a word processor differs from a text editor (which has Paragraph, Style and Format concepts, but usually no robust Line concept) and from a desktop publishing app (which has a TextFlow concept and a real Page concept).\n90% of design is reuse. Designing an app is now mostly about combining existing concepts in creative ways. Gordon’s Noosphere takes concepts like Notebook, ContentAddressableStorage, PublicKeyNaming, ImmutableDocument, Version, etc and makes an entirely new platform for peer-to-peer knowledge sharing.\nBehavior = state + actions. Sketching behavior as a tentative UI is very evocative, but it only goes so far. If you think about behavior as states and actions, with the help of concepts to modularize it, you can go a lot further and still avoid the complexities of code.\nConcepts can help you think more deeply about design:\nSynergies. The best concept compositions bring synergies. Making the Macintosh Trash a Folder was not inevitable, and it has some complex consequences (if you understand both concepts, you’ll be able to construct a scenario in which emptying the trash doesn’t delete the items in it). Design moves. You can think about design in steps that correspond to a repertoire of known design moves, with analogies in physical design. Creating synergy by tightening the synchronization of two concepts is an example. Overloading. Every concept should have exactly one purpose. You can diagnose subtle flaws by analyzing concepts that are “overloaded” with more than one purpose (see the knots Epson tied itself into by including tray selection in PaperSize). Radical independence. We’re used to letting all our concepts depend on each other. But what if you made them truly independent? How would file systems change if names were disentangled from identities? For one, Git would know when you renamed a file. At the organizational level, concept design can have even bigger impacts:\nCataloging assets. Your products’ concepts are your key assets, differentiators (and liabilities). Alignment across products. Many companies have a family of products that drift apart over time. Aligning concepts helps users and saves dev effort. Shared language. Concepts offer a way to bridge between engineers and designers, and to bring your whole organization (marketing, strategy, IP) aligned on a clear understanding of your products. And this is just the beginning. There are lots of exciting things to apply concepts to, such as\nDark patterns. Deviation from known concepts can offer a more solid way to identify dark patterns, and what Cory Doctorow calls the “enshittification” of apps can be explained in terms of dark concepts. New modularization. Concepts suggest a new way to structure code that avoids some of the entanglements of OOP. Using LLMs. We’re looking at using LLMs to help with concept design. A catalog of concepts may also help an LLM generate better code. Want to learn more? My book The Essence of Software explains all this and much more with hundreds of examples from popular apps. The website includes a bunch of blog posts, and a talk.\nHot off the press: a paper I wrote with colleagues at Palantir describing how they organized their development process around concepts.\nThere’s a concept design forum too, which is a great place to post reactions, questions, comments, etc. Feel free to email me too.\n","permalink":"https://essenceofsoftware.com/posts/three-stages/","summary":"A brief intro to concept design in three parts.","title":"Three Stages of Enlightenment"},{"content":"As I developed the framework of concept design, I thought that concept integrity would be central. Violations of integrity would be like feature interactions in telephony, and at the very core of software design.\nI have to admit that I was wrong. Violations of concept integrity are just so egregious that they can usually not be tolerated. Consequently, they occur rarely, and other design criteria (in particular concept specificity) turn out to be more useful in practice.\nWhen integrity violations do occur, however, they can cause major trouble for users. In EOS, I give the example of the violation of Google Drive\u0026rsquo;s synchronization concept, one so painful that an angry user bought the domain googledrivesucks.com (now no longer active) to vent his frustration at losing all his files due to the design bug.\nRecently, I\u0026rsquo;ve come across three more examples of integrity violation, causing varying degrees of surprise.\nViolation of Undo in Apple Mail In a recent update to macOS, Apple extended the basic Message concept with an undo send action. This is handy: it gives you 30 seconds after you\u0026rsquo;ve sent a message to change you mind.\nAppropriately, Apple synchronized this action with the undo action of the general Undo concept. So if you send a message and then immediately execute an undo, it is interpreted as undo send, even without clicking the special transient button carrying that label.\nBut Apple went further. The undo action is synchronized with undo send for as long as the message has not been sent, even if another action has been performed in the meantime.\nI noticed this because I have a standard way of dealing with messages in my inbox. I respond to the message, and then I either delete it, or move it to another folder. With the new undo send, something surprising and annoying kept happening. I would reply to a message, delete that message, and then execute undo, having decided I would prefer to file it instead. Mail would undo the send, and not the deletion.\nInitially, I thought that the effect of the undo was to pop the undo-stack all the way back to the send. In fact, it\u0026rsquo;s not that bad, and what it does instead is to keep the send action on the top of the stack even as other actions are performed. Either way, this violates the simple and intuitive operational principle of Undo that executing undo reverses the last action performed.\nViolation of Playlist in Spotify The operational principle of the Playlist concept says that if you start playing the playlist by selecting a song and clicking play then, until you stop playing, it will play the tracks of the playlist in order from that song onwards.\nSpotify aficionados will know that this isn\u0026rsquo;t true. If you add a song to the queue, that song will be played before the next song in the playlist.\nArguably, there\u0026rsquo;s nothing wrong with the this. The Queue is a nice concept that brings more flexibility to users. Moreover, the Playlist concept is arguably overloaded if its purposes include both organizing tracks into groups and scheduling their playing, so regarding the scheduling as a feature of only the Queue concept is not only a more accurate reflection of how Spotify works, but also eliminates the overloading.\nNevertheless, the app shows signs of some design struggles here. Strangely, if the interposed song is in the current playlist, it won\u0026rsquo;t be highlighted with the little playing animation that would be shown if it were next in the playlist. And the entries in the queue that show upcoming tracks from a playlist disappear en masse if you start playing a song from a different playlist. The Queue concept, in other words, isn\u0026rsquo;t exactly a queue, and includes two distinct kinds of tracks which are treated differently: the ones added implicitly when you play a playlist and the ones added explicitly through the add to queue action. You might wonder: can you move songs from the explicit list to the implicit one? Yes, it seems you can. Hmmm.\nViolation of Ride in Uber An operational principle may have caveats: that the typical outcome may not follow in extraordinary circumstances. For example, The Order concept promises that if you order some items, they will eventually be shipped and delivered. But customers understand that this principle cannot be a cast iron promise, and an inventory glitch may prevent its fulfillment.\nIf the expected outcome fails to follow with high enough frequency, perhaps it becomes an integrity violation.\nUsing Uber\u0026rsquo;s ride service, I\u0026rsquo;ve noticed that rides are increasingly canceled. You request a ride, select the service level, confirm the pickup location and expect the driver to arrive at the prescribed time or shortly thereafter. But just before the expected arrival time, you get a message saying that the driver has canceled. (On several occasions, that message has been followed immediately by an email from Uber telling me that I have been charged a fee for canceling the ride myself.) At other times, I\u0026rsquo;ve received a message that the driver already arrived but couldn\u0026rsquo;t find me, even though I could see that no car had passed.\nThe driver must surely have the ability to cancel a confirmed ride in some circumstances (such as unexpected traffic). But it seems that drivers are canceling more frequently than necessary, presumably so that they can switch to more lucrative rides.\nWhen integrity is lost How do typical users respond to broken integrity? I doubt they say to themselves \u0026ldquo;ah, this app seems to be violating the essential integrity of this concept.\u0026rdquo; I suspect that instead they just get confused: they don\u0026rsquo;t really know what to expect when they order an Uber, what happens when they queue songs in Spotify, or what undo actually does in Apple Mail.\nFor the designer, however, integrity is a yearning for what might have been: how if things had been different, everything would have been so simple…\n","permalink":"https://essenceofsoftware.com/studies/small/broken-integrity/","summary":"Integrity violations aren\u0026rsquo;t as common as I once expected. But when they happen, they\u0026rsquo;re bad.","title":"Breaking Integrity: Three Examples"},{"content":"The risks of computing From the very birth of computing, enthusiasm for its benefits has been accompanied by concerns about its risks. These concerns can be placed into three broad categories, corresponding to three periods in which they came to the fore: from the advent of computing to the late 1980s; from the 1990s through to around 2010; and from then until today.\nEarly on, the primary concern was the possibility of catastrophes resulting from unreliable systems operating in critical domains such as aviation, defense, space exploration, medicine, etc. Because of investments in safety (and perhaps also because of sheer good luck), in most of these domains, catastrophes caused by computers have been extremely rare. One major exception is in medicine, where devices routinely fail with fatal consequences, and many of the companies that make them seem not to have adopted the methods and techniques that have worked well in other domains.\nIn the 1990s, when most systems (including many critical ones) became accessible through the public internet, malicious attacks became widespread. This changed the calculus: whereas previously the potential cost of a catastrophe might be mitigated by the low probability of its occurrence, now, with sufficient financial incentive (for individual actors) or political value (for state actors) the likelihood of a vulnerability being exploited became a near certainty. Miraculously, despite the widespread of occurrence of attacks over the last 30 years, almost all consequences have been financial and none have resulted in major loss of life.\nSince about 2010, a different kind of concern has emerged from a confluence of three developments: pervasive use of smart phones, growth in social media platforms, and the advent of AI algorithms. These all happened at roughly the same time. The iPhone came out in 2007, and had 100m owners by 2011. YouTube was the first platform to exceed 100m users in 2007; Facebook had 100m in 2008 and passed a billion) by 2013. Facebook introduced likes in 2009, and began to develop algorithms to determine the content of a users’ feeds; reportedly, they adopted a machine learning approach in 2011, one year before the demo of AlexNet, the image classifying network that inaugurated the era of deep learning.\nThe concern now is damage to our social fabric: that AI algorithms used to make decisions in hiring, financial services, policing, sentencing, and so on, perpetuate or even promote bias; that chat platforms cause pressure, especially in teens, leading to higher rates of depression, anxiety and suicide; and that the engagement-chasing tactics of social media encourage the spread of disinformation and exacerabte political fragmentation and polarization.\nAnother theme, less dramatic but perhaps as significant economically, is the propensity of software technology to sap attention from users, distracting them from productive work and burdening them with mundane and time-consuming data processing tasks. This wastage might explain in part the “digital stagnation” in which increases in productivity in the US slowed since the start of the 1980s (when the personal computer was introduced).\nHow to save the day? How can we prevent further damage to our social fabric, without abandoning the advantages of software?\nSome people believe that adding more technology can counteract the problems of existing technology: the decentralized web, blockchain, multiparty computation, end-to-end encryption, differential privacy, and so on. Interestingly, despite the excitement about the potential of machine learning, almost nobody thinks that it’s a technology that will make the social problems of software easier to solve; in fact, it will probably make things worse. Some of these technologies are indeed promising but none is likely to be a panacea. And most of them provide means to limit communication and increase privacy, which does not always align with how users want to use software.\nOthers believe that regulation can force companies to eliminate the social risks of software. Terminating some bad practices (eg, the selling of users’ personal data without consent) would undoubtedly help, but it’s become clear that legislating away the problems of social media is a pipe dream. And the sad example of GDPR-mandated cookie controls shows how design-by-government can backfire.\nThe problems caused by software are not surprises, but are direct consequences of its design. This suggests that the most direct path to fixing software is via design, first by identifying problematic design elements and then devising remediations to fix them. Technology and regulation may play supporting roles, but changing software designs must be central.\nProblematic design elements: dark concepts Almost all of the negative impacts of software on our lives—whether in stealing valuable attention, or tearing apart the social fabric—can be attributed to a small number of design elements.\nThese elements can be described as concepts, units of functionality that are independent of one another (and hence can be understood one at a time), and reusable (and hence can do damage in many different settings).\nI call concepts that cause trouble dark concepts, using the terminology of dark patterns. But whereas dark patterns are often quite egregious attempts to deceive, dark concepts are usually more subtle. They lie deeper within an application, and are harder to identify and root out, in part because dark concepts often also fulfill legitimate purposes.\nUpvote: a dark concept An example of a problematic concept—and one that has been much discussed—is what I call Upvote. The apparent purpose of this concept is to use the wisdom of the crowd to evaluate the quality of items. Like most concepts, Upvote is polymorphic, and can be applied to any kind of item: photos in Instagram, comments on New York Times articles, tweets in Twitter. The concept is perhaps most effective in Q\u0026amp;A apps such as StackExchange and Quora, where the number of upvotes often corresponds reliably to the credibility of an answer.\nThe Facebook incarnation of Upvote, in particular, has been the subject of much hand-wringing. Upvotes, originally called “likes” in Facebook (on launch in 2009), were problematic from the start, since users had a tendency to “like” photos of cats more than posts containing thoughtful life reflections.\nIn 2016, Facebook introduced “reactions.” This was actually a synchronization of two distinct concepts: a Reaction concept which, like that concept in apps such as Slack, allowed a reader to select an emoji in response to a post, and the Upvote concept, which used the number of likes to promote popular posts.\nControversially, the design did not allow Upvote and Reaction to be controlled independently, but instead automatically turned reactions into upvotes. Worse, it seems that emotional reactions were converted into upvotes with higher weightings than simple likes. Leaked reports from Facebook claimed in particular that reacting with the angry emoji was equivalent to five regular likes in terms of the impact on a post’s promotion.\nThe Upvote concept therefore plays a central role in the dissemination of negative and sensationalist content. Like other dark concepts, it has a plausibly beneficial purpose (using crowdsourcing to promote higher quality content) which might have motivated its original design, and which may still be primary in some applications. But in Facebook, and many other apps, that purpose has been supplanted by a far less innocuous one—increasing engagement at all costs—so that a seemingly straightforward concept has turned dark.\nOther dark concepts As apps, and social networking apps in particular, have evolved, a cluster of concepts has emerged, shaping users’ experiences in often negative ways.\nThe Feed Concept Every concept has an “operational principle”: an archetypal behavior that users have in mind. For example, the principle for the concept of EmailMessage is (roughly) that if you send a message to someone whose email address is listed in the to-field, then sometime later a copy of that message will appear in their inbox.\nA similar kind of principle once applied to the Feed (or Timeline) concept in social media apps such as Facebook and Twitter: that users would create posts, and their connections would subsequently see them in their feeds.\nOf course, it was never quite so simple: the Upvote concept determined the ranking of posts, and the Friend and Follower concepts determined which posts were included. These other concepts were perhaps sufficient to explain the behavior, and it was not clear that a separate Feed concept was needed.\nBut over time, the implicit operational principles of these concepts faded, and social media companies exploited their unpredictable behavior to their own advantage. The Upvote concept no longer determines ranking of posts in any simple way; nor does Friend limit the posts you see to your friends. Instead, these concepts provide input to the Feed concept which, by way of inscrutable and undocumented algorithms, shows you the posts the company wants you to see.\nThese posts include, of course, “sponsored posts” (aka advertisements), and they are selected not only on the basis of your explicit upvotes but also on implicit aspects of your behavior, the wording of your posts, your profile and demographics, and probably also various forms of personal information obtained without your consent from third-party data brokers.\nOn most platforms, there is no way to override this unpredictable and uncontrollable behavior, for example by asking to see posts from friends ordered by upvotes and/or time of posting.\nTwitter reluctantly allowed users to switch to a “chronological timeline” but the company-curated timeline is the default. In March 2022, Twitter decided to no longer allow users to switch their default, but then reversed after complaints.\nThe Playlist Concept In Spotify, a dark concept similar to Feed has become prominent. The seemingly innocent Playlist concept has apparently become so important to Spotify’s business model that Corey Doctorow coined the term “playlistification” to describe the pernicious impact of this concept on the app as a whole.\nGo to your home page in Spotify, and you’ll find that it’s completely dominated by playlists: playlists you recently played, playlists of artists you might like, playlists curated specially for you, and so on. You’ll see that many of the recommended playlists are marked “by Spotify.” The rest are marked as “public playlist,” suggesting that they might be listener-curated lists that were made public. Look carefully, however, and you’ll see that almost all of these name Spotify as the curator.\nIn practice, this means that, through the Playlist concept, Spotify not only decides what playlists will be recommended to you, but even what the content of those playlists will be. Doctorow cites this as a prime example of “chokepoint capitalism” and alleges that Spotify extracts payment from record labels in return for placing their artists’ tracks on popular playlists.\nThis kind of strategy might be called “reintermediation.” The web brought disintermediation, in which companies eliminated go-betweens, allowing consumers to purchase services directly from suppliers. This model is at the core of the gig economy (eg, in Uber and Lyft) and the marketplaces created by online retailers (such as Amazon and Etsy).\nBut then the companies that provided the disintermediation realized there was money to be made by reinserting themselves, and controlling the relationship between producer and consumer. This reintermediation is seen not only in concepts such as Spotify’s playlist but also in the concept of Recommendation in which a company recommends products or services to a user supposedly on the basis of that user’s prior activity, but also based on the company’s preferences of which suppliers and products to favor.\nThe Rating and Review concepts, in which concepts provide numerical ratings and textual reviews of products and services, are ripe for abuse. Ratings and reviews have enormous impact on consumer choices, and must inevitably be filtered to prevent fraudulent use, so the simplest and most transparent algorithms are not feasible.\nYelp, in particular, has been the subject of many allegations, including that advertising salespeople offered to suppress negative reviews in return for a company purchasing advertising. Yelp’s business practices were the subject of a documentary called Billion Dollar Bully. None of these accusations has been substantiated in court, however.\nThe Tag Concept Tag is another concept whose purpose seemed to be innocuous at first. Allowing users to identify friends in photos offered a simple way to bring photos to their subject’s attention, and to make it easier to search for images in which particular friends appeared.\nIt became clear, however, that the real purpose of the concept is more insidious. In fact, there seem to be two distinct dark purposes. One is to expand the density of the social graph, by sharing a user’s posts beyond their friend group: in Facebook, by default, if a user is tagged in a post, that post is shared with the user’s friends, including those who are not friends of the original poster. The other is to harvest more personal data from users by tracking who appears together in photos, and in whose photos.\nNot surprisingly, Facebook was unsatisfied with relying on manual tagging and was eager to streamline the process, so it began to use facial recognition algorithms to detect faces automatically and make tagging suggestions to users. A class action suit was filed in Illinois in 2015 alleging a violation of the state’s Biometric Information Privacy Act; in 2021, the suit was settled, with Facebook paying $650m to affected users.\nThe Cookie Concept Dark uses of the Cookie concept have been a concern for more than a decade. As in all these examples, the original purpose was reasonable. In this case, it was to compensate for the “stateless” nature of HTTP, the underlying protocol of the web.\nBecause HTTP, in contrast to other protocols (such as FTP) treats each request independently, rather than as part of a session, it offers no built-in way to remember a user’s prior interactions with a server. Any context must therefore be maintained by a separate mechanism. This is where cookies come in. When you first access a website, it sends a cookie back that is installed in your browser; this cookie is then forwarded back to the website’s server with every subsequent request. The interaction thus becomes “stateful,” with the state held in your browser (in the form of the latest version of the cookie), and in the server (typically in a database table indexed on cookie identifiers).\nSuch a mechanism is indispensable. It’s what allows sessions (allowing you to log in just once and have requests that follow be treated as authenticated); it supports preferences that govern how a website behaves; it allows more context-specific behavior; and it makes it possible to add items to a shopping cart before you’ve logged in to an online store.\nThe cookie concept supported another more controversial purpose: letting websites track users. By depositing a persistent, unique identifier in your browser, a website can track you across sessions, whether or not you are logged in (or even have an account). I noticed a slick use of this strategy just a few weeks ago: I visited a page on a tea store website (teabox.com) and within minutes received an email coincidentally informing me that the tea I had looked at was on sale. I was not logged in at the time, but since I had an account, the server was able to correlate my cookie with my email address.\nThe privacy implications of cookies were exacerbated by advertisers. Although a cookie can only be sent back to the domain that served it, a single webpage can contain fragments (most notably advertisements) loaded from a different domain, with their own cookies. By around 2014, it was not unusual for webpages to carry hundreds of these so called “third party cookies.”\nIn 2020, Apple started blocking third-party cookies in its Safari browser, and Google is promising to follow suit in Chrome by the end of 2024. But first-party cookies are still ripe for abuse. The European Union’s GDPR forces companies to obtain explicit consent for cookie usages, but the consent mechanism is ignored by most users and has just become another annoyance.\nThe Notification Concept The Notification concept has many legitimate uses. For example, if you post a question on a forum that you rarely visit, you will likely want to be notified when a response appears. Likewise, most users appreciate notifications that items they’ve ordered have been shipped, or that a returned item has been accepted.\nIncreasingly, however, companies use notifications to draw users to an app with much greater frequency than they would willingly choose. LinkedIn, for example, sends notifications whenever a user receives a message. Insidiously, such messages are often advertisements that are posted as if they were communications from contacts. And by omitting details about the message—its sender, subject or any part of its contents—from the notification (which appears, for example, as an email), the user is forced to visit the app to discover that the interruption is unwarranted.\nOther Problematic Concepts Other concepts might be implicated in a variety of social and psychological harms, but have been less consciously manipulated by companies for their own benefit.\nFor example, the Presence concept included in apps such as Slack, which allow users to indicate whether they are available, is ripe for abuse in work environments in which managers track their employees and encourage them to be constantly online and interruptible. More troubling versions of this concept have been deployed. A friend reported that Skype for Business used to determine presence automatically from mouse movements, and that she and her colleagues felt obligated to move the mouse every couple of minutes to ensure that they would be seen to be working.\nThe Reaction concept, in which users can respond to social media posts with emojis signaling approval or disapproval, seem to be a mere convenience, and provide some benefit in reducing the burden of lengthier responses. But for teenage girls, in particular, the pursuit of online approval has devastating impacts on mental health.\nResponding to Skepticism Complaints about the negative impact of apps and social media reasonably evoke some skepticism. A few questions especially come to mind.\nFirst, one might ask: is the damage inflicted by these dark concepts enough to matter? In response, I would note first that the impact of dark concepts should be assessed in aggregate across all users. Even if most individual users do not suffer extensive harm, the overall damage (to the social fabric, and in some cases to total productivity) may be significant. Moreover, some users will suffer more than others; the dramatic growth of depression and anxiety in young people has been attributed by some to use of phones and social media.\nSecond, one might wonder: do users really care? Perhaps most users are satisfied with their apps and they feel that they get more than enough benefit to outweigh the costs. As habits and cultural assumptions shift slowly over time, however, users might not be aware of the extent to which their judgments are conditioned by their environment. In some cases (for example, loss of privacy) the damage may not be seen at the time, and only becomes evident later (for example, when personal data is used in a bad way). Users may also be unaware of the costs they are suffering: they might be less pleased about Spotify’s recommendations if they come to realize that those recommendations were made to all of their peers and were paid for by record labels. And of course sometimes the users are not the ones who suffer. In the case of Spotify’s playlists, Doctorow argues that the prime victims are artists and record labels for whom payments to promote their material are a kind of legal bribe.\nSome commentators (such as Bryan Caplan) argue that the very fact that consumers continue to use platforms and apps indicates their “revealed preferences” and shows that, whatever they might say, they are happy with the tradeoff they have made (often accepting dark behaviors in return for not having to pay). This argument would be stronger if good alternatives were available so we could actually observe users’ preferences for one option over another.\nMoreover, this argument militates against any concerns about product quality so long as companies are transparent. It would suggest, for example, that companies should be able to offer food products that are contaminated so long as purchasers realize that they are receiving a discounted price in return for a higher chance of being poisoned. But of course the key point with social media companies is that they are not transparent, and they go to considerable lengths to present dark concepts as being beneficial to the consumer alone.\nIf some of these questions could be formulated more precisely, it might be possible to design experiments or data analyses to answer them more rigorously. Companies that make software might be interested to know if dark concepts actually pay off. They might be very successful in the short term in meeting goals for engagement or advertising, while nevertheless annoying consumers enough to be more negative than positive in the long term. It would be interesting to know whether the metrics-driven culture of many companies (in which each unit is evaluated in terms of easily measurable but not necessarily reliable metrics of success) is a factor in the development and prominence of dark concepts.\nClosing Thoughts Dark patterns. Harry Brignull, a British UX designer, coined the term dark pattern back in 2010, and started cataloging examples of websites that used a variety of common patterns to deceive users, from “Roach Motel” (the subscriptions that were easy to start but near impossible to end) to “Privacy Zuckering” (sharing more personal information than intended).\nNow, more than a decade later, although these kinds of dark pattern are still common, they seem almost quaint in comparison to the insidious schemes that lie more deeply beneath the user interface. These schemes, which I’ve described in this post as a handful of dark concepts, go beyond getting us to perform actions that we might not have intended. They take control away from us, and shape our experiences in ways that are often opaque and contrary to our true interests.\nData science rubrics. Many of the dark concepts are enabled by the data science technologies that have emerged over the last decade as companies have collected and exploited vast banks of data about their customers. A recent new book (Data Science in Context) presents rubrics to help designers identify and address the kinds of concerns that I have raised here. A section of the book on personalization, for example, outlines the risks posed by algorithms for targeting advertising and making recommendations. This offers a dual perspective to mine, focusing on the developers’ objectives and the data science methods used to achieve them, where I have instead pointed to concepts—units of functionality which, while broadly motivated by those objectives, form the building blocks of modern applications.\nFixing social media. How can a concept design lens help fix social media? First, viewing social media apps through their concepts helps make clear not only how pervasive the problems are, but how similar they are across apps. Second, they dispel the misapprehension that the problems are complicated and inscrutable. While the algorithms that lie behind dark concepts may be subtle, the concepts themselves, whose design makes room for those algorithms—the Feed concept for example priming the user to abandon all expectations of how the posts displayed will be selected—are easily grasped by users, and indeed, experienced daily. Third, concepts give a point of leverage for improvement; each concept can be addressed in isolation and fixed without impacting other concepts. Fourth and finally, concepts offer a way to codify best practices—for example, a Tag concept that enriches search while respecting the privacy of users—thus supporting industry-wide efforts to reform application design (and to indemnify developers eager to do the right thing but not sure how to do so).\nAI-augmented applications. As machine learning becomes an important part of all applications, the kinds of issues I’ve discussed here will become even more important. As a recent op-ed put it:\n“Social media was the first contact between A.I. and humanity, and humanity lost… Large language models are our second contact with A.I. We cannot afford to lose again.”\nPerhaps the concept framework can help not only articulate the roles that AI algorithms play in apps, but also contain them, placing them in sandboxes that allow users (and society more broadly) to understand, if not exactly how they work, then at least the boundaries of their functionality, which behavioral expectations can be relied upon, and which plausible assumptions that users adopted in their naivety will have to be relinquished.\n","permalink":"https://essenceofsoftware.com/posts/dark-concepts/","summary":"The building blocks that social media apps use to control us.","title":"Dark Concepts in Software Design"},{"content":"Now translated into Japanese by Takeo Imai! You can also read his book review in English or Japanese.\nRereading a great book Don Norman’s influential book, The Design of Everyday Things, talked about doors, light switches and refrigerators, and had little to say about software. But many of Norman’s diagnoses and prescriptions applied nicely in our domain too.\nWe recognized, for example, that in software the gulfs of execution and evaluation—which separate the user’s intent and interpretation from the system’s actions and outputs—are especially troublesome, and we took on Norman’s notion of affordances for guiding the selection of user interface widgets.\nPerhaps more significantly, we were inspired by Norman’s compelling arguments that things could be easier to use. We became groupies in the usability cult, expressing (sometimes smug, but often justified) indignation at the apparently incompetent design of most artifacts, and we vowed to do better ourselves in our software work.\nRereading the book, however, I’ve come to think that its prescriptions are in some key respects poorly matched to the design of software. In the 2013 edition of the book, twenty five years after the original was published, Norman added a new chapter on design thinking with this revealing paragraph:\nHow can one person work across so many different domains? Because the fundamental principles of designing for people are the same across all domains. People are the same, and so the design principles are the same.\nBut design is not just about people; it’s about technology too, and what happens at the meeting point of people and their artifacts. Different artifacts present radically different challenges. Urban design, graphic design, industrial design and software design might all be forms of design, but they require different skills and sensibilities.\nNorman’s fridge I’ve always been puzzled by Norman’s fridge example. In a section on conceptual models, he explains how users misunderstand the temperature controls on the standard American fridge. The two controls, often labeled “freezer” and “fresh food,” seem to offer independent temperature adjustments. But it turns out that one sets the level of the compressor and the other changes the ratio of cold air that flows to each of the compartments.\nNorman rightly notes that this is design is flawed, since it’s almost impossible to make a reliable temperature adjustment. Make the freezer setting colder, and you turn up the compressor, making your fresh food colder too and freezing your salad. Try and make the fresh food colder, and you’ll divert more cold air away from the freezer, causing your ice cream to defrost. Worst of all, temperature changes in a fridge usually take a day to stabilize, so making a series of adjustments becomes a week-long exercise.\nThroughout the discussion, however, Norman hints repeatedly that the problem isn’t primarily the design of the fridge’s cooling mechanism but rather the way the controls present it—what Norman calls the “system image.” Here’s his summary:\nIt was extremely difficult to regulate the temperature of my old refrigerator. Why? Because the controls suggest a false conceptual model.\nThis strikes me as a very strange statement. The reason the temperature is hard to regulate is that it doesn’t provide suitable controls. It’s true that the controls suggest a false model, but if that were the problem, one could fix it by changing the labels, for example to “overall coldness” and “fridge/freezer ratio.” That’s not easy to understand, but it’s arguably no longer false. Does it help? Not at all.\nThe assumption behind conceptual models There is a fundamental assumption at play here that bears more explicit articulation. Norman is willing to take the design of the underlying mechanism largely for granted. His interest is not in reshaping mechanisms—he leaves that to engineers—but to improve the interface, through which the mechanism is presented to users. Here’s a diagram from the book showing how the interface projects a “system image” of the designer’s conceptual model, in turn generating the model in the user’s head:\nIn this respect, the original title of Norman’s book, The Psychology of Everyday Things, was more apt. Norman, as psychologist, is interested in how users understand (or misunderstand) mechanisms, and the valuable contribution of his book is the realization that psychological insights can be put to work in designing the interface, so that the “gulfs” between users and mechanisms are bridged.\nTo Norman, a conceptual model is a psychological construct distinct from any concrete reality. Designers have only so much leeway in shaping the conceptual model, because the reality is fixed, and if the model is to be accurate, it can’t stray too far from it.\nAn online lecture that Norman appears in makes this point abundantly clear. The canonical example it gives of a conceptual model? The rain cycle.\nThermostats revisited A classic example of a confused conceptual model: the traditional thermostat. Many people incorrectly believe that if the room isn’t warm enough, you should turn the dial all the way up to the top. But this won’t cause the heating system to generate more or hotter air. The setting on the thermostat simply governs the temperature at which the heating will turn off. So you’ll get warm just as quickly by setting the actual desired temperature.\nNorman uses this example too, and in my view it’s a more convincing one for his argument than the fridge. Once you adopt the correct model, you can operate the thermostat more effectively. You don’t need to turn the dial all the way up and then turn it back down again when you’re happy with the temperature; you just set it once.\nThere’s nothing really wrong with the underlying mechanism. It serves the user’s needs, and can be built cheaply using simple parts (typically a bimetallic strip that bends as it warms up and an attached mercury bulb that switches the heating system on and off as it tips and the mercury flows). It makes some sense, given the low cost and effectiveness of this mechanism, for the designer to take this reality as given, and focus instead on the user’s conception and operation.\nBut now consider a modern thermostat, such as the Google Nest. This device has a sensor that reads the temperature, but its control mechanism is defined in software. Without the need to make an affordable physical mechanism that includes both sensing and switching, the designer is freed to make it behave in any way they choose. There can be different modes and schedules; awareness of human presence; anticipated preheating periods; and so on.\nThese behaviors can be organized into concepts (schedule, vacation mode, zone, etc), but these concepts are not psychological abstractions. They’re not part of a model. They are the reality of the software design, and are just as mutable as the user interface.\nSoftware conceptual design To speak of a software system or app as having a “conceptual model” is misleading. It suggests there is some model distinct from the reality of the software itself. On the contrary, the conceptual design is the design of the software. So I prefer instead to talk not of the software’s “conceptual model” but rather its “conceptual design.”\nThis is a major shift in perspective. It moves our attention away from a software system’s user interface towards its underlying semantics. It suggests that psychology, while important, might not be so central in achieving usability. And most importantly, it points to the conceptual structure of software as its essence, and challenges us to figure out what that structure should be.\n","permalink":"https://essenceofsoftware.com/posts/conceptual-models/","summary":"Don Norman’s book helped improve interface design, but it also misled us.","title":"Conceptual models missed the point"},{"content":"Innovation and software are almost synonyms. It’s often software that\u0026rsquo;s changing the way we live and work (for good and for bad), and new applications of software abound. And then there’s AI which is advancing in leaps and bounds and might be a net benefit (but the jury is still out on that).\nHow strange then that software development, as a field, is so conservative. We do things pretty much the way we\u0026rsquo;ve always done them: mostly by trial-and-error, hacking the code until it sort of works. Agile development brought us more project discipline (in tracking progress, breaking the work into reasonable chunks, etc) and saved us from the often pointless work of writing massive documents that few people read anyway. At the same time, by trumpeting the joy of code it sapped much of the enthusiasm we might have had for thinking carefully about what we were trying to do before writing code (although the advent of design thinking, with its emphasis on need finding, offered a small corrective on that).\nOur favorite 20th century design methods User-centered design, emerging in the late 1970s and popularized by Don Norman’s book in 1986, brought with it a collection of new practices: need finding (Robert McKim, 1970s); paper prototyping (1980s); user personas (Alan Cooper, 1980s); heuristic evaluation of user interfaces (Jakob Nielsen, 1990); wireframing; and so on.\nAnd as companies began to realize how much the UX of their products mattered, they started to hire visual designers and UX designers.\nI don’t think it’s an exaggeration to say that the problem of usability of user interfaces is now solved. Sure, there are still bad UIs around, but most UIs are pretty slick. I rarely look at any product by a major software company and see a serious flaw in the UI. For one thing, flaws don’t last long: if your users are struggling because a button’s mislabeled or in the wrong place, you just fix it.\nSiloing software UX design Does that mean that we don’t have any more usability problems? Far from it. But, having eliminated the UI flaws, the remaining issues run deeper, in the underlying concepts. As I explain in the opening of my book, for example, what confuses people about Dropbox sharing isn’t the UI (which is generally excellent) but the underlying concepts of folder sharing in Dropbox, which have some rather strange properties.\nThe UX techniques that we have from user-centered design and design thinking are useful and important, but they aren’t sufficient for addressing these issues because they don’t pay enough to attention to the functionality of the software itself.\nDesign, as Mitchell Kapor explained, is what happens at the meeting point of technology and people. Rather than doing what Kapor called for—namely creating a role for a software designer who would mediate between the two—we’ve instead siloed our development teams into those who work on the user side and don’t address the software deeply (UX and visual designers) and those who work on the technology side and don’t address the users (software architects and programmers). Product managers can help bridge the gap, but in many organizations they are relatively junior employees and don’t have the clout or expertise they need to be effective.\nA remedy: concept designers That’s why organizations need designers who work at the conceptual level, with a deep understanding of both users and their needs and the software concepts that are built to support them.\nBack to the Dropbox example. Such a person would want to understand what users need from a file sharing app; how people work together, and how they organize their own “digital gardens”; what risks they face regarding security and privacy; and so on. But just as importantly, such a person would want to understand deeply the existing concepts used for file sharing, from Unix folders to file synchronization, and concepts for controlling access, from access control lists to capabilities and authentication concepts.\nInstead, what we do today is to divide the design work between two siloed groups. The requirements analysts and UX designers create the outlines of the product, and the programmers fill in the details. But, as the Eames’s famously put it, the details aren’t just details: they are the design. So with this approach you end up with concepts that have rough edges and don’t have the power, flexibility and robustness that they might have had.\nUnderstanding software, not code Now perhaps you’re thinking this is impossible: nobody could have the expertise of both a UX designer and a programmer. But that’s missing the point. You can understand software concepts without being a programmer.\nTo know what a hard link in Unix is (and how it can break the operational principle of the Trash concept) requires only a conceptual understanding; you don’t need to be able to read the Unix source code.\nWhat concepts offer is a way to talk about software functionality in a precise and sophisticated way that is implementation independent.\nGoing back to Dropbox and file sharing: to be a good UX designer, you’ll need to understand folder hierarchies, path names, aliases and links, etc., but the breaking of files into storage blocks can be completely ignored. That’s the software engineer’s responsibility: if it’s done right, the user need not ever know it’s happening. But the user can’t ignore the distinction between soft and hard links, because they behave completely differently.\nOne final example: as a UX designer, you shouldn’t need to know anything about distributed algorithms. It’s the implementer’s job to make sure that data is moved around and replicated in an appropriate way. But notions of consistency and asynchrony are another matter: if your programmers insist that eventual consistency is good enough, you’ll need to be able to figure out the impact on users.\nWhere to start? A new educational strategy How then can we break down the UX design siloes and create designers with the right combination of skills?\nWithin companies today, UX designers, product managers and programmers can work more closely together. UX designers can expand their training in software concepts; and product managers and programmers can learn more about social, psychological and ethical factors. Of course the very best practitioners are already crossing these artificial boundaries.\nThe future will be impacted most, I believe, by a new kind of software design education, in which students are taught how to work at Kapor’s interface between human and machine: combining the technical aspects of software with an appreciation of human needs and purposes.\nA new software design class These are the principles that have guided the development of a new class that I’m teaching with Arvind Satyanarayan at MIT. In Software Studio (6.1040*), we teach students how to think about software design through the lenses of concept design and value-sensitive design.\nIn our class, students learn (and practice!) a collection of standard user-centered design techniques, including needfinding, brainstorming, sketching and wire framing, paper prototypes, user testing and heuristic evaluation of UIs. But they also learn how to formulate and analyze designs precisely in terms of concepts, and how to explore the social and ethical impacts of their work.\nThese ideas are not taught in the abstract, but through concrete examples. Students apply them first in weekly individual assignments (in which they design and built Fritter, a Twitter clone that addresses design problems they identify in Twitter itself), and then in a major team project, in which they build an app of their own choosing.\nIt remains to be seen how successful this will be, but the evidence of growing enrollment is encouraging. The class began as a combination of a traditional programming class (in which we taught functional programming in JavaScript, Node.js, etc.) and a design class. Over time, we’ve increased the design component and reduced the programming component—aided by the adoption of TypeScript as the language of our prerequisite programming class.\nWe were initially concerned that this move towards design would damage the appeal of the class. But in fact the opposite has been true, and the class has grown dramatically from year to year. This fall we will have our largest enrollment ever, and we’re excited to see how it will go. MIT students, it seems, aren’t only interested in learning how to hack code for that summer job. They want to design software that will change the world, and to do that, they realize that paying some attention to the world matters too.\n* Our class number was previously 6.170. How the number changed, and how we ended up with 5 digits, is itself an interesting story in conceptual design.\n","permalink":"https://essenceofsoftware.com/posts/ux-design-silo/","summary":"There’s no single role for UX design. Instead it’s split up, with bad consequences.","title":"The siloing of UX design"},{"content":"Sometimes an app would be enormously improved if a single essential concept were added.\nAll email clients, for example, would benefit from a robust concept for identifying the people you communicate with. In Apple Mail, searching for messages from particular users is a nightmare: I type the user’s name into the search box and it matches multiple email addresses, which I then have to filter on one by one. And often even that fails to reveal the message I was looking for, because the from field of the message happened not to include the person’s name.\nIn this example, Apple is not to blame (although the design could surely be improved). Without a universal scheme (such as a public key infrastructure for email names and addresses), there’s no easy way to identify users. Even within a more closed network like Gmail you still have the problem of identifying users from outside the system.\nIt’s more surprising when a crucial concept is missing for no apparent reason. I’ve been thinking about Zoom recently and how, despite being a generally wonderful service, it has some design flaws that make the entire experience of using it much less pleasant.\nA new concept for Zoom The missing concept that I have in mind might be called MeetingList. Its purpose would be to let you join meetings more easily, without having to store and recall a meeting identifier externally. The operational principles might be:\nWhen you join a meeting, the meeting identifier is added to the meeting list, and after the meeting ends, you can restart or rejoin the meeting by clicking on it. When you schedule an upcoming meeting, the meeting identifier is added to the list, and you can start or join when the time comes around by clicking on it. If someone else schedules a meeting, they can send you the meeting identifier, and you can add the identifier to your meeting list, and then subsequently start or join by clicking on it. This concept would surely not be difficult to implement. OPs (1) and (3) do suggest that a meeting identifier might carry (either within the identifier or by a binding in the cloud) the date and time, host and title of a meeting; OP (2) doesn’t require this, since the meeting has just been created by this user.\nZoom already includes a fledgling form of this concept that supports only OP (2), but it’s not very useful since only those meetings you happened to create yourself are included:\nThe fuller concept would offer many advantages:\nWhen a meeting ends prematurely, you would be able to rejoin easily without having to recall the meeting identifier. To join a meeting scheduled by someone else, you wouldn’t need to dig around in your email or calendar to find the link. Zoom could provide notifications for meetings about to happen; you wouldn’t need to rely on separate calendar notifications (or give Zoom access to your calendar, which only works anyway for Google and Outlook calendars). This concept would be easy for users to understand; it’s a simple extension of the familiar CallHistory concept used by all communication apps (and by Zoom itself for its phone call feature).\nAn Achilles heel for Zoom? Including this concept might be important not only because it makes Zoom easier to use. Without it, Zoom is more vulnerable to competition from related products (such as Microsoft Teams, Google Meet and Apple FaceTime).\nWhen Zoom first became popular, its concept of a meeting with a persistent identifier was critical to widespread adoption. Unlike other video calling apps at the time, notably Skype and FaceTime, you could connect with someone without them having to have an app installed and being ready to receive your call. You simply sent a URL containing the meeting identifier, and if they accessed it at a pre-arranged time, they could connect to you through the Zoom app in a browser.\nThis was a brilliant idea, and made Zoom feel lightweight and easy. There was no commitment to a platform. In contrast, Teams (for example) required you to register an account and join a team, and it was hard to take calls in more than one team. A common piece of advice at the time recommended assigning a separate browser to each of your teams in order to work around bugs in switching between teams!\nSince then, Microsoft has fixed many of the problems in Teams, and incorporated Zoom’s meeting identifier concept.\nZoom’s very advantage over its competitors—that it was easier to use because there was no platform commitment—may become its downfall. If users don’t have a way to manage all their Zoom meetings within Zoom itself, and have to resort to using external calendars and tools, Zoom becomes simply a video calling technology.\nZoom’s advantage in video quality (and better handling of bandwidth glitches) is shrinking as its competitors improve their technology. If Apple, Google and Microsoft continue to unify and integrate the features they offer within their own platforms, it will be easier and more tempting just to schedule meetings there, leaving Zoom out in the cold.\nThis post was inspired by a discussion with David Jackson who pointed out to me the risk of Zoom\u0026rsquo;s lack of stickiness in its platform, which helped me understand the larger significance of a conceptual flaw that had always bothered me.\n","permalink":"https://essenceofsoftware.com/studies/small/zoom-missing/","summary":"How the omission of a single concept may threaten the success of an app, with Zoom as an example.","title":"Zoom's Missing Concept"},{"content":"In an old interview, unfortunately no longer online, Jony Ive explained how his team at Apple had designed the MacBook\u0026rsquo;s rounded corners. \u0026ldquo;I suppose that\u0026rsquo;s pretty obsessive, isn\u0026rsquo;t it?\u0026rdquo; he ended, with a self-deprecating smile.\nIt struck me as a perfect metaphor for software design: that so often the real work is in the smoothing of rough edges. Each small protrusion and snag may seem inconsequential by itself, but in aggregate they create a constant stream of irritation and confusion for the user. Focusing on them may seem obsessive, but the result can be dramatic. As the Eames told us, the details are not the details: they are the design.\nConcept design offers a new way to control irregularities, the rough edges of software. In this post, using Zoom as an example, I\u0026rsquo;ll explain some simple tactics for identifying and then eliminating them.\nZoom\u0026rsquo;s Reaction Concept When analyzing an existing application, its concepts may not be readily apparent. In a well-designed app, the concepts will be mapped directly to user interface elements, but this is not always so (especially for subtle concepts). Nevertheless, when evaluating an app, it makes sense to start by assuming that the key concepts are aligned with the primary features as they present themselves to users.\nSo we\u0026rsquo;ll assume for now that the prominent button labeled Reactions corresponds to a concept, and we\u0026rsquo;ll treat it as such for now.\nNotice that there are three rows of buttons: (a) the top row, comprising emojis such as clapping, thumbs up, etc; (b) the middle row with yes and no buttons, left and right arrows, and a coffee cup; and (c) the bottom row with a single \u0026ldquo;raise hand\u0026rdquo; button.\nIrregularities in Reactions Clicking on any button, in any row, causes the corresponding icon to appear, as you\u0026rsquo;d expect. It actually appears in multiple places in the user interface, most prominently in the top-left-hand corner of your box in the meeting grid.\nWhat might not be so obvious is that the different buttons behave in slightly different ways.\nFirst, there\u0026rsquo;s a disappearing effect on some reactions but not others. Clicking on a button on the top row makes the emoji appear, but then disappear automatically after ten seconds; this doesn\u0026rsquo;t happen with the reactions on the other rows.\nSecond, some have an undo action. Clicking any button also produces a small icon above the Reactions dialog; for all but the emojis, these icons turn out to be buttons you can click to turn off the effect.\nThird, the buttons enforce some disjointness rules. The top row buttons operate as a radio button group; clicking any one replaces another (if it hasn\u0026rsquo;t yet disappeared). Likewise, the middle row is a group. Surprisingly, the Raise Hand button in the bottom row belongs to this group: if you\u0026rsquo;ve clicked the check mark, for example, clicking the Raise Hand button will make it go away.\nIn Search of Purposes Having identified these irregularities, our next step is to try and justify them—essentially reverse engineering the designer\u0026rsquo;s rationale.\nLet\u0026rsquo;s think about purposes. The Reactions concept as a whole has a simple purpose: to allow users to convey certain common feelings in a lightweight fashion. Digging more deeply, though, we can identify different sub purposes with each row.\nThe top row lets users convey emotions, presumably in reaction to something said or shown by someone else. For the middle row, it\u0026rsquo;s hard to find a single subpurpose. The yes/no buttons are for responding to questions from the speaker; the arrow buttons are to suggest that the speaker slow down or speed up; and the coffee cup button indicates that the user is away from the meeting. The Raise Hand button signals a desire to speak.\nWith these more refined purposes, we can now consider the irregularities and see if they make sense.\nThe disjointness idea is plausible: for example, you wouldn\u0026rsquo;t want to tell a speaker to slow down and speed up at the same time. The automatic disappearance is reasonable, too: clapping, for example, is a momentary gesture and it might be annoying to have to turn it off.\nBut when we examine these more closely, we see that there are some problems. It makes sense that speed up and slow down are disjoint, and that laughing and crying are too. But why can\u0026rsquo;t you clap and send a heart at the same time? And why does stepping away make your yes/no response disappear? And raising your hand cancel your request that the speaker slow down or speed up?\nWe can summarize the irregularities we\u0026rsquo;ve found in two tables. First, a table of the features of each reaction type, with a check mark showing that a feature is present, and a red checkmark in parens showing that it is present but probably shouldn\u0026rsquo;t be:\nReaction Disappears Counted Cancel by host Emojis ✔︎ (✔︎) Yes/no ✔︎ ✔︎ Slow/speed ✔︎ ✔︎ Away (✔︎) (✔︎) Hand (✔︎) ✔︎ (Counted refers to whether a count is displayed for the reaction in the participants panel; cancel by host refers to that panel\u0026rsquo;s \u0026ldquo;clear all feedback\u0026rdquo; action. Both are discussed later.)\nAnd a table showing disjointness between reaction types:\nReaction Emojis Yes/no Slow/speed Away Hand Emojis ✔︎ Yes/no ✔︎ (✔︎) (✔︎) (✔︎) Slow/speed (✔︎) ✔︎ (✔︎) (✔︎) Away (✔︎) (✔︎) ✔︎ (✔︎) Hand (✔︎) (✔︎) (✔︎) ✔︎ The large number of red marks in these tables suggests we have work to do!\nFixing Irregularities The complicated differences between reactions are no doubt intended to save the user trouble (so you don\u0026rsquo;t need to unclap your hands, for example, or turn off slow down when you want the speaker to speed up). But like any automation, they come at the expense of reduced flexibility.\nIn some cases, the flexibility is minor: being able to send more than one emoji at a time, for example. But in others, it\u0026rsquo;s more significant. I can\u0026rsquo;t imagine that any speaker really wants the yes/no responses to a question to disappear when participants raise their hands.\nThis suggests a concept design that lets the developer adjust the rules over time. Rather than hardwiring the fact that claps disappear but yes/no answers don\u0026rsquo;t, we could design the Reaction concept with an action makeDisappearing (reaction), which, when executed, causes a particular reaction to become a disappearing one. In a preamble in the code, a series of calls can then configure the reactions to be disappearing or not.\nA similar strategy can be applied to disjointness: an action addToGroup (reaction, group) might add a reaction to a radio button group with the effect that all reactions in the group are mutually disjoint.\nWith this richer concept in hand, it would be easy to fix the problems in Zoom. We can assign the yes and no reactions to one group, and the slow down and speed up reactions to another.\nNote also that a concept that has been parameterized in this way is properly generic. Our Reaction concept need know nothing about emoji or yes/no answers; it simply has a generic notion of reactions. Keeping a concept generic saves us from introducing domain-specific behaviors, and thinking about using the concept in other contexts helps us evaluate its plausibility.\n(It would be interesting to know if the design of Zoom already incorporates this more flexible concept, or if the programmers have instead hardwired a collection of behaviors—which might explain why what seem to be some undesirable cases have not been fixed. It also seems that these irregularities have changed between versions. When the disappearing behavior was introduced, it applied to the second row of reactions too, leading one frustrated teacher to recommend that you use the raise hand reaction for counting responses to a question instead!\nMore controversially, we could expose these configuring actions to users, letting each user decide what behavior they want: for example, whether they want to be allowed to display multiple emojis at once.\nAnother possibility is to simply eliminate the irregularities. We could design the Reaction concept with two distinct actions: toggleReaction (reaction), which turns a reaction on or off; and transientReaction (reaction), which shows a reaction for ten seconds or extends it for ten further seconds if it\u0026rsquo;s already showing. One could map these actions to the user interface in various ways: a double-click might be a toggleReaction and a single-click a transientReaction, for example.\nOf course, such a design introduces its own complications. I\u0026rsquo;m not recommending it, but just illustrating the range of possibilities.\nDoubling Down: Expanding Differences When you encounter an irregularity in a design, one response is to try and minimize it or even eliminate it. But the opposite may be plausible too: to actually expand the irregularity. If differences are justified by purposes, then distinct purposes may call for amplifying differences amongst the functions that fulfill them.\nHaving noticed that the reactions behave differently, we identified at least two different subpurposes: sending emotional reactions (clapping, smiley faces, etc) and providing feedback (slow down, speed up, etc). One difference between these is that reactions in the first class disappear automatically.\nRecognizing these distinct subpurposes, we might now wonder whether they suggest additional differences in behavior.\nFirst, we might consider differences of audience: perhaps feedback should be shown only to the speaker; a student in a class might not want to share with all the other students a request that the lecturer slow down (and might want to be anonymous to the lecturer too). And perhaps users would like to select the targets of their emojis, sending a heart to just one favored participant, say.\nSecond, noticing that the participant window shows counts for reactions:\nwe might decide to only count some kinds of reaction. Counting how many participants replied yes vs. no, or how many said speed up vs. slow down makes sense. But do we need a count of the number of participants who have sent hearts?\nThird, we might consider differences in interactions with other concepts. Perhaps emotional reactions should be included in the chat window, for example, the way they are in Slack:\nReaders familiar with concept design will detect a bad smell in this discussion. Expanding irregularities amongst reactions will make the overall Reaction concept more complicated and fragile. To keep it generic, we\u0026rsquo;ll now have to have classes of reactions that are counted and classes that are not; and classes that can have targets of particular users; and classes of reactions that are reported in the chat.\nThese irregularities have arisen because a single concept is now serving multiple purposes—and those purposes are to some degree in conflict with each other. This is what I call overloading, and it suggests that we should consider splitting Reaction into multiple concepts.\nEven Larger Differences: Concept Splitting Once the possibility of concept splitting comes up, we start to see opportunities for even larger differences between the reactions (and we\u0026rsquo;ll be reaching the point where it will no longer make sense to even talk of them as \u0026ldquo;reactions\u0026rdquo; as if they were a uniform set with a single purpose).\nThe Raise Hand function, for example, seems to be in a different category from the others, because of the role it plays in negotiating who\u0026rsquo;s speaking. People often forget to lower their hands, making us wonder if a concept is called for that manages the queue of requests to speak, letting a moderator select the next speaker and automatically lower their hand.\nThe emoji reactions might be absorbed in a more conventional reaction concept associated with the chat window: sending emojis could still be easily accessible via buttons on the toolbar, but these might be shortcuts for emoji messages that are similar to other messages in a chat, but just shorter.\nWe\u0026rsquo;ve gone full circle in a sense, here: we accepted the irregularity of emoji reactions (as distinct from yes/no, for example), and even embraced it. Now we\u0026rsquo;re proposing moving the emoji functionality out of the Reaction concept into another concept. But recognizing that the Chat concept might be the right place for it, we\u0026rsquo;re ending up having actually eliminated the irregularity! We might retain the distinction that emojis, unlike regular text messages, show in a user\u0026rsquo;s box in the meeting grid. Even this might be worth reconsidering, however. Maybe chat messages could be displayed there too; I could imagine asking a small class to post a brief response to a question, and preferring to see those responses next to the faces of my students rather than in the chat window.\nLikewise, the \u0026ldquo;I\u0026rsquo;m away\u0026rdquo; button suggests a concept in its own right. It brings to mind the concept of Presence in social media apps such as Slack, in which a user can indicate their availability and readiness to participate.\nAs to the yes/no, slow down/speed up buttons: they too suggest a separate Feedback concept. The Feedback concept is what you might call a fledgling concept in Zoom. It\u0026rsquo;s not quite there, but there are hints in various places that the developers have such a concept in mind. There is a \u0026ldquo;clear all feedback\u0026rdquo; action in the Participants window; unfortunately, the word \u0026ldquo;feedback\u0026rdquo; does not appear in the Reaction window so it will be unclear to most users that it refers to all but the first row of reactions. And it\u0026rsquo;s strange that \u0026ldquo;I\u0026rsquo;m away\u0026rdquo; and a raised hand are treated as feedback, so when this action is executed, those flags disappear too.\nOne might also wonder whether the yes/no buttons are intended to provide a kind of ad hoc poll. If so, perhaps they should be integrated into Zoom\u0026rsquo;s Poll concept. This concept is rather heavyweight: you have to create questions in advance, give a title to the poll as a whole, and only multiple-choice (and not yes/no) questions are supported.\nA User Interface Mockup Our concept analysis of the Reaction concept has led us to a realization that there are actually multiple concepts behind this one dialog. Ideally, concepts are mapped directly to the user interface. Here\u0026rsquo;s a mockup of how things might look, with tooltips showing all at once:\nThe raise-hand and coffee-cup reactions have been moved into a Presence concept (far left) with which the user indicates their participation (requesting to speak, speaking, watching/listening or away). The audio and video settings might be set implicitly: when you switch to speaking, for example, your audio is unmuted. This might address the common problem that users often start speaking without unmuting. Also, some (like me!) are anxious in a large meeting that they might have their audio on inadvertently, so having an explicit indicator that you\u0026rsquo;re not speaking may help. Less common settings (eg, speaking with video off, or speaking with video on and audio off for an ASL presenter) could be available in a submenu.\nThe emoji reactions have been moved into the Chat concept, which is now directly accessible on the toolbar. And the slow down/speed up reactions are now in a Feedback concept of their own, along with visible counts. These controls might appear only when a host activates them.\nOf course, this is all speculative, and this design undoubtedly has flaws yet to be discovered. My intent in showing this mockup is not to claim that it would necessarily work better than Zoom\u0026rsquo;s current arrangement, but simply to show the way in which concept analysis can drive user interface design.\nSummary of principles The approach I\u0026rsquo;ve described here is guided by a few basic principles:\nIrregularities, or differences between similar features, usually impose a cognitive burden, because there\u0026rsquo;s more to understand than when things behave the same way. Simplicity therefore suggests eliminating irregularities and a preference for uniformity. An irregularity may, however, serve a legitimate purpose, and its value may exceed the cost. So any evaluation of irregularities must make purposes explicit. For each irregularity, then, we must determine whether it is justified (and should be tolerated) or not (and should be eliminated). If justified, we should ask whether the irregularity is sufficient to support the purpose that motivates it, or should even be extended. A concept can accommodate a certain amount of irregularity in its design, but the irregularity can be organized in a systematic and generic way. Once purposes diverge significantly, it is better to split into multiple concepts for each of the variant forms of behavior, rather than attempting to contain them within a single concept. Paradoxically, this final extension of differences typically reduces irregularities, because each concept is now more regular. If the new concepts already exist in the app, or are familiar from other apps, the perceived gain in simplicity is even greater. Summary of tactics The process I\u0026rsquo;ve outlined here is summarized in the diagram below:\nThe process may be executed as a routine design review, applied to each concept in turn. Alternatively, it may be prompted by a misfit that is observed (to use Christopher Alexander\u0026rsquo;s term). In my case, the misfit was the common experience of participants in meetings forgetting to lower their hands after they spoke, which led me to take a closer look at Zoom\u0026rsquo;s Reaction concept.\nOnce the relevant concepts have been identified, we look for irregularities. For each irregularity, we try to find either a justification (eg, that emoji, unlike raised hands, are transient gestures so it makes sense that they should disappear spontaneously), or a problematic scenario (for example, a lecturer losing track of yes/no answers because raised hands clear them). These lead to a classification of which irregularities should be retained and which should be eliminated.\nThe next phase involves a design intervention: refining the concept at hand (eg, defining configurable disjointness rules for reactions), splitting into multiple concepts (eg, factoring out the raised hand reaction into a Moderation concept), or merging some functionality into an existing concept (eg, moving emoji into Chat).\nThe outcome of the phase is a modified design. Hopefully it\u0026rsquo;s an improvement, but we might have introduced new problems, or failed to solve the initial misfit. Some evaluation is called for, whether by user studies, design review or some combination. As with any design process, some iteration is inevitable.\nHow Concepts Help The basic idea of identifying and eliminating irregularities is not specific to concept design. But concepts help in some key respects:\nThe task of reviewing a design for irregularities is eased by considering one concept at a time. Concept design\u0026rsquo;s focus on motivating purposes helps us clarify which irregularities are justified and which are not. Concepts give modular boundaries within which to add new functionality (such as the addToGroup configuring action). Concepts help structure refactorings in which functionality is moved (eg, emojis becoming part of the Chat concept). An Invitation I\u0026rsquo;m sure many of my readers have conducted analyses similar to the one I describe here. If you have one you can share, please post it in the concept forum. And as always, comments, criticisms and reactions (!) welcome.\nA Concept Integrity Violation You may wonder why I included parens around the red checkmarks in the tables. Winter Ferguson pointed out that those checkmarks appear uncolored on Android phones. Android automatically converts text characters to corresponding emoji, on which CSS colors are then ineffective. This is a nice example of a concept integrity violation and is a known problem.\n","permalink":"https://essenceofsoftware.com/studies/larger/zoom-reaction/","summary":"Removing irregularities is perhaps the best way to achieve simplicity. This post explains some tactics for how to do it.","title":"Redesigning Zoom's Reaction Concept"},{"content":"The secret of great design Watch great designers at work, or study their designs, and you may wonder: what\u0026rsquo;s their secret? Are they preternaturally inventive? Do they have better taste and judgment than the rest of us? Perhaps, but I don\u0026rsquo;t believe that this can account for the best designs. Based on my own experience studying design and designers (mostly but not exclusively for software), I\u0026rsquo;d attribute the success of great designers to two factors.\nFirst, they\u0026rsquo;ve done it before. Contrary to the picture of design implicit in the design thinking movement—in which extraordinary ideas emerge out of the blue, and each design problem is considered afresh—most designs are adaptations, extensions and compositions of design ideas that came before. A great designer is able to see the essence of a problem, find analogies to problems previously encountered, and rework old solutions.\nSecond, they refine their work. Nobody can solve a challenging problem in one step. What distinguishes great designers from mediocre ones isn\u0026rsquo;t that their first attempts are so much better, but that they critique their work ruthlessly, and keep polishing it until no more improvement seems possible. This observation is not unique to design; George Saunders says it\u0026rsquo;s the secret of great writing too (and perhaps it\u0026rsquo;s a general life strategy).\nImplications for software design Each of these has its implications. The first suggests we might codify a collection of reusable design ideas—both patterns (reusable solutions) and design moves (reusable tactics)—for software. Patterns have been very influential in software engineering (that is, in shaping implementation structures), and concept design is an attempt to find patterns in software design (that is, in shaping behavioral structures). Design moves have been discussed less but are no less important.\nThe second suggests that we need language, structures and design criteria for expressing and evaluating software designs, and that we need to engage in intensive critique of our designs. User testing is valuable, because you usually learn surprising things from people who approach your designs without your preconceptions. But like any form of testing, it doesn\u0026rsquo;t directly help you make a better product. Think about testing code: it can expose bugs (although rarely the subtle ones), but if you just fix each bug as you find it, you\u0026rsquo;re not going to end up with reliable and robust code. Whereas testing encourages you to focus on local details, and to fix problems with patches that can even reduce clarity and uniformity, critique focuses your attention on the essential and more structural aspects of a design and tends to lead to adjustments that produce better overall alignment and consistency.\nAlthough design thinking has done good things, raising awareness of design and encouraging people in all walks of life to see new opportunities for design, it has also tended—perhaps because of its emphasis on democratization of design—to undermine expertise and the value of critique. Natasha Jen, a Pentagram designer, pokes fun at design thinking for these flaws.\nConcept design moves Thinking about design moves, I came to realize that many of the case studies that I analyze in EOS involve one or more simple tactics. I\u0026rsquo;ve classified them into 3 pairs of design moves, each pair comprising a move and its dual. A design move isn\u0026rsquo;t a panacea; it trades off one quality of the design for another. If applied skillfully though, a single move can transform a design from good to great.\nI outline the design moves and give examples of their application in a paper that I\u0026rsquo;ll be presenting in a keynote at the NASA Formal Methods conference in May. (Also see slides on this topic from my recent talk at the Boston ACM/IEEE chapter.) Here, I\u0026rsquo;ll just give a taste of the moves.\nSplit/merge. Split breaks a concept into multiple smaller concepts; merge forms a composite concept from distinct ones. The tradeoff here is between simplicity and power: with more than one concept, the user has more control but things are also more complicated.\nUnify/specialize. Unify takes a collection of concepts that are variants on a theme and unifies them in a single concept specialize breaks a single concept into more specialized variants. The tradeoff here is between generality and problem fit; when you unify, you end up with fewer and more general concepts, but they don\u0026rsquo;t fit the specific applications quite so well.\nTighten/loosen. Tighten increases the synchronization between concepts; loosen weakens it. The tradeoff here is between automation and flexibility. With tight synchronization, the user does less but also has fewer options.\nA language of design: an example Design moves—and concepts—don\u0026rsquo;t guarantee that your designs will be good. But they give you a language for thinking about design, and a way to turn what might otherwise be an intimidating pile of seemingly arbitrary design options into a structured design space that can be navigated more systematically.\nRather than repeating examples from the paper, I\u0026rsquo;ll illustrate this idea with a design move that has yet to be applied. In EOS, I briefly discussed the problem of raised hands in Zoom. People in a meeting raise their (virtual) hands to request to talk, and when their turn comes, they unmute themselves but often forget to lower their hands again. The host, on later spotting the hand still raised, has to ask: is that from before, or a new request to speak?\nWhen I think about this problem, two design moves come to mind. One is the tighten move: we could synchronize the RaisedHand and Mute concepts so that (for example) when you unmute yourself your hand is automatically lowered. This would allow the two concepts to continue to be used mostly independently, but it would have the annoying consequence of lowering your hand (and losing your place in the queue) if you momentarily (and unintentionally) unmute yourself.\nAnother option is to apply the merge move to combine the RaisedHand and Mute concepts into a single concept, Moderation say, for moderated discussions. When the host switches to moderation mode, all non-hosts are muted by default and have to raise a hand to speak. The moderator chooses each speaker in turn, and their hand is lowered and they are simultaneously unmuted. This is arguably a simpler solution, but it eliminates the ability to use muting in a more ad hoc way.\nThis second approach suggests to me that the Mute concept might be serving two distinct purposes: that is, in concept lingo, it\u0026rsquo;s overloaded. One purpose is to let participants in a meeting turn off their microphone for privacy, and the other is to control who is allowed to speak. This suggests applying a split move in which Mute is split into Privacy, a concept that lets users control what other users can see and hear of them, and Permission, which grants users the right to be seen and heard. Our merge move may then be applied to Permission and RaisedHand, leaving Privacy as a distinct concept, which might support some needs more directly. For example, Privacy might provide a toggling action for \u0026ldquo;going private\u0026rdquo; when you want to take a brief break.\nAs always, the challenge will be to balance richness of functionality and automation on the one hand with maintaining simplicity and not overwhelming the user with controls on the other.\nLet\u0026rsquo;s help Zoom! Here\u0026rsquo;s an experiment in community design. Let\u0026rsquo;s together develop a design to address the Zoom raised-hands problem in the concept forum. I\u0026rsquo;ll post these initial ideas to get the topic going.\nFrom my newsletter: archives and signup here.\n","permalink":"https://essenceofsoftware.com/posts/design-moves/","summary":"Or how to become a great software designer.","title":"Design Moves for Software"},{"content":"Dark patterns are no longer just yucky strategies that companies get away with, tempting you to buy insurance you don\u0026rsquo;t need or sign up for a free-for-a-while trial that can only be canceled by calling a phone number that is always busy. Soon they might be illegal. Regulators are taking a closer look, and lawyers in the US and the UK are starting to warn about the liabilities they bring.\nBut what exactly makes a pattern dark? Where is the line between an aggressive but legitimate sales tactic and an immoral or illegal scheme?\nA new project In a research project funded by a new NSF program about designing accountable software systems, I\u0026rsquo;m working with my students Geoffrey Litt and Josh Pollock (and my co-PIs Danny Weitzner and Joan Feigenbaum and their students) on answering this question.\nOur initial take involves—surprise!—software concepts. To date, dark patterns work has focused on user interface tricks: getting you to click the big green button, for example, when it\u0026rsquo;s really that small text field that doesn\u0026rsquo;t even look like a button that you want to click instead. But the more serious dark patterns run deeper, and are conceptual in nature.\nAs bad as it gets For example, in one of the most egregious scams I\u0026rsquo;ve ever seen (thanks so Eunsuk Kang for telling me about this), Intuit advertises a version of the Turbotax tax filing app as \u0026ldquo;guaranteed free.\u0026rdquo; In fact, however, as Propublica has documented, the app is far from free, and if you fail to fall within a narrow range of cases, you will be required to pay (after, of course, you\u0026rsquo;ve taken all the trouble to enter your private and personal tax data). Intuit and other tax software companies had lobbied to prevent the IRS from offering its own free solution (as the revenue agency does in many other countries), and in return promised to provide one itself. The free one, it turns out, is not called the \u0026ldquo;Free Edition,\u0026rdquo; but rather the \u0026ldquo;Freedom Edition,\u0026rdquo; and is not reachable from the Turbotax website.\nDark patterns in Facebook Most cases of dark patterns are more subtle. Take Facebook, for example. Suppose Alice posts a photo of Bob, and Carol (a friend of them both) then tags Bob in the photo. This seemingly innocent action has an unexpected and pernicious consequence: it makes the photo visible to all of Bob\u0026rsquo;s friends, even those who are not friends of Alice or Carol. Perhaps the photo shows Bob at a job fair talking to a competitor and his friends include his work colleagues. There are all kinds of reasons that Bob might not want the photo to be shared in this way.\nNow Facebook would likely respond that its users should know how the app behaves, and that they are free to choose more stringent privacy settings. Bob could have turned on a \u0026ldquo;tag review\u0026rdquo; setting that would have required his approval before the photo was shown to all his friends. In practice, of course, most users don\u0026rsquo;t have a clue what\u0026rsquo;s going on, or how to change their settings. And not surprisingly, the defaults seem to favor Facebook rather than its users.\n(I\u0026rsquo;ve written a piece for a lay audience about how Facebook uses \u0026ldquo;dark concepts\u0026rdquo; to further its sometimes nefarious business practices: you can find it here.)\nThe key idea Here\u0026rsquo;s our idea. What\u0026rsquo;s wrong with simple UI dark patterns is that they confound the expectations of the user: you think that big green button is the normal approval of your purchase, but it turns out to be the one that signs you up with extra insurance that you didn\u0026rsquo;t want to buy. Patterns that operate more deeply also confound expectations, not about the meaning of user interface widgets, but about the very behavior of the concept at hand. When you see a familiar concept like a shopping cart, you bring with you all your prior experiences of shopping carts, so you assume this shopping cart will be no different. You don\u0026rsquo;t expect items to be added to the cart spontaneously without your approval; shopping carts just don\u0026rsquo;t do that.\nThe use of a common concept thus establishes a kind of implicit contract between the company and the consumer. In exactly the same way, if an employee of a supermarket came and dropped items into your cart because they \u0026ldquo;thought you might like them,\u0026rdquo; your grounds for objection should not be that such an action is inherently evil. It\u0026rsquo;s that shopping carts just don\u0026rsquo;t work that way.\nFacebook\u0026rsquo;s concepts Let\u0026rsquo;s apply this idea to Facebook. Our task is to come up with plausible concept definitions that capture our expectations of activities such as posting, friending and tagging. This will not be easy because apps like Facebook are so baroque that arguably our expectations are uncertain and inconsistent. So instead we\u0026rsquo;ll try to imagine how an app like Facebook might work if it were designed to be as simple and clear as possible.\nOur first task is to identify the essential concepts, and assign each a compelling purpose and a role with respect to privacy. For example, we might come up with the following concepts and purposes:\nPost: author content Comment: respond to content by others Friend: control access to your content Tag: identify users in images Even these apparently simple purposes are not uncontentious. The Friend concept, for example, is also used to filter content (so that you see content that you\u0026rsquo;re interested in). This purpose arguably belongs to the Follower concept, however, which in Facebook is merged into the Friend concept. But this issue need not bother us here, since privacy is our primary concern.\nNow let\u0026rsquo;s assign privacy roles to concepts:\nPost: determines who can edit content (the author) Comment: determines who can edit response (its author) Friend: determines who can see content (friends of the content publisher) Tag: no role The privacy roles of Post and Content are straightforward: they control only who can edit the post or the comment (namely the author who wrote them in the first place). Deciding that Tag has no privacy role is obviously inconsistent with the way it behaves in Facebook, but we\u0026rsquo;re laying out the simple case for now. Friend is the interesting one here, and as we\u0026rsquo;ll see, what\u0026rsquo;s tricky is defining who publishes given content.\nAs we enumerate these concepts and their roles, we realize that we have covered reading and editing (and implicitly creating, assuming that each concept governs the creation of its namesake items), but not deletion. So we add one more concept\nOwner: manage deletion of items that will determine who is allowed to delete which items.\nDefining states Our next task is to outline the behavior of the individual concepts. The interesting case is Friend. We start with the state components (aka the data model):\npublishes: User -\u0026gt; set Item friends: User -\u0026gt; set User sees: User -\u0026gt; set Item = friends.publishes The publishes relation will track which items have been published by which users. Note that I wrote Item and not Post or Comment or Tag. No concept should depend on the existence of any other; we should be able to understand Friend without knowing what form the items that are published will take. From Friend\u0026rsquo;s point of view, only the identity of an item matters, and Item can be viewed as a type variable (so that Friend is generic over all possible items).\nNote also that the Friend concept maintains its own record of which item was published by which user; it can\u0026rsquo;t refer to a relation in some other concept (such as which items are owned by which users in the Owner concept, or which posts are authored by which users in the Post concept). This might seem to be redundant, but in fact it isn\u0026rsquo;t: we\u0026rsquo;ll see that the publishes relation does not neatly align with these other relations in the other concepts. You can own an item but not publish it, for example.\nFinally, the sees relation defines which items a particular user can see. It is not a state component that can be independently updated, but is instead defined in terms of the other state components. The expression friend.publishes uses the Alloy join operator, and associates with a user the set of items published by that user\u0026rsquo;s friends.\nDefining actions To complete the behavior of the individual concepts, we define actions. For Friend, for example, we might have the following actions, each with a comment giving its effect on the state:\npublish (u: User, i: Item) // add i to u.publishes friend (u, u’: User) // adds u to u’.friends and vv unfriend (u, u’: User) // removes u from u’.friends and vv read (u: User, i: Item) // blocks unless i in u.sees delete (i: Item) // remove item Of course these specs can be written precisely in a language like Alloy and—in addition to being precise—that would let you do automatic simulation and analysis. But that\u0026rsquo;s a story for another day. The important things to note here are: (a) that the read action is a kind of placeholder; it doesn\u0026rsquo;t really do anything except block if the user isn\u0026rsquo;t allowed to see the item; and (b) that the publish action is not called \u0026ldquo;post\u0026rdquo; or anything that like, because it will be in the composition with other concepts that we get to decide what comprises publishing.\nDefining synchronizations Now we can put things together, and define the overall behavior by synchronizing the actions of the individual concepts. I\u0026rsquo;ve only shown you the actions of the Friend concept, but I think you\u0026rsquo;ll be able to figure out what the actions of the other concepts do from their names.\nHere are two synchronizations:\napp fb includes Post, Comment, Tag, Owner, Friend, … // when user u adds a comment c to post p sync Comment.add_comment (u, p, c) // the author of the comment becomes its owner Owner.create (u, c) // the owners of the post become owners of the comment Owner.add_owners (p.(Owner/owners), c) // the comment is published by the publisher of the post Friend.publish (p.(Friend/publishes), c) // when user u adds tag t to item i for user m sync Tag.tag (u, m, t, i) // creator of tag becomes owner Owner.create (u, t) // mentioned user becomes owner too, so can delete Owner.add_owners (t, m) // tag itself is published by publisher of item Friend.publish (i.(Friend/publishes), t) The first one describes what happens when the user adds a comment. That action produces three additional actions: the user who adds the comment becomes an owner of the comment; the owners of the post (on which the comment is made) become owners of the comment; and the comment is published by the publisher of the post. The expression p.(Friend/publishes) uses Alloy\u0026rsquo;s nifty dot operator to navigate backwards from the post p through the publishes relation in Friend to obtain the set of users that are the publishers of p.\nAs a result of these actions, when Alice adds a comment to Bob\u0026rsquo;s post, only Alice can edit the comment (since authorship is maintained in the Comment concept); both of them can delete the comment (since they are both owners, and the Owner concept lets owners delete items); and the comment is deemed to have been published not by Alice but by Bob. This last part is critical for our exploration of privacy in Facebook: it\u0026rsquo;s what ensures that it\u0026rsquo;s Bob\u0026rsquo;s friends, and not Alice\u0026rsquo;s friends, who see the comment.\nThe second sync says what happens when a user tags an item. Both the tagger and the tagged person are owners, so the tagged person can delete the tag if they don\u0026rsquo;t like it. More interestingly, the effect of tagging includes making the tag published by the publisher of the item. So this means that the tag is visible to the same set of users as the item it\u0026rsquo;s attached to.\nModeling the way Facebook really is This is not how Facebook works by default, although you can get this behavior by tightening your privacy settings. Here is a sync that describes Facebook\u0026rsquo;s default behavior:\n// when user u adds tag t to item i for user m sync tag.tag (u, m, t, i) ... // owners as before // tag itself is published by publisher of item friend.publish (i.(Friend/publishes), t)\t// tagged person becomes a publisher of the item friend.publish (m, i)\t// tagged person becomes a publisher of the tag friend.publish (m, t) In this variant, the person tagged (m) additionally becomes a publisher of both the tag and the item tagged: that means that all their friends will see the tagging.\nWhat have we gained? By describing the Facebook design in terms of concepts, we\u0026rsquo;ve achieved a few things:\nWe\u0026rsquo;ve factored the behavior into understandable pieces. In particular, the Friend concept is responsible for managing visibility, and encapsulates a very simple rule: you can see what your friends publish.\nBecause concepts can be combined by synchronization, we can describe behaviors in which actions happen not only because they are directly initiated by the user (such as adding a comment), but also because they occur implicitly, in combination with other events (such as the publishing of the comment by the author of the associated post).\nWe can now disentangle things, When Alice tags Carol in Bob\u0026rsquo;s post, who is publishing what? In the design we proposed as the sane one, Bob is the publisher of the tag, just as he would be the publisher of a comment that Alice made on his post. And thus the audience of the post is not expanded by the act of tagging. In contrast, in the Facebook design, when Alice tags Carol in Bob\u0026rsquo;s post, Carol is deemed to be publishing not only the tag but also the original post!\nWe can now judge the various design options, distinguishing sane and straightforward designs from those that are at best baroque and at worst malicious. For example, the case for arguing that any follow-up to a post is treated as published by the author of the post is straightforward, and corresponds to the idea of the whole discussion being \u0026ldquo;hosted\u0026rdquo; by that author. The case for the actual Facebook design, in which a tagged person is deemed to republish the post as a whole, seems much harder to defend.\nWhere does this all go? The upshot of this rather lengthy (and in places complicated) discussion can be stated simply. A user interacts with a software system by executing a series of actions. In an honest design free of dark patterns, which actions occur, and the effect of those actions, align with the user\u0026rsquo;s expectations. Those expectations are set by the assumptions that the user brings to the interaction, based on prior experiences with similar systems. They include simple expectations about the user interface itself (for example, that green means go and red means stop), and deeper expectations about the underlying functionality (for example, that items don\u0026rsquo;t spontaneously get added to a shopping cart). Concepts provide a way to identify recurring units of functionality, and thus to articulate these assumptions.\nThese ideas could be used to make apps such as Facebook more transparent and accountable to their users. An industry standards body could agree on the definition of concepts such as Friend, and Facebook would then be encouraged (or perhaps required) either to conform to that concept, or make it very clear that their concept was not compliant. Along with this, Facebook would be required to explain in simple terms who is publishing what, so that a user can figure out how actions in other concepts are interpreted as publishing actions from the point of view of Friend.\nAdmittedly, this isn\u0026rsquo;t simple. But it\u0026rsquo;s surely simpler than the status quo, in which the user is faced with a complex data model of posts, comments, replies, tags and friends, and very little structure that would allow them to understand what\u0026rsquo;s really going on.\nThis post has been edited, based on very helpful discussions with Rebecca Jackson and Eunsuk Kang. Thank you to both of them. Thank you also to Jimmy Koppel for pointing me to Joel Spolsky\u0026rsquo;s story in which he experienced \u0026ldquo;sneak into basket\u0026rdquo; in a physical store.\n","permalink":"https://essenceofsoftware.com/studies/larger/facebook/","summary":"What\u0026rsquo;s really going on with dark patterns?","title":"Facebook and Dark Patterns"},{"content":"Basic sys admin tasks can be surprisingly hard. You find some magic incantations online, which you dutifully type into a terminal, but then they don\u0026rsquo;t have the desired effect. What now? At that point, you\u0026rsquo;re usually stuck with no recourse (except to start an endless descent into online forums).\nI had this experience recently trying to install Jekyll, a static website generator. Despite trying for a few hours, I just couldn\u0026rsquo;t do it. So I switched to Hugo, a similar tool that offers an executable image so you don\u0026rsquo;t have to build it yourself. Downloading a file and executing it is fortunately something I\u0026rsquo;m capable of doing (although even that has become harder with Apple\u0026rsquo;s security protections).\nWhy are these simple tasks often so challenging? In large part, it\u0026rsquo;s because the underlying concepts are both complicated and fragile, so you need to know a lot—and understand them more deeply than should be necessary— to work around their rough edges.\nDNS and DKIM Here\u0026rsquo;s an example, involving two concepts—one you\u0026rsquo;re probably familiar with and one that might be new to you. The first is domain name resolution, and is the central concept of DNS (the Domain Name System). Its purpose is to decouple long-lasting domain names from the ephemeral machine addresses that serve them. The operational principle, roughly, is that the owner of a domain provides the domain name and an IP address, and subsequent lookups for that domain resolve to that IP address. The concept works hand-in-hand with the domain registration concept, which manages the process by which an owner acquires a domain (along with permission to update its DNS records), and the concepts corresponding to the various protocols that use DNS, such as http.\nThe second is domain keys identified mail (DKIM), a concept whose purpose is to reduce email spoofing. The operational principle is that the owner of a domain signs the from-address of an outgoing email message (along with some other fields from the email header) with the domain\u0026rsquo;s private key, and places the signature in the header; the recipient then uses the domain\u0026rsquo;s public key to check the signature, rejecting the message if the check fails. Assuming no other server has access to the domain\u0026rsquo;s private key, this ensures that only the legitimate servers of that domain can successfully sign messages, and messages that purport to come from that domain but are actually spoofing the from-address will be rejected.\nGetting the Key How does the recipient obtain the domain\u0026rsquo;s public key? It could send a request to the outgoing email server, but that would require the server to support a special protocol. Instead, the key is stored as a DNS record. In addition to the records used for the domain name resolution concept, the domain name system includes a special \u0026ldquo;TXT\u0026rdquo; record type that is used to hold protocol-specific details associated with a domain. Thus a domain owner can create a record for DKIM that holds the DKIM public key.\nTXT records have proliferated with the growth of new applications. Unfortunately, the DNS query that returns TXT records returns all the TXT records associated with a domain, and the recipient has to sort through them to find the relevant record. For this reason (and others), a new approach has become popular, in which the DKIM key is placed not in a TXT record, but in a CNAME record. CNAME, which stands for \u0026ldquo;canonical name\u0026rdquo; is a form of record used for domain name aliasing: it lets you map one host name within your domain to another. It is commonly used, for example, to redirect a name such as www.dnj.photo to dnj.photo.\nSo how is this done? Here\u0026rsquo;s the trick. You create a CNAME record that maps a name like s1._domainkey.foo.com to the name of a host that, when queried, will provide the DKIM key (thus also introducing a useful layer of indirection, so you can change the DKIM key without changing the DNS record). In this name, s1 is called a \u0026ldquo;selector\u0026rdquo; and is used by DKIM to select one key over another (so that you can rotate keys for example, or have different keys for different email services), and foo.com is the email-sending domain that is being authenticated. The funny part is _domainkey. That\u0026rsquo;s an actual string, not a placeholder, and its purpose is to ensure that the whole name is interpreted as a label for a DKIM key, and not as a regular domain name.\nUnderscoring the Problem This tactic risks polluting the namespace of your domain, and that\u0026rsquo;s why the DKIM designers included the underscore. Perhaps you have a host name called domainkey, but surely you wouldn\u0026rsquo;t want a host name that starts with an underscore? In fact, underscores cannot appear in host names according to IETF standards. But then, since CNAME records map host names, shouldn\u0026rsquo;t an underscore be prohibited there?\nRFC2181 attempted to clarify this issue, insisting that DNS is a general database, and is not just for domain name resolution, and the names that appear as labels in DNS records can thus be arbitrary strings. That\u0026rsquo;s all very well, but it\u0026rsquo;s not consistent with the intended role of CNAME records, which is precisely to map host names! Another article noting this confusion explains that \u0026ldquo;there are many DNS entries that are not host names,\u0026rdquo;\u0026quot; which seems to imply that CNAME records, whose labels generally are host names, should indeed be excluded from the looser rule.\nNot surprisingly, some DNS providers reject underscores in CNAME records, and this means that if your mail service needs DKIM keys to be stored in CNAME records, you\u0026rsquo;re hosed. ()\nWhat\u0026rsquo;s going on here? In short, to support DKIM, the domain name resolution concept has been overloaded. This is a form of overloading I call \u0026ldquo;piggybacking\u0026rdquo; in my book, in which a developer wanting to add some function finds some existing concept to support it, even though it doesn\u0026rsquo;t quite fit.\nHere, the new function is using DNS for mapping application-specific attributes to values. By squeezing this functionality into the existing domain name resolution concept, and the CNAME record in particular, we now have a mess in which some \u0026ldquo;host names\u0026rdquo; aren\u0026rsquo;t in fact names of hosts, and DNS providers differ on how they interpret the rules about whether such a name can have an underscore.\nWhat\u0026rsquo;s the alternative? Instead of piggybacking, DNS might have been extended with a new concept, allowing lookups in which you give a domain name (foo.com) and an application-specific attribute (dkim, say), and the DNS server returns the associated value (the DKIM public key). A new standard for TXT records could support such a concept.\nNo Big Deal? Now you might say that I\u0026rsquo;m making a mountain out of a molehill, and that these are small complications. But in practice, it\u0026rsquo;s a mass of small, self-inflicted wounds of this sort that make many systems so complex and fragile. It turns out that DKIM—as well as SPF and DMARC, the other protocols used to prevent mail spoofing—can often be broken by hackers, because of exactly this kind of non-uniformity (for example, in parsing email addresses).\nPiggybacking will always seem cheaper and easier than modifying a concept or creating a new one. But the eventual price may be much higher.\nFrom my newsletter: archives and signup here.\nUpdate A few updates and corrections, following some further investigation and input from DNS experts:\nDNS as a general database. The earliest RFCs mention DNS holding information beyond host addresses—including phone numbers for CSNET, for example—and make it clear that the resource records were not to be limited to the initial types. It wasn\u0026rsquo;t until later, though, that the idea of DNS as a general key/value store seems to have emerged explicitly. Jerry Saltzer, who developed a name service for Athena at MIT called Hesiod, told me that Paul Mockapetris added the TXT resource type to support more general lookups, as required by applications such as Hesiod.\nDomain names as intentional names. Domain names that included property labels go back at least to Hesiod, which used an @-symbol to separate the property-specifying part from the rest, eg. finger-server@berkeley.mit.edu. A project at MIT in 1999 explored this general idea, in which a name does not designate a service directly, but rather specifies the properties for a desired service, and called it intentional naming. In 2000, RFC 2782 described the addition of the SRV resource type, which mapped domain names of the form _service._protocol.name to server/port names, allowing intentional names such as _ldap._tcp.foo.com.\nDomain names that include property keys. A domain name like domainkey.foo.com is not an intentional name that specifies a service. The DKIM protocol does not require a service; all that\u0026rsquo;s needed is for the DKIM key to provided for the domain. Instead, this domain name is a combination of a domain name (foo.com) and a key to be looked up in the DNS records of that domain name.\nUnderscores in domain names. The use of underscores in these extended forms of name prevented conflicts with hostnames, but introduced the new risk of the new labels conflicting. In 2019, RFC 8552 described the convention of naming with underscored labels, and introduced a registry to avoid collisions.\nUnderscore confusions. The early RFCs said that domain names should be as general as possible, but confusing wording misled many people. A much-quoted statement from RFC 882 seems to say that underscores are not permitted in the labels that comprise domain names: \u0026ldquo;The labels must follow the rules for ARPANET host names. They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen.\u0026rdquo; This statement, however, seems on closer reading to be an informal explanation for a grammar that is not intended to be mandatory: \u0026ldquo;The preferred syntax of domain names is given by the following BNF rules. Adherence to this syntax will result in fewer problems with many applications that use domain names (e.g., mail, TELNET).\u0026rdquo; This complicated position is elaborated in RFC 1035 which states:\nThe DNS specifications attempt to be as general as possible in the rules for constructing domain names. The idea is that the name of any existing object can be expressed as a domain name with minimal changes. However, when assigning a domain name for an object, the prudent user will select a name which satisfies both the rules of the domain system and any existing rules for the object, whether these rules are published or implied by existing programs.*\nNot surprisingly this has confused even experts; a ballot amongst a consortium of companies voted to sunset the use of underscores in DNS names appearing in certificates, citing the statement in 1035 that \u0026ldquo;labels must follow the rules for ARPANET host names\u0026rdquo; which it took, incorrectly, to specify \u0026ldquo;the characters which may be used in DNS domain names.\u0026rdquo;\u0026quot;\nCNAME records for DKIM. Use of CNAME resource records for DKIM is not (I believe), as I originally suggested above, to avoid the use of TXT records, but rather to provide an extra level of indirection so that a domain can delegate to a hosting service the job of assigning (and rotating) DKIM keys. The use of the _domainkey prefix alone would limit the number of TXT records returned, since the extended domain name has a different set of resource records associated with it than the base domain name.\nRefining the concept design analysis. In summary, there are (at least) three distinct concepts in play here. First there is the concept of a HierarchicalName, which allows a name space to be divided into separately managed zones. This concept is familiar from file systems, and from the structure of many web APIs (which use so called \u0026ldquo;RESTful\u0026rdquo; names for resources). Second is the concept of IntentionalName, in which a name becomes a kind of specification. This also existed prior to DNS, although the concept has been less widely adopted. Third is the concept of Metadata in which an object has a collection of properties associated with it; a photo, for example, has its capture time and exposure; a file has its creation time; a DNS domain has its domain key.\nThe lens of concept design helps us recognize that much of the richness of DNS that we have explored comes from the fact that three distinct concepts are being offered. The familiarity of these existing concepts should make DNS easier to understand.\nWhat is unusual is that all three concepts are implemented by the same mechanism. In concept lingo, the second and third concepts are \u0026ldquo;piggybacked\u0026rdquo; onto the first concept, with the properties and specifiers of Metadata and IntentionalName respectively both represented as labels in a prefix of a domain name. Like many piggybacking designs, this is ingenious and solves some problems. In particular, it allowed DNS itself to remain unchanged, without the need for new resource types or mechanisms (although it did require the creation of a new registry to avoid name clashes).\nThe downside is that piggybacked concepts generally cannot be fully supported by a mechanism that was not designed for them. A full implementation of IntentionalName, for example, would allow wildcard specifiers, so that for example one could request not only a color printer _printer._color.local.foo.com or a monochrome printer _printer._mono.local.foo.com but also any printer _printer.*.local.foo.com whether color or monochrome. As noted in RFC8552, DNS cannot support such wildcards. Another price paid for the piggybacking is some additional complexity involved in squeezing the new functionality into a Procrustean bed—here, the underscore, and all the confusion created about whether it is permitted.\nWhether the DNS design is bad or good is not the main issue here—and now I have a better appreciation of the tradeoffs I am less inclined to insist that this piggybacking is a mistake. What my analysis shows, I hope, is that concept design can reveal the underlying issues and make clearer whatever tradeoffs are being made.\nMany thanks to Dave Crocker, Jerry Saltzer and Garrett Wollman who generously shared with me their expertise on DNS.\n","permalink":"https://essenceofsoftware.com/studies/larger/dns-dkim/","summary":"Or why basic sysadmin tasks are so hard.","title":"Concept Piggybacking in DNS"},{"content":"I\u0026rsquo;m always excited to find examples outside software that concept design can be applied to. Here\u0026rsquo;s one I came across this week.\nAs part of a restructuring of our curriculum, we\u0026rsquo;re fixing up our class numbers. Currently, our programming sequence looks like this:\n6.0001 Introduction to CS and Programming using Python 6.009 Fundamentals of Programming 6.031 Software Construction 6.170 Software Studio After renumbering, it might become:\n6.100 Introduction to CS and Programming using Python 6.101 Fundamentals of Programming 6.102 Software Construction 6.103 Software Studio Surprisingly, making such a change actually turns out to be really hard. One major problem is that (because we have a large department with many classes) this would require reassigning some existing numbers. But MIT\u0026rsquo;s registrar has a rule that you can\u0026rsquo;t reassign a class number unless it has been unused for five years!\nWhat\u0026rsquo;s going on here? The concept of class number has multiple, conflicting purposes: in concept lingo, it\u0026rsquo;s overloaded. One purpose is to be a shorthand that signals the role that the class plays in degree requirements. This includes the level of the class (thus hinting that 6.101 would be taken before 6.102) and, in the case of our renumbering, also the general area the class belongs to (thus 6 says it\u0026rsquo;s a class in EECS, and 6.1 that it\u0026rsquo;s in the software track).\nThe other purpose is that the number acts as a name that persists from term to term (so students and prospective employers can assume that 6.103 is consistent from year to year, for example). It\u0026rsquo;s because of this second purpose that the registrar forbids reassignment of numbers without a five-year gap.\nWhat\u0026rsquo;s the solution to this dilemma? The standard concept design move would be to split into multiple concepts, one for each purpose. We could add a mnemonic name (or commandeer the course title) to serve the naming purpose, and reserve the number for signaling the place in the curriculum. So then we might have, for example:\n6.100 Software Intro 6.101 Software Fundamentals 6.102 Software Construction 6.103 Software Studio Is this a reasonable solution? I\u0026rsquo;m not sure. It would require a culture change at MIT to refer to a class by a name (\u0026ldquo;software studio\u0026rdquo;) rather than a number (\u0026ldquo;103\u0026rdquo;). On the other hand, encouraging more mnemonic names might be a good thing and would help beyond the university: I remember that Tim Nelson\u0026rsquo;s class that teaches Alloy at Brown is called \u0026ldquo;Logic for Systems\u0026rdquo; and not that its number is \u0026ldquo;CSCI 1710\u0026rdquo;. And the alternatives are not much better. One ingenious proposal is to satisfy the registrar\u0026rsquo;s rule by appending a zero to each number, and then remove it after five years.\nI wonder if there\u0026rsquo;s a better conceptual solution. Let me know if you can think of one!\nFrom my newsletter: archives and signup here.\n","permalink":"https://essenceofsoftware.com/studies/small/class-numbers/","summary":"Who knew such a simple thing could be so challenging?","title":"The Class Number Dilemma"},{"content":" This article summarizes the key ideas in the book The Essence of Software. It is not intended to be understandable by itself. It gives almost no examples, makes claims without justifying them, and cites almost no related work. And it\u0026rsquo;s not nearly so much fun to read as the book :-). But, hey, at least it\u0026rsquo;s short(ish)!\nIf you haven\u0026rsquo;t read the book, I recommend that you don\u0026rsquo;t start with this, but watch my ACM talk instead. But perhaps you don\u0026rsquo;t really want to read the book, and just want to be able to hold your own at a cocktail party showing off your sophistication in software design. In that case, this may be for you.\nIf you\u0026rsquo;ve read the book, this summary should be a useful reminder of the key ideas, to help you solidify them in your mind and relate them to other approaches. In this sense, it augments the shorter list of provocative questions in Chapter 12 of the book.\nA final warning before you jump in: this summary assumes a fairly extensive background in software design and development. The book itself is aimed at a broader audience, introduces the ideas more gently by way of example, and relegates the harder technical stuff to end notes.\nDefining Software Design In most fields, \u0026ldquo;design\u0026rdquo; means shaping an artifact to meet the needs of users, thus sitting at the boundary between humans and machines. Despite Kapor\u0026rsquo;s notable manifesto in 1990, and the book edited by Winograd that followed in 1996, little attention has been paid to software design.\nEnormous effort, in contrast, has been devoted to software engineering (whose interest instead is software\u0026rsquo;s internal structure and means of construction). This disparity of attention has resulted in great advances in programming, represented by a major body of knowledge and well-known design principles.\nThe field of human-computer interaction has likewise produced an impressive body of knowledge about user interfaces and how they shape the way we understand and use software. But for software design, where the focus is on the fundamental abstractions that underlie both the interface and the implementation, our knowledge is much more limited, and we have had to be content with being guided instead by vaguer notions, and (albeit sensible) appeals to simplicity and clarity.\nThe Book\u0026rsquo;s Aims and Approach The aims of the book are to highlight a central aspect of software design; to lay out a way to structure and express software designs from this perspective; to provide some heuristics and principles for design; and, more generally, to inspire a renewed focus on software design as a discipline.\nThe book is driven by examples drawn from over 100 apps. By focusing on widely used software produced by the best companies, it seeks to show that serious problems are rife, and challenge even the most talented developers, and that the book\u0026rsquo;s ideas and techniques apply to real software.\nThe book presents a design approach comprising simple textual and diagrammatic notations, a collection of readily applicable heuristics, and some deeper principles. In order to explain the approach in more detail, and to show how it differs from many prior approaches, a collection of end notes is included covering topics from design thinking to formal methods.\nThe Problem Software is harder to use than it needs to be. We spend an increasing portion of our lives engaged with software apps and systems, so improving the design of software impacts the quality of our lives and our ease of working effectively together.\nAs software becomes an ever more critical part of our civic infrastructure, we rely on apps and systems to behave predictably and reliably. A high proportion of failures are due to user errors, for which poor design is usually to blame. And even when a failure is attributed to a bug in the code, it is likely that the bug is due to lack of clarity in design (rather than a simple failure to meet a specification).\nLack of clarity in software design also makes software harder to build, and leads to degradation over time as accretions to a codebase compromise its modularity yet further.\nLevels of Design Software design activities and criteria can be assigned to three levels: physical, which concerns the choice of colors, type, layout, etc, and is influenced by particular human anatomical and cognitive capabilities; linguistic, which concerns the use of icons and labels, terminology, etc, and is dependent on shared cultural and linguistic assumptions; and conceptual, which concerns the underlying semantics of the application, including the actions that can be performed and the effects they have, and the structure and interpretation of the state as viewed by the user.\nThe conceptual level is the most fundamental; different physical and linguistic designs for an app might be equally effective, but even small changes to the conceptual design are usually very disruptive to users. Users\u0026rsquo; problems with apps arise more often from incorrect conceptual understandings than from an inability to interpret the physical and linguistic signals of the user interface.\nThis is due in part to the great advances that have been made in the last decades at the physical and linguistic levels. Many books and online collections of heuristics teach user interface design very effectively, and given also that many client-side frameworks now provide professionally designed widgets, there is little excuse nowadays for a design that fails at these levels.\nPrior Work on Conceptual Design The importance of the conceptual level has been recognized for more than half a century. From early on, researchers noted the importance of a user\u0026rsquo;s \u0026ldquo;mental model\u0026rdquo;, and the need for the design to construct such a model explicitly, so that the user\u0026rsquo;s model and the system model coincide. Fred Brooks coined the term \u0026ldquo;conceptual integrity\u0026rdquo; and argued that the conceptual aspects of software design represented the essence of the field, as opposed to the accidental aspects, to which he relegated the concerns of \u0026ldquo;representation.\u0026rdquo; The fields of conceptual modeling, domain modeling and formal methods all emphasized the centrality of an abstract model of state (and, in the case of formal methods, also behavior) in the design of software.\nAnd yet none of these fields expressly addressed the problem of designing the conceptual structure of software in order to meet the needs of the user and to align the user\u0026rsquo;s understanding. Formal methods focused primarily on the problem of correctness, and ensuring conformance of the implementation to the model. Conceptual modeling and domain modeling focused primarily on the representation of knowledge about the context of operation of a system, rather than on the structures that the designer invented.\nMost curiously missing was a well defined notion of \u0026ldquo;concept.\u0026rdquo; Even in the field of conceptual modeling, there is no shared understanding of what a concept might be, or even well-known candidate definitions. An entire conceptual model seems to be too large to count as a single concept, and its constituent entities (or classes or objects) are too small, especially since an informal understanding of a concept tends to involve relationships amongst multiple elements.\nA New Definition of Concept A concept is a reusable unit of user-facing functionality that serves a well-defined and intelligible purpose. Each concept maintains its own state, and interacts with the user (and with other concepts) through atomic actions. Some actions are performed by users; others are output actions that occur spontaneously under the control of the concept.\nA concept typically involves objects of several different kinds, holding relationships between them in its state. For example, the Upvote concept, whose purpose is to rank items by popularity, maintains a relationship between the items and the users who have approved or disapproved of them. The state of a concept must be sufficiently rich to support the concept\u0026rsquo;s behavior; if Upvote lacked information about users, for example, it would not be able to prevent double voting. But the concept state should be no richer than it need be: Upvote would not include anything about a user beyond the user\u0026rsquo;s identity, since the user\u0026rsquo;s name (for example) plays no role in the concept\u0026rsquo;s behavior.\nConcept Reuse and Familiarity Most concepts are reusable across applications; thus the same Upvote concept appears for upvoting comments in the New York Times and for upvoting answers on Stack Overflow. A concept can also be instantiated multiple times within the same application.\nThis archetypal nature of concepts is essential. From the user\u0026rsquo;s perspective, it gives the familiarity that makes concepts easy to understand: a user encountering the same context in a new setting brings their understanding of that concept from their experience in previous settings.\nFrom a designer\u0026rsquo;s perspective, it allows concepts to be repositories of design knowledge and experience. When a developer implements Upvote, even if they can\u0026rsquo;t reuse the code of a prior implementation, they can rely on all the discoveries and refinements previously made. Many of these are apparent in the behavior of the concept itself, but others are associated with the implementation, or are subtle enough to need explicit description. The community of designers could develop \u0026ldquo;concept catalogs\u0026rdquo; that capture all this knowledge, along with relationships between concepts (for example, that Upvote often relies on the Session concept for identifying users, which itself is associated with the User concept for authenticating users).\nConcept Independence Perhaps the most significant distinguishing feature of concepts, in comparison to other modularity schemes, is their mutual independence. Each concept is defined without reference to any other concepts, and can be understood in isolation.\nEarly work on mental models established the principle that, in a robust model, the different elements must be independently understandable. The same holds in software: the reason a user can make sense of a new social media app, for example, is that each of the concepts (Post, Comment, Upvote, Friend, etc) are not only familiar but also separable, so that understanding one doesn\u0026rsquo;t require understanding another.\nConcept independence lets design scale, because individual concepts can be worked on by different designers or design teams, and brought together later. Reuse requires independence too, because coupling between concepts would prevent a concept from being adopted without also including the concepts it depends on.\nPolymorphism is key to independence: the designer of a concept should strive to make the concept as free as possible of any assumptions about the content and interpretation of objects passed as action arguments. Even if a Comment concept is used within an app only for comments on posts, it should be described as applying comments to arbitrary targets, defined only by their identity.\nPushing polymorphism as far as possible is a useful strategy: for getting to the essence of a concept; for finding other opportunities to use a concept within the same app (and in other apps); for discovering that what appears to be an app-specific concept is actually a general and familiar one; and, when it\u0026rsquo;s not possible, for identifying concept subtleties or design flaws.\nConcepts can be implemented independently also; in the Deja Vu platform, Santiago Perez De Rosso showed how apps could be constructed by glueing together reusable, full-stack concepts drawn from a library.\nA Structure for Describing Concepts To work with concepts, we need a simple structure for describing them. A concept is defined by its behavior, which comprises its state space and a set of actions. These can be defined using standard methods and notations.\nFor defining the state space, a data model is given that consists of a collection of components, each of which is a scalar/option, set or relation (of any arity). This model can be recorded with textual declarations or as an extended entity-relationship diagram. The details of the data model are not fundamental to concepts, and different models could be used. What is important is that the state of a concept is (by default) visible to users, so states should make sense to users (and at the very least should be defined explicitly in the spirit of \u0026ldquo;model-based specification\u0026rdquo; rather than implicitly by algebraic axioms).\nActions can be initiated by the user or by the system, and a single action can have both inputs and outputs. A single action can abstract what would be an entire use case in object-oriented modeling approaches, allowing a much terser form of description. An action can also abstract away (that is, not represent in detail) the creation of complex inputs; for example, an action input may be a rich-text object that in the implementation would be produced by an extended interaction with a rich-text editor.\nActions read and write states. An action may have a precondition that makes it applicable only in certain states; when the precondition does not hold, the action is blocked and may not occur. Actions can be non-deterministic, resulting in more than one possible outcome for a given pre-state/input combination. But any non-determinism must be exposed in output arguments, so that the state of a concept is always a function of the trace of actions that have occurred so far. For example, an airline reservation concept could have an assignSeat action that picks an arbitrary seat and assigns it to the customer, but the seat must be represented as an output of the action (in addition to appearing in the post-state of the relation that maps customers to seats).\nFrom a data modeling perspective, there is no global data model. Instead, there is a collection of local data models, one for each concept. Each concept\u0026rsquo;s data model is just rich enough to support its actions. (In teaching data modeling, I have found that this makes it much easier for programmers to scope their models. In traditional modeling, it\u0026rsquo;s all too easy to get carried away with building a data model based on ontological observations that are not actually relevant to the design of the system.)\nIf the state and action definitions are written in a formal notation such as Alloy, the concept behavior can be analyzed automatically, with generation of sample executions, comparison of action versions, and checking properties. The modularity that concepts provide amplify the effectiveness of automated analysis. Such analysis is always inherently intractable (the number of executions to check rising super-exponentially with the number of objects in the state), so researchers have always looked for ways to decompose a system into smaller parts. Concepts provide such a decomposition, aligned naturally with the functionality boundaries of the system.\nThe definition of a concept augments this basic behavioral description with two more novel parts explained in more detail below: the purpose and the operational principle.\nConcept Purposes The idea that an artifact should have a purpose distinct from its specification is hardly novel, and many researchers have recognized the importance of purposes, and the impossibility of expressing them fully and precisely (most notably Christopher Alexander in Notes on the Synthesis of Form).\nWhat is novel in concept design is the idea that it is not sufficient for the system or app as a whole to have a purpose (or a collection of purposes). Each concept in its design should have its own purpose. A concept\u0026rsquo;s purpose defines, in a general setting, the reason for its invention and what benefits it offers; in the setting of a particular app, it defines the motivation and justification for including the concept.\nPurposes also clarify subtle distinctions between related concepts. In social media, for example, there are several concepts that may sit behind a \u0026ldquo;thumbs up\u0026rdquo; widget: Upvote, whose purpose is to rank items by popularity so that users (purportedly) see the most valuable content first; Reaction, whose purpose is to convey an emotional reaction to the author of an item; Recommendation, whose purpose is to learn a user\u0026rsquo;s preferences so that subsequent items can be recommended more reliably; and Profile, whose purpose is to track the user\u0026rsquo;s interests in order to target advertising.\nBecause purposes sit at the boundary of the human/computer interface, they cannot be judged formally (the way correctness can, eg). Nevertheless, the book provides criteria for determining whether a purpose is compelling, and when two distinct purposes may be masquerading as one.\nThe Operational Principle A concept definition also includes its operational principle (OP), an archetypal scenario that shows how the concept fulfills its purpose.\nSuperficially, the OP is like a use case, but it plays a very different role. Use cases are a specification notation, and a full spec typically requires many use cases. Use cases are often expressed at a low level too, in terms of the micro-interactions of the user interface (clicking buttons etc). Since a concept\u0026rsquo;s actions fully define its behavior, nothing more is needed to predict how the concept will behave.\nInstead, the OP captures the dynamic essence of the concept, telling the most basic story about how the concept works. Sometimes the OP is so simple it barely even needs stating: when you add a comment to a post, your comment will subsequently appear along with the post (the OP of the Comment concept). But often the OP is more interesting. It may require two distinct scenarios: when you delete a item, you can restore it from the trash; once you empty the trash, though, it\u0026rsquo;s gone forever and its space is reclaimed (Trash). It may need to be quite elaborate in order to demonstrate the purpose and distinguish the concept from similar but less powerful concepts: if you assign a style to two items, and then you update the style, both items will be updated in concert (Style). And it may involve multiple users: after users have upvoted items, the items will be ranked by the number of times they were upvoted (Upvote).\nThe OP may also be aspirational, applying only in ideal circumstances (even if they are typical). It is not true that if you reserve a restaurant table, and then turn up, a table will necessarily be available (since guests at your table may have stayed longer than expected); nor that a successful authentication associated with a user account must have been performed by the same user that registered the account (since someone may have stolen your password). Like many aspects of concepts, the OP may appear to be obvious until you consider things more deeply.\nThe OP often provides the clearest way to explain a concept. To use the PORT elevator system, a rider selects their destination floor on the device in the lobby, which responds with the identifier of the lift car to be taken; the rider enters that car and is taken to their floor. This is complicated enough—and different enough from standard elevators—for the maker (Schindler) to advertise that it\u0026rsquo;s \u0026ldquo;as easy as 1-2-3\u0026rdquo; (namely involves an OP with multiple steps).\nParadoxically a full behavioral description is often less helpful, since (a) it fails to distinguish the essential aspects of the concept design (how style updates effect items) from more arbitrary design decisions (what happens when a style is deleted); and (b) it conveys the motivation for the behavior. For Michael Polanyi, from whose work the idea of the OP is taken, this latter consideration is the key to understanding the difference between design/engineering on the one hand and science on the other (put bluntly, why physics cannot explain how a clock works).\nConcept Synchronization Within an app, concepts can operate largely without interaction: commenting on posts, for example, may be orthogonal to upvoting them. But often concepts need to be coupled together to achieve the app\u0026rsquo;s goals. For example, it may be problematic for users to edit posts after they have been upvoted (since the upvotes responded to content that may have changed). To mitigate this the upvote action of the Upvote concept could be synchronized with the edit action of the Post concept, to prevent edits after a post has been upvoted or to remove upvotes when a post is edited.\nFor some concepts, synchronization is part of their intended usage. An access control concept is intended to suppress some actions in another concept when the user lacks the right permissions; a subscription concept is intended to react to certain actions by generating subsequent notification actions. In these cases, the concept typically offers \u0026lsquo;placeholder\u0026rsquo; actions that are synchronized with the real actions in other concepts: an access control concept may have an access action, for example, which is then pinned to the action to be controlled in another concept.\nSynchronizations are defined reactively: when an action occurs in one concept, some actions should occur in other concepts. Arguments can be passed between actions, so that synchronization includes data flow. Each synchronization is atomic and happens in its entirety or not at all, so if one of the reactive actions is blocked by its concept, the initial action cannot occur either.\nThis model of communication and interaction is inspired by Hoare\u0026rsquo;s CSP, and can be formalized in its terms. A crucial property of the model is that the behavior of each individual concept is preserved. Synchronization can prevent a concept from executing an action, and it can limit the arguments presented to an action, but it can never cause an action to occur (or an output to be produced) that would not be possible for the concept in isolation. As will be explained below, composing concepts maintains their integrity.\nThe synchronization mechanism is also essential for maintaining concept independence and avoiding the need for one concept to \u0026lsquo;call\u0026rsquo; another. At the code level, composition requires mediators to implement the synchronization; these mediators make reference to the concepts, but the concepts themselves remain free of mutual references.\nSynchronization and Automation Synchronization of concepts is often a form of automation, in which the user is saved the trouble of executing a concept action because it follows execution of an action in another concept automatically. Omission of a desirable automation can be attributed to under-synchronization of an app\u0026rsquo;s concepts: in Zoom, for example, the raised hand concept might be synchronized with the audio muting concept so that participants\u0026rsquo; hands are automatically lowered after they have their turn.\nAutomation preempts the user\u0026rsquo;s manual control and thus, if not configurable, can be problematic. Such automation can be attributed to over-synchronization of an app\u0026rsquo;s concepts: in some calendar apps, deleting in the event concept leads undesirably to declining in the invitation concept.\nSynchronization and Decomposition What an app presents as a single concept may be better understood as a synchronization of multiple component concepts. A hint that such a decomposition is warranted is often found in the presented concept having multiple, conflicting purposes.\nFacebook, for example, appears to offer a \u0026rsquo;like\u0026rsquo; concept, but on closer examination, this concept is a synchronization of several of the concepts described above: Upvote, Recommendation, Reaction and Profile. This synchronization is responsible for some confusions and critical commentary about Facebook\u0026rsquo;s design (notably, that an \u0026lsquo;angry\u0026rsquo; reaction produces a positive upvote—by some accounts counted for more than a simple \u0026rsquo;like\u0026rsquo;). With this decomposition in mind, different design options are easy to see: having separate buttons for \u0026ldquo;I\u0026rsquo;m angry\u0026rdquo; and \u0026ldquo;I want to see more of this kind of post\u0026rdquo;, for example.\nConcept Synergy In most compositions, concepts bring additive value. But in some designs, concepts can take advantage of each other so that one concept achieves its own functionality in part by relying on another concept. This is synergy in the setting of concept design, where the value of putting two concepts together is more than the sum of their individual values.\nAn example is Apple\u0026rsquo;s synergistic composition of the Folder and Trash concepts. By making the trash a folder, the design allows the action of moving items between folders to be used to restore items from the trash. Achieving this synergy is non-trivial, and the book explains its subtle evolution over time.\nAttempted synergies can backfire. A version of Outlook placed system logs (reporting for example on connection failures between client and server) in mail folders as if they were messages, but this led to numerous problems (for example that reports of connection failure could not be delivered to clients if\u0026hellip; connections failed).\nConcept Dependence Diagrams As explained above, concepts are inherently uncoupled and free standing, so they can be understood, designed, evaluated and reused independently. Bringing concepts together in an app does not couple them either, since the synchronization mechanism ensures that each concept will conform to its own behavioral expectations even if its actions are tied to the actions of other concepts. Furthermore, implementation need not introduce any dependencies between concepts.\nThere is a kind of dependence between concepts, however, that is useful to analyze, and that arises in the context of usage in a particular app or system. A concept C1 is said to depend on a concept C2 in an app A when the inclusion of C1 only makes sense if C2 is also present. Note that \u0026ldquo;makes sense\u0026rdquo; doesn\u0026rsquo;t mean that C1 would somehow break if C2 were missing, so there is no traditional software-engineering dependence here. Rather, the dependence reflects an understanding of the role of C1 in A. In a social media app with a Comment concept and a Post concept, for example, Comment might depend on Post because it was included to allow users to comment on posts, and if there are no posts, there\u0026rsquo;s not much point in having comments.\nAs with conventional dependencies, the concept dependence relation can be depicted as a dependence graph or diagram. Dependences have several uses. They determine which orders of explanation of an app\u0026rsquo;s concepts will be intelligible; you\u0026rsquo;d explain posts before comments on posts, for example, They define, implicitly, an entire application family (as the set of subgraphs in which some subset of the concepts appears and a concept never appears without those it depends on). The dependence diagram can thus be used to make scoping decisions about which concepts to include or exclude. Dependencies also suggest a development order: it\u0026rsquo;s better to build Post before Comment, so that comments have targets to be tested on.\nConcept Mapping In an implementation of a concept design, the concepts must be mapped to the user interface, connecting the conceptual level of design to the physical and linguistic levels. Actions might be executed by gestures or button presses, for example, and the states of the concepts will be displayed in various views.\nThe standard techniques and heuristics of user interface design apply directly to concept mapping. The lens of concept design helps focus this work.\nSome concepts present tricky mapping challenges, and good mappings can be part of the design knowledge associated with a concept. For the filtering view in the Label concept, in which a collection of items is filtered by a selected label, for example, it might seem desirable to maintain an invariant that the items displayed are exactly those carrying the label, but this turns out not to be a good idea.\nIn Gmail, the Label concept is used to classify messages, but the user interface shows messages in the context of conversations and associates labels with conversations instead. This mapping design is troubled, and leads to a variety of odd behaviors.\nTesting and Prototyping Are Not Enough A concept design is ultimately evaluated in the context of use, and misfits (in which a design fails to fulfill its intended purpose) are never fully predictable, because they depend on properties of the human environment which may not even be knowable until the design is deployed.\nUser testing offers some limited value, especially for evaluating concept mappings, and for catching egregious flaws, but (unlike real deployment) will rarely encounter the corner cases that are most troublesome in a design. Prototyping is a valuable strategy for exploring candidate designs early on, and indeed many of the practices of design thinking can be fruitfully combined with concept design, and are made more useful by the separation of concerns that concepts offer.\nBut generate-and-test is not a viable method for finding a good design in a large design space, let alone a great design. To do that requires an expert designer who can apply prior knowledge of likely problems and known solutions, often acquired in very different settings. Concepts provide a framework for recording and retrieving such knowledge.\nA note for computer scientists: the known problems of standard concepts are sometimes grounded in technology (eg, the difficulty of achieving consistent views of data in a distributed system) but are more often a result of human behavior (eg, that ticket sales invite scalpers who then drive up prices).\nThe Need for Concept Design Principles Over the last few decades, a rich body of UX design principles has been developed. These include prescriptive adaptations of psychological ideas, such as the Gestalt principles of grouping, which can be used to guide layout design, and James Gibson\u0026rsquo;s notion of affordance, made applicable in user interfaces by Don Norman\u0026rsquo;s principle that affordances should be explicitly \u0026ldquo;signified\u0026rdquo;. Norman\u0026rsquo;s book The Design of Everyday Things introduced several additional principles, such as the idea of \u0026ldquo;mapping\u0026rdquo; (a different usage from concept mapping) in which a user interface mirrors the structure of the domain being controlled. Other pioneers produced explicit collections of principles: Ben Shneiderman\u0026rsquo;s golden rules, Jakob Nielsen\u0026rsquo;s heuristics, and Bruce Tognazzini\u0026rsquo;s principles of interaction design.\nEspecially in the hands of experts, these principles make it possible to design a user interface that is likely to be highly usable, with a low probability of serious usability flaws. Indeed, there is no excuse nowadays for poor user interface design, and user testing is much less important for routine design work (although it\u0026rsquo;s still valuable of course, especially for very novel or critical designs).\nAlmost all of these principles, however, are focused on the physical and linguistic levels of design. New principles are needed at the conceptual level. The book presents three such principles and explains them in detail with many examples from contemporary apps.\nConcept Design Principles Familiarity. When possible, a familiar concept should be preferred to a new, unfamiliar one. With familiar concepts, users can rely on their prior experience, and don\u0026rsquo;t need to learn a concept afresh. And designers can take advantage of the body of knowledge associated with a known concept, reducing the risk of a design with unexpected misfits. The familiarity principle can be seen as an application of a meta principle of consistency: when a purpose arises in an app that has arisen before, the same solution should be used.\nSpecificity. If you draw up a list of the purposes purportedly served by an app, the purposes and the concepts that aim to fulfill them should be in one-to-one correspondence. This implies first that every purpose should have at least one concept that fulfills it, and that every concept has some purpose that motivates it. So much is obvious, although there are examples in real apps (which the book explains) of unhelpful concepts that serve no user purpose, and purposes that are essential to an app that are fulfilled by no concept.\nThe more subtle implications are: no redundancy, namely that each purpose should be fulfilled by at most one concept, and no overloading, namely that each concept should serve at most one purpose. Redundancy is clearly undesirable because it involves a waste of resources (in the designer\u0026rsquo;s work and in the user\u0026rsquo;s understanding). Overloading is a more subtle notion: that a concept that tries to serve multiple purposes is pulled in different directions, and cannot serve any one purpose effectively. This idea is related to the independence axiom in Nam Suh\u0026rsquo;s theory of mechanical design, and to the common observation in programming that each segment of code should have a single goal.\nThe book gives many examples of concepts doomed by overloading, classifying them into different causes for the overloading:\nfalse convergence, when a concept is designed for two different functions that were assumed (wrongly) to be aspects of the same purpose; denied purpose, when a purpose was ignored by the designer, despite the desires of users; emergent purposes, in which new purposes arise for old concepts, often invented by the users themselves; and piggybacking, when an existing concept is adapted or extended to accommodate a new purpose. Integrity. This principle says that when concepts are put together into an app, the each concept should continue to behave according to its (app-independent) concept definition. If concepts are composed by synchronization, integrity will be preserved by design. But if composition is more ad hoc, or if concepts are adjusted to suit the larger context of the app, there is a risk of violation. Google Drive, for example, offers a sync concept (not to be confused with concept sync!) in which local files on disk are kept in sync with files in the cloud. But in an egregious violation of integrity of this concept, only conventional files are properly synced, and files associated with Google Apps (such as Google Docs) are treated specially, and are represented on the local disk not by their contents but by a URL. A lack of awareness of this limitation has been catastrophic for some users.\nComparisons to Other Approaches Concept design builds on more than 50 years of advances in a variety of fields, and contributes some new ideas. First, some work that sounds similar but is not really relevant:\nConcept maps are diagrammatic representations of collections of facts, with each edge representing a proposition that applies a predicate (the label of the edge) to two atoms (the nodes it connects). Concept maps are an application of knowledge graphs; they were proposed as an educational tool. They are \u0026ldquo;conceptual\u0026rdquo; in the sense that predicate logic lets you formalize all kinds of relationships, at any level of abstraction; and since the predicates don\u0026rsquo;t need to be designated (using Michael Jackson\u0026rsquo;s term), they can express ideas that would be hard to nail down precisely. The very flexibility of concept maps seems to limit their leverage in software design.\nConcept lattices are representations of partial orders between classifications, where the order is set inclusion. Classification is a useful activity but is limited in the context of software design because it doesn\u0026rsquo;t address relationships, and is usually static, so does not address behavior.\nAt the other end of the spectrum, the most closely related work is:\nConceptual modeling. The field of conceptual modeling is broad and has many motivations, but part of it is concerned with explicit representations of the structures that underlie both applications and the mental models of users. The emphasis of the field has tended in the direction of ontologies (eg, for reasoning and knowledge representation), or in the direction of domain modeling (eg, for understanding the environment in which a software system operates), and in these respects conceptual structures are usually discovered rather than invented. In contrast, concept design focuses more on concepts as socio-technical inventions that serve a purpose. Most conceptual models are essentially abstract data models (in the spirit of entity-relationship diagrams), although there is work on dynamic models too (although these are hard to distinguish from standard kinds of dynamic models such as Statecharts). What concept design seeks to provide, which seems to be missing from conceptual modeling, is the notion of a identifiable concepts. Without this, there can be no modularity in conceptual models, and it is not possible to talk about concepts as separable contributions, to identify common concepts between models, or to reuse concepts. The terminology of conceptual modeling sometimes seems to imply that the entities of a data model are concepts, but these are not good concepts, since concepts typically involve relationships between objects. Some approaches seek to identify concepts with objects or classes, but this has an implementation flavor to it, since there is rarely a natural way to assign relationships and behavior to a single object or class.\nObjects and classes. In the early days of object-oriented programming and development, the idea that objects naturally mirrored or modeled the real world was seen as a major benefit. Over time, the idea became less plausible, first because object-oriented programming took on more arcane forms moving further away from domain structures (under the influence of the Gang of Four patterns and other advanced programming techniques), and second because the inherent implementation bias in object-oriented design became clearer. The key problem is that, in the real world, the properties and behavior of objects involve relationships between them, and can rarely be satisfactorily assigned to individual objects. In practice, this is usually addressed by defining objects that hold relationships between other objects, but such a style is not really object-oriented. Worse, objects cannot generally be defined independently of one another (and in fact, as explained in the book, tend to produce code dependences that violate Parnas\u0026rsquo;s principles).\nDomain-driven design. DDD is a very popular and successful approach to software development, created by Eric Evans. It can be viewed as a modern incarnation of the idea of building a software system on an explicit model of its problem domain, which goes back to Simula and JSD. A key innovation of DDD is the idea of \u0026ldquo;bounded context\u0026rdquo;: that each system or app has its own domain model, which serves its own functionality, reflects the world view of its development team, and may differ from (and even be inconsistent with) the domain models of other systems that operate in the same (larger) domain. Concept design proposes a more fine-grained structure, focusing on modularity within an app, each concept in a sense having its own bounded context (defined by its state/data model). In addition to the domain modeling aspect, DDD embodies an extensive collection of patterns and practices for building more flexible software. One pattern in particular suggests you \u0026ldquo;look for the underlying conceptual contours\u0026rdquo;, and it seems likely that concept design should thus be a good match for DDD. Finally, one key difference: a domain-driven design typically starts by constructing a model of the specific domain in a bottom-up fashion; one popular approach called \u0026ldquo;event storming\u0026rdquo; starts by classifying key events in the domain (just as JSD did). Concept design instead starts with recognizing standard concepts that can be assembled for the app at hand, and thus places more emphasis on reuse of domain models. This difference should not be fundamental however, and it seems likely that the two approaches could be used profitably together.\nFeature-oriented development. Features are increments of functionality; a feature model comprises a set of features and constraints on which subsets of features can be combined together, implicitly defining a product family. Feature-based frameworks (such as Don Batory\u0026rsquo;s AHEAD tool suite) automate the configuring of features and merging of code fragments. In contrast to concepts, features are more flexible, and can not only model arbitrary increments of functionality but can also address aspects of a system that are not user-facing; a caching feature, for example, might improve performance without producing any observable change in behavior. The price paid is that features, unlike concepts, are not generally independent of one another, and cannot be easily reused across applications.\nCross-object modularity mechanisms. A variety of mechanisms have been developed to accommodate functionality that cross-cuts the traditional object boundaries in OO programming. These include aspect-oriented programming, subject-oriented programming and role-based programming. The latter two are similar to concepts in factoring out behaviors that serve distinct purposes, separating them from the existence of particular objects. Unlike concepts, however, these notions are not generally independent of one another, and (like features) are intended to be implemented in the context of a particular system and not transportable between systems.\nFeature interaction. In the context of telephony, \u0026ldquo;features\u0026rdquo; have a different connotation, and are (like concepts) user facing and typically intended to be independent of one another. The \u0026ldquo;feature interaction\u0026rdquo; problem arises when a system includes features that prescribe conflicting behaviors. In concept design terms, feature interaction is a violation of integrity, and would be resolved by not allowing conflicting features to be active at the same time. This is one approach taken in telephony, but more flexible approaches are taken too (for example, giving priority to one feature over another). This flexibility might be achieved with concepts by designing concepts with non-determinism that can be resolved in composition.\nMicroservices. Most backend applications are nowadays structured as a collection of \u0026ldquo;microservices\u0026rdquo;, each providing an API and its own internal logic and storage. A concept can be viewed, at least from a functionality perspective, as a \u0026ldquo;nanoservice\u0026rdquo;—like a microservice but with a much more limited scope (with a single focused purpose rather than a collection of purposes around some area of functionality such as billing or advertising). Microservices are not generally independent of one another, and because they aggregate app-specific collections of functions, are not reusable across apps.\nStrategies Each chapter of the book includes an outline at the end of some practical strategies, and a final chapter presents a set of questions for each kind of person (program manager, consultant, UX designer, etc) who might use concept design. Here are some highlights of ways to use concept design in your work:\nDesign concepts for a new app, or for new functionality in an existing app, using the concept design structure, formulating purposes and operational principles to bring design focus. Avoid reinventing the wheel when designing an app, by always looking for ways in which existing, familiar concepts might suffice (and by spotting concepts that are small variations away from familiar concepts). Inventory the concepts in your app (or app family) to get a bird\u0026rsquo;s eye view of its functionality. Construct a dependence diagram to show what subsets are possible. Do this as a retrospective review of an existing app, in designing a new app, or in planning extensions or digital transformations. Identify concepts that are the most valuable (eg, product differentiators), the most troublesome (in terms of user confusion/complaints, development costs, etc), the least valuable (which might be dropped or replaced by more powerful concepts), etc. Genericize one or more concepts by reformulating its purpose independently of the type of objects it works on, and consider how making it more polymorphic might allow it to be simplified and applied in more contexts within your app (and across your app family or suite). Decompose concepts that serve multiple purposes into single-purpose concepts that are synchronized together. Find familiar concepts lurking behind the concepts of your app, and adjust your concepts or decompose them to expose familiar concepts that will make the app more intelligible to users. Look for synchronization opportunities that would increase the degree of automation in your app; conversely look for ways in which synchronization is excessive and eliminates manual controls that users would welcome. Identify redundant concepts that could be removed by being replaced by existing concepts (which may need some enrichment). Identify overloaded concepts that are complex or brittle, and split them into separate concepts. When users have trouble understanding your app, first consider overloading as a possible cause. Examine the data model or schema of your app and consider how you might decompose it into the local data models of individual concepts. Create help and training materials that are concept-driven so they can be grasped more easily by users: follow the dependence diagram for ordering; use consistent names for a concept throughout; note when a familiar concept is being used; present concepts with purposes and operational principles first (before the details of all their actions). Fun things The end notes include mini-essays on some serious topics, including:\nEmpiricism and its pitfalls Design thinking and \u0026ldquo;content-free\u0026rdquo; process Verification and its pernicious consequences Christopher Alexander, patterns and misfits Inevitability as a design criterion Mental models and gulfs of evaluation and execution Normal and radical design The pitfalls of object-oriented programming and also include some lighter topics such as:\nMy favorite pasta sauce recipe How to prevent ice dams and water leaks in your house An Easter egg in Don Norman\u0026rsquo;s book Why pixels and inches in CSS don\u0026rsquo;t mean what you think Comments \u0026amp; reactions? Join the discussion at forum.essenceofsoftware.com!\n","permalink":"https://essenceofsoftware.com/posts/distillation/","summary":"Save yourself the trouble of actually reading the book.","title":"The Essence of the Essence"},{"content":"Here is a lovely sketchnote by the wonderful artist and designer MJ Broadbent, which she created live during a keynote I gave about concept design at the SATURN conference in 2016. Concept design has come a long way since then, but many of the key ideas are here.\n","permalink":"https://essenceofsoftware.com/posts/sketchnote/","summary":"Early concept design ideas, drawn by MJ Broadbent.","title":"Concept Design Sketchnote"},{"content":"Yes, I know you get too much email too. But I wanted a chance to be in touch with people who are interested in concept design, so if you\u0026rsquo;d like to be included in my occasional mailings:\n","permalink":"https://essenceofsoftware.com/posts/subscribe/","summary":"Join an ultra-low-volume email list to hear new ideas about concepts and design.","title":"Design Updates in Your Mailbox"},{"content":"Concepts provide a framework for recording design experience. Designers often find themselves reinventing the wheel, designing and deploying concepts that have been built many times before, and then rediscovering a raft of subtle problems that were already known.\nMy dream is to have a concept catalog (or a collection of concept catalogs, perhaps organized by domains) that would hold a collection of concept definitions, and with each the accumulated experience and design wisdom from previous deployments of that concept.\nAs an example of what such a catalog entry might look like, here is a sample entry for the Upvote concept. Comments, suggestions and reactions welcome in the forum topic for this example.\nThis idea is inspired, of course, by the success of design patterns in programming.\nName: Upvote [Item, User]\nPurpose: Track relative popularity of items\nAlso known as: Like\nRelated concepts:\nReaction (send emotional reaction to author of post)\nRecommendation (track user approvals for future recommendations)\nModerate (maintain quality of forum by approving submissions)\nKarma (grant privileges to users based on good behavior)\nFlag (crowdsource moderation by having users mark bad items)\nState:\nupvotes, downvotes: Item -\u0026gt; set User count: Item -\u0026gt; one Int // item's count is defined as number of ups minus downs // all i: Item | i.count = #i.upvotes - #i.downvotes Actions:\nupvote (i: Item, u: User)\u2028u not in i.upvotes i.upvotes += u i.downvotes -= u downvote (i: Item, u: User)\u2028u not in i.downvotes i.downvotes += u i.upvotes -= u unvote (i: Item, u: User)\u2028u in i.(upvotes+downvotes) i.upvotes -= u i.downvotes -= u Operational principle:\nafter a sequence of upvote and downvote actions, the count of an item is equal to the number of users who upvoted minus the number who downvoted General notes\nSyntax. Since this is the first concept that I\u0026rsquo;m adding to the catalog, here are a few notes on the syntax I\u0026rsquo;m using. This is the structure I use in EOS, but with more formal definitions of the actions. I\u0026rsquo;m using Alloy-like syntax for expressions and statements, which is hopefully intuitive even to those unfamiliar with Alloy. The state and actions part of the concept model is not new to concept design; it\u0026rsquo;s just a definition of a state machine activated by atomic actions as you\u0026rsquo;d find in any formal modeling language (such as TLA+, B, Z, VDM, OCL).\nDeclarations. The declaration r: A -\u0026gt; B declares a relation from A to B, constrained by any multiplicities (using the keywords one for exactly one, lone for zero or one, some for one or more, and set for zero or more).\nAction definitions. Action definitions can include constraints on the pre-state, as well as update statements. For example, in upvote, the first line is a precondition saying that you can\u0026rsquo;t upvote an item if you\u0026rsquo;ve already upvoted it. An update statement changes a relation, using +/- for adding and removing tuples. For example, in upvote, the second line says that the set of users associated with the item i in the upvotes relation is increased by u (or, equivalently, the tuple i-\u0026gt;u is added to the relation).\nType parameters. The list of types after the name of the concept gives the type parameters. These are types that are polymorphic, and can be bound to other types when the concept is composed. In this case, all the types are parameters. The Item type, for example, may be bound to Comment or Post in another concept, and the User type may be bound to any kind of principal (as the term is used in the security community).\nOperational principle. The OP is given informally. As explained in EOS, it\u0026rsquo;s an archetypal scenario, and unlike use cases, doesn\u0026rsquo;t need to cover all the functionality (so note that unvote is not mentioned). In this case, it looks a bit pathological because the concept is pretty simple: it basically just counts votes.\nDesign issues\nNegative count. This design allows counts to go negative. An alternative is to make a minimum of zero (even if the number of downvotes is still tracked implicitly).\nUnvote as own action. In some deployments, you may use downvote to let a user reverse their upvote action instead of unvote, but this won\u0026rsquo;t work if downvoting is limited (eg, by Karma) to certain users (as in Hacker News).\nUsing count. The count of an item can be used in different ways. Often it is shown to users, and is used to determine the rank order in which items appear. Sometimes the count is used directly for item ordering (as in comments in the NY Times), but more complex schemes are common: in Hacker News, for example, the rank of a post is computed from the number of points, the post\u0026rsquo;s age, and other factors.\nPreventing double voting. This design prevents a user from voting twice on a single item. In practice, this requires either tracking the identity of the user (eg, with session tokens), or by using a proxy for the user, such as the IP address, MAC address or browser id.\nVisibility. Usually only the count relation (and not the upvotes and downvotes relations) is made visible, so users cannot see who voted for an item. For Facebook \u0026ldquo;likes\u0026rdquo;, usernames are visible, but that can be seen as an artifact of combining the Upvote concept with the Reaction concept.\nEventual consistency. Getting exact counts is generally not required, so common implementations scale by not requiring strong consistency. This means that a user who upvotes an item will generally see that item\u0026rsquo;s count increase, but may not see the effect of a simultaneous (or even earlier) upvote from another user until later.\nKnown uses. Universally used in all social media apps, typically for posts and comments. Facebook combines the Upvote, Reaction and Recommendation concepts into a single \u0026ldquo;Like\u0026rdquo; concept, with some confusing outcomes (for example, that an angry reactions counts as an upvote).\nSynchronizations. Commonly synchronized with authoring concepts (such as Comment and Post), and with Karma (where an upvote may be synchronized with a reward action, so that a user needs to obtain some number of upvotes before reaching a certain karma level).\n","permalink":"https://essenceofsoftware.com/studies/small/upvote/","summary":"An example of a concept catalog entry that defines a concept\u0026rsquo;s purpose, behavior and design issues.","title":"Upvote: An Example Concept"},{"content":"I\u0026rsquo;m excited to have been invited by the ACM Practitioner Board to give a tech talk at noon Eastern Time on Dec 1, 2021.\nUpdate: recording of the talk here\nThe Essence of Software (Or Why Systems Often Fail by Design, and How to Fix Them)\nWe’ve made great strides in software, but many systems are still hard to use or behave badly. Traditionally, we’ve looked to bugs in code to explain why systems go wrong—or to flaws in the user interface that may lead to misuse. In this talk, I’ll argue that the real problem often lies elsewhere: in the very concepts of the design. I’ll present a variety of surprising snags with familiar applications, from Dropbox to Gmail, and I’ll show how concepts can diagnose them and suggest fixes. I’ll explain in concrete and actionable terms what concepts are—essentially free-standing “nanoservices” that factor the behavior of a system into independent and reusable parts—and how you can apply them in your work, whether you’re a coder, program manager, software architect, UX designer, or consultant.\n","permalink":"https://essenceofsoftware.com/posts/acm-tech-talk/","summary":"Daniel explains concept design in a single video.","title":"ACM tech talk"},{"content":"You can download two free chapters from the book:\nWhy I Wrote This Book Concept Integrity ","permalink":"https://essenceofsoftware.com/posts/sample-chapters/","summary":"Download free sample chapters.","title":"Sample book chapters"},{"content":" Strong concepts. Having one or a few core concepts that are embodied in scenarios that deliver what the user needs, overcoming points of pain and friction. For Zoom, that’s the meeting concept. Conceptual clarity. The core concepts of an app have to be conveyed, through the UI and all explanatory material, with clarity and simplicity. It’s probably not possible to do this for a “dual use” app (like Hangouts) that offers multiple concepts (eg, video call and meeting) for the same function. User are also very sensitive to needless complications and distractions in a UI (in Skype’s case, spam and advertisements). Robustness. If the concept implementations are buggy or unreliable, or fail to scale sufficiently, it won’t matter that their design is good. Users will abandon them anyway. Skype suffered from a reputation for bugs and low reliability. Robustness might seem orthogonal to concept design, but it isn’t. Bugs arise because developers get confused about what they’re trying to do, so having fewer, simpler concepts leads to more robust code.\ndiagrams for these scenarios\nlabeling the painful bits: actions and states in old scenario. do as annotations. and use same ones on new scenario. whether all steps must be included (eg end of call?); generally would be good to show completion but may not be necessary. whether user actions not in concept included (send link, talk); yes, helpful to show in grey. whether have to resolve how call ends or host leave early; no, show one example and leave full range to SM. can i lift the meeting concept? yes, because create action is still allowed (but how to specify that link returned is unique? does this really doom the lifting? and even join has to find the object, so it’s really not so easy to lift; mahybe a story here about why objects aren’t good?) ","permalink":"https://essenceofsoftware.com/drafts/concepts-redux/strong-concepts/","summary":"\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eStrong concepts.\u003c/strong\u003e Having one or a few core concepts that are embodied in scenarios that deliver what the user needs, overcoming points of pain and friction. For Zoom, that’s the meeting concept.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eConceptual clarity.\u003c/strong\u003e The core concepts of an app have to be conveyed, through the UI and all explanatory material, with clarity and simplicity. It’s probably not possible to do this for a “dual use” app (like Hangouts) that offers multiple concepts (eg, \u003cem\u003evideo call\u003c/em\u003e and \u003cem\u003emeeting\u003c/em\u003e) for the same function. User are also very sensitive to needless complications and distractions in a UI (in Skype’s case, spam and advertisements).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRobustness\u003c/strong\u003e. If the concept implementations are buggy or unreliable, or fail to scale sufficiently, it won’t matter that their design is good. Users will abandon them anyway. Skype suffered from a reputation for bugs and low reliability.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eRobustness might seem orthogonal to concept design, but it isn’t. Bugs arise because developers get confused about what they’re trying to do, so having fewer, simpler concepts leads to more robust code.\u003c/p\u003e","title":""},{"content":"The operational principle Using a term coined by Michael Polanyi (and brought to software engineering by Michael Jackson), I call a scenario that explains how a product “just works” its operational principle (OP).\nThe operational principle is just a sample scenario, but it includes enough richness to convey typical usages. The example I gave above for Zoom could be written in a more textual way like this\n\u0026lt;create, start, join+, end\u0026gt; where join+ means one or more join actions. Another scenario would be this\n\u0026lt;create, start, end\u0026gt; in which the meeting is started but nobody joins, so the poor host ends it without talking to anyone. That’s a perfectly valid scenario, but it’s not useful, so it wouldn’t count as an OP.\nAnother example: To use a traditional elevator, you call it (by pressing the up or down button), you enter the first elevator that arrives, select your floor, and exit when you arrive:\n\u0026lt;call, enter, select, exit\u0026gt; Some modern elevators work differently: you select the floor in the elevator lobby, enter the elevator at the indicated bank, and then exit at your floor:\n\u0026lt;select, enter, exit\u0026gt; Note that we could enrich the OP to make it clear that you have to enter the correct elevator, writing something like\n\u0026lt;select(f):b, enter(b), exit(f)\u0026gt; to say that when you select floor f, the system responds with elevator bank b, and you then enter an elevator in bank b, and exit at floor f. Likewise we could have indicated that the participants in our Zoom meeting use the link it was started with:\n\u0026lt;create():m, start(m), join(m)+, end(m)\u0026gt; A restaurant reservation system lets you reserve a table, then start using it at the prescribed time, and then end using it some time later:\n\u0026lt;reserve, start, end\u0026gt; (How much later the end action might occur is a difficult design challenge: if diners stay too long, the restaurant fails to deliver on its OP, but if the reservation slot is longer than necessary, tables will sit empty.)\nThe OP of a software app represents its innovative essence. Some OPs were invented just for software (or for computerized systems)\u0026mdash;the elevator’s, for example. Many, like the restaurant reservation OP, were invented as a social protocol that existed long before computers (back in the 19th century, in fact). And some, like Zoom’s, are a kind of hybrid: naming a meeting place in advance is not so different from what people have always done when they say “let’s meet at the pub at 7pm” (although, unlike the pub, Zoom’s meeting place is created on demand from an unlimited pool).\nThe non-obviousness of an OP may be a symptom of cleverness: the elevator that begins with floor selection, for example, allows better load balancing (by directing travelers to particular elevators).\nBut a trivial OP can be a significant innovation too. Take the OP for blogging platforms, which is little more than this:\n\u0026lt;post, read+\u0026gt; That is, the blogger posts an article and then multiple people read it. This seems almost too silly to specify. But it’s more subtle than it seems. The action of reading an article involves specifying\nhow is selection specified???\nsomeone performs a read, they\nread actions are read\nactually relies on a\nthe converse doesn’t hold: a trivial\nit’s a mistake to think that a trivial OP is evidence of no ad\ndon’t be the converse is not true\niPod\npay, park, leave all these OPs have interesting subtleties if a one space zone, don’t need car license (but they may still ask so they can prevent you sharing?)\nIf you don’t believe that use is part of the system, see what happens if you make lots of OpenTable reservations and don’t turn up.\nOPs are often computerizations of social protocols. parking, reservations, calls, meetings in places.\nCan describe socio-technical systems too. The OP for social security:\n\u0026lt;assign, contribute+, retire, withdraw+\u0026gt; as a simple sequence\nComparisons drinking band to simple age verification Can be very simple, novelty is in comparison to more complex one. Social media posting.\nThis was the argument that Apple’s detractors used This argument has become weaker over the years as Apple’s detractors (who claimed that design didn’t matter and that Apple’s fancy products would never compete against the cost effectiv\n","permalink":"https://essenceofsoftware.com/drafts/concepts-redux/the-operational-principle/","summary":"\u003ch1 id=\"the-operational-principle\"\u003eThe operational principle\u003c/h1\u003e\n\u003cp\u003eUsing a term coined by Michael Polanyi (and brought to software engineering by Michael Jackson), I call a scenario that explains how a product “just works” its \u003cem\u003eoperational principle\u003c/em\u003e (OP).\u003c/p\u003e\n\u003cp\u003eThe operational principle is just a sample scenario, but it includes enough richness to convey typical usages. The example I gave above for Zoom could be written in a more textual way like this\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e\u0026lt;create, start, join+, end\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003ewhere \u003cem\u003ejoin+\u003c/em\u003e means one or more \u003cem\u003ejoin\u003c/em\u003e actions. Another scenario would be this\u003c/p\u003e","title":""},{"content":"Consent concepts\nowner,\nGPT suggests data custodian authorized party service provider\nlet’s go with provider or party.\nconsent applies to a resource instance (SSN) or type (bank account transaction). but can treat instance as a pathological type that has just one instance.\ndo we need actions allowed if party is specified? party might share. or purposes/uses\nmarketing, sharing, etc\ncan owner provide blanket consent to any party? or any party in some category? or a party in some group (eg, as defined by Consumer Reports companies)? can one party inherit consent from another?\nactions for concept request consent grant consent; (and party confirms/attests?) revoke consent exploit consent\nshould exploiting consent be part of this concept? what if exploit does not meet consent constraint?\nshould concept keep record of exploited consent actions? this seems to extend it to a logging concept. maybe just have an action for approving of disapproving of an exploitation.\nduration, expiration rights assertion: not attestation requesting data be fixed, deleted, etc consent important element of control tracing essential for accountability\n","permalink":"https://essenceofsoftware.com/drafts/consent-concepts/","summary":"\u003cp\u003eConsent concepts\u003c/p\u003e\n\u003cp\u003eowner,\u003c/p\u003e\n\u003cp\u003eGPT suggests\ndata custodian\nauthorized party\nservice provider\u003c/p\u003e\n\u003cp\u003elet’s go with provider or party.\u003c/p\u003e\n\u003cp\u003econsent applies to a resource instance (SSN) or type (bank account transaction). but can treat instance as a pathological type that has just one instance.\u003c/p\u003e\n\u003cp\u003edo we need actions allowed if party is specified? party might share. or purposes/uses\u003c/p\u003e\n\u003cp\u003emarketing, sharing, etc\u003c/p\u003e\n\u003cp\u003ecan owner provide blanket consent to any party? or any party in some category? or a party in some group (eg, as defined by Consumer Reports companies)? can one party inherit consent from another?\u003c/p\u003e","title":""},{"content":"deemph: data model and action definitions for coding. extending: modularity, but probably needs data. essence: OPs. relevance: LLMs\n","permalink":"https://essenceofsoftware.com/drafts/deemph--data-model-and-action-definitions-for-coding/","summary":"\u003cp\u003edeemph: data model and action definitions for coding.\nextending: modularity, but probably needs data.\nessence: OPs.\nrelevance: LLMs\u003c/p\u003e","title":""},{"content":"Design is necessary but not sufficient One final response to another possible skepticism. Some might say that none of this design stuff matters if the software isn’t reliable, scalable, performant, etc.\nThe answer to this objection is easy: yes. Design is necessary but not sufficient. Just as great architecture won’t make a building successful if the civil engineers fail to ensure that it can withstand the wind, so a well-designed software product can’t succeed without good software engineering.\nThat doesn’t mean design doesn’t matter, or that design doesn’t have a big impact on these other factors; it just means that design isn’t everything. But to those of us who are designers, it might be still be the most enjoyable and exciting thing\u0026hellip;\n","permalink":"https://essenceofsoftware.com/drafts/design-is-necessary-but-not-sufficient/","summary":"\u003ch1 id=\"design-is-necessary-but-not-sufficient\"\u003eDesign is necessary but not sufficient\u003c/h1\u003e\n\u003cp\u003eOne final response to another possible skepticism. Some might say that none of this design stuff matters if the software isn’t reliable, scalable, performant, etc.\u003c/p\u003e\n\u003cp\u003eThe answer to this objection is easy: yes. Design is necessary but not sufficient. Just as great architecture won’t make a building successful if the civil engineers fail to ensure that it can withstand the wind, so a well-designed software product can’t succeed without good software engineering.\u003c/p\u003e","title":""},{"content":"function cascadeDel (p: PostConcept, c: CommentConcept) { return i -\u0026gt; { p.del (*, i); c.del (i) } }\nfunction authDel (p: PostConcept, s: SessionConcept) { return i -\u0026gt; { u = s.getUser ();\np.del (u, i); } }\nfunction authCascDel (\u0026hellip;)\nsy = new Sync ();\nsy = sy.add (\nnew Link (“Post.del”,”Comment.del”) ) sy = sy.add (\\ // when Sess.user, Post.del (u) )\n","permalink":"https://essenceofsoftware.com/drafts/function-cascadedel-p--postconcept-c--commentconcept-i/","summary":"\u003cp\u003efunction cascadeDel (p: PostConcept, c: CommentConcept) {\nreturn i -\u0026gt; {\np.del (*, i);\nc.del (i)\n}\n}\u003c/p\u003e\n\u003cp\u003efunction authDel (p: PostConcept, s: SessionConcept) {\nreturn i -\u0026gt; {\nu = s.getUser ();\u003cbr\u003e\np.del (u, i);\n}\n}\u003c/p\u003e\n\u003cp\u003efunction authCascDel (\u0026hellip;)\u003c/p\u003e\n\u003cp\u003esy = new Sync ();\u003cbr\u003e\nsy = sy.add (\u003cbr\u003e\nnew Link (“Post.del”,”Comment.del”)\n)\nsy = sy.add (\\\n// when Sess.user, Post.del (u)\n)\u003c/p\u003e","title":""},{"content":"A Concept Model of Noosphere Warning Heavy-duty concept design ahead! This note exploring the concept design of Noosphere is not for the faint-hearted. For those who aren\u0026rsquo;t very familiar with the concepts of decentralized storage (which includes me!) or who are learning concept design for the first time, this is likely not the best introduction to concepts.\nWhy conceptualize Noosphere? This note applies my theory of concept design to Gordon Brander\u0026rsquo;s Noosphere. The reasons for expressing Noosphere in terms of concepts are:\nSeparation of concerns: to separate design aspects as cleanly as possible so they can be more easily understood, and reuse of existing concepts identified; Design innovation: to capture the novelty of the design, which I expect to be not in the invention of new concepts but in the combination of old concepts (demonstrating what Margaret Boden calls \u0026ldquo;combinational creativity\u0026rdquo;); Mental model: to provide a compelling and tractable mental model for users (in this case probably programmers building atop the platform); Product family: to show, from the independence of concepts, that the design represents a family of possible designs generated by the possible concept subsets; Design decisions: to highlight key design decisions and suggest other points in the design space that might be considered; Design rationale: to convey the rationale for the design by articuating the purpose of individual concepts and exploring the properties that concepts assure. In these respects, Noosphere is no different from any other systems I have considered in the past. But unlike most of them, Noosphere is a platform and not an application or a service, and this presents useful challenges to concept design:\nInterface vs. implementation: Concepts capture observable behavior and eschew any implementation details. In an app, this distinction is straightforward, but for a platform it\u0026rsquo;s murkier. For example, Noosphere\u0026rsquo;s use of public key cryptography for signing objects requires that they be constructed in a particular order (for example, with version pointers inside the signed envelope so that their validity can be assured too); arguably this is an implementation detail.\nGeneral vs. specific usage: An application generally dictates a particular usage, but a platform is intended to support a variety of different usages. Some line needs to be drawn between the most particular and the most general. Notably, I\u0026rsquo;ve taken the view that Noosphere should be independent of Subconscious (Brander\u0026rsquo;s notebook application that it supports) but that peer-to-peer sharing of objects is essential to the platform.\nBecause of these challenges, Noosphere offers a useful case study of applying concept design in a different domain, and a chance to explore some of the complications that arise.\nBasic Concepts Noosphere assumes a setting of decentralized file sharing, so we start there; without this, it\u0026rsquo;s hard to motivate Noosphere\u0026rsquo;s concepts. Because Noosphere itself doesn\u0026rsquo;t prescribe any particular file sharing protocol, we\u0026rsquo;ll make the concept pretty minimal:\nconcept Stash \u0026lt;Content, Principal\u0026gt;: Memo, Stash purpose decentralized, replicated storage of immutable content principle // chain of copies retains owner and content new (s1); new (s1, c, o, m1); copy (m1, s2, m2); copy (m2, s3, m3) {m3.content = c and m3.owner = o} state memos: Stash one -\u0026gt; set Memo owner: Memo -\u0026gt; one Principal content: Memo -\u0026gt; one Content actions new (out s: Stash) // create a new stash new (s: Stash, c: Content, o: Principal, out m: Memo) // create a new memo in a stash from given contents and owner copy (m: Memo, to: Stash, out m': Memo) // create copy m' of memo m in stash to with same owner and content as m delete (m: Memo) delete (s: Stash) Notation notes The header gives a name to the concept (Stash) and lists its parameter types (Content and Principal) and the types of objects that the concept generates (Memo and Stash). The operational principle is an archetypal scenario illustrating how the concept fulfills its purpose. In this case, there\u0026rsquo;s a sequence of actions followed by a (Hoare-style) assertion. It says: if you create a stash s1, and then a new memo m1 in that stash, and you copy the memo to another stash s2 and then to a further stash s3, then afterwards the content of the final copied memo m3 will be the original content inserted into m1, and its owner will be the original owner. In other words, just what you\u0026rsquo;d expect: memos can be copied around and retain content and ownership. The state is a bunch of relations marked with multiplicities in Alloy style. So memos, for example, maps elements of the set Stash to the set Memo, and the multiplicities say that each stash maps to a set (any number) of memos, and each memo is mapped to by exactly one stash (that is, memos are not shared across stashes). The actions define possible updates on the state. Observer actions aren\u0026rsquo;t generally needed because the state is assumed to be visible. I like to overload names to make things succinct, so Stash is both the name of the concept and the name of a type (set of items) generated by it. Note that a concept is not like an OO class, but is (computationally) just a state machine, although it\u0026rsquo;s common for a concept to bear the name of a significant type of item that it manages. Action names are overloaded too: the actions for creating new stashes and memos have the same name, as do the deletion actions. Design notes\nGenericity. Concepts should be as generic as possible. So this concept does not constrain the type of memo content (it could be text files, videos, etc) nor who the principals are that act as owners. In Noosphere, the content will turn out to be text and the principals will be represented by public keys. The unconstrained types are given as parameters to the concept, using the standard kind of parameteric polymorphism familiar from languages like ML. Stashes and memos. I\u0026rsquo;ve coined the term stash for the memo container. In Noosphere, it\u0026rsquo;s called a sphere, but I\u0026rsquo;m avoiding that term for now since a sphere includes more than a collection of memos, and I\u0026rsquo;ll want to describe it as a composition of concepts. The term memo is Noosphere\u0026rsquo;s. Stash vs. principal. Note that stashes aren\u0026rsquo;t owned, and there is no constraint that all the memos in a stash have the same owner. This flexibility allows a stash to be used both as an owner\u0026rsquo;s repository for the memos they create, and as storage for memos received from others. A stash is not necessarily a peer in the P2P file-sharing sense either, since a user might want to store both their own stash of owned memos and a stash of received memos on the same machine. No discovery protocol. The concept is intentionally minimal and does not describe how memos are found; that will be addressed in part by other concepts later. Ownership preservation. The copy action preserves ownership. A user can copy the text of a memo and make a new memo that they own, but can\u0026rsquo;t claim ownership of a memo copied from another stash. In the simplest intended application of Noosphere, memos are text notes:\nconcept Note: Buffer, Note purpose provide editable text notes principle // editing and saving creates a series of immutable notes new (b); edit (b, t1); save (b, n1); edit (b, t2); save (b, n2) {n1 != n2 and n1.content = t1 and n2.content = t2} state content: Note -\u0026gt; one Text current: Buffer -\u0026gt; one Text actions new (out b: Buffer) // create new empty buffer open (n: Note, out b: Buffer) // open a new buffer with the text content of note n edit (b: Buffer, t: Text) // replace current text of buffer b with t save (b: Buffer, out n: Note) // create new note n whose content is current text in b Design notes:\nImmutability. This describes a minimal concept in which the user edits a mutable buffer and every save generates a fresh, immutable note. Editing abstracted away. Editors are complicated things whose details aren\u0026rsquo;t relevant here, so we model editing as an action that just replaces the text of the buffer with a new value. Even though notes are immutable, the history of a note will be represented in Noosphere by a chain of versions:\nconcept Version \u0026lt;Item\u0026gt; purpose let users access older versions of items principle // if you derive a new version then rollback, // you're where you started derive (i1, i2); rollback (i2, i) {i = i1} state private pred: Item -\u0026gt; lone Item actions derive (p, i: Item) // when i has no pred, make p the pred of i rollback (i: Item, out p: Item) // when i has pred p, return it Design notes:\nImmutability. The derive action is the only way to update the state, so existing predecessor bindings cannot be altered. The intent (suggested by the action being called derive rather than register, eg) is that the derive action is executed when the new item is created, so its predecessor is established from the start. Multiple successors. Although an item has at most one predecessor, it can have multiple successors. Navigating back only. Perhaps most significantly, the concept does not offer any way to traverse from an item to its successor(s); although this is perhaps even more useful than navigating backwards, the way that versioning is implemented (by embedding a predecessor pointer) does not permit it. There will be another means of finding more recent versions in Noosphere through another concepts. (To enforce this, I\u0026rsquo;ve marked the state component as private and provided an observer action rollback that reads it only in one direction.) In a decentralized storage system, in which the object you\u0026rsquo;re looking for can migrate from peer to peer, location-based addressing won\u0026rsquo;t work. Instead a scheme is used in which addresses of objects are based on their contents:\nconcept ContentAddress \u0026lt;Object, Content\u0026gt;: Address purpose support addressing of objects in decentralized storage principle // if you register two objects with same content // they will have the same address and a lookup on that // address over a set including both will return one of them register (o1, c, a1); register( o2, c, a2); lookup ({o1, o2}, a2, o) {a1 = a2 and (o = o1 or o = o2)} state addr: Object -\u0026gt; one Address content: Object -\u0026gt; one Content hash: Content lone -\u0026gt; one Address actions register (o: Object, c: Content, out a: Address) // register new name a for object o with content c lookup (s: set Object, a: Address, o: Object) // return any object o in set s with address a Design notes:\nAddresses not names. A content-based address is sometimes called a \u0026ldquo;name\u0026rdquo;. But the term \u0026ldquo;address\u0026rdquo; is more accurate and will be less confusing in Noosphere, since we\u0026rsquo;ll be introducing names through a different concept. Addresses unique over content. The hash state component represents the fixed relationship between contents and their hashes. The multiplicities of this relation say that each content maps to one address, and each address is mapped to by at most one content. Hash collisions are in theory possible, but are assumed not to happen. Addresses not unique over objects. In contrast, the addresses of objects are not unique, since two distinct objects may have the same contents and therefore the same address. In fact, this is essential in a decentralized storage system, because there will be many copies of a single object, and each must have the same address. Non-deterministic lookup. The lookup action takes a set of objects within which to conduct a search, and arbitrarily returns an object within that set with the requested address. If no such object exists, the action fails. Content addresses can act as primitive names for memos, and have the advantage of simplicity (being derivable directly from content) and persistence (always identifying the same thing). But as names they suffer from two disadvantages. First, persistence is not always what\u0026rsquo;s wanted: the author of a memo might want to update the content and have existing names refer to the new content. Second, addresses are not human readable.\nNoosphere uses petnames to overcome these problems:\nconcept PetName \u0026lt;Object\u0026gt;: Dir purpose decentralized human-readable naming principle // immutable directories built in a series that preserves old bindings new (d1); bind (d1, n1, o1, d2); bind (d2, n2, o2, d3); lookup (d3, n1, o) {o = o1} state bindings: Dir -\u0026gt; String -\u0026gt; lone Object actions new (out d: Dir) // create an empty directory bind (d: Dir, n: String, o: Object, out d': Dir) // create new directory which adds binding of name n to object o lookup (d: Dir, n: Name, out o: Object) // return object bound to name in directory Design notes:\nPolymorphic target. The concept is polymorphic in the target type of a name, so anything can be named (even a directory). Immutable directories. Directories are immutable, so each binding action creates a new directory. This might seem burdensome since binding a collection of n new objects will produce n new directories, but this will be mitigated by the ability of users to publish only occasionally (see User concept below). String names. For simplicity, I\u0026rsquo;ve made names strings, but a richer design would make the concept polymorphic in the name type, allowing a separate path name concept (for example) which could then support hierarchical naming. Our final concept represents users:\nconcept User \u0026lt;Asset\u0026gt;: User purpose centralize some assets principle // after a series of asset replacements, // the latest asset is the one given in the // last replacement new (a1, u); replace (u, a2) {u.asset = a2} state asset: User -\u0026gt; one Asset actions new (a: Asset, out u: User) // create a new user with a given asset replace (u: User, a: Asset) // set u's asset to be a Design notes:\nMutable state. In a conventional app, the User concept is just to provide unique identifiers for authentication (typically an additional concept), and might include basic profile fields such as display name and email address. In a decentralized system like Noosphere, there\u0026rsquo;s a more significant role because owners of content can act as arbiters identifying the latest (and thus most authentic) versions. Introducing users as mutable stores is thus a small concession to centralization. Just one asset. For simplicity, each user has only a single associated asset. This will turn out (when we instantiate the concept) to be the user\u0026rsquo;s latest petname directory. More realistically, each user might have a collection of assets tied to keys (in the style of a domain\u0026rsquo;s DNS bindings). Concept identification tactic. One simple way to identify missing concepts is to anticipate the concept composition, and note type parameters for which concrete, instatiating types have yet to be provided. The Stash concept, for example, introduce a type parameter for the content of memos; this will be instantiated with the Note type from the Note concept. Since Stash has a Principal type parameter, there will need to be some concept binding it to a concrete type of uses or cryptographic keys. This User concept will play that role. Concept Composition With the concepts in hand, we can now assemble them into a coherent system:\napp Noosphere includes concept Note: Buffer, Note concept Version \u0026lt;Address\u0026gt; concept Stash \u0026lt;Note + Dir, User\u0026gt;: Memo, Stash concept ContentAddress \u0026lt;Memo, Note\u0026gt;: Address concept PetName \u0026lt;Address\u0026gt;: Dir concept User \u0026lt;Dir\u0026gt;: User Notation notes:\nType instantiation. The included concepts have their type parameters instantiated with concrete types that correspond to the types generated by other concepts. For example, the Note concept exports a Note type; this then appears as the second argument in the instantiation of ContentAddress, saying that the notes of the Note concept will be the contents of the ContentAddress concept. Design notes:\nVersioning addresses. The items that are versioned by the Version concept are content addresses, these addresses in turn resolve to memos which contain notes. An alternative would have been to version notes instead, but this seemed too abstract. Indeed, regarding notes as directly obtainable would eliminate the role of content addresses. In the implementation, the predecessor fields of memos hold content addresses that are then used to find memos whose contained notes have the corresponding predecessor content. By mapping addresses to addresses, the conceptual model of versioning does deviate slightly from the implementation, whose versioning representation is not homogeneous, and instead maps a memo to a content address. This non-homogeneity is tolerable only because the versioning concept is not separated out. Directories as assets. The asset held by each user is a petname directory holding the user\u0026rsquo;s most recent bindings of petnames to (content addresses of) memos. Directories in stashes. The Stash concept is instantiated with a union type for the content of memos, allowing memos to hold either petname directories or notes. All stash objects are owned, so directories will be owned too. This is important because our concept design allows any user to create new petnames for memos, including memos owned by other users, and in practice one might want to rely only on petnames created by owners. Addressing notes, not text. The contents of the ContentAddress concept are notes and not the text inside them, since two notes that happen to have the same text must be regarded as distinct items (with distinct predecessors) and must therefore have distinct addresses. Names for addresses. The PetName concept is instantiated with the target of the names being the addresses of the ContentAddress concept. Both content addresses and petnames can be used to locate memos. When used in links, the content addresses will be suitable for permanent and unchanging references, and the petnames for allowing references to the most recent versions of memos. Action synchronizations The actions of the composed system are described as synchronizations of the actions of the constituent concepts. We\u0026rsquo;ll consider just a few of the more interesting and central synchronizations.\nThe fundamental action involves creating memos:\nsync save_edit_as_memo (owner: User, buf: Buffer, stash: Stash, pred: Note, name: String, out note: Note, memo: Memo, addr: Address, dir: Dir) Note.save (buf, note) Stash.new (stash, note, owner, memo) ContentAddress.register (memo, note, addr) Version.derive (ContentAddress.addr[pred], addr) PetName.bind (User.asset[owner], name, addr, dir) User.replace (owner, dir) The context is assumed to be a user (who will become the owner of the new memo) who saves text in a buffer they are editing into a (presumably local) stash. A predecessor and a petname for the new note are also required; these might be provided by default, by using the last note that was saved as the predecessor, and keeping its petname as the name of the new note.\nThe synchronization causes actions to be performed in each of the other concepts:\nNote: The saving of the buffer results in creation of a new note; Stash: The new note is inserted into the stash, creating a memo; ContentAddress: A content address is computed based on the note and associated with the memo; Version: The (content address of the) predecessor note is recorded to be the predecessor of the (content address of the) new note; PetName: The petname is bound to the (content address of the) new note in the current petname directory of the user, creating a new directory; User: the new petname directory is stored as the user\u0026rsquo;s current directory. How does the newly created memo reach other users? Rather than imagining the user interface design for a completed peer-to-peer system, we can just specify a few key actions.\nA user can get a memo\u0026rsquo;s content address by looking up a petname in a directory that they have in a local stash:\nsync resolve_name_to_address_local (owner: User, dir: Dir, name: String, stash: Stash, memo: Memo, out addr: Address) memo in Stash.memos[stash] dir = Stash.content[memo] owner = Stash.owner[memo] PetName.lookup (dir, name, addr) Notation notes:\nPrecondition. The sync includes a precondition saying that the directory in which the name lookup occurs is the content of a memo in the given stash, and its owner is the expected owner. You can also go to a user\u0026rsquo;s current petname directory for the most up-to-date binding of a petname:\nsync resolve_name_to_address_auth (owner: User, name: String, out addr: Address) PetName.lookup (User.asset[owner], name, addr) Notation notes:\nState queries. The directory that is presented as the first argument to the lookup action is obtained by a query of the User concept, looking up the asset associated with the user owner. A similar state query was used in the first sync above to find the content address of the predecessor note. With a content address in hand, a memo can now be located by resolving the address within the set of memos of a given stash:\nsync get_memo_by_address (s: Stash, a: Address, out m: Memo) ContentAddress.resolve (Stash.memos[s], a, m) Notation notes:\nFailure. In concept design, an action may fail (or more exactly fail to occur) if its preconditions are not satisfied. So implicitly this action will only occur when a memo actually exists in the given stash with the given name. Of course in an implementation appropriate errors should be reported. For memos to be available in a local stash, they must be copied from peer stashes:\nsync copy_memo_between_peers (from, to: Stash, memo_from: Memo, out memo_to: Memo) memo_from in Stash.memos[from] Stash.copy (memo_from, to, memo_to) You can find the address of the predecessor of a note:\nsync get_predecessor_addr (m: Memo, a: Address) Version.rollback (ContentAddress.addr[Stash.content[m]], a) Reflections and next steps Errors. The concept model no doubt contains errors arising from my lack of knowledge of Noosphere and decentralized storage. Modeling liberties. It also takes some modeling liberties, most notably in treating content addresses as being derived directly from the identities of notes (rather than from the bits that comprise them). Immutability modeling. The immutability of various objects is modeled here by simply having immutable relations in concept states; in the implementation, the immutability is assured as a side-effect of the use of content addresses (since modifying the content of a memo would change its address). Since content, for the purpose of content addressing, includes predecessor pointers, the content address also ensures the immutability of predecessors. I considered representing this more explicitly in the concept model but it created modularity problems (and anyway seems to be an implementation detail). Role of versioning. The Version concept seems weak, as evidenced by its not very compelling operational principle. It\u0026rsquo;s not clear to me how versioning is used. One scenario might be that, when a memo is copied between stashes, the receiving stash checks to see if the new memo is a newer version of an existing memo in the stash, and if so replaces it. How this would work with naming, however, is not clear. Links. In Subconscious, the note storage app built on Noosphere, the text of a note can include links to other notes. A link might refer to a content addresses or a petname (or perhaps both, allowing an archival referent and an updatable one). Public keys. In Noosphere, public keys are used as principals and also to ensure the integrity and authenticity of memos. I don\u0026rsquo;t think this would be hard to model. I left it out for now, because it\u0026rsquo;s mostly an implementation detail, with the concept aspects captured by the User concept and the owner relation in Stash. Hierarchical storage. I noted above that by introducing a Pathname concept, and using path names as petnames instead of strings, the name space could simulate a hierarchical folder structure (as in Git). Spheres. A sphere in Noosphere combines the roles of User and Stash. In the concept model these are separated to allow the possibility that a user has multiple stashes. ","permalink":"https://essenceofsoftware.com/drafts/noosphere-v2/","summary":"\u003ch1 id=\"a-concept-model-of-noosphere\"\u003eA Concept Model of Noosphere\u003c/h1\u003e\n\u003ch2 id=\"warning\"\u003eWarning\u003c/h2\u003e\n\u003cp\u003eHeavy-duty concept design ahead! This note exploring the concept design of Noosphere is not for the faint-hearted. For those who aren\u0026rsquo;t very familiar with the concepts of decentralized storage (which includes me!) or who are learning concept design for the first time, this is likely not the best introduction to concepts.\u003c/p\u003e\n\u003ch2 id=\"why-conceptualize-noosphere\"\u003eWhy conceptualize Noosphere?\u003c/h2\u003e\n\u003cp\u003eThis note applies my \u003ca href=\"https://essenceofsoftware.com\"\u003etheory of concept design\u003c/a\u003e to \u003ca href=\"https://substack.com/@gordonbrander\"\u003eGordon Brander\u003c/a\u003e\u0026rsquo;s \u003ca href=\"https://subconscious.substack.com/p/noosphere-a-protocol-for-thought\"\u003eNoosphere\u003c/a\u003e. The reasons for expressing Noosphere in terms of concepts are:\u003c/p\u003e","title":""},{"content":"Some notes about syncs Daniel Jackson\nSept 16, 2024\nSeparating Semantic Actions from HTTP Request All web app frameworks separate details of HTTP requests (HTTP verb, URL, serialization formats) from underlying data model calls. In our previous architecture (the one we used in Kodless and in the 6.1040 starter code), we used the HTTP “route” (that is, the endpoint API call corresponding to the HTTP request) also as the function that performs concept synchronization.\nThis was a good tradeoff for the class because it closely mimics traditional app structure (which doesn’t have synchronizations), but it mixes the two concerns. Instead, we should have a synchronization layer that is separate from the HTTP layer, with each HTTP endpoint making only a single call to the concept sync layer.\nThe Need for Additional Actions In some cases, the HTTP endpoint might be able to call a concept action directly. But often the concept actions will require arguments that are provided by sync with other concepts.\nFor example, consider an endpoint for creating a new comment. The Comment concept might have an action with the signature\nComment.new (author: User, post: Post, content: String, out c: Comment) (The types User and Post would actually be type parameters, but we can ignore that complication for now.)\nOne could imagine an app in which comment authors aren’t authenticated. In that case, the User type might be just a string, and the HTTP request might pass that on directly. But in a typical app, the author will be the user in the session, obtained by sync with a Session concept. To achieve that, we define a new interface action\nnewComment (session: Session, post: Post, content: String) in which the author is no longer specified (and the resulting comment object is not used).\nThis could then be defined by a synchronization:\nsync when newComment (s, p, c) u = Session.getUser (s) c = Comment.new (u, p, t) Modularity of Synchronizations In our previous thinking, it was fine to define synchronizations in a completely application-specific way. That is, we consider the application at hand, and ask which actions amongst all its concepts should happen together.\nBut in the framework we’re developing, there will be a catalog of standard synchronizations for the user to choose from. These will often connect concepts pairwise. Syncs that involved more than two concepts would not work in some applications because they might require the presence of concepts that are not included.\nFor example, one application might have notifications, so would want to sync Comment with Notification; another might have karma points, and want to sync Comment with Karma. These should be separate syncs so they can be selected independently of one another.\nHere’s an example of a sync for notifications\nsync when Comment.new (u, p, t) Notification.new (u, \u0026quot;your new comment: \u0026quot; + t) Notification.new (p.author, \u0026quot;new comment on your post: \u0026quot; + t) Note that the sync requires access to the state of the concepts (obtaining the author of the target post as the subject of the notification).\nThis sync illustrates also the chaining of synchronizations: the transaction would begin with the user executing newComment, which then fires Comment.new, which itself then fires Notification.new.\n","permalink":"https://essenceofsoftware.com/drafts/some-principles-regarding-concept-synchronization/","summary":"\u003ch1 id=\"some-notes-about-syncs\"\u003eSome notes about syncs\u003c/h1\u003e\n\u003cp\u003eDaniel Jackson\u003cbr\u003e\nSept 16, 2024\u003c/p\u003e\n\u003ch2 id=\"separating-semantic-actions-from-http-request\"\u003eSeparating Semantic Actions from HTTP Request\u003c/h2\u003e\n\u003cp\u003eAll web app frameworks separate details of HTTP requests (HTTP verb, URL, serialization formats) from underlying data model calls. In our previous architecture (the one we used in Kodless and in the 6.1040 starter code), we used the HTTP “route” (that is, the endpoint API call corresponding to the HTTP request) also as the function that performs concept synchronization.\u003c/p\u003e","title":""}]