[{"content":"There\u0026rsquo;s a pattern I often see in Python code that feels woefully un-Pythonic.\n1 2 3 4 5 6 7 8 9 10 class ConcreteBuilder(AbstractBuilder): def __init__(self, context): self._context = context def build(self): return do_something_with_context(context) def uses_builder(builder: AbstractBuilder): builder.build() It\u0026rsquo;s effectively defining a function the long way. The cause for this, of course, is \u0026ldquo;abstraction\u0026rdquo;. This might make sense in Java or C#, where you cannot define free-standing functions, but in Python it makes no sense. If you want to pass a function to another function - you just do it. If you want to type-check it, use protocols.\n1 2 3 4 5 6 class BuildFunction(Protocol): def __call__(self): pass def uses_builder(build: BuildFunction) build() But once you do that, you run into a new issue. While it is very easy to find all implementations of an ABC (grep will do), it is a lot harder to find all the functions implementing a protocol.\nSpecifically, VSCode (with PyLance) allows you to look for implementations of a protocol, but it will only find classes implementing it, not functions. While PyCharm finds no implementations at all.\nI will say it clearly - this is a tooling problem. And as such, it should be solved by tooling. But\u0026hellip; Tooling is hard (though I\u0026rsquo;m hoping ty or Pyrefly will make writing them simpler) and writing misguided code is more fun.\nSingle Abstract Method (SAM) Interfaces in Python So, with no further ado, allow me to introduce - SAM (Single Abstract Method) interfaces1 for Python!\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 class AbstractBuilder(ABC): @abstractmethod def build(self): ... @implement_sam(AbstractBuilder) def my_concrete_builder(): return \u0026#34;Properly built string!\u0026#34; assert isinstance(my_concrete_builder, AbstractBuilder) print(my_concrete_builder.build()) # \u0026gt; Properly built string! With this we get the best of both worlds! We define simple functions to implement our interfaces, and we can find all implementations using grep! I guess there\u0026rsquo;s nothing quite like bringing Java-inspired patterns into Python\u0026hellip;\n\u0026ldquo;How does this work?\u0026rdquo; you might ask, and for you, I provide the implementation.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 from abc import ABC from functools import wraps def implement_sam(interface: type[ABC]): if not interface.__abstractmethods__: raise TypeError(\u0026#34;No abstract method found.\u0026#34;) sam, *others = iter(interface.__abstractmethods__) if others: raise TypeError(f\u0026#34;Expected single abstract method, found {1 + len(others)}.\u0026#34;) def decorator(f): @interface.register class SAMAdapter: def __repr__(self): return f\u0026#34;\u0026lt;SAMAdapter(interface={interface}, impl={f})\u0026gt;\u0026#34; @wraps(f) def method_wrapper(self, *args, **kwargs): return f(*args, **kwargs) setattr(SAMAdapter, sam, method_wrapper) return SAMAdapter() return decorator We first make sure that the provided interface is indeed a SAM, then proceed to implementing it. We create an adapter class so that we can register it as a subclass of our ABC, and assign our function to be called as the interface method. We instantiate our new class, and return. Simple and effective.\n\u0026ldquo;And what about type annotations?\u0026rdquo; you might ask. Well, tough luck. We have multiple issues on that front. First and foremost - mypy does not allow passing ABCs to functions taking type[T]. This means that we\u0026rsquo;ll have to add a type: ignore[type-abstract] comment at every callsite. Second, as far as I know there\u0026rsquo;s no way to force type-checking of the signature of f against the interface, as there is no way to explicitly access the type of the specific method. So in lieu of static type-checking, we\u0026rsquo;ll have to change the signature to interface: Any and rely on testing.\n\u0026ldquo;I don\u0026rsquo;t think this is a good idea\u0026rdquo; you may conclude, after readin this post. And you\u0026rsquo;ll be right. But good ideas are rare, and experimentation is fun.\nInspired by Java\u0026rsquo;s lambdas and Functional Interfaces\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/2026-04-python-sam/","summary":"There\u0026rsquo;s a pattern I often see in Python code that feels woefully un-Pythonic.\n1 2 3 4 5 6 7 8 9 10 class ConcreteBuilder(AbstractBuilder): def __init__(self, context): self._context = context def build(self): return do_something_with_context(context) def uses_builder(builder: AbstractBuilder): builder.build() It\u0026rsquo;s effectively defining a function the long way. The cause for this, of course, is \u0026ldquo;abstraction\u0026rdquo;. This might make sense in Java or C#, where you cannot define free-standing functions, but in Python it makes no sense.","title":"Single Abstract Method (SAM) Interfaces in Python"},{"content":"A few weeks ago I saw some people chatting about the benefits of 20-20-20 timers. The idea is that to reduce eye strain (as we tend to stare at screens all day\u0026hellip;) you set a timer so that every 20 minutes you spend 20 seconds looking at something that\u0026rsquo;s 20 feet away.\nSo I looked for an existing timer, and there are quite a lot of them, but they all have their issues. The main one, that is shared between all timers I could find, is that they use notifications to tell you the time\u0026rsquo;s up. And the problem with notifications is that I always ignore them. Well, not always, but if I\u0026rsquo;m focused enough to need a timer to look away from the screen, I will definitely ignore a notification. So not finding an existing solution, I went and created1 my own, with my own specifications.\nThe main thing, of course, was figuring out the notification problem. I wanted something that I can\u0026rsquo;t ignore easily, so that I actually stop and look away when I need to; while not too annoying, so that I don\u0026rsquo;t turn it off and forget about it. What I ended going for is a gradient overlay for the screen, making it dark on the edges and almost-normal in the middle. To ensure I can still click whatever I might need to, it\u0026rsquo;s transparent around the mouse pointer as well.\nWhite screen with a black gradient overlay. The edges are dark while the center is bright. And you can see a brighter area around the mouse pointer as well. When the 20 minute timer expires, the overlay darkens gradually, letting me know I need to pause. Sure, I can keep working, but it is a little annoying and a constant reminder to stop. I\u0026rsquo;ve been using it for over a week now, and it seems to work well for me.\nTo turn hide the overlay, I need to lock the screen for 20 seconds or longer. I get a small \u0026ldquo;beep\u0026rdquo; sound letting me know when I\u0026rsquo;ve done it.\nAnother issue is that there are some cases where I don\u0026rsquo;t actually want to look away. I want to make sure none of these cases makes me turn the app off, as I will never remember to turn it back on. So the first two options I added (and stuck with) are:\n\u0026ldquo;Pause until unlock\u0026rdquo; - keeps the overlay from showing until I unlock the screen again. This is sometimes a very long time, especially at home, but still ensures the timer will start again at some point. \u0026ldquo;Pause for\u0026hellip;\u0026rdquo; - guarantees I have until %d minutes before it shows the overlay, effectively extending my timer. These have worked well for me (especially as I can use them even after the overlay is shown), and I managed to completely avoid exiting the app.\nI\u0026rsquo;m also working on some automatic \u0026ldquo;I\u0026rsquo;m busy!\u0026rdquo; detection to avoid annoying overlays. These are promising, but need more work:\n\u0026ldquo;In a meeting\u0026rdquo; - detect camera/microphone usage as an indicator for video-meetings. I generally don\u0026rsquo;t want to lock my screen during a meeting, but would like to get the overlay shown as soon as it is over. \u0026ldquo;Watching a movie\u0026rdquo; - no-one wants to have the screen go darker right at the climax of a movie. To avoid this, I\u0026rsquo;m experimenting with hueristics for \u0026ldquo;a movie is playing\u0026rdquo;. This proves a little tricky. While using the app, I ran into another small issue relating to locking the screen - if I have media playing, I want to pause it (at least in most cases). So I added it as a feature - whenever I lock the screen now, the currently playing media will pause. It doesn\u0026rsquo;t work for VLC (I am using the Windows SMTC API and VLC does not), but it works for YouTube, so I\u0026rsquo;m happy.\nThe app is available at https://github.com/tmr232/easy-eyes and you\u0026rsquo;re free to use it, but keep in mind that it is a personal project aimed to make me happy.\nYes, I vibe-coded it. I feel that for small projects like this, where most of the work would be looking up Windows APIs and copying examples from SO until something works, vibe-coding is a reasonable solution. It also makes it time-effective enough to do a project I couldn\u0026rsquo;t justify otherwise. The icon is hand-crafted based on public-domain art.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/2026-04-easy-eyes/","summary":"A few weeks ago I saw some people chatting about the benefits of 20-20-20 timers. The idea is that to reduce eye strain (as we tend to stare at screens all day\u0026hellip;) you set a timer so that every 20 minutes you spend 20 seconds looking at something that\u0026rsquo;s 20 feet away.\nSo I looked for an existing timer, and there are quite a lot of them, but they all have their issues.","title":"20-20-20 Timers"},{"content":"I\u0026rsquo;m signing out from Mastodon.\nI love the platform, and I\u0026rsquo;ve had many good interactions on it. But it no longer feels like a safe space for me. There is only so much that content filters can do to fend off hostility. And while that hostility is not directed at me, and the views behind it may sometimes align with my own, it is still smothering. We could all do with a bit more nuance and empathy.\nIf you want to chat, you can email me at \u0026ldquo;blog\u0026rdquo; at this website.\n","permalink":"https://tamir.dev/posts/2026-04-signing-out-for-now/","summary":"I\u0026rsquo;m signing out from Mastodon.\nI love the platform, and I\u0026rsquo;ve had many good interactions on it. But it no longer feels like a safe space for me. There is only so much that content filters can do to fend off hostility. And while that hostility is not directed at me, and the views behind it may sometimes align with my own, it is still smothering. We could all do with a bit more nuance and empathy.","title":"Signing out, for now"},{"content":"After creating Function Graph Overview some users suggested that I add the actual source code to the graphs. I didn\u0026rsquo;t particularly want to do that, feeling that it doesn\u0026rsquo;t serve my goal with this project. Additionally, I had no real idea on how to do it. Not any practical idea, at least.\nSo, lacking ideas and lacking motivation, I mostly let it be. But a couple of weeks ago I saw \u0026lt;foreignObject\u0026gt; mentioned somewhere and realized it can be used here. A bit of work and experimentation, and the mechanics are done (and available my graphviz-embed repo, including a demo). It works by pre-calculating the size of the HTML elements you want to embed, and feeding it to Graphviz for the DOT rendering. A more complete explanation is in the repo itself.\nWhy not in the actual project? Well\u0026hellip; While it works, technically, I could not figure out an attractive way to represent source code in the graph. You see, when I create the graph-overview I have some artistic freedom. Since there\u0026rsquo;s no source-code displayed I can make changes similar to what a compiler does to get a better visual representation of the flow. But with source-code these changes no longer look good.\nConsider a C-style for-loop:\n1 2 3 for (int k = 0; k \u0026lt; 10; ++k) { puts(\u0026#34;Hello, World!\u0026#34;); } The graph-overview for it looks something like this:\nMight take a minute to get used to, but it is clear and readable.\nNow look at the same thing with source-code filled in:\nThis makes no sense. We have the for header broken up and merged with the body and the lines before it in all sorts of weird ways. Additionally, it\u0026rsquo;s hard to make sense of a line out of context like this.\nWe might be able to get something clearer with clusters or other graphical elements, but I am not currently interested in pursuing this. I will, however, be posting this in the GitHub issue for this, and if anyone comes up with a good representation, I\u0026rsquo;ll be more than happy to integrate it.\nIn the meantime, you can copy my code1 and use it in your own projects.\nI considered releasing an NPM package, but realized two things. One is that releasing a TypeScript NPM package is way more work than I anticipated, and the tutorials are bad. The second is that this is basically one function, and it makes no sense to have it as an opaque dependency when you can just copy it and adapt it to your needs.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/graphviz-embed-html/","summary":"After creating Function Graph Overview some users suggested that I add the actual source code to the graphs. I didn\u0026rsquo;t particularly want to do that, feeling that it doesn\u0026rsquo;t serve my goal with this project. Additionally, I had no real idea on how to do it. Not any practical idea, at least.\nSo, lacking ideas and lacking motivation, I mostly let it be. But a couple of weeks ago I saw \u0026lt;foreignObject\u0026gt; mentioned somewhere and realized it can be used here.","title":"Embedding HTML in Graphviz Graphs"},{"content":"I\u0026rsquo;ve read a lot of code. Hand-written code, generated code, compiled, disassembled and decompiled code. I read bad code, decent code, and even some good code. LLM code is differnt. It\u0026rsquo;s\u0026hellip; Fake.\nSure, it is code. It gives the right instructions to the computer, it achieves the goal it was \u0026ldquo;written\u0026rdquo; for. But it feels different. Uncanny.\nThe first thing you notice when you start reading it is that it looks good. And I mean, really good. It is well formatted; it is documented; there are comments around the parts that look scary. There are even tests! Long, detailed, extensive tests! Hell, it even does error handling! On a first draft! If it was written by a human, I\u0026rsquo;d assume this code has been through a lot. That it\u0026rsquo;s been experimented with, written, re-written, debugged, fixed, and eventually documented and tested. So when I come to review the LLM code, my guard is down and it looks really good. That is, until I look a bit deeper.\nIt does so much of the things that \u0026ldquo;make code look good\u0026rdquo;, without actually making it, well, good. The code doubles-down on exception-handling, using catch-all blocks all the way down the call-stack; it checks already-satisfied conditions, seemingly unable to keep track of the flow. The documentation, comments included, is well phrased and structured but seems to lack any real insights. Some functions are used in a way that, well, \u0026ldquo;works\u0026rdquo;, but is definitely not intended. The tests definitely test, well\u0026hellip; things, but there\u0026rsquo;s no clear cut as to what within the clutter of assertions.\nAnd it feels like you\u0026rsquo;re reading busywork. Like no programmer would ever actually write this code. Maybe a few lines of it, maybe even a function - but never the whole thing. Programmers are lazy, in the best way. And they (usually) don\u0026rsquo;t entirely forget what their code is doing halfway between a function definition and the call to it. And I don\u0026rsquo;t like it.\nEven if I can\u0026rsquo;t quite put my finger on it, even if I can\u0026rsquo;t explain it, the code feels off, wrong, uncanny. And it pisses me off. It pisses me off because as I read the code, I know I am spending more time and effort on this code than the person who \u0026ldquo;wrote\u0026rdquo; it. But more than that, it pisses me off because when I see something that looks off, I can\u0026rsquo;t ask \u0026ldquo;hey, why did you do this?\u0026rdquo;. There is no-one to ask. No choices were made by anyone. No-one lived in this code. No-one toiled, no-one struggled.\nAnd I like choices. Choices are the things that make code fun. The things that make code interesting. Choices and tradeoffs and intent. When people write code they make a lot of choices. Some are purely \u0026ldquo;functional\u0026rdquo; choices, and some are stylistic. Some choices are for the computer, and some are meant for the reader. Meant to lead them from line to line, from concept to concept. Grouping some lines together and keeping other apart. Using a specific language construct to hint at something. Making something explicit, and another thing implicit. All those choices have meaning. They all reflect intent.\nWhen I used to do security research, I used to hunt for intent. For the meaning behind the code, for its purpose. Then I\u0026rsquo;d take that purpose, and extrapolate, and map that onto the code asking \u0026ldquo;which cases fit the purpose, but aren\u0026rsquo;t handled in the code?\u0026rdquo;\nAnd I feel like that\u0026rsquo;s missing now. Sure, there are still bugs aplenty. But something is missing.\n","permalink":"https://tamir.dev/posts/ai-vibe-check-2025/","summary":"I\u0026rsquo;ve read a lot of code. Hand-written code, generated code, compiled, disassembled and decompiled code. I read bad code, decent code, and even some good code. LLM code is differnt. It\u0026rsquo;s\u0026hellip; Fake.\nSure, it is code. It gives the right instructions to the computer, it achieves the goal it was \u0026ldquo;written\u0026rdquo; for. But it feels different. Uncanny.\nThe first thing you notice when you start reading it is that it looks good.","title":"AI Coding Vibe Check"},{"content":"I recently started at a new job. After 2 years of solo projects, I\u0026rsquo;m working with many developers on the same project. And after years of local Windows dev, I\u0026rsquo;m suddenly doing remote dev on Linux. Naturally, I needed some new tools for a decent expernience.\n(tl;dr: Lazygit, Atuin, Starship, gh-dash)\nLazygit This is probably the tool I\u0026rsquo;ve been most excited about in a while.\nLazygit \u0026ldquo;a simple terminal UI for git commands\u0026rdquo;, and it\u0026rsquo;s phoenomenal.\nWhen you open it (out of the box and without any configuration), you\u0026rsquo;ll see several panels. If you look at the \u0026ldquo;Branches\u0026rdquo; panel, you\u0026rsquo;ll see your local branches sorted by last update, and you can check them out by pressing Space. This is almost enough for me to recommend the tool. The \u0026ldquo;Files\u0026rdquo; panel shows you all the modified/staged files, and allows you to stage them, stage parts of them (interactively, with easy undo!), or revert changes. Everything is accessible via both keyboard and mouse, and works really fast.\nIn addition to all the obvious git operations (staging, committing, pushing, automatically updating the default branch, and so on and so forth), it has some really advanced features.\nIf you go to the \u0026ldquo;Commits\u0026rdquo; panel, you can modify them. You can re-order or drop commits entirely; reword a commit; or even undo a specific file-change in a specific commit. And all if it through a super simple interface and without the need to ever write git rebase on your own.\nIt\u0026rsquo;s a great tool, really easy to start using, yet absolutely full of advanced features. I highly recommend it!\nAtuin I use Atuin to manage my shell history across multiple remove machines, and it does a fantastic job. It is easy to set up, easy to use, and avoids a whole mess of \u0026ldquo;what did Bash do with my history?\u0026rdquo; issues.\nStarship Starship is a cross-shell prompt. I use it with PowerShell on my Windows machine, and I could just take the same config and use it on my Bash shell on Linux.\ngh-dash If you\u0026rsquo;re working a lot with GitHub you might be familiar with gh (the GitHub CLI). And gh is great, but it isn\u0026rsquo;t interactive. You run a command to see all your open PRs, then a command to check their status, then a command to check them out\u0026hellip;\nLuckily, there are great tools wrapping it, and gh-dash is one of them. It creates a fancy TUI for GitHub, allowing me to see my open PRs and the PRs waiting for my review, and check them out with a single click (you might need to configure the location of your local repos), making it a great time-saver for me.\n","permalink":"https://tamir.dev/posts/new-tools-remote-git/","summary":"I recently started at a new job. After 2 years of solo projects, I\u0026rsquo;m working with many developers on the same project. And after years of local Windows dev, I\u0026rsquo;m suddenly doing remote dev on Linux. Naturally, I needed some new tools for a decent expernience.\n(tl;dr: Lazygit, Atuin, Starship, gh-dash)\nLazygit This is probably the tool I\u0026rsquo;ve been most excited about in a while.\nLazygit \u0026ldquo;a simple terminal UI for git commands\u0026rdquo;, and it\u0026rsquo;s phoenomenal.","title":"New Tools I'm Using (May 2025)"},{"content":"I usually think of SVGs as a \u0026ldquo;simple\u0026rdquo; graphical files, but that\u0026rsquo;s probably because I never had to edit them programmatically.\nI am currently working on adding overlay-notes to Function Graph Overview. The idea is that you use start- and end-comments in your code to define a region, and have that region reflected in the graph as a block surrounding the relevant nodes. Hopefully, this will allow better understanding, as whole regions of the graph can now be labelled. While the project uses Graphviz DOT to define and render the graphs themselves, I decided to add the overlay-notes via post-processing to minimize their effect on graph layout1.\nThis means I need to take the SVG output from Graphviz, and \u0026ldquo;just\u0026rdquo; add the new graphics to it. This involves a couple of steps:\nDetermine the position and size of the notes Add them to the SVG Positioning our Notes One of the wonderful things about SVGs is that they can contain non-visual metadata. Specifically - when exporting to SVG, Graphviz can export node-IDs as SVG id attributes which we can access programmatically, matching the visual nodes with the CFG nodes we used to create them.\nI use svg.js to: get the graphical node positions; find their bounding-box; create the graphics and insert them into the SVG at the wrong place entirely. SVG Transforms You see, SVG elements can have a transform attribute, similar to a CSS transform, changing their position in the document. And Graphviz does indeed use them when exporting graphs2. To correct for this, we copy the transform from the graph element onto our new note elements:\n1 note.transform(graph.transform()); And get our next problem: SVG Viewbox SVG separates document-space from user-space. All of our graphical elements are positioned in document-space, and then we use a viewbox to specify the user\u0026rsquo;s view into the document.\nWe\u0026rsquo;ve just added new elements to the document, so we must adjust the viewbox to include them. To do this, we calculate the bounding box of all our graphical elements (the graph and the notes), manually apply the transformation to the bounding-box, and then adjust the viewbox. But there\u0026rsquo;s another little catch.\nSVGs, like HTML, can use all sorts of units of length. Graphviz uses pt for the width and height of the svg element, but neglects the units for everything else (defaulting to px). This is a great choice by them, as it means the SVG can be specified and manipulated without considering units at all, then scaled using the width and height. So to maintain the original rendered scale of the SVG, we take the final viewport width and height in px, and use that value as the SVG\u0026rsquo;s new dimensions in pt (so if the viewbox is (100px, 200px), the SVG will be (100pt, 200pt)).\nSVG Background Now, there\u0026rsquo;s one last thing to consider - the document background. By default, any part of the SVG that does not display an element, is transparent. This means that to have a solid background, we need to add an element with that color. When we use the bgcolor attribute, Graphviz does this for us. As Graphviz groups graphs in SVG G elements, it puts the background element into the graphs group. This is all good behaviour, as it maintains a good SVG structure, but it is something we must consider when adding our notes.\nFirst, we must make sure our notes end up between the graph elements themselves and the background. For this purpose, we move the background element from the graph G to the top-level SVG. Additionally, having changed the viewbox, we must scale the background element to fill it.\nText Wrapping And with that, we\u0026rsquo;re done3.\nI tried using DOT clusters, but they impact the layout of the graph too significantly to be a real option.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI don\u0026rsquo;t know why they do it. But I definitely don\u0026rsquo;t wanna rewrite that code, so I\u0026rsquo;ll just be happy they give me high-quality outputs and document my findings.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nText-wrapping is not part of the SVG1.1 spec, so there is no easy fix here. To achieve it, we can either split it ourselves into lines; or embed an HTML P element inside the SVG. Preferably the first option.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/graphviz-dot-svg-transforms/","summary":"I usually think of SVGs as a \u0026ldquo;simple\u0026rdquo; graphical files, but that\u0026rsquo;s probably because I never had to edit them programmatically.\nI am currently working on adding overlay-notes to Function Graph Overview. The idea is that you use start- and end-comments in your code to define a region, and have that region reflected in the graph as a block surrounding the relevant nodes. Hopefully, this will allow better understanding, as whole regions of the graph can now be labelled.","title":"Graphviz DOT \u0026 SVG Transforms"},{"content":"A few days ago I created CFGBot - a small Bluesky bot that publishes hourly control-flow-graphs.\nWhile technically simple, this bot is something I wanted to create for a few years now. Originally, I wanted to write one that publishes IDA Pro graphs of random functions in system binaries. But due to logistical issues (how to run a bot that uses software I\u0026rsquo;m only allowed to run on my own PC) and my concerns about copyright (can I freely post the disassembly of copyrighted software?) I never quite got around to it.\nDespite those roadblocks, I kept wanting to share the visual wonder of control-flow-graphs with the world. And a few days ago it dawned on me that now, after writing my control-flow-graph generator, I can finally make this happen. So I glued some pieces together, and here we are.\nA bit of internals The core of the bot is the function-graph-overview project, used to generate the graphs themselves. To choose what function to show every hour - I pre-process a corpus of source-code (currently - CPython\u0026rsquo;s implementation and standard library, but more to come soon!) to get an index of all the functions in it.\nThe index contains basic information:\nLocation of the function (filepath, location in the file) Number of CFG nodes The function definition (this accounts for overloads and anonymous functions better than function names) Then, on every run, I randomly select a function, biased slightly towards higher node-counts, as those are more interesting.\nI then render this function to a PNG image, and post to Bluesky.\nHigh-res images The whole part described above was achieved rather easily (once I had the pieces!) using a bit of Python (to run the scripts and post to Bluesky) and Github-Actions (to \u0026ldquo;host\u0026rdquo; the bot and run it hourly). But there is a problem - Bluesky only supports relatively small images. And control-flow-graphs, especially the cool-looking ones, tend to be very large.\nTo resolve this, I added the /render/ page to function-graph-overview. It takes a GitHub URL (one that includes a line number) as an argument, and renders the SVG in the browser (this way I don\u0026rsquo;t need hosting for those images!). I added basic panning and zooming to make it a tad more useful, and an option to download the SVG. Thus solving the fidelity issue with relative ease.\n","permalink":"https://tamir.dev/posts/hello-cfgbot/","summary":"A few days ago I created CFGBot - a small Bluesky bot that publishes hourly control-flow-graphs.\nWhile technically simple, this bot is something I wanted to create for a few years now. Originally, I wanted to write one that publishes IDA Pro graphs of random functions in system binaries. But due to logistical issues (how to run a bot that uses software I\u0026rsquo;m only allowed to run on my own PC) and my concerns about copyright (can I freely post the disassembly of copyrighted software?","title":"Hello CFGBot 🤖"},{"content":"After giving some thought to the beauty of sorting in Python, I wanted to see how close to it I can get with Go. As always, when writing library code, we need to balance three competing axes: the readability of the application code using our library; the complexity of our library code; and the runtime-performance of the solution.\n\u0026ldquo;Normal\u0026rdquo; Go In Go, the best way to achieve all three is usually to stick with \u0026ldquo;normal\u0026rdquo; Go code, and avoid any advanced trickery. In this case, unfortunately, this was not good enough.\n1 2 # The Python \u0026#39;gold standard\u0026#39; sorted(people, key=attrgetter(\u0026#34;age\u0026#34;, \u0026#34;name\u0026#34;)) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 type Person struct { Name string Age int } people := []Person{ {\u0026#34;Gopher\u0026#34;, 13}, {\u0026#34;Alice\u0026#34;, 55}, {\u0026#34;Bob\u0026#34;, 24}, {\u0026#34;Alice\u0026#34;, 20}, } // Go\u0026#39;s standard solution for comparison // See https://pkg.go.dev/cmp#example-Or-Sort slices.SortFunc(people, func(a, b Person) int { return cmp.Or( strings.Compare(a.Name, b.Name), cmp.Compare(a.Age, b.Age), ) }) Go\u0026rsquo;s solution works, it performs well, and has very low library complexity. But it leaves a lot to be desired on the readability front. It\u0026rsquo;s 4 lines, declares a function closure, and repeats every attribute name twice.\nReflection Since we can\u0026rsquo;t do much better with \u0026ldquo;normal\u0026rdquo; code, we turn to the next candidate - reflection. Go\u0026rsquo;s reflect library seems intentionally designed to make reflection a conscious choice; you can\u0026rsquo;t just \u0026ldquo;fall into it\u0026rdquo;. And for good reason! My solution using reflection turned out to be 20 times slower than the \u0026ldquo;normal\u0026rdquo; solution, and using ~2000 allocations when the first solution used none. It did, however, provide us with a very readable call-site:\n1 slices.SortFunc(people, CmpByFields[Person](\u0026#34;Name\u0026#34;, \u0026#34;Age\u0026#34;)) This is more or less a direct translation of the Python syntax to Go, with the only exception being that [Person] type argument, as Go\u0026rsquo;s type deduction can\u0026rsquo;t do it automatically. Sadly, being very slow and very complex, it does not meet our standards.\nNow we have two solutions. One is fast and simple, but makes for a very verbose call-site. The other, while clean and elegant at the call-site, is slow and hard to maintain. To close that gap, we\u0026rsquo;re going to use another type of Go code - code generation.\nCode Generation We\u0026rsquo;re going to build it step-by-step, so that we know how all the parts interlink and interoperate.\nThe first step of any code-generation journey is, of course, manual generation. In this case, we can write several implementation of cmp for our Person type. They\u0026rsquo;ll look something like this:\n1 2 func CmpByFields_Person_Name_Age(a, b Person) int func CmpByFields_Person_Age_Name(a, b Person) int This works, and we can generate those fairly easily, but this is not quite the syntax we\u0026rsquo;re going for.\nNext, we\u0026rsquo;ll write a CmpByFields and use it to return the relevant function based on the arguments we get:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func CmpByFields[T any](fields ...string) func(a, b Person) int { if reflect.TypeOf(*new(T)) != reflect.TypeOf(*new(Person)) { panic(\u0026#34;not implemented\u0026#34;) } if len(fields) != 2 { panic(\u0026#34;not implemented\u0026#34;) } if fields[0] == \u0026#34;Name\u0026#34; \u0026amp;\u0026amp; fields[1] == \u0026#34;Age\u0026#34; { return CmpByFields_Person_Name_Age } if fields[0] == \u0026#34;Age\u0026#34; \u0026amp;\u0026amp; fields[1] == \u0026#34;Name\u0026#34; { return CmpByFields_Person_Age_Name } panic(\u0026#34;not implemented\u0026#34;) } Yeah, that\u0026rsquo;s not great. And it will only get worse as we add more types and field names to sort by. So instead of this, we\u0026rsquo;ll store them in a map and perform lookups:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 // The key must hold both the struct type and the field names. type mapKey struct { Type reflect.Type Fields string } func newMapKey[T any](fields ...string) mapKey { return mapKey{ Type: reflect.TypeOf(*new(T)), // Slices can\u0026#39;t be used in map keys. Fields: strings.Join(fields, \u0026#34;, \u0026#34;), } } var cmpMap = make(map[mapKey]any) func init() { cmpMap[newMapKey[Person](\u0026#34;Name\u0026#34;, \u0026#34;Age\u0026#34;)] = CmpByFields_Person_Name_Age cmpMap[newMapKey[Person](\u0026#34;Age\u0026#34;, \u0026#34;Name\u0026#34;)] = CmpByFields_Person_Age_Name } func CmpByFields[T any](fields ...string) func(T, T) int { key := newMapKey[T](fields...) if cmpFunc, ok := cmpMap[key]; ok { return cmpFunc.(func(T, T) int) } panic(\u0026#34;not implemented\u0026#34;) } That\u0026rsquo;s a lot better. A proper lookup1 instead of ad-hoc if-statements. With this, it is time to start generating code!\nFor code-generation, we need to know a few things: what to generate, how to generate it, and where to put it. As we\u0026rsquo;ve already written a couple of \u0026ldquo;generated\u0026rdquo; functions by hand, we already know the \u0026ldquo;how\u0026rdquo;, and can convert it to code. For the \u0026ldquo;how\u0026rdquo; we need to know all the different ways the code uses CmpByFields to sort slices of structs. This requires static analysis of the code2, and the detection of all calls to CmpByFields, as well as the types and values used in those.\nWith \u0026ldquo;how\u0026rdquo; and \u0026ldquo;what\u0026rdquo; known, we need to answer the \u0026ldquo;where\u0026rdquo;. In our previous code, we had the init() function in the same file that called CmpByFields. Once we generate code, this can\u0026rsquo;t work as it will require the code-generation to modify our hand-written code, and we don\u0026rsquo;t want that. Instead, we\u0026rsquo;ll generate a new file for every file we analyze. So for main.go, we\u0026rsquo;ll generate main_codegen.go which will contain the comparator implementations and the init() function that are required for main.go3. Since all files in a directory are the same Go package, this\u0026rsquo;ll work perfectly.\nPackaging We have managed to achieve our goal. Our code is highly readable at the call-site; it performs similarly to the \u0026ldquo;normal\u0026rdquo; code; and while the code-generation itself may be a little tricky, the generated code is as simple as can be. We can now use code-generation based sorting freely within our code. But there still is one problem. We can only do this within our code. since the code-generation depends on the CmpByFields function, it is not portable, and we should change that.\nAfter all the work we\u0026rsquo;ve done, this is a simpler matter of moving some code around. We\u0026rsquo;ll create a new package, called cmpgen. It will contain our CmpByFields function, as well as the lookup map and the map-key type. But in addition, it will include a Register function for registering new comparator-functions into the map:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 package cmpgen import ( \u0026#34;fmt\u0026#34; \u0026#34;reflect\u0026#34; \u0026#34;slices\u0026#34; \u0026#34;strings\u0026#34; ) type mapKey struct { Type reflect.Type Fields string } func newMapKey[T any](fields ...string) mapKey { return mapKey{ Type: reflect.TypeOf(*new(T)), Fields: strings.Join(fields, \u0026#34;, \u0026#34;), } } var cmpMap = make(map[mapKey]any) func CmpByFields[T any](fields ...string) func(T, T) int { key := newMapKey[T](fields...) if cmpFunc, ok := cmpMap[key]; ok { return cmpFunc.(func(T, T) int) } panic(\u0026#34;not implemented\u0026#34;) } func Register[T any](fn any, fields ...string) { registry[newMapKey[T](fields...)] = fn } By doing this, we\u0026rsquo;ll have one central map for all the comparators, for all files we analyze. And all the init() functions will use Register to populate it. With this in place, we\u0026rsquo;ll be able to use cmpgen.CmpByFields in any package we wish.\nInvisible Code-Generation Personally, I like referring to this as \u0026ldquo;invisible code generation\u0026rdquo;. When you read code that uses CmpByFields there is no indication of any code-generation. What\u0026rsquo;s more, you can start writing your code before you run the code-generation, and your IDE will be perfectly happy, giving you all the completion and highlights you need.\nIf you want to the entire code, or even use cmpgen.CmpByFields in your own projects, head over to the cmpgen repo.\nYes, we are using a bit of reflection her to compare struct types. But happens during the lookup phase, and not in the comparator passed to SortFunc, so the performance impact is minimal.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nStatic analysis requires quite a bit of code, so I\u0026rsquo;ll skip it here. If you\u0026rsquo;re interested, take a look at the collection of calls using the callector helper package.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThere are likely to be duplicate implementation in the generated code, as multiple files will sort the same way. This is not a problem because generation implementation for identical call-sites will be identical. The performance penalty for adding a duplicate key to a map will be truly negligible.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/invisible-codegen-go/","summary":"After giving some thought to the beauty of sorting in Python, I wanted to see how close to it I can get with Go. As always, when writing library code, we need to balance three competing axes: the readability of the application code using our library; the complexity of our library code; and the runtime-performance of the solution.\n\u0026ldquo;Normal\u0026rdquo; Go In Go, the best way to achieve all three is usually to stick with \u0026ldquo;normal\u0026rdquo; Go code, and avoid any advanced trickery.","title":"Invisible Codegen in Go"},{"content":"Python has what is probably the most elegant way of sorting a collection of objects by their attribute values:\n1 sorted(people, key=attrgetter(\u0026#34;age\u0026#34;, \u0026#34;name\u0026#34;)) Let\u0026rsquo;s break it down.\nsorted() We start with the sorted() function. It returns a new sorted list from the items in its first argument.\n1 2 \u0026gt;\u0026gt;\u0026gt; sorted([3, 1, 2]) [1, 2, 3] sorted() also takes an optional key function. key, when provided, is used to extract a comparison key from the items being sorted.\n1 2 3 # Sort numbers by last digit \u0026gt;\u0026gt;\u0026gt; sorted([13, 21, 32], key=lambda x: x % 10) [21, 32, 13] This also means that if we want to sort custom objects by a given attribute, we can write:\n1 2 3 4 5 6 from dataclasses import dataclass @dataclass class Person: name: str age: int 1 2 3 4 5 6 7 8 9 \u0026gt;\u0026gt;\u0026gt; sorted([ Person(name=\u0026#34;Alice\u0026#34;, age=32), Person(name=\u0026#34;Bob\u0026#34;, age=28), Person(name=\u0026#34;Alice\u0026#34;, age=28), ], key=lambda x: x.age) [Person(name=\u0026#39;Bob\u0026#39;, age=28), Person(name=\u0026#39;Alice\u0026#39;, age=28), Person(name=\u0026#39;Alice\u0026#39;, age=32)] Which is great, but what if we want to sort by age and by name?\ntuple Tuples are usually immutable sequences of heterogeneous elements. Python\u0026rsquo;s tuple has the added property of being compared lexicographically; the first items are compared; if they are the same then the second items are compared, and so on.\nThis allows us to adjust our code to account for both age and name. But remember to put the more significant member first!\n1 2 3 4 5 6 7 8 9 \u0026gt;\u0026gt;\u0026gt; sorted([ Person(name=\u0026#34;Alice\u0026#34;, age=32), Person(name=\u0026#34;Bob\u0026#34;, age=28), Person(name=\u0026#34;Alice\u0026#34;, age=28), ], key=lambda x: (x.age, x.name)) [Person(name=\u0026#39;Alice\u0026#39;, age=28), Person(name=\u0026#39;Bob\u0026#39;, age=28), Person(name=\u0026#39;Alice\u0026#39;, age=32)] With this, we can sort our objects by their attribute values. But we can do better.\nattrgetter() operator.attrgetter() is a higher-order function; it takes one-or-more attribute names, and returns a function that can be used to extract those named attributes from an object.\n1 2 3 4 \u0026gt;\u0026gt;\u0026gt; from operator import attrgetter \u0026gt;\u0026gt;\u0026gt; get_age_and_name = attrgetter(\u0026#34;age\u0026#34;, \u0026#34;name\u0026#34;) \u0026gt;\u0026gt;\u0026gt; get_age_and_name(Person(name=\u0026#34;Bob\u0026#34;, age=28)) (28, \u0026#39;Bob\u0026#39;) We can use this to replace our previous lambda, and get:\n1 2 3 4 5 6 7 8 9 \u0026gt;\u0026gt;\u0026gt; sorted([ Person(name=\u0026#34;Alice\u0026#34;, age=32), Person(name=\u0026#34;Bob\u0026#34;, age=28), Person(name=\u0026#34;Alice\u0026#34;, age=28), ], key=attrgetter(\u0026#34;name\u0026#34;, \u0026#34;age\u0026#34;)) [Person(name=\u0026#39;Alice\u0026#39;, age=28), Person(name=\u0026#39;Alice\u0026#39;, age=32), Person(name=\u0026#39;Bob\u0026#39;, age=28)] Which, to me, seems as straight-forward as can be. I really enjoy the way different language and library features combine to create such beautiful patterns.\n","permalink":"https://tamir.dev/posts/python-beautiful-sorting/","summary":"Python has what is probably the most elegant way of sorting a collection of objects by their attribute values:\n1 sorted(people, key=attrgetter(\u0026#34;age\u0026#34;, \u0026#34;name\u0026#34;)) Let\u0026rsquo;s break it down.\nsorted() We start with the sorted() function. It returns a new sorted list from the items in its first argument.\n1 2 \u0026gt;\u0026gt;\u0026gt; sorted([3, 1, 2]) [1, 2, 3] sorted() also takes an optional key function. key, when provided, is used to extract a comparison key from the items being sorted.","title":"Beautiful Sorting in Python"},{"content":"Over the last few weeks I\u0026rsquo;ve been working on a VSCode extension for visualizing the control-flow-graph (CFG) for the function you\u0026rsquo;re reading.\nIf you want, you can play with the live demo, install the extnsion, or look at the code.\nWhile some choices for the visualization are rather straightforward, some are not. So I want to go over the choices I made and the reasoning for them.\nBasic Control Flow First - to represent a block of code, we use a block. The more code we have in the block, the taller the block will become. This helps give a basic sense of \u0026ldquo;how much code is there?\u0026rdquo;\nNext, to connect two blocks we use arrows. The default color is blue. Since the graph can get quite large, we mark the entry-point in green and all the exits (return statements, mostly) in red. Then we have conditions (if). We use green for the consequence branch (when the condition holds) and red for the alternative (when the condition evaluates to false). For switch statements, we have two possible representations: chained-ifs, or flat. Since both have their benefits, choosing between them is a configurable option.\nChained Flat For loops we use the same structures we used before. Green for the consequence, leading us into the loop body; red for the alternative case, leaving the loop. You might notice that the arrow pointing back to the top of the loop has a thicker line. We make all back-linking arrows thicker so that they are easily distinguished from other arrows, as they are structurally distinct.\nThese are the basic blocks of our graph, showing the flow of the program. But there are some structures that break the flow of the code.\nExceptions \u0026amp; Context Managers While most control-flow structures translate well to graphs, exceptions are an exception. Since every line of code can potentially raise an exception, drawing all the arrows would make the graph entirely unreadable. Instead, we use clusters. We surround the entire try-except-else-finally construct with a pale-blue background to let us know everything within it is part of the same thing. For the try block we use a green background, letting us know that this is the \u0026ldquo;happy path\u0026rdquo;. except blocks are separately surrounded by pale-red backgrounds, letting us know that they can be reached separately.\nThe else block, if present, is placed in the surrounding blue background. This separates it clearly from the try block, letting us know that exceptions from it are not handled in this blue cluster. Then, with finally, things get a little tricky. In a simple case, we just give it a yellow background and call it a day. But it\u0026rsquo;s interaction with return statements in the preceding blocks can result in interesting flows.\nBelow, we have the flow for a try-except block, where the except clause returns from the function. Next to it, we have the same structure if a finally clause is added:\ntry-except(return) try-except(return)-finally In this case, there are 2 separate flows out of the try-... block. To keep them separate, we duplicate the finally block for every flow. While this is useful, it can get a bit crazy when the finally block has logic inside it: Context Managers The with statement gets a similar treatment, this time with a light-fuschia background: When nested, we also add a white border to separate the levels. The same border technique is used for exceptions as well. Special Nodes In addition to the function entry-point and return statements, raise and yield also get special treatment:\nraise statements get a triangle in the same color as the except blocks. The triangle is meant to be similar to the \u0026ldquo;house\u0026rdquo; shape of return nodes, while the color is meant to clarify the connection to except.\nyield gets a hexagonal shape, conceptually the combination of the exit and entry node shapes (as you both leave and entry the function through the yield statement), and the color is light-blue, making it distinct from other nodes in the graph.\n","permalink":"https://tamir.dev/posts/cfg-visualization-legend/","summary":"Over the last few weeks I\u0026rsquo;ve been working on a VSCode extension for visualizing the control-flow-graph (CFG) for the function you\u0026rsquo;re reading.\nIf you want, you can play with the live demo, install the extnsion, or look at the code.\nWhile some choices for the visualization are rather straightforward, some are not. So I want to go over the choices I made and the reasoning for them.\nBasic Control Flow First - to represent a block of code, we use a block.","title":"Visualizing Code in CFGs"},{"content":"Version Numbers Current software versioning schemes conflate two different axes of progress - \u0026ldquo;features\u0026rdquo; and \u0026ldquo;bugfixes\u0026rdquo; - into a single \u0026ldquo;number\u0026rdquo;. This works well as long as only a single version is supported by the maintainers, but breaks apart when more than one version is supported.\nAs long as we have a single version, we can generally say \u0026ldquo;bigger is better\u0026rdquo;. The larger the version number, the better the software is. More features, less bugs. But as soon as we support more than one version, this logic breaks. A bugfix version (1.1.14) will never be \u0026ldquo;bigger\u0026rdquo; than the next minor or major version (1.2.0 or 2.0.0), but it may include critical security fixes you need.\nThis means that instead of specifying \u0026gt;=1.1.14 as the required version, users must specify something equivalent to 1.1.14 \u0026lt;= v \u0026lt; 1.2.0 || 1.2.3 \u0026lt;= v \u0026lt; 1.3.0 || 2.0.1 \u0026lt;= v \u0026lt; 2.1.0, and add any new minor or major version they want to support as it comes out. If a new critical bugfix is released, they need to update all the conditionals, hoping they don\u0026rsquo;t get them wrong.\nThis is maintenance hell. In addition to being error-prone, this fine-grained dependency specification increases the chances of conflicting dependencies; especially if one package updates a dependency to address a security issue, while another doesn\u0026rsquo;t. We need a better solution, one that properly addresses the different concerns of feature updates vs. bugfixes \u0026amp; security updates. We need another axis in our versioning.\nPatch Levels Security patch levels can provide this missing axis. Just like two versions with the same major and minor versions are expected to have the same features, two versions with the same patch-level are expected to have the same critical bugfixes.\nNow, we can change our dependency specification to v \u0026gt;= 1.1.14 \u0026amp;\u0026amp; patch \u0026gt;= PL3. This is shorter, simpler, and more flexible. If 1.3 were to come out, we won\u0026rsquo;t have to touch our dependency specification.\nWhile these can work with any type of patch, they require more work on the package-maintainer side, as they now need to specify them for all the released versions. As such, it seems good practice to stick to critical bugfixes, or security bugfixes.\nIn Practice An example of a project using security patch levels is the Android operating system, using them in their Android Security Bulletins, and use dates as security levels (instead of a running number). Unfortunately, I am unaware of any package/dependency managers using a similar mechanism.\n","permalink":"https://tamir.dev/posts/thoughts-about-security-patch-levels/","summary":"Version Numbers Current software versioning schemes conflate two different axes of progress - \u0026ldquo;features\u0026rdquo; and \u0026ldquo;bugfixes\u0026rdquo; - into a single \u0026ldquo;number\u0026rdquo;. This works well as long as only a single version is supported by the maintainers, but breaks apart when more than one version is supported.\nAs long as we have a single version, we can generally say \u0026ldquo;bigger is better\u0026rdquo;. The larger the version number, the better the software is.","title":"Some Thoughts About Version Numbers \u0026 Security Patch Levels"},{"content":"I recently read An app can be a home-cooked meal by Robin Sloan and found it highly relatable. They talk about how not all software needs to scale, or be general, or even work for more than one person. Software can be small, and quirky, and targeted at a single-digit audience. And I love that.\nAs a kid, I used to write code like that. Small things that worked only for me, and \u0026ldquo;works on my machine\u0026rdquo; was basically all I needed. It was fun, and exciting, and sometimes helpful. I wrote scripts, and small GUI apps, and very one-thing-specific browser plugins.\nAs I got into the industry, and became a \u0026ldquo;senior developer\u0026rdquo;, I kinda stopped doing that. I expect all my software to work in all cases, scale, be \u0026ldquo;clean\u0026rdquo; and upgradable and generally never actually get written, as there\u0026rsquo;s just too much to do before a \u0026ldquo;side project\u0026rdquo; becomes \u0026ldquo;viable\u0026rdquo;. In recent months, I\u0026rsquo;ve realized that I went through this change. That was a rather sad realization, and since then I\u0026rsquo;ve been working hard to change that. Sure, there\u0026rsquo;s production software. But there\u0026rsquo;s also small, personal software. Home cooked software.\nAnd finally, after a long time of not doing any useful side-project, I finally wrote some software that is small, simple, and makes me super happy.\nI wrote a tiny static-site generator for my cooking recipes, which is as minimal as can be and has absolutely zero fool-proofing or fault tolerance; but it lets me save \u0026amp; share my recipes.\nI wrote a remote-control for my laptop, so that I can control YouTube or VLC or Netflix when I connect it to my TV. It detects the top-most window, and shows a web-interface with simple buttons. When you click them, it sends keypresses to that window. Since Netflix doesn\u0026rsquo;t have a \u0026ldquo;next-episode\u0026rdquo; key, I also wrote a Firefox Addon to add one, so that I can use it in my remote.\nIt\u0026rsquo;s all small, simple, totally won\u0026rsquo;t pass code-review software. And it works. My partner and I use the remote daily, and I am finally documenting my recipes. And most importantly - this software makes me happy.\n","permalink":"https://tamir.dev/posts/home-cooked-software/","summary":"I recently read An app can be a home-cooked meal by Robin Sloan and found it highly relatable. They talk about how not all software needs to scale, or be general, or even work for more than one person. Software can be small, and quirky, and targeted at a single-digit audience. And I love that.\nAs a kid, I used to write code like that. Small things that worked only for me, and \u0026ldquo;works on my machine\u0026rdquo; was basically all I needed.","title":"Home Cooked Software"},{"content":"At the time of writing this, I am not aware of any problem this solves in idiomatic Go code. But I thought of it, so I am sharing it with you. Now, you can use it in your own code, and have a new problem.\nGo\u0026rsquo;s solution for polymorphism is interfaces. An interface is essentially a collection of methods. If a type implements those methods - it matches the interface, and can passed to functions that expect that interface. Note that you don\u0026rsquo;t need to explicitly specify the interface you\u0026rsquo;re implementing - if your type has the right methods, it\u0026rsquo;ll match. Be it intentionally or by accident.\nWhen you create your type, it\u0026rsquo;s likely you\u0026rsquo;ll have more methods than those that match specific interface. Sure, your dog implements the Eater interface, but it also has Bark and Bite methods. Makes perfect sense. But some time in the future, the library exposing Eater adds a new interface - VenomousBiter - which requires the Bite method. Now your Dog matches VenomousBiter as well. Unfortunately, the author of the library intended VenomousBiter to only be used for venomous animals (and Snakebite as a Snake method seemed all too specific), and now your dog is venomous. Better design and naming of interfaces and methods can probably protect you from this mess. But can you really trust everyone to be sensible enough, or would you rather put in some work to avoid venomous dogs?\nThe problem stems from Go\u0026rsquo;s interfaces being a form of structural subtyping ,matching based on the structure of a type - it\u0026rsquo;s methods, this case; as opposed to nominal subtyping, where types are matched based on their names (think OOP inheritance). To fix it, we\u0026rsquo;ll tag our interfaces, ensuring only types with matching tags match the interface.\nTo make tags work, we need to make sure they cannot be added by accident. The presence of a tag must always indicate intent. Interfaces only check for methods - so we need to add a method that cannot be matched by accident. Since any method name can be matched by accident, we\u0026rsquo;ll use something stronger - package boundaries. By using a non-exported method name in the interface, we can guarantee that no type outside the current package can match the interface.\n1 2 3 4 type VenomousBiter interface { venomous() Bite(...) } Great. Now our dog is no longer venomous. On the other hand, we have no way to make our VenomousSnake venomous. So we introduce another type in the library - the other half of the tag!\n1 2 3 type Venomous struct{} func (Venomous) venomous() {} And use that for our snake:\n1 2 3 4 type VenomousSnake struct { animals.Venomous // ... } Now, our snake is venomous, and our dog is not. Mission accomplished.\nFull Code You can read the full code below, or run it in the Go Playground\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 // file: animals/animals.go package animals import \u0026#34;fmt\u0026#34; type Eater interface { Eat(food string) } type Biter interface { Bite(prey string) } type VenomousBiter interface { venomous() Bite(prey string) } type Venomous struct{} func (Venomous) venomous() {} func Feed(e Eater, food string) { e.Eat(food) } func Poke(b Biter) error { if _, isVenomous := b.(VenomousBiter); isVenomous { return fmt.Errorf(\u0026#34;poking venomous animals is ill-advised\u0026#34;) } b.Bite(\u0026#34;your finger\u0026#34;) return nil } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 // file: main.go package main import ( \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;animals\u0026#34; // Change this to match your environment! ) type Dog struct { name string } func (d *Dog) Eat(food string) { fmt.Println(d.name, \u0026#34;ate\u0026#34;, food, \u0026#34;and wagged it\u0026#39;s tail.\u0026#34;) } func (d *Dog) Bite(prey string) { fmt.Println(d.name, \u0026#34;bit\u0026#34;, prey, \u0026#34;and barked.\u0026#34;) } type VenomousSnake struct { animals.Venomous name string } func (s *VenomousSnake) Eat(food string) { fmt.Println(s.name, \u0026#34;ate\u0026#34;, food, \u0026#34;and fell asleep.\u0026#34;) } func (s *VenomousSnake) Bite(prey string) { fmt.Println(s.name, \u0026#34;bit\u0026#34;, prey, \u0026#34;and injected it with venom.\u0026#34;) } func main() { dog := Dog{name: \u0026#34;Barky\u0026#34;} snake := VenomousSnake{name: \u0026#34;Hissy\u0026#34;} animals.Feed(\u0026amp;dog, \u0026#34;a bone\u0026#34;) animals.Feed(\u0026amp;snake, \u0026#34;a mouse\u0026#34;) err := animals.Poke(\u0026amp;dog) if err != nil { log.Fatal(err) } err = animals.Poke(\u0026amp;snake) if err != nil { log.Fatal(err) } } ","permalink":"https://tamir.dev/posts/go-tag-interfaces-venomous-dogs/","summary":"At the time of writing this, I am not aware of any problem this solves in idiomatic Go code. But I thought of it, so I am sharing it with you. Now, you can use it in your own code, and have a new problem.\nGo\u0026rsquo;s solution for polymorphism is interfaces. An interface is essentially a collection of methods. If a type implements those methods - it matches the interface, and can passed to functions that expect that interface.","title":"Tag your Go Types"},{"content":"Writeup of my talk about the cpppy library\nA few years ago I gave a talk about implementing C++ semantics (namely, destructors) in Python. After giving the talk, I published my speaker notes and slides, to make it as accessible as possible, even to people who don\u0026rsquo;t wanna watch a video. I was quite pleased with myself, until a few weeks ago when I wanted to look up a detail so that I can reference it in a new blogpost. I went to the speaker notes, started reading, and oh. Oh no. I could get through it, but it really doesn\u0026rsquo;t hold up on its own despite my memory of it being \u0026ldquo;everything I said during the talk\u0026rdquo;. That\u0026rsquo;s quite unfortunate, as I am very pleased with the work I done for this talk, and want people to happen upon it. Which brings us here - this is that talk, but as a blogpost.\nBut Why? When coming to implement C++ semantics in Python, the first question we need to ask ourselves is \u0026ldquo;why?\u0026rdquo;. Why would we want to take anything (other than performance) from C++ into Python? After all, C++ is a low-level language, mostly \u0026ldquo;expert oriented\u0026rdquo;, and is slowly becoming more \u0026ldquo;Pythonic\u0026rdquo;. And on the other hand, Python is a high-level language, it is beginner friendly, and has far fewer footguns than C++.\nThe answer to that question is resource-management. In C++, all resources are handled the same way. Be it allocated memory; a file; a lock; or your own resource - they are all handled using the same mechanism. In Python, on the other hand, different resources are handled differently. While memory is managed for you by the garbage collector, all other resources must be managed by the programmer.\nBefore you say anything - no. Python is not C. We don\u0026rsquo;t have to call the cleanup functions manually. Instead, Python gives us context managers.\nContext Managers Context-managers are relatively straight-forward. To use them, we use the with statement as follows:\n1 2 with FileReader(path) as f: print(f.read()) Then, the language uses our FileReader object to wrap the indented block, roughly as follows:\n1 2 3 4 5 6 7 8 9 _tmp = FileReader(path) f = _tmp.__enter__() try: print(f.read()) except: if not _tmp.__exit__(\u0026lt;exception info\u0026gt;): raise finally: _tmp.__exit__(\u0026lt;None exception info\u0026gt;) Before entering the indented block, we create our FileReader and call its __enter__ method. Then, we run the contents of the block. Then, on leaving the block, we call __exit__. If an exception was raised from the block, we\u0026rsquo;ll get the exception info and have a choice to either silence it (by returning a True value) or re-raise it (by returning False). If no exception was raised - we\u0026rsquo;ll get get no exception information.\nTo define a context manager, all we have to do is add an __enter__ method and an __exit__ method to our class:\n1 2 3 4 5 6 7 8 class FileReader: def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.close() ... Real Code This is well and nice, but let\u0026rsquo;s look at some real code, a redacted version of something we had running in production:\n1 2 3 4 5 6 7 8 9 10 class ArchiveReader: def __init__(self, path: str): self.data = {} with ZipFile(path) as zipfile: for name in zipfile.namelist(): with zipfile.open(name) as f: self.data[name] = f.read() def read(self, name): return self.data[name] We had a zip file containing many small files. So to make reading fast and simple, we read all the files into memory when initializing the reader. Then, when we needed a specific file - we just accessed it in the data dict. This way we only unzip once, which was a significant performance gain.\nThe usage of the class was also very straightforward:\n1 2 3 4 5 reader = ArchiveReader(\u0026#34;corecpp.zip\u0026#34;) print(reader.read(\u0026#34;2021\u0026#34;)) # Prints: # Hello, CoreC++! We were fairly pleased.\nBut, as time went by, we ran into issues. The data files we were reading changed from being tiny (\u0026lt;10MiB) to huge (\u0026gt;5GiB). As a result, we could no longer keep the files unzipped in memory.\nThe \u0026ldquo;easy\u0026rdquo; solution would keep the path, and open the zip file whenever we read a file, unzipping only the desired file. This works, but due to the need to parse zip metadata, it adds a significant overhead in execution time.\nThe solution we went with was to create the ZipFile object (thus parsing the metadata) and hold it in our ArchiveReader class. Then, when we read a file, we open and unzip only the said file:\n1 2 3 4 5 6 7 class BigArchiveReader: def __init__(self, path: str): self.zipfile = ZipFile(path) def read(self, name: str): with self.zipfile.open(name) as f: return f.read() Unfortunately, now that we hold a ZipFile object, we need to manage it. To do that, we had to make our archive reader a context-manager:\n1 2 3 4 5 6 7 8 class BigArchiveReader: ... def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.zipfile.close() Which, in turn, changes the usage of our reader:\n1 2 with BigArchiveReader(\u0026#34;corecpp.zip\u0026#34;) as big_reader: print(big_reader.read(\u0026#34;2021\u0026#34;)) In turn, this demonstrates a bigger issue. Holding a context manager as a member changes the interface of our objects, forcing them to be context managers as well. This change propagates up to all respective owners, and up the stack to the point where the top-level object is created. As this is a breaking change, we must be able to change the code that uses our object. In the general case, this is not possible.\nC++ Destructors C++, however, has a solution to those issues - destructors. Destructors are what makes C++ resource management work, and they have 3 key properties we\u0026rsquo;re interested in. They are:\nAutomatic Composable Implicit. Automatic Invocation The invocation of a destructor is automatic. When we leave a scope, they are called automatically, no matter how we leave the scope.\n1 2 3 4 5 { auto reader = FileReader(path); std::cout \u0026lt;\u0026lt; reader.read() \u0026lt;\u0026lt; \u0026#39;\\n\u0026#39;; } // \u0026lt;-- The destructor is called here! Seamless Composition Destructors of member fields are called automatically, making composition easier. If we add a new member, we know that it\u0026rsquo;s destructor will be called when it is needed.\n1 2 3 4 5 6 7 8 9 class ArchiveReader { ... }; ... auto reader = ArchiveReader(path); } // \u0026lt;-- Destructor is called! // ~ArchiveReader(); 1 2 3 4 5 6 7 8 9 10 11 12 class BigArchiveReader { ZipFile zipfile; ... }; ... auto big_reader = BigArchiveReader(path); } // \u0026lt;-- Destructor is called! // ~BigArchiveReader(); // Followed by member destructor: // ~ZipFile(); Implicit Interfaces Last but not least - destructors are implicit in object interfaces. Objects with user defined destructors and objects without them are all used the same way.\n1 2 3 4 5 6 7 8 9 // Object with no destructor { auto object = ObjectWithoutDtor(); } // Object with destructor { auto object = ObjectWithDtor(); } This is even if we don\u0026rsquo;t define them, they always exist, for all objects. This means that when we write our own destructors, our interfaces don\u0026rsquo;t change, and no change is propagated.\nOur Goal With that in mind, we want to bring destructors from C++ to Python. Converting our archive reader from this (11 lines of code, 4 are dedicated to resource management):\n1 2 3 4 5 6 7 8 9 10 11 12 13 class BigArchiveReader: def __init__(self, path: str): self.zipfile = ZipFile(path) def read(self, name: str): with self.zipfile.open(name) as f: return f.read() def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.zipfile.close() To this (7 lines of code, 0 dedicated to resource management):\n1 2 3 4 5 6 7 8 class BestArchiveReader: zipfile: ZipFile def __init__(self, path: str): self.zipfile = ZipFile(path) def read(self, name: str): with self.zipfile.open(name) as f: return f.read() And ensure that our usage remains the same as the original ArchiveReader, with no context-managers and no interface pollution:\n1 2 reader = BestArchiveReader(\u0026#34;corecpp.zip\u0026#34;) print(reader.read(\u0026#34;2021\u0026#34;)) Don\u0026rsquo;t Try This At Work Seriously, just don\u0026rsquo;t.\nIt\u0026rsquo;s all fun, and standard, and truly painful in a code review or debugging session.\nWith that in mind, let\u0026rsquo;s start!\nImplementing C++ Destructors in Python Automatic Since our target class is a bit complicated, we\u0026rsquo;ll be joined in our implementation journey by a simple class called Greeter. The greeter is a simple class. It takes a name on construction, and says \u0026ldquo;Hello\u0026rdquo;. On destruction, it says \u0026ldquo;Goodbye\u0026rdquo;\nThis will allow us to keep track of ctors and dtors as we progress through this post.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class Greeter: def __init__(self, name): self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) def close(self): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) def main(): greeter = Greeter(1) print(\u0026#34;We have a greeter!\u0026#34;) greeter.close() main() Hello, 1! We have a greeter! Goodbye, 1. Our first implementation is as straight-forward as can be. With a constructor and a \u0026ldquo;close\u0026rdquo; method to act like our dtor.\nThe next step is pretty straight-forward. Since Python already provides us with context-manages and the with statement, we\u0026rsquo;ll use them and see where it gets us.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 class Greeter: def __init__(self, name): self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) return False def main(): with Greeter(1): print(\u0026#34;We have a greeter!\u0026#34;) main() Hello, 1! We have a greeter! Goodbye, 1. Stacking Dtors What if we want more than one Greeter? No problem! Just stack the context-managers!\n1 2 3 4 5 6 7 8 def main(): with Greeter(1): print(\u0026#34;First\u0026#34;) with Greeter(2): print(\u0026#34;Second\u0026#34;) main() Hello, 1! First Hello, 2! Second Goodbye, 2. Goodbye, 1. Then again, this type of nesting can get unwieldy quick. We need something better.\nSince we\u0026rsquo;re already stacking context-managers, we can use a proper stack:\n1 2 3 4 5 6 7 8 9 10 11 12 13 class DtorScope: def __init__(self): self.stack = [] def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): while self.stack: self.stack.pop().__exit__(exc_type, exc_val, exc_tb) def push(self, cm): self.stack.append(cm) The DtorScope will hold all of our Greeter objects in place, then call their __exit__ methods when we leave the with context:\n1 2 3 4 5 6 7 8 9 10 def main(): with DtorScope() as dtor_stack: greeter1 = Greeter(1) dtor_stack.push(greeter1) greeter2 = Greeter(2) dtor_stack.push(greeter2) main() Hello, 1! Hello, 2! Goodbye, 2. Goodbye, 1. And with that, we\u0026rsquo;ve rid ourselves of the nesting issue. It works, destruction is automatic, it\u0026rsquo;s fairly straight-forward, and all too explicit.\nImplicit Now that our destructors get called automatically on scope exit, we want to make sure that we don\u0026rsquo;t need to write any code to make it happen.\nIf we could, we\u0026rsquo;d like our function to look like this:\n1 2 3 def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) And have it implicitly do all the plumbing we managed earlier.\nFirst, since we always want to push our objects onto the dtor stack, let\u0026rsquo;s make it part of their construction.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class Greeter: def __init__(self, name, dtor_stack): dtor_stack.push(self) self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) return False def main(): with DtorScope() as dtor_stack: greeter1 = Greeter(1, dtor_stack) greeter2 = Greeter(2, dtor_stack) main() That\u0026rsquo;s a good start. We can no longer forget to push our greeters onto the DtorScope, and we saved a couple of lines.\nThat said, we\u0026rsquo;re explicitly repeating and passing around a construct that should be implicit.\nTo pass the dtor_stack implicitly to the Greeter class, we need to store it somewhere. In our case, we\u0026rsquo;ll use a global variable! And just like function calls go into a stack so that we know where to return, so will our DtorScopes. So instead of a single global variable, we\u0026rsquo;ll have to use a global stack.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 _dtor_stack = [] def get_dtor_stack(): return _dtor_stack class DtorScope: def __init__(self): self.stack = [] # Push current scope onto global scope stack get_dtor_stack().append(self) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): # Pop current scope from global scope stack get_dtor_stack().pop() while self.stack: self.stack.pop().__exit__(exc_type, exc_val, exc_tb) def push(self, cm): self.stack.append(cm) def push_dtor(cm): return get_dtor_stack()[-1].push(cm) This is the same as our previous dtor-scope, but now we keep a global stack of scopes. This allows us to always tell which dtor-stack to push our instances into without naming the stack.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 class Greeter: def __init__(self, name): self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) # Push the greeter onto the current dtor-scope push_dtor(self) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) return False def main(): with DtorScope(): greeter1 = Greeter(1) greeter2 = Greeter(2) main() Much better, but we still need to explicitly create the scope inside every function.\nDecorators However, since the dtor scoping mechanism has nothing to do with the function itself, we can use it at the callsite instead of inside the function, and it\u0026rsquo;d work exactly the same.\n1 2 3 4 5 6 def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) with DtorScope(): main() This will have to be done on every callsite, so we add a utility function to help with that.\nThe *args, **kwargs syntax is there to pass along any and all function arguments unchanged. Think of it as Python\u0026rsquo;s version of perfect-forwarding.\n1 2 3 4 5 def call(f, *args, **kwargs): with DtorScope(): return f(*args, **kwargs) call(main) Alternatively, we can take the function and return a closure that includes the scoping.\n1 2 3 4 5 6 7 8 9 def cpp_function(f): def _wrapper(*args, **kwargs): with DtorScope(): return f(*args, **kwargs) return _wrapper scoped_main = cpp_function(main) scoped_main() Inside _wrapper we capture f from the parent scope. As Python is reference-based, we don\u0026rsquo;t need to specify how to capture.\nNext, since Python is dynamic, we can replace the original main function with the scoped one.\n1 2 3 4 5 6 7 def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) main = cpp_function(main) main() Here the wrapper holds the previous version of the function, while the name main is bound to the wrapped version.\nIn Python, wrapping a function and replacing its original is a common operation. Because of that, we have some syntactic sugar called \u0026ldquo;decorators\u0026rdquo;.\n1 2 3 4 5 6 @cpp_function def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) main() This is functionally equivalent to the previous snippet, but is cleaner and more expressive. And with that, the inside of main looks the way we want it.\nImport Hacks We still have a problem, though. Even if main looks nice, we had to decorate it with cpp_function. Having to do that to every function we write for it to work is not very \u0026ldquo;implicit\u0026rdquo;, is it?\nWhat if instead, we could just write from cpp import magic, and have that \u0026ldquo;magic\u0026rdquo; decorate our functions for us?\n1 2 3 4 5 from cpp import magic def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) Nice, isn\u0026rsquo;t it? So let\u0026rsquo;s get to it!\nThe Magic Function As a first step, we\u0026rsquo;ll call a function at the end of our module to decorate everything:\n1 2 3 4 5 6 7 8 9 10 11 from cpp import magic from greeter import Greeter def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) magic() main() Our magic() function needs to do 2 things:\nGet the module it was called in; and decorate all the functions. 1 2 3 def magic(): calling_module = get_calling_module() decorate_module_functions(calling_module) To get the calling module, we use inspect.stack to traverse the callstack and find the right module\n1 2 3 4 5 6 7 import inspect def get_calling_module(): # Use 2 here because we need to go 2 frames up. stack_frame = inspect.stack()[2].frame module = inspect.getmodule(stack_frame) return module We get the callstack, take the frame 2 level above us (the caller to magic()), and use inspect.getmodule to get the relevant module.\nThen we decorate our functions.\n1 2 3 4 5 6 7 8 9 10 11 def decorate_module_functions(module): for name, value in inspect.getmembers(module): if not inspect.isroutine(value): continue # Only convert functions that were defined in the importing file. # We don\u0026#39;t want to convert library imports and the likes of those. if inspect.getmodule(value) != module: continue setattr(module, name, cpp_function(value)) Our function takes a module and modifies it.\nTo do this, we\u0026rsquo;re using some of Python\u0026rsquo;s reflection capabilities. inspect.getmembers(obj) returns all the member variables of a given object. In our case - a module. inspect.isroutine(obj) tells us whether a value is a function. inspect.getmodule(obj) returns the module an object was defined in. setattr(obj, name, value) sets an object attribute named name to value.\nAnd with that our magic() function is operational!\nFor our next trick, we\u0026rsquo;ll make the call to magic() disappear as well. This means that we need to somehow make the from cpp import magic line do the actual magic.\n\u0026ldquo;But Tamir! An import is not a function call!\u0026rdquo; you might say. Well, let\u0026rsquo;s see what a Python import actually does.\nImport Internals When we run from cpp import magic we go through the following steps.\nFirst, we look for a module named cpp in the global module cache, sys.modules. If it is present, the module is already loaded, and we can skip to name binding.\nIf it is not present in the cache, we find it on disk, create a module object from it, place the module object in the cache, and then execute the module.\nNote that we first store it in the cache, and only then execute the module. This is important as Python allows for cyclic imports, and we want to avoid recursion. It will also come in handy later.\nOnce we finish executing the module, we need to bind the relevant names. In this case - magic.\nPython takes the cpp module from sys.modules and looks for magic inside it. If it finds it, it binds that to the name magic.\nLastly, Python modules may define a __getattr__(name) function. If it is defined, it is called whenever we try to import a name that isn\u0026rsquo;t present in the module.\n1 magic = cpp.__getattr__(\u0026#34;magic\u0026#34;) So, as you can see - import can be a function call!\n1 2 3 4 5 6 7 8 9 def _magic(): calling_module = get_calling_module() decorate_module_functions(calling_module) def __getattr__(name): if name != \u0026#34;magic\u0026#34;: raise AttributeError() _magic() This converts the import to a call and allows C++ to work it\u0026rsquo;s magic.\n1 2 3 4 5 6 7 8 9 from cpp import magic from greeter import Greeter def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) main() Well, almost\u0026hellip;\nYou see, since we\u0026rsquo;re imported on the first line of the module, the module is empty. The functions we want to decorate are not yet defined. To fix this, we can do one of two things.\nThe first option is to import our magic at the end of the file\n1 2 3 4 5 6 7 8 9 from greeter import Greeter def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) from cpp import magic main() This works, but feels far from magical.\nThe other option, then, is to import the modules ourselves! With the fully imported module at hand, we can modify it as we wish.\nImport Cycles This means that our _magic() function is going to change a little:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import importlib.util import sys def import_by_path(name: str, path: str): spec = importlib.util.spec_from_file_location(name, path) module = importlib.util.module_from_spec(spec) sys.modules[name] = module spec.loader.exec_module(module) return module def _magic(): calling_module = get_calling_module() name = calling_module.__name__ path = calling_module.__file__ imported_module = import_by_path(name, path) decorate_module_functions(imported_module) Using Python\u0026rsquo;s import mechanisms, we import another instance of the module that imported us.\nWe use it\u0026rsquo;s name and path to import it again, then store it in the global module cache instead of the original.\nThis is where the fact that the cache is filled prior to module execution comes in handy!\nWe execute the module, to define all the types, and then decorate the functions.\nWith this change, we can now write the import at the top again:\n1 2 3 4 5 6 7 8 9 from cpp import magic from greeter import Greeter def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) main() And recurse infinitely.\nOur module runs the magic() function. The magic() function imports our module. The module runs the magic() function. The magic() function imports our module.\nAnd so on and so forth.\nTo fix that, we add a flag to all the modules we import, before executing them. Then, in our magic() function, we check for the flag.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 IMPORT_FLAG = \u0026#34;__magically_imported__\u0026#34; def import_by_path(name: str, path: str): spec = importlib.util.spec_from_file_location(name, path) module = importlib.util.module_from_spec(spec) sys.modules[name] = module setattr(module, IMPORT_FLAG, True) spec.loader.exec_module(module) return module def _magic(): calling_module = get_calling_module() name = calling_module.__name__ path = calling_module.__file__ if hasattr(calling_module, IMPORT_FLAG): return imported_module = import_by_path(name, path) decorate_module_functions(imported_module) This breaks the recursion, but we still have an issue.\nOnce we finish all of our import magic, we return to the module that triggered the magic. This module has not yet been modified. Once we return to it, it\u0026rsquo;ll run to completion. In the case of our main module - main() will run twice. First, when we import it inside magic(). Second, when we return to the main module and let it execute. In both cases, we\u0026rsquo;ll be running the non-decorated version. This is not at all what we want.\nTo avoid this sort of thing, Python code usually uses the following:\n1 2 if __name__ == \u0026#34;__main__\u0026#34;: main() __name__ always holds the name of the current module. In the case of the main module, it\u0026rsquo;ll be \u0026quot;__main__\u0026quot;.\nThis ensures that when a module is imported (and used as a library) it will not run the main() function.\nIn our case, this will not be enough. First, we actually do import the module. Second, we need to ensure that once we\u0026rsquo;re done, the original doesn\u0026rsquo;t run.\nSo once again, we modify our magic function!\n1 2 3 4 5 6 7 8 9 10 11 12 13 def _magic(): calling_module = get_calling_module() name = calling_module.__name__ path = calling_module.__file__ if hasattr(calling_module, IMPORT_FLAG): return imported_module = import_by_path(name, path) decorate_module_functions(imported_module) if imported_module.__name__ == \u0026#34;__main__\u0026#34;: sys.exit(imported_module.main()) Instead of preventing main() from running, we call it explicitly. That means that, like in C++ code, we don\u0026rsquo;t need to call main() explicitly in our code.\n1 2 3 4 5 6 7 from cpp import magic from greeter import Greeter def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) When we\u0026rsquo;re done, we call sys.exit to terminate the process, and never actually reach the module that initially imported us.\nClass Boilerplate Another thing we want to address is the code inside our Greeter. We are currently writing a lot of boilerplate there. We have the __enter__ method, the unused __exit__ arguments, and pushing the instance into the dtor stack.\nWith a bit of inheritance, we can move all that boilerplate out of Greeter.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 from cpp import magic class CppClass: def __init__(self, *args, **kwargs): push_dtor(self) ctor = getattr(self, self.__class__.__name__, None) if ctor: ctor(*args, **kwargs) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): dtor = getattr(self, \u0026#34;_\u0026#34; + self.__class__.__name__, None) if dtor: dtor() class Greeter(CppClass): def Greeter(self, name): self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) def _Greeter(self): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) That\u0026rsquo;s great. Now as we add more classes, we don\u0026rsquo;t need to handle all that annoying dtor-stack stuff. As a bonus - our constructor is now named Greeter, as it would be in C++; and our destructor is _Greeter, as ~ is not valid in Python identifiers.\nBut\u0026hellip; We\u0026rsquo;re still missing something. We decorated all free functions with cpp_function, but we still need to decorate all of our member functions.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 def decorate_object_methods(obj): for name, value in inspect.getmembers(obj): # Ignore magic methods if name.startswith(\u0026#34;__\u0026#34;): continue if not inspect.isroutine(value): continue setattr(self, name, cpp_function(value)) class CppClass: def __init__(self, *args, **kwargs): push_dtor(self) decorate_object_methods(self) ctor = getattr(self, self.__class__.__name__, None) if ctor: ctor(*args, **kwargs) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): dtor = getattr(self, \u0026#34;_\u0026#34; + self.__class__.__name__, None) if dtor: dtor() This is similar to what we did with the modules, but this time we check for magic methods, as we don\u0026rsquo;t wanna decorate them.\nLast but not least - we want to make it truly implicit.\nCurrently, we use inheritance explicitly to extend our Greeter class with the CppClass methods. In essence, we\u0026rsquo;re injecting 3 methods into our Greeter class - __init__, __enter__, and __exit__. We\u0026rsquo;re using inheritance, but we can use a decorator just as well. We take the class, make the relevant modifications, then return the modified version.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 def cpp_class(cls): def __init__(self, *args, **kwargs): ... def __enter__(self): ... def __exit__(self, exc_type, exc_val, exc_tb): ... cls.__init__ = __init__ cls.__enter__ = __enter__ cls.__exit__ = __exit__ return cls @cpp_class class Greeter: ... This might look a bit funky, but it integrates well with our previous work decorating all free functions.\nA few extra touches are moving the method decoration out of the __init__ method, so that it only happens once; and adding a __cpp_class__ attribute so that our code can tell it\u0026rsquo;s actually a C++-style class.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 def cpp_class(cls): decorate_object_methods(cls) def __init__(self, *args, **kwargs): ... def __enter__(self): ... def __exit__(self, exc_type, exc_val, exc_tb): ... cls.__init__ = __init__ cls.__enter__ = __enter__ cls.__exit__ = __exit__ cls.__cpp_class__ = True return cls @cpp_class class Greeter: ... Lastly, we modify _magic to decorate classes as well:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 def decorate_module_classes(module): for name, value in inspect.getmembers(module): if not inspect.isclass(value): continue # Only convert functions that were defined in the importing file. # We don\u0026#39;t want to convert library imports and the likes of those. if inspect.getmodule(value) != module: continue setattr(module, name, cpp_class(value)) def _magic(): ... decorate_module_classes(module) ... And with that, are classes are automatically converted to C++ classes!\n1 2 3 4 5 6 7 8 9 10 11 12 13 from cpp import magic class Greeter: def Greeter(self, name): self.name = name print(f\u0026#34;Hello, {self.name}!\u0026#34;) def _Greeter(self): print(f\u0026#34;Goodbye, {self.name}.\u0026#34;) def main(): greeter1 = Greeter(1) greeter2 = Greeter(2) Composable We\u0026rsquo;ve come a long way so far, and our destructors are called automatically and implicitly. The next step is making our classes composable. Or, more specifically - handling members properly.\nBefore we do that, let\u0026rsquo;s see where we are now:\n1 2 3 4 5 6 7 8 9 10 11 12 class BetterArchiveReader: zipfile: ZipFile # This is a type annotation, it does nothing. def BetterArchiveReader(self, path: str): self.zipfile = ZipFile(path) def read(self, name: str): with self.zipfile.open(name) as f: # Resource management return f.read() def _BetterArchiveReader(self): self.zipfile.close() # More resource management We have fewer lines, and fewer resource-management lines. That\u0026rsquo;s great. But we also have a new issue - double free! When we create the ZipFile object, it gets pushed into the dtor-scope of the BetterArchiveReader ctor. Later, when we get to the dtor and call .close() on it, it is already closed!\nTo fix that, we need to somehow remove the ZipFile from the function scope, and let the BetterArchiveReader handle it instead. So something like:\n1 2 3 4 5 6 7 8 class BetterArchiveReader: ... def BetterArchiveReader(self, path: str): self.zipfile = ZipFile(path) remove_dtor(self.zipfile) ... Which would call a .remove method on the current DtorScope:\n1 2 3 4 5 class DtorScope: stack: list ... def remove(self, cm): self.stack.remove(cm) Which will, unfortunately, fail. list.remove() checks for equality, not identity. So if we have multiple object that compare equal - we\u0026rsquo;re in for some painful debugging. To remedy this, we\u0026rsquo;ll borrow a trick from the C++ playbook and use a comparison object:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class IdentityComparator: def __init__(self, obj): self.obj = obj def __eq__(self, other): return self.obj is other class DtorScope: ... def remove(self, cm): self.stack.remove(IdentityComparator(cm)) def remove_dtor(cm): get_dtor_stack()[-1].remove(cm) IdentityComparator wraps the object we provide, replacing the equality check with an identity check, resolving the issue.\nAnd while ZipFile does have a .close method, not all context-managers do. So we\u0026rsquo;ll make our code a bit more generic with a call to __exit__:\n1 2 3 ... def _BetterArchiveReader(self): self.zipfile.__exit__(None, None, None) That\u0026rsquo;s better, but we\u0026rsquo;re back to managing the dtors of our objects manually again. Instead, we want assignment to a class member to handle all that automatically.\nDescriptors Unlike C++, Python does not have any form of assignment operators. So we can\u0026rsquo;t use those. It does, however, have setters and getters. Before we dive into that bit of syntax, let\u0026rsquo;s look at what we want them to do:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 def is_cpp_class(cls): # Use the flag we set earlier to detect C++ classes return getattr(cls, \u0026#34;__cpp_class__\u0026#34;, False) def get_zipfile(self): # Nothing to do on get - the class maintains ownership. return getattr(self, \u0026#34;zipfile\u0026#34;) def set_zipfile(self, zipfile): old = getattr(self, \u0026#34;zipfile\u0026#34;, None) # If the old object a C++ class, we destruct it if is_cpp_class(old): old.__exit__(None, None, None) # Then we remove the new object from the scope of the # function it was assigned in. if is_cpp_class(zipfile): remove_dtor(zipfile) setattr(self, \u0026#34;zipfile\u0026#34;, zipfile) Naturally, we don\u0026rsquo;t wanna be writing and calling those methods everywhere. To circumvent that, we\u0026rsquo;ll use \u0026ldquo;descriptors\u0026rdquo;, Python\u0026rsquo;s flavour of getters and setters.\nAny class member variable (class, not instance) that implements __set__ and __get__ is a descriptor. When we later access those variables through an instance, those methods are called to set or get the value from the member. Adjusting our code to use them, we get:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 class CppMember: def __set_name__(self, owner, name): # Called on class creation self.private_name = \u0026#34;_\u0026#34; + name def __get__(self, instance, owner=None): # Called on an instance, when getting the value return getattr(instance, self.private_name) def __set__(self, instance, value): # Called on an instance, when setting the value old = getattr(instance, self.private_name, None) if is_cpp_class(old): old.__exit__(None, None, None) if is_cpp_class(value): remove_dtor(value) setattr(instance, self.private_name, value) class Reader: # Create zipfile as a class member variable zipfile = CppMember() def Reader(self, path): self.zipfile = ZipFile(path) __set_name__ is another descriptor method. It is called on class creation (when Python creates the class object, not an instance) to pass the variable name to the descriptor. Then, the descriptor can derive a unique name from it and use it to store the actual value in the instance.\nThis takes care of the function-scope. The next thing to do is make sure when our destructor is called, we call all the member destructors as well. So in cpp_class, we add a call the member destructors after dealing with the class destructor.\n1 2 3 4 5 6 7 8 9 10 11 12 13 def __exit__(self, exc_type, exc_val, exc_tb): ... # Handle class dtor first # Then handle member dtors in reversed definition order for name, value in reversed(inspect.getmembers(self)): if name.startswith(\u0026#34;_\u0026#34;): continue if not is_cpp_class(value): continue value.__exit__(None, None, None) This leaves us with much nicer code:\n1 2 3 4 5 6 7 8 class BetterArchiveReader: zipfile = CppMember() def BetterArchiveReader(self, path): self.zipfile = ZipFile(path) def read(self, name: str): ... We just assign a ZipFile to a member, and we\u0026rsquo;re good to go. Our member destructors are called automatically.\nAnnotations But\u0026hellip; That zipfile = CppMember() still bothers me. It shows some plumbing that is better hidden. And we can do better.\nPython has type annotations. They look something like:\n1 2 class SomeClass: name: SomeType They don\u0026rsquo;t really do anything, but once we use them, Python stores a mapping between name and SomeType inside our SomeClass object. And if we have a mapping, we can query it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 def create_members(cls): # Query the the member variable annotations for the given class member_names = list(getattr(cls, \u0026#39;__annotations__\u0026#39;, {})) for name in member_names: # Create a new CppMember Descriptor object member = CppMember() # Since we\u0026#39;re modifying an already-created class, # we need to call `__set_name__` manually here member.__set_name__(cls, name) # Assign the newly created descriptor to the class member. setattr(cls, name, member) # This will be useful later setattr(cls, \u0026#39;__member_names__\u0026#39;, member_names) def cpp_class(cls): ... create_members(cls) ... def __exit__(self, exc_type, exc_val, exc_tb): ... # We have a list of all members now, so no need # to use reflection to get it. for name in reversed(self.__member_names__): value = getattr(self, name, None) if is_cpp_class(value): value.__exit__(None, None, None) And with that, we\u0026rsquo;ve reached our goal! We can now write our BestArchiveReader as if it was a C++ class:\n1 2 3 4 5 6 7 8 9 class BestArchiveReader: zipfile: ZipFile def BestArchiveReader(self, path: str): self.zipfile = ZipFile(path) def read(self, name: str): with self.zipfile.open(name) as f: return f.read() Wrapping Up We started by looking at Python\u0026rsquo;s take on destructors, and the issues it poses. Then, we took inspiration from C++ and gradually built a \u0026ldquo;better\u0026rdquo; solution. We focused on 3 main goals, and achieved them:\nAutomatic: destructors are called when / where they are needed. Composable: members work automatically, with no added boilerplate. Implicit: no extra code; no changes to interfaces; no interface pollution. But\u0026hellip; Some of you may have noticed a glaring issue. Once a value is created in a function, or assigned to a member variable - we can no longer move it. See, we have ownership, but no move semantics.\nThere are some extra tricks we can use (and are used in the library) for returning values, but I don\u0026rsquo;t think the general case is elegantly solvable.\nFurther Reading First, thank you for reading this entire post. If you want to play with the concepts I presented here, check the full implementation, you can check it out on my Github. You\u0026rsquo;ll also find there an implementation of access specifiers, which you\u0026rsquo;re sure to enjoy.\nThis was a fun experiment when I first started playing with this, and a fun talk to present. I hope you have fun with this too.\n","permalink":"https://tamir.dev/posts/cpppy/","summary":"Writeup of my talk about the cpppy library\nA few years ago I gave a talk about implementing C++ semantics (namely, destructors) in Python. After giving the talk, I published my speaker notes and slides, to make it as accessible as possible, even to people who don\u0026rsquo;t wanna watch a video. I was quite pleased with myself, until a few weeks ago when I wanted to look up a detail so that I can reference it in a new blogpost.","title":"Implementing C++ Semantics in Python - a writeup"},{"content":"Once I define a model for thinking, I find that it is good to keep running it over in your mind, and see how it applies to different situations. To see if it holds for all cases, or if there are edge-cases that make it fall apart; or to see if there are interesting things to learn when applying it to real-world scenarios.\nSo, having talked about resources in my previous post, I spend some time thinking about interesting edge-cases with resource-management in GC languages. After a while, I thought about mutexes and locks. They are an interesting example as they are usually made up of 2 separate resources.\nLocks \u0026amp; Mutexes When we think about locks, there is one obvious acquire-use-release cycle we think about. We acquire the lock, use the resource it guards, then release the lock when we\u0026rsquo;re done.\n1 2 3 with thing.lock: # Acquire the lock thing.modify() # Use the guarded resource # Release the lock as we exit the scope But in most cases, we have another resource - the lock itself. On Windows, for example, a mutex can be creates using the CreateMutex function. That, in turn, returns a handle to the OS-level mutex. When we\u0026rsquo;re done using it, we need to call CloseHandle to close the handle and have the OS release the mutex. But\u0026hellip; I never released a mutex in Python. So what\u0026rsquo;s going on?\nPython\u0026rsquo;s threading.Lock First, I wanted to make sure Python\u0026rsquo;s threading.Lock is indeed backed by an OS-level mutex. I could read the code, but there are simpler ways.\nProcess Explorer allows us to see the number of handles used by a process. So if we create some locks, we should see the handle-count increasing.\n1 2 3 4 5 6 7 8 from threading import Lock # Wait for input, so that we can open the process in ProcExp input() # Create 100 locks locks = [Lock() for _ in range(100)] # Wait for input, so that we can see the increase input() At first, we see that we have 92 handles: Then we press Enter, and see the number increase to 192: Which is what we expected. This indicates that Python\u0026rsquo;s threading.Lock is indeed backed by an OS mutex in Windows. But\u0026hellip; We never release those mutexes ourselves, so what is going on?\nFirst, we need to make sure Python does not simply leak those resources. By adding the following lines to our code:\n1 2 3 # Delete the locks, so that they can be released del locks input() We can see that after del locks, the handle count drops back down to 92. So how does this happen?\nFinalizers \u0026amp; The __del__ Method To handle such cases, Python uses finalizers, implemented using the __del__ method. The __del__ method is called when an object is about to be destroyed. As Python is a GC language, the language runtime owns the objects, and is responsible for destroying them at its own time. This means that while it can be called immediately after we run del locks, it can also be called much later, or not at all. Our only guarantee is that if and when the GC reclaims the memory of our object, it\u0026rsquo;ll also call the __del__ method to clean it up properly.\nThis is bad for unique-resources, and is why we use a context-manager to manage the lock itself. But for counted-resources, like the OS handle representing the mutex itself - this is a good solution. Even if the finalizer is not called before our process terminates, the OS itself will be sure to release those handles upon process termination.\nLessons Learned From this we learn a new lesson about Python, and our model. For unique-resources, we want to use context-managers, to ensure release at the right time. For counted-resources, we can use finalizers, and know that it\u0026rsquo;s good enough. In real code, we\u0026rsquo;ll see that finalizers are usually used to wrap OS resources, and almost never user-defined-resources.\nOther Languages Now that we know how Python handles counted-resources, we can have a look at some more languages.\nJava \u0026amp; C# In Java, things are roughly the same. It\u0026rsquo;s another object-oriented language, and it uses finalizers for the same purposes. C# behaves similarly, being another object-oriented language. I will venture a guess, and say that most GC object-oriented languages use finalizers.\nBut, not all GC languages are object-oriented. Go, for example, is not. As such, it is interesting to see how it tackles the same problem.\nGo Looking at A Tour of Go, we see that mutexes are used as simple structs. There\u0026rsquo;s no NewMutex() function nor CloseMutex(). This is odd, as Go does not have finalizers. But, looking at the implementation of the sync.Mutex struct things become clear:\n1 2 3 4 type Mutex struct { state int32 sema uint32 } Go mutexes are not backed by OS mutexes. This is probably because Go uses goroutines instead of OS-level threads, and manages its own concurrency. Nice going, Go!\nBut what of other OS-level objects? Go can make its own mutexes, but sockets, for example, must go through the OS.\nSockets In Python, sockets are managed the same way mutexes are. You can change the threading.Lock to socket.socket in the code we used above to see the effects. In Go, however, they opted for a different solution. Instead of having a \u0026ldquo;socket\u0026rdquo; object, they have \u0026ldquo;connection\u0026rdquo; objects. Those connection objects are obviously unique-resources, as one needs to close their connections (that\u0026rsquo;s true in Python as well!). So by modifying the API, they remain consistent and avoid leaks.\nFinal Words By taking our model and applying to it different situations, we managed to learn some new things and reach new insights.\nWe learned about finalizers, and how they are used to manage counted-resources in GC languages. And we learned about Go\u0026rsquo;s scheduling, with their own mutexes; and their clever choices in the socket API to handle the lack of finalizers.\nAs we keep pushing our understanding of programming languages, and trying to apply it to more situations, I\u0026rsquo;m sure we\u0026rsquo;ll learn a lot more.\n","permalink":"https://tamir.dev/posts/resource-management-02/","summary":"Once I define a model for thinking, I find that it is good to keep running it over in your mind, and see how it applies to different situations. To see if it holds for all cases, or if there are edge-cases that make it fall apart; or to see if there are interesting things to learn when applying it to real-world scenarios.\nSo, having talked about resources in my previous post, I spend some time thinking about interesting edge-cases with resource-management in GC languages.","title":"Finalizing Mutexes"},{"content":"In the last couple of years, I\u0026rsquo;ve been thinking a lot about resource management in different programming languages. The parts I\u0026rsquo;m happy with, the parts I wanna rant about, and the tradeoffs at play. I have some topics in that space that I want to talk about, but it constantly feels like I need to give way too much context to even get started. So today, I want to share some of my mental model for resource management. Both to enable me to draw on it later; and to let others discuss it in similar terms.\nAs we\u0026rsquo;re going to be talking about resources, the first thing to do is clarify what I mean when I write \u0026ldquo;resource\u0026rdquo;, and a few other useful definitions.\nDefinitions A resource is anything that has a acquire-use-release cycle. It can be memory, allocated, used, then released; it can be a lock, acquired to access a shared state; it can be a file, opened for a read operation, then closed. If it fits the pattern, it\u0026rsquo;s a resource.\nResources can generally be split into two categories: counted resources and unique resources. Counted resources are resources with a finite supply, but where the identity of the resource itself is immaterial. Memory is a counted resource - it\u0026rsquo;s a problem if it runs out, but I don\u0026rsquo;t generally care about the specific memory addresses. Another example, specifically on Windows, is handles1. We only care about them when they run out. With unique resources, however, we care deeply about the identity of the resource. Think about locks, or files, or database transactions. No two are the same. This leads to a bit of a difference in handling them. While counted resources can be released at any time before we run out without negative effects; unique resources need to be released as soon as we\u0026rsquo;re done using them, or problems are likely to occur.\nResources are, from acquisition to release, owned. Ownership of a resource is the unique responsibility to release it.\nResource management is the art of wrangling resources and their ownership. There are many approaches to resource management, and we\u0026rsquo;ll review some of them. Our motivation, as we do so, is to reach an ideal of resource management. Our goal is to have clear and transferrable ownership of resources. Clear ownership means that the question \u0026ldquo;do I own this resource?\u0026rdquo; can be answered locally, and without ambiguity. This is critical for the proper operation of a program, as ambiguity leads to confusion and bugs. Transferrable ownership is the ability to move the ownership from one owner to another. This is a key component in being able to return resources from functions. As we move forward, we\u0026rsquo;ll see that there\u0026rsquo;s often a tradeoff between clarity and transferability.\nDifferent Models Now, with our definitions and goals in mind, we can start looking into different techniques for managing resources. Specifically, we\u0026rsquo;ll focus on the solutions provided by programming languages, rather than coding techniques to work with them.\nManual Manual management of resources is the simplest model. As such, it is available in most any programming language. It gives the programmer two primitives - acquire and release - and leaves them to manage everything on their own.\nOn the positive side - we have full control. We can easily transfer ownership of a resource as a function argument, or a return value.\n1 2 3 4 5 6 7 8 9 10 def open_config(): config_file = open(\u0026#34;config path\u0026#34;) return config_file # Here we transfer ownership to the caller. def close_config(config_file): config_file.close() def main(): config_file = open_config() # Transfer ownership from callee close_config(config_file) # Transfer ownership to callee On the other hand - we can\u0026rsquo;t do local reasoning to deduce the owner. We have to read documentation, or other code, and do complicated bookkeeping, just to know if we need to release a resource.\nUsually, when we think of the downsides of manual management, we think of C\u0026rsquo;s manual memory management, and the issues that arise there. Considering the following examples, can you easily tell which memory needs to be freed?\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 void example(char* ptr) { // ptr is a function argument. We must document whether we take // ownership when called, as there\u0026#39;s no way to deduce it. // p is the sole pointer to the memory, so we must call `free(p)` // before leaving the scope. void* p = malloc(10); // Define a static string. const char* greeting = \u0026#34;Hello, Bugs!\u0026#34;; // strdup allocates internally, so we need to `free()` the memory. char* greeting2 = strdup(greeting); // strstr returns a pointer to our static string, // so we don\u0026#39;t `free()` it. char* bugs = strstr(greeting, \u0026#34;Bugs\u0026#34;); // inet_ntoa returns a pointer to a _shared_ buffer. // Not only we can\u0026#39;t free it, but the next call to the function // will change the value we\u0026#39;re already holding. char* text_address = inet_ntoa(ip_address); } Scoped With scoped ownership, a resource is owned by a lexical scope. This is usually a block or a function, depending on the language. It is expressed differently in different languages, but the idea is the same. The resource is:\nis acquired before or on entering the scope; can be used inside the scope; automatically released on leaving the scope. In Python2, this is expressed as:\n1 2 3 4 5 6 # The call to `open` acquires the file, # then the `with` statement binds it to the scope. with open(\u0026#34;my file\u0026#34;) as f: # The file is used inside the scope. f.read() # Upon leaving the scope, the file is closed. This makes ownership very clear. Look at the code - if you see an owning scope, it owns the resource. We never have to go beyond the current function to infer ownership.\nOn the other hand, this makes transferring resources impossible. Once a scope owns a resource, it will be sure to release it. If we return to our previous Python example, we\u0026rsquo;ll see that we can no longer return our resource from our function:\n1 2 3 4 5 6 def open_config(): with open(\u0026#34;config path\u0026#34;) as config_file: return config_file # We return a file, but it\u0026#39;ll be closed. def main(): config_file = open_config() # Here the file is already closed! We can easily pass resources to our callees, without transferring ownership:\n1 2 with open(\u0026#34;my file\u0026#34;) as file: print_file(file) # No transfer of ownership. But the other direction is now impossible.\nWe traded transferability for clarity. As a result, we have fewer bugs, but also reduced control. So while this is a good approach, it cannot fully replace manual management.\nGarbage Collection With garbage collection we take an entirely different approach, circumventing the issue. Instead of transferring ownership from one place to the other, we have a single owner for everything. The language runtime owns the memory.\nScoped management kept you from returning resources from functions? That\u0026rsquo;s no longer an issue - the runtime will hold them for you!\nManual management had you confused about ownership? Not an issue - the runtime owns your memory!\n1 2 3 4 5 6 7 def f(): # Return a value, not an issue! return \u0026#34;This is a string, backed by runtime-owned memory!\u0026#34; def g(x): # The runtime owns `x`, so don\u0026#39;t worry about it! pass But you\u0026rsquo;ve probably noticed me cheating here. Before, we talked about resources. Now, we\u0026rsquo;re only talking about memory.\nThis is because memory, being a counted resource, is inherently different from the unique resources that make up what we usually think of when we say \u0026ldquo;resource\u0026rdquo;. Since we only care about having enough memory, it\u0026rsquo;s ok if the GC take a moment, or even a long moment, before it releases it. There are even situations where it\u0026rsquo;s ok if it never releases that memory.\nWith unique resources, this doesn\u0026rsquo;t work. If you hold a lock, you need it to be released now. If you\u0026rsquo;re managing a database transaction, you want to finalize it before the next one. You need control, and a GC (Garbage Collector) doesn\u0026rsquo;t give you that.\nIn this case, the tradeoff between clarity \u0026amp; transferability, between simplicity and power, is entirely untenable for some use-cases. Unlike the manual-or-scoped tradeoff, this one cannot be circumvented by clever design. That\u0026rsquo;s why we only have it for memory management.\nMove Semantics (or \u0026ldquo;whatever Rust does\u0026rdquo;) Instead of going into the weeds of C++\u0026rsquo;s move semantics or Rust\u0026rsquo;s ownership model, I\u0026rsquo;ll lay out the basic principles, as a combination of concepts discussed before.\nAll resources are scope-managed and scope-owned. Always and by default. Ownership can be transferred clearly to a different scope. This means that we get our goal - clear and transferrable ownership. We can easily deduce the owner when looking at a piece of code, and we can transfer the ownership to a different scope if needed. Unfortunately, this doesn\u0026rsquo;t come free. We have to explicitly reason about, and decide, on things that were implicit before.\nConsider a linked list for example. With a singly-linked list, we can say that the head owns the first node, which owns the next node, and so on. But with a doubly-linked list, there is no \u0026ldquo;obvious\u0026rdquo; solution. Is a node owned by the next node, or the previous one? Being owned by both would be an issue, as we already said there can be only one owner. Such issues make this model considerably harder to learn and adapt to.\nConclusion Now that we\u0026rsquo;ve seen all four models, we can put them in a nice table to compare them:\nModel Clear Transferrable Simple Complete Manual Yes Yes Scoped Yes Yes Maybe? GC Yes Yes Move Yes Yes Maybe? I added two columns that we did not discuss explicitly. \u0026ldquo;Simple\u0026rdquo; is for simplicity, which we mentioned throughout. \u0026ldquo;Complete\u0026rdquo; is whether we can do with that model alone, and not require the others. Scoped get\u0026rsquo;s a \u0026ldquo;maybe\u0026rdquo; here because if you use enough Dependency Injection you might just be able to get it working. That said, I don\u0026rsquo;t recommend it.\nWith no single model getting a \u0026ldquo;yes\u0026rdquo; on all columns, we\u0026rsquo;ve yet to find the \u0026ldquo;best solution\u0026rdquo;. Until then, we\u0026rsquo;ll have to keep mixing and matching to solve the problems we\u0026rsquo;re facing to the best of our ability.\nSee Why is the limit of window handles per process 10,000? and Pushing the Limits of Windows: Handles\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSee Python with statement\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/resource-management-01/","summary":"In the last couple of years, I\u0026rsquo;ve been thinking a lot about resource management in different programming languages. The parts I\u0026rsquo;m happy with, the parts I wanna rant about, and the tradeoffs at play. I have some topics in that space that I want to talk about, but it constantly feels like I need to give way too much context to even get started. So today, I want to share some of my mental model for resource management.","title":"Resource Management Models"},{"content":"range over func A few days ago, Russ Cox published a proposal for range over func in Go. Given my deep interest in iteration (slides, code \u0026amp; full speaker notes are available here), I had to take a look. At first, it struck me as a bit odd. If you want to have a look yourself, take a look at the proposed spec. I\u0026rsquo;ll just show an example - a Go equivalent of Python\u0026rsquo;s range \u0026ldquo;function\u0026rdquo;:\n1 2 3 4 5 6 7 8 9 10 func Range(start, stop, step int) func(func(int) bool) bool { return func(yield func(int) bool) bool { for i := start; i \u0026lt; stop; i += step { if !yield(i) { return false } } return true } } Which, translated to Python, would be\n1 2 3 4 5 6 7 8 9 10 11 12 def go_range( start: int, stop: int, step: int ) -\u0026gt; Callable[[Callable[[int], bool]], bool]: def _impl(yield_: Callable[[int], bool]) -\u0026gt; bool: i = start while i \u0026lt; stop: if not yield_(i): return False i += step return True return _impl Which is quite a mouthful. It works by having the Go compiler convert a loop:\n1 2 3 for i := range Range(4, 10, 2) { fmt.Println(i) } Into a function call:\n1 2 3 4 5 yield := func(i int) bool { fmt.Println(i) return true } Range(4, 10, 2)(yield) With some interesting semantics:\nWhen breaking from the loop in any way (be it break, return, or goto), the generated yield function returns false to signal the iterator function to stop; The iterator function should return true if iteration completed successfully, or false if it was stopped; Any call to yield would trigger another iteration of the loop. That last part is of particular interest - it means that the iterator function can completely ignore break, and keep iterating indefinitely.\nContext Managers in Go While that last part felt wrong to me (as it can cause bugs that take forever to discover), it also felt a bit familiar. It took me some time, playing with the code, before it clicked! This guarantee that the iterator function must return before leaving the loop is akin to Python\u0026rsquo;s context managers. Looking at the Go iterator function structure, you can see some similarities to context-managers written with contextlib.contextmanager (I know, I know. But it looked similar to me.) The main difference being that unlike in Python, in Go we also control iteration (yes, if any of you want \u0026ldquo;context managers that can re-run the code\u0026rdquo;, push for this feature in Go).\nExamples with file(\u0026hellip;): This means that I can finally write context-managers in Go, and get rid of that terrible (sorry Gophers) create-and-defer-close pattern that\u0026rsquo;s everywhere!\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 func createFile(path string) func(func(*os.File, error)bool)bool { return func(yield func(*os.File, error) bool) bool { file, err := os.Create(path) defer file.Close() return yield(file, err) } } func main() { for file, err := range createFile(\u0026#34;SoCool.txt\u0026#34;) { if err != nil { fmt.Println(err) break } fmt.Fprintln(file, \u0026#34;Hello, World!\u0026#34;) } } with Transaction(\u0026hellip;): If we want to go beyond that, we can have context managers for transactions (commit on successfully leaving the loop, cancel on panic):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 func WithTransaction(name string) func(yield func(*Transaction) bool) bool { return func(yield func(*Transaction) bool) bool { t := Transaction{} // Create the transaction object defer func() { if r := recover(); r != nil { // Cancel \u0026amp; \u0026#34;re-raise\u0026#34; on panic cancel(name, t) panic(r) } else { // Commit on \u0026#34;normal\u0026#34; loop exit commit(name, t) } }() return yield(\u0026amp;t) // Pass transaction to the \u0026#34;loop\u0026#34; } } type Transaction []string func (t *Transaction) Add(item string) { *t = append(*t, item) } func commit(name string, t Transaction) { fmt.Printf(\u0026#34;%s: committing...\\n\u0026#34;, name) for _, item := range t { fmt.Printf(\u0026#34;%s: %s\\n\u0026#34;, name, item) } } func cancel(name string, t Transaction) { fmt.Printf(\u0026#34;%s: cancelled\\n\u0026#34;, name) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 func main() { // This transaction will commit for t := range WithTransaction(\u0026#34;Commit successfully\u0026#34;) { t.Add(\u0026#34;Item 1\u0026#34;) t.Add(\u0026#34;Item 2\u0026#34;) } // And this will be cancelled for t := range WithTransactionPanic(\u0026#34;Panic\u0026#34;) { t.Add(\u0026#34;Item 1\u0026#34;) t.Add(\u0026#34;Item 2\u0026#34;) panic(\u0026#34;at the Disco\u0026#34;) } } with suppress(\u0026hellip;): Or emulate Python\u0026rsquo;s contextlib.suppress context-manager (despite it being entirely un-Go-like):\n1 2 3 4 5 6 7 8 9 10 11 12 func Suppress(err error) func(yield func() bool) bool { return func(yield func() bool) bool { defer func() { if r := recover(); r != nil { if r, ok := r.(error); !ok || !errors.Is(r, err) { panic(r) } } }() return yield() } } 1 2 3 4 5 for range Suppress(SomeError) { panic(SomeError) } fmt.Println(\u0026#34;This will run!\u0026#34;) Go and play! I don\u0026rsquo;t know if this feature and design will make it into Go or not. Even if it does, I don\u0026rsquo;t know what patterns of use will emerge. All that said - I think being able to write Python-style context-managers in Go is worth playing around with, and I hope more people would experiment with it!\n","permalink":"https://tamir.dev/posts/context-managers-in-go/","summary":"range over func A few days ago, Russ Cox published a proposal for range over func in Go. Given my deep interest in iteration (slides, code \u0026amp; full speaker notes are available here), I had to take a look. At first, it struck me as a bit odd. If you want to have a look yourself, take a look at the proposed spec. I\u0026rsquo;ll just show an example - a Go equivalent of Python\u0026rsquo;s range \u0026ldquo;function\u0026rdquo;:","title":"Context Managers in Go"},{"content":"This is another post converted from a twitter thread. The original was published in 2018.\nReading a lot of C++ code lately, there are quite a few features I miss from tools for binary reverse engineering.\nInstant renaming Want to give a function a name that is more indicative for you? Go ahead. Just hit \u0026lsquo;rename\u0026rsquo;, rename, and you\u0026rsquo;re done. No waiting, no missed spots. Everything renamed instantly. Even in projects with \u0026gt;300k functions.\nInstant xrefs (uses) for identifiers Both globally and inside functions. Just click, and you have it. No waiting. Sure, it can miss in some cases if your analysis isn\u0026rsquo;t good enough, but that\u0026rsquo;s just because it\u0026rsquo;s binary analysis.\nRepeatable function comments Tired of going into a function, or hovering it, to get the documentation?\nRepeatable functions show on every call site. Write the relevant notes there and see them everywhere. Jumping into functions is mentally expensive.\nGraph-view / function flow graphs You can instantly get an estimate of the complexity \u0026amp; flow of a function. You can visually see loops and branches. This allows you to quickly scan through functions, then zoom in on the interesting parts.\nEven in text, the gutter contains arrows indicating jumps in the code\nColors I can easily use color to add meaning to the flow-graph. I can mark lines or basic blocks in whatever color I want to instantly see meaning.\nIn the picture, all the nodes that can reach the blue node are painted green.\nNamed bookmarks Not much to show here, but being able to have a list of interesting locations + the reason they are interesting is super handy.\nComments are linkable I can \u0026ldquo;go to definition\u0026rdquo; by double-clicking on a function name in a comment. This is super useful when reading a long explanation and jumping around a bit to make sure you get it.\nAdd Missing Info Since this is a research tool, I can also teach it things it didn\u0026rsquo;t know. Like function signatures or indirect-call it failed to deduce on its own\nRename and comment without modification! A major difference from code-reading tools is that here, the code is completely static. It is what it is. You don\u0026rsquo;t change it, you don\u0026rsquo;t compile it. Comments \u0026amp; renames don\u0026rsquo;t change the code - they are only added in my view.\nSo unless you\u0026rsquo;re fixing analysis issues, you don\u0026rsquo;t need to re-analyze as much as with code-editors.\nThis also allows for more expressive function names\nSummary Instant navigation + the visual impact of the graphs make for a super-fast workflow. You can easily traverse large amounts of code while hardly reading it.\nJump, see a familiar pattern, rename function for future reference, jump to next location.\nThis kind of visual information, seeing the \u0026ldquo;shape\u0026rdquo; of a function, is missing from code-editors. You can\u0026rsquo;t zoom out to get a better perspective.\nPeter Hilton did a great job discussing the visual shortcomings of code editors in his talk Beautiful code: typography and visual programming.\nScripting support as a priority Additionally, scripting is a first-class citizen in the UI. The default workspace contains a Python CLI that can be used to query and analyze the codebase, or to automate boring tasks. CLI vs. plugins means that you can code-as-you-go, and don\u0026rsquo;t need a polished plugin with a GUI\n","permalink":"https://tamir.dev/posts/binary-re-feature-overview/","summary":"This is another post converted from a twitter thread. The original was published in 2018.\nReading a lot of C++ code lately, there are quite a few features I miss from tools for binary reverse engineering.\nInstant renaming Want to give a function a name that is more indicative for you? Go ahead. Just hit \u0026lsquo;rename\u0026rsquo;, rename, and you\u0026rsquo;re done. No waiting, no missed spots. Everything renamed instantly. Even in projects with \u0026gt;300k functions.","title":"Binary Reverse Engineering Features"},{"content":"I originally posted this as a series of tweets. With the state of Twitter, I decided to convert it to a blog post.\nSo\u0026hellip; A floating point question.\nHow many items are expected in a set (Python\u0026rsquo;s set, C++\u0026rsquo;s std::set, Go\u0026rsquo;s map[float64]bool, etc.) when I fill it with NaN values?\nThis seems to work differently in different languages.\nC++ If we run the following C++ code:\n1 2 3 4 5 6 7 8 9 10 11 12 #include \u0026lt;set\u0026gt; #include \u0026lt;cmath\u0026gt; int main() { std::set\u0026lt;float\u0026gt; floatSet; for (auto i = 0; i \u0026lt; 10; ++i) { floatSet.insert(NAN); } return floatSet.size(); // 1 } We get a single value in the set. This is a bit unexpected, as NAN is not equal to itself.\nIf we try and add more items to the set after the NAN, we\u0026rsquo;ll also see that the set is effectively broken:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 #include \u0026lt;set\u0026gt; #include \u0026lt;cmath\u0026gt; int main() { std::set\u0026lt;float\u0026gt; floatSet; floatSet.insert(NAN); for (auto i = 0; i \u0026lt; 10; ++i) { floatSet.insert(i); } return floatSet.size(); // 1 } Returns 1 as well. Note that if we first add non-NaN values to the set, it seems to work ok.\nPython In Python code, things behave a little differently.\nFilling a set with the same NaN object will give us a set with a single element:\n1 2 3 4 5 6 7 floats = set() nan = float(\u0026#34;NaN\u0026#34;) for _ in range(10): floats.add(nan) print(len(floats)) # 1 While filling it with different NaN objects will give us a set with multiple objects:\n1 2 3 4 5 6 7 floats = set() for _ in range(10): nan = float(\u0026#34;NaN\u0026#34;) floats.add(nan) print(len(floats)) # 10 You see, objects in Python sets are only required to be hashable. There is no requirement for implementing equality. So, if 2 values are the same object (spelled a is b, or id(a) == id(b)), they\u0026rsquo;ll only appear once in a set. If they are not the same - equality (if possible) will be checked.\nTotally expected behaviour.\nGo Go code seems to be the only one that behaves as expected:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 package main import \u0026#34;math\u0026#34; import \u0026#34;fmt\u0026#34; func main() { floatSet := make(map[float64]bool) nan := math.NaN() for i := 0; i \u0026lt; 10; i++ { floatSet[nan] = true } fmt.Println(len(floatSet)) // 10 } We insert \u0026ldquo;the same\u0026rdquo; NaN values, and still get 10 values in the set.\nNaN Bonus! Go is now getting a clear function added to clear maps. This is mostly required because there\u0026rsquo;s no other way to remove NaN keys from a map.\nMore Words Floating point numbers are weird. They are weird when they work correctly (thanks Go) and weirder when they don\u0026rsquo;t.\nIf you have any choice in the matter, never use them as keys. If you do, well, be prepared to have some interesting bugs.\n","permalink":"https://tamir.dev/posts/sets-of-floats/","summary":"I originally posted this as a series of tweets. With the state of Twitter, I decided to convert it to a blog post.\nSo\u0026hellip; A floating point question.\nHow many items are expected in a set (Python\u0026rsquo;s set, C++\u0026rsquo;s std::set, Go\u0026rsquo;s map[float64]bool, etc.) when I fill it with NaN values?\nThis seems to work differently in different languages.\nC++ If we run the following C++ code:\n1 2 3 4 5 6 7 8 9 10 11 12 #include \u0026lt;set\u0026gt; #include \u0026lt;cmath\u0026gt; int main() { std::set\u0026lt;float\u0026gt; floatSet; for (auto i = 0; i \u0026lt; 10; ++i) { floatSet.","title":"Sets of Floats"},{"content":"Recently I decided that I want to have comments in my blog.\nBut\u0026hellip; I don\u0026rsquo;t wanna host them. For various reasons. From maintenance and moderation to GDPR and the likes.\nInstead, I am using GitHub discussions and a bit of hacky Python to generate discussions for my posts. It is hosted in a Github action, but I really don\u0026rsquo;t recommend using it yet. It is super ad-hoc and experimental.\nBut\u0026hellip; if all went well, you\u0026rsquo;ll see a link to the discussion below the post!\n","permalink":"https://tamir.dev/posts/github-discussions/","summary":"Recently I decided that I want to have comments in my blog.\nBut\u0026hellip; I don\u0026rsquo;t wanna host them. For various reasons. From maintenance and moderation to GDPR and the likes.\nInstead, I am using GitHub discussions and a bit of hacky Python to generate discussions for my posts. It is hosted in a Github action, but I really don\u0026rsquo;t recommend using it yet. It is super ad-hoc and experimental.\nBut\u0026hellip; if all went well, you\u0026rsquo;ll see a link to the discussion below the post!","title":"Github Discussions for Hugo posts"},{"content":"Often, we write code to complete a task. Other times, we write code to learn something new. But there are also those rare occasions where our goals are simpler. When we have a dumb idea stuck in our head, and the only way to get it out is to code it. Or when we just want to see, to prove to ourselves, that we can write a certain piece of code.\nThis post is about one of those wonderful occasions. Specifically - I had to see whether I can implement C++ Scoped Enumerations in Python. And, since you\u0026rsquo;re reading this post, it\u0026rsquo;s fair to assume that I succeeded.\nC++ Scoped Enumerations If you\u0026rsquo;re not a C++ programmer (good for you!) you may be wondering - what are scoped enumerations?\nWell, first and foremost - they are enumerations. A nice syntactic sugar to define a group of named, constant integer values. They look like this:\n1 2 3 4 5 enum class Colors { Red, Green, Blue }; (Here Colors::Red is 0, Colors::Green is 1, and Colors::Blue is 2.)\nSecondly, they are scoped. This might seem obvious to anyone not coming from C (or older C++), but this means that the names only exist under the enumeration that defines them. You can write Colors::Red to get the Red value, but just writing Red somewhere won\u0026rsquo;t get it. Yes, this seems obvious, but it wasn\u0026rsquo;t the case until C++11.\nThat\u0026rsquo;s about it.\nOf course, we can talk about the fact that they are a form of a strong typdef, and disallow implicit conversions, and more\u0026hellip; But we\u0026rsquo;re to bury implement enums not to praise them.\nScoped Enums in Python Why implement this in Python, you might ask. After all, we have enum.Enum, and that\u0026rsquo;s plenty good:\n1 2 3 4 class Colors(Enum): Red = auto() Green = auto() Blue = auto() And, well, you\u0026rsquo;re right. There\u0026rsquo;s no reason to do it. It\u0026rsquo;s a bad idea. But\u0026hellip; It\u0026rsquo;s plenty fun to try.\nSo in this post we\u0026rsquo;ll make sure we can write the following example, and then go even further.\nSimple Enums 1 2 3 4 class Colors(ScopedEnum): Red Green Blue This is our first goal. We want to make sure that Colors.Red would be 0, Colors.Green is 1, and Colors.Blue is 2.\nThis might look like an impasse, if we try to access Red before defining it, we get a NameError exception. But Python is a very versatile language, and while it does its best to make well-behaved code act reasonably, it also allows for some very fun behaviour when you abuse its mechanisms.\nAmong those fun-to-abuse mechanisms, we have metaclasses. By implementing the __prepare__ method in our metaclass (read about it in the Python docs and on Brett Cannon\u0026rsquo;s blog), we can specify the namespace object to use when constructing the actual class (instead of the default dict). This means that during class construction, every name lookup will go through our object.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 class Namespace(dict): def __init__(self): super().__init__() self.last_value = -1 def __getitem__(self, name): # Ignore some special names Python uses and we don\u0026#39;t care about if name.startswith(\u0026#34;__\u0026#34;) and name.endswith(\u0026#34;__\u0026#34;): return super().__getitem__(name) self.last_value += 1 self[name] = self.last_value return class ScopedEnumMeta(type): @classmethod def __prepare__(metacls, name, bases): return Namespace() class ScopedEnum(metaclass=ScopedEnumMeta): pass By creating our own implementation of __getitem__, we can define any new variable we encounter. Thus avoiding the NameError, and defining all the members we wanted.\nAssigning Literals The next step would be allowing a bit more control for the programmer. You see, some C++ programmers want to set explicit values in their enumerations:\n1 2 3 4 5 enum class Numbers { One = 1, Five = 5, Six }; And if C++ can do it, so should Python:\n1 2 3 4 class Numbers(ScopedEnum): One = 1 Five = 5 Six To do this, we need to extend our Namespace implementation. You see, if we run it now, we get:\n1 2 \u0026gt;\u0026gt;\u0026gt; print(Numbers.One, Numbers.Five, Numbers.Six) 1 5 0 That\u0026rsquo;s because we forgot to update our last_value member on assignment. To do that, we\u0026rsquo;ll add some code into __setitem__ as well:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 class Namespace(dict): def __init__(self): super().__init__() self.last_value = -1 def __getitem__(self, name): if name.startswith(\u0026#34;__\u0026#34;) and name.endswith(\u0026#34;__\u0026#34;): return super().__getitem__(name) self.last_value += 1 self[name] = self.last_value return def __setitem__(self, name, value): # Ignore special names if name.startswith(\u0026#34;__\u0026#34;) and name.endswith(\u0026#34;__\u0026#34;): return super().__setitem__(name, value) self.last_value = value return super().__setitem__(name, value) Assigning Constants The next natural step in our progression is to allow assigning various constants into our enumeration. This includes global values, as well as members previously defined in the same enum.\n1 2 3 4 5 enum class Error { MyError, MyOtherError, SomeOsError = SOME_OS_ERROR } Or in Python:\n1 2 3 4 class Error(ScopedEnum): MyError MyOtherError SomeOsError = SOME_OS_ERROR If we try to print the values now, we get a weird result:\n1 2 \u0026gt;\u0026gt;\u0026gt; print(Error.MyError, Error.MyOtherError, Error.SomeOsError, Error.SOME_OS_ERROR) 0 1 None 2 There are two problems here. Let\u0026rsquo;s understand them together, and see what they mean for our next steps.\nFor one thing, SomeOsError is None instead of the value of SOME_OS_ERROR. For another, we have SOME_OS_ERROR as a member variable! How did that happen?\nIf we look at our enum class definition, we can classify name usages as reads and writes:\n1 2 3 4 5 class Error(ScopedEnum): MyError # read MyOtherError # read # write read SomeOsError = SOME_OS_ERROR For every read, we call __getitem__, and for every __write__ we call __setitem__. So, step by step:\nWe try to read MyError, calling __getitem__ and defining it We do the same for MyOtherError We call __getitem__ with \u0026quot;SOME_OS_ERROR\u0026quot;, defining the member We assign None (the value we return from __getitem__) into SomeOsError. With that, we know what went wrong. But fixing it is going to be a bit tricky.\nDetecting Member Definitions So far, we treated every read, or __getitem__ call, as a member definition. As soon as we allow assigning from existing variables, however, this falls apart. We need to find a new logic to it, a way to differentiate the places where a read means a new member, from where a read just means \u0026ldquo;this is the right-hand-side of an assignment\u0026rdquo;.\nLet\u0026rsquo;s consider some representative cases:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 X = ... class E1(ScopedEnum): A class E2(ScopedEnum): A B = A class E3(ScopedEnum): A = X class E4(ScopedEnum): A B = X + A A B X E1 1 read. Defined. - - E2 2 reads. Defined and assigned from. Assigned to - E3 Assigned to - 1 read. Assigned from E4 2 reads. Defined and assigned from. Assigned to 1 read. Assigned from From this, we can deduce some rules. The first is straightforward - if a name is assigned to, we need to create that variable. The second is trickier. We need to count the number of times a variable is read, and the number of times it is used in an assignment. If the number of reads is larger than the number of usages in assignments, we need to define it.\nThis means that we need to go line-by-line, counting, before we know what names we actually define. This will require post-processing, which we\u0026rsquo;ll get to later. But first, we must learn to count!\nCounting Names Essentially, there are two different types of name-reads in our enums. One defines a new member, and one is used in an assignment:\n1 2 3 class E(ScopedEnum): A # A is read, defines a new member B = A # A is read, used in an assignment into B To correctly define members, we need to tell those cases apart. While it\u0026rsquo;s easy to see the difference, the mechanics of code evaluation make it a bit tricker. For reads, regardless of their being a member-definition or a use-in-assignment, we get a call to __getitem__. For every assignment, regardless of it\u0026rsquo;s inputs, we get a call to __setitem__.\nTo properly visualize this, we can write a \u0026ldquo;logging namespace\u0026rdquo;:\n1 2 3 4 5 6 7 8 9 10 11 12 13 class LoggingNamespace(dict): def __getitem__(self, name): if name.startswith(\u0026#34;__\u0026#34;) and name.endswith(\u0026#34;__\u0026#34;): return super().__getitem__(name) print(f\u0026#34;__getitem__({name!r})\u0026#34;) def __setitem__(self, name, value): # Ignore special names if name.startswith(\u0026#34;__\u0026#34;) and name.endswith(\u0026#34;__\u0026#34;): return super().__setitem__(name, value) print(f\u0026#34;__setitem__({name!r}, {value!r})\u0026#34;) And get the following result:\n1 2 3 __getitem__(\u0026#34;A\u0026#34;) __getitem__(\u0026#34;A\u0026#34;) __setitem__(\u0026#34;B\u0026#34;, None) As we can see - the __setitem__ calls only give us the name of the target variable. We may be tempted to say this is enough, but consider the following code:\n1 2 3 A = 1 class E1(ScopedEnum): B = A + A This is a valid enum definition, and it gives the exact same get-set sequence. We need more information.\nThe missing piece, as visible in the output, is the variable inputs into assignments. We need to have the names of all the variables used in an assignment visible to us when we call __setitem__.\nPlaceholders The value passed into __setitem__ is going to be controlled by:\nThe values we return from __getitem__ calls in out namespace The literals used in the assignments The operations used to combine the above. And constrained by the need to produce the correct result, and not just a list of names.\nAmong those 3 inputs (__getitem__ results, literals, operations) we have full control over __getitem__ results. With that, we need to somehow preserve both the names provided and the values calculated. We\u0026rsquo;re going to do this by cheating.\nWe\u0026rsquo;re going to change our __getitem__ to return a special object - a placeholder object - that holds the name of the variable we read. We don\u0026rsquo;t need (and actually can\u0026rsquo;t) return the \u0026ldquo;true\u0026rdquo; value of the named variable, as that requires telling definitions and assignments apart. In addition to holding the name, our placeholder objects will also implement all the operators we want to allow in our enum definitions. Once more - we can\u0026rsquo;t calculate actual values. Instead, we\u0026rsquo;ll construct a tree of operations.\nA simplistic implementation of a placeholder, allowing only addition, will look something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 import operator from typing import Any from dataclasses import dataclass @dataclass class Placeholder: names: list[str] lhs: Any op: Any rhs: Any def __add__(self, other): names = self.names[:] if isinstance(other, Placeholder): names += other.names return Placeholder(names, self, operator.add, other) def __radd__(self, other): names = self.names if isinstance(other, Placeholder): names = other.names[:] + names return Placeholder(names, other, operator.add, self) Note that we are maintaining a list of names, as we want to be able to count occurrences. Additionally, be mindful of the reverse order in __add__ and __radd__, as it is critical for calculating the correct values later.\nLiterals will always appear in an operation, so always within .lhs or .rhs. Named variables, however, need a \u0026ldquo;trick\u0026rdquo;. So we\u0026rsquo;ll represent them by assigning the name to the .lhs, and None as the operator.\nWith the placeholder mechanism in place, we can now tell which reads go to define new members, and which are assignment uses. This allows us to translate any sequence of calls to __getitem__ followed by a call to __setitem__ (or not, if we reached the end of the enum definition) into a sequence of member definitions and assignments.\nPost Processing A part that I only alluded to before, is that we\u0026rsquo;re about to have a post-processing step. Due to the way we differentiate member definitions from assignments, we cannot define our members \u0026ldquo;as-we-go\u0026rdquo;. Instead, our __getitem__ and __setitem__ implementations only collect information, without creating any new variables. When the class definition is done, our ScopedEnumMeta.__new__ method will be called, and we\u0026rsquo;ll get our namespace object as an input. At that point, we\u0026rsquo;ll be able to convert the information we collected into a proper enum.\n1 2 3 4 5 6 7 8 class ScopedEnumMeta(type): @classmethod def __prepare__(metacls, name, bases): return Namespace() def __new__(cls, name, bases, namespace): classdict = populate_enum(namespace.member_info) return type.__new__(cls, name, bases, classdict) Calculating Values There are 2 types of members we need to handle. The first type is member definitions. They will work just as they did before - take the last assigned value, and increment it by one.\n1 2 3 4 5 6 7 8 9 def populate_enum(member_info): last_value = -1 members = {} for member in member_info: if isinstance(member, Definition) last_value += 1 members[member.name] = last_value continue The second type - assignments - are a bit more involved. An assignment can either be a regular value (if only literals were used), or it can be a Placeholder. If it is a placeholder, we need to traverse our tree of operations, and calculate a value:\n1 2 3 4 5 6 7 8 def calculate(value): if not isinstance(value, Placeholder): return value return value.op( calculate(value.lhs), calculate(value.rhs), ) But we\u0026rsquo;re missing a piece - recall that the operation for named variables is None, and the stored value is a name, not an actual value. To get those, we need to do an extra step, and perform a lookup:\n1 2 3 4 5 6 7 8 9 10 11 def calculate(value, namespace): if not isinstance(value, Placeholder): return value if value.op is None: return namespace[value.names[0]] return value.op( calculate(value.lhs, namespace), calculate(value.rhs, namespace), ) As for getting the namespace - we\u0026rsquo;ll have to use the inspect module and peek at our parent stack-frames. But I won\u0026rsquo;t get into this here.\nAnd with this, we\u0026rsquo;re done.\nRead the full implementation here.\nParting Words Python is a wonderful language. It allows us to ask whether we can, without worrying about whether we should. In this post, I shared the implementation details and some of the thought process of implementing C++ Scoped Enums in Python. I hope you found this entertaining, and maybe learned a thing or two. And I sincerely hope you\u0026rsquo;ll never use this in any production code.\n","permalink":"https://tamir.dev/posts/snake-eyes-cpp-scoped-enums/","summary":"Often, we write code to complete a task. Other times, we write code to learn something new. But there are also those rare occasions where our goals are simpler. When we have a dumb idea stuck in our head, and the only way to get it out is to code it. Or when we just want to see, to prove to ourselves, that we can write a certain piece of code.","title":"Snake Eyes: C++ Scoped enums"},{"content":"Why? The first question you should ask yourself when seeing this title is \u0026ldquo;why?\u0026rdquo;. Why would we want to add wikilink1 support to a Markdown formatting library?\nWell, recently I started using Obsidian (a Markdown-based personal wiki). Since my writing is mostly code-related, it includes a lot of code-snippets. As I like having my code neatly formatted, and hate formatting it by hand, I wanted a tool to do that for me. The best tool I found was mdformat. It\u0026rsquo;s Python based, formats Python, Rust, and Go code snippets, and even formats the Markdown itself.\nThe only problem being: it formats a lovely wiki [[link]] as \\[\\[link\\]\\], breaking it in the process. To mitigate that, I had to add wikilink support2.\nIf you just want to see the code, go to mdformat-wikilink.\nMdformat Plugins mdformat is a markdown formatting tool written in Python. It is based on the wonderful markdown-it-py library, and has plugin support.\nThere are 2 types of mdformat plugins:\nCode formatter plugins, used for formatting the code inside fenced blocks; Parser extension plugins, used to add support for new nodes. Since we\u0026rsquo;re adding support for a new type of syntax, a wikilink, we\u0026rsquo;ll be writing a parser extension plugin. To do so, we\u0026rsquo;ll follow the mdformat guide for developing plugins.\nThat said, our mdformat plugin will only do the rendering. For parsing, we need to write a markdown-it-py plugin.\nMarkdown-it-py Plugins Luckily for us, markdown-it-py has very good support for plugins as well. Documentation includes design principles (which make for a good architecture overview), API documentation, and existing plugins. You can also check the markdown-it live demo (using the Javascript library that was later ported to Python) to interactively see a token stream.\nActual Code After a bit of reading, experimenting, and finding out - it seems that we only need very little code to make things work. We can do something more complex, but for our needs (ensuring mdformat doesn\u0026rsquo;t modify wikilinks) we can hack something quick. We\u0026rsquo;re going to create a new parser token for wikilinks, and make mdformat render it as-is.\nThe code will consist of 3 parts:\nmarkdown-it-py plugin, to parse the wikilinks as a new token mdformat plugin, to use the previous plugin, and render the new token as-is A bit of pyproject.toml config to make mdformat recognize the plugin. Parsing Wikilinks Since we\u0026rsquo;re only parsing wikilinks to keep them unmodified, we\u0026rsquo;re not going to break them up into parts. Instead, we\u0026rsquo;ll just keep them as a block of text. This means that we can write a simplistic parser that does the following:\nUsing regex, we check whether the string currently fed to the parser is a link If it isn\u0026rsquo;t, we do nothing and report that it isn\u0026rsquo;t. If it is, we push a wikilink token with the entire link as its content, and increment the parser position part the link. The last (and most important) part of the code is registering the parser we just wrote. We register it as an \u0026ldquo;inline\u0026rdquo; rule (as it is an inline element, not a block), and we register it last, as it doesn\u0026rsquo;t replace any other elements (well, plain text\u0026hellip;).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 import re from markdown_it import MarkdownIt from markdown_it.rules_inline import StateInline LINK_PATTERN = re.compile(r\u0026#34;\\[\\[([^[|\\]\\n])+(\\|[^]\\n]+)?]]\u0026#34;) def _wikilink_inline(state: StateInline, silent: bool) -\u0026gt; bool: match = LINK_PATTERN.match(state.src[state.pos :]) if not match: # Not a wikilink! return False # Push the wikilink token token = state.push(\u0026#34;wikilink\u0026#34;, \u0026#34;\u0026#34;, 0) token.content = match.group() # Increment parser location state.pos += match.end() # Found a wikilink! return True def wikilink_plugin(md: MarkdownIt) -\u0026gt; None: # Register the parser! md.inline.ruler.push(\u0026#34;wikilink\u0026#34;, _wikilink_inline) Formatting Wikilinks Our mdformat plugin is even more simplistic.\nThe update_mdit function is used to load our wikilink-parsing plugin into the current instance of markdown-it-py The _render_wikilink function returns the content of the wikilink token (or node, in mdformat terminology) that we pushed That\u0026rsquo;s it.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 from collections.abc import Mapping from markdown_it import MarkdownIt from mdformat.renderer import RenderContext, RenderTreeNode from mdformat.renderer.typing import Render from mdformat_wikilink.mdit_wikilink_plugin import wikilink_plugin def update_mdit(mdit: MarkdownIt) -\u0026gt; None: # Load the markdown-it wikilink plugin to parse wikilinks mdit.use(wikilink_plugin) def _render_wikilink(node: RenderTreeNode, context: RenderContext) -\u0026gt; str: # Render as-is. return node.content # Register the render function RENDERERS: Mapping[str, Render] = {\u0026#34;wikilink\u0026#34;: _render_wikilink} A Tiny Bit of Config The last thing we need to do is register the right entry-point for our plugin, so that mdformat will know to load and use it. We can do it in our pyproject.toml file (I\u0026rsquo;m using Poetry, other tools have similar options).\n1 2 [tool.poetry.plugins.\u0026#34;mdformat.parser_extension\u0026#34;] \u0026#34;wikilink\u0026#34; = \u0026#34;mdformat_wikilink.mdformat_plugin\u0026#34; And with that, we\u0026rsquo;re done.\nYou can see the whole project at mdformat-wikilink.\nWikilinks the link markup you see in wikis like Wikipedia. They are of the form [[Target]] or [[Target|Alias]], and generally create links inside the wiki.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAnd a blog post documenting it, for future reference. Which you are now reading.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/adding-wikilink-support-to-mdformat/","summary":"Why? The first question you should ask yourself when seeing this title is \u0026ldquo;why?\u0026rdquo;. Why would we want to add wikilink1 support to a Markdown formatting library?\nWell, recently I started using Obsidian (a Markdown-based personal wiki). Since my writing is mostly code-related, it includes a lot of code-snippets. As I like having my code neatly formatted, and hate formatting it by hand, I wanted a tool to do that for me.","title":"Adding Wikilink Support To Mdformat"},{"content":"Recently I was asked about Python tools \u0026amp; libraries that I use. After taking the time to write a lengthy reply, I figured I might as well write a blog post, to make it more widely available.\nThis post is going to go over some Python tools \u0026amp; libraries that I use and that I like.\nFormatting Automatic formatting gives you the gift of not ever caring about formatting again. Not when you resolve conflicts, not when you review code, not when you write code.\nI currently use 3 different formatters to get it done. I run autoflake, then isort, then black.\nBlack The uncompromising Python code formatter\nBlack is a code-formatter for Python with minimal (line length) configurable settings. It\u0026rsquo;s a really good code formatter, and will help avoid all the \u0026ldquo;but I like my code this way\u0026rdquo; complaints. Additionally, it\u0026rsquo;s now more-or-less the de-facto standard for formatting Python code.\nTo paraphrase Rob Pike, \u0026ldquo;Black\u0026rsquo;s style is no one\u0026rsquo;s favorite, yet Black is everyone\u0026rsquo;s favorite.\u0026rdquo;\nisort isort sorts your imports. It sorts them alphabetically, and also groups them separating standard-library, third-parties, and your own code.\nEven if you don\u0026rsquo;t care about sorting imports - it\u0026rsquo;s going to solve a ton of import-duplication \u0026amp; merge-conflicts for you.\nautoflake autoflake removes unused imports. That\u0026rsquo;s it. Use it.\nLinting Just like automatic formatting - you don\u0026rsquo;t need to think about anything you can lint for. This also means you never need to argue about it.\nmypy mypy is a static type-checker for Python. It uses type annotations \u0026amp; type inference to tell whether your types match or not. It is not perfect, but it catches often catches major issues in my code.\nI highly recommend using it in your project from the get-go. Writing types as you code is easy. Writing types for an existing project is exhausting.\nWhile the default settings are ok, there are some extras I recommend:\ncheck-untyped-defs makes sure you get warnings even if you didn\u0026rsquo;t annotate a function at all. This one, in particular, can be annoying to turn on late into a project. strict-equality warns you if you compare non-overlapping types. Those comparisons are usually bugs. warn-return-any warns if you return an unknown type from a function annotated to return a specific type warn-no-return warns if you only have return statements in some branches of your function. This is annoying sometimes, but saves you in the cases where you actually forget to return a value. warn-unreachable warns you when a piece of code is deemed unreachable. This one is super helpful in realizing that you made a typing error. Like checking a variable that\u0026rsquo;s never None against None. And the make the output friendlier, use:\npretty show-error-context pylint Pylint, with it\u0026rsquo;s catchy tag-line (\u0026ldquo;It\u0026rsquo;s not just a linter that annoys you!\u0026rdquo;), is for the most part just a linter that annoys you.\nIf you set it up, and disable all the errors you don\u0026rsquo;t care about (I have over 20 disabled globally), it can give some value. But it is also painfully slow.\nIt is currently here because I still use it, but I am more and more hesitant about including it in new projects. I just don\u0026rsquo;t feel like its value is worth the time and hassle. Especially with mypy catching a lot of the true errors pylint catches.\nflake8 flake8 is a style-guide-enforcer for Python. It is fast and capable, and has a large assortment of plugins that can catch actual bugs as well.\nRuff Ruff is \u0026ldquo;an extremely fast Python linter, written in Rust.\u0026rdquo; And it lives up to that. I don\u0026rsquo;t have much experience with it in production, so I don\u0026rsquo;t know how much it actually catches compared to the previous tools.\nRuff is now in very active development, and I plan to integrate it soon. Even if it\u0026rsquo;s missing some features you currently want, I think it\u0026rsquo;s worth keeping an eye out for it.\nTesting \u0026amp; Automation pytest Pytest is my test-framework of choice. The fixture-based design takes a while to get used to, and there is a bit of \u0026ldquo;magic\u0026rdquo; going on. But once you get the hang of it, it is extremely capable and easy to use and extend.\nnox nox makes it easy to automate your tests for multiple Python environments (think multiple versions of Python, multiple OSs, etc.).\nI find it straight-forward and easy to use and extend, as the configuration is entirely in Python.\nIn addition to tests, I use it to automate code formatting \u0026amp; linting, code-generation, and various CI tasks. That way I know that what I run locally and in the CI uses the same code and configuration.\nYou can also read Hynek Schlawack\u0026rsquo;s post on nox.\nProfiling \u0026amp; Benchmarking memray Memray is a memory profiler for Python. It does what it says on the tin, and does it well. It shows you which parts of your code allocated the most memory, and allows you to easily analyze that using multiple \u0026ldquo;reporters\u0026rdquo;. My most-used reporters are the Flame Graph Reporter and the Tree Reporter.\nThe only major downside is that it does not support Windows (unless you\u0026rsquo;re using WSL).\nAustin Austin is a sampling profiler for Python. It is fast and capable, and I use it a lot.\nBe aware that some related tools (austin-web) are not always as up-to-date and may cause issues.\nI recommend using it to profile an entire run, and then use Speedscope to analyze it. If you need to analyze a part of a run [austin-tui][] can show a live view, and then save it to a trace file.\nThe default output format for Austin is supported by Speedscope and is also easily editable using scripts (if you want to filter out specific parts of a run).\npy-spy py-spy is another sampling profiler for Python. For me, it\u0026rsquo;s main benefit is it\u0026rsquo;s top view, which gives a good overview of a process and lets you know \u0026ldquo;what\u0026rsquo;s taking so long?\u0026rdquo;. Additionally, it can show you stack traces for all the currently running threads.\nI usually use it when a running process misbehaves and I want to know why.\nThe Python Profilers The Python Profilers are for when sampling profilers aren\u0026rsquo;t enough. You need to know the call-count, and not just the durations. They don\u0026rsquo;t sample, so they will severely affect your program\u0026rsquo;s runtime. But there is no real alternative.\nFor viewing and analyzing the data, I highly recommend using KCacheGrind (or possible QCacheGrind if you\u0026rsquo;re on Windows and don\u0026rsquo;t wanna bother with setting up GUI for WSL). It is extremely fast even with very large profiles, and has some very good visualizations for analyzing the data. To convert the data to a fitting format, use pyprof2calltree\nMemory Usage Over Time Sometimes the simplest solution is the best one.\nWhen I need to find the part in my code that suddenly allocates way to much - I often log my memory usage over time, graph it, then compare it with a log or a sampling-profiler run to see which part of the run correlates with the spike in memory usage.\nIt\u0026rsquo;s crude, but it works.\nI tend to use the psutil library for it.\npytest-benchmark pytest-benchmark allows you to easily run short-benchmarks in your test-suite. It takes care of all the complicated stuff - repeating the runs, and calculating statistics - and gives you easy-to-read results.\nI also wrote a small tool on top of it to perform comparative-benchmarks and compare different implementations for the same code. You can find it at tmr232/python-benchmark.\npytest-json-report pytest-json-report generates a JSON report for a pytest run. It has 2 main benefits:\nMakes it easy to parse the test results and create custom reports Makes it easy to add extra information into the test results I use it to add peak-memory-usage into the test-results, so that I can keep track of that.\nPackaging \u0026amp; Deps Poetry Poetry is my go-to for managing my Python dependencies.\nThe main advantage of Poetry over other tools is that it automatically maintains a lock-file for you. By committing the lock-file into your repo, you ensure that package versions will be identical across all your environments. That way you know that what you develop with locally is the same as what\u0026rsquo;s tested in the CI, same as what\u0026rsquo;s deployed to staging, and also the same as what you deploy to production.\nI think that this alone should be enough to convince you.\nLibraries rich Rich is a Python library for rich text and beautiful formatting in the terminal.\nIf you print anything to the terminal - use Rich. It\u0026rsquo;s better than any other library in the category.\ntyper Typer, from the creator of FastAPI, makes building advanced CLIs easy and painless.\nIn it\u0026rsquo;s most basic - it takes the argument names and type annotations from a function, and converts it to a fully-features CLI (flags, arguments, documentation, completions).\nattrs attrs is my preferred way to write Python classes. It uses type annotations to declaratively define your classes, and does a fantastic work of deducing extra functionality from it (comparisons, equality, hash\u0026hellip;).\nTo keep things short - I don\u0026rsquo;t remember the last time I wrote an __init__ method.\nAltair If you ever need to draw a graph, use Altair.\nFor it\u0026rsquo;s philosophy and a brief intro, I recommend watching How to Think about Data Visualization by Jake VanderPlas.\nThe only downside I found so far is that exporting an image usually requires a web-browser in the process. But if your final output is HTML - you should be good.\nPandas Yes, pandas, the data-frame library.\nIf you\u0026rsquo;re ever dealing with numeric data, it\u0026rsquo;s worth spending the time (a couple of days, in my case) to develop basic competency with pandas. Once you have that, a lot of annoying tasks that you used to do in Excel become easy and straightforward.\n","permalink":"https://tamir.dev/posts/python-tools-and-libs/","summary":"Recently I was asked about Python tools \u0026amp; libraries that I use. After taking the time to write a lengthy reply, I figured I might as well write a blog post, to make it more widely available.\nThis post is going to go over some Python tools \u0026amp; libraries that I use and that I like.\nFormatting Automatic formatting gives you the gift of not ever caring about formatting again. Not when you resolve conflicts, not when you review code, not when you write code.","title":"Python Tools \u0026 Libraries"},{"content":"All whitespace is significant.\nIt might not always matter to your computer, or compiler, or piece of code. But to you, a human reading the code, it is significant.\nI often here people complaining about significant whitespace. They say it makes no sense, that it makes working with the code harder. That whitespace, specifically in code, should not be significant. But the unavoidable truth is that whitespace is always significant, regardless of the language you use.\nWhitespace Primer Before we talk about its significance, we need to define what whitespace is.\nWe\u0026rsquo;ll start with the Wikipedia definition of a whitespace character:\n[\u0026hellip;] any character or series of characters that represent horizontal or vertical space in typography. When rendered, a whitespace character does not correspond to a visible mark, but typically does occupy an area on a page.\nNext, we\u0026rsquo;ll divide it into 2 categories - visible and invisible.\nVisible Whitespace Visible whitespace is the whitespace you can see. Indentation, spacing, line breaks\u0026hellip; All of them make for visible whitespace.\nThe words in this line are separated by spaces. This line is indented using 4 spaces. An empty line precedes this line! Invisible Whitespace Invisible whitespace is all the whitespace you can\u0026rsquo;t see. This is not because it\u0026rsquo;s using different characters, but because it is positioned where it does not directly move other characters.\nThis line ends with 4 spaces. There are 2 line breaks after this line. Since you cannot see it, it\u0026rsquo;s hard to make sense of it. This is the source of many complaints.\nIndistinguishable Whitespace In addition to visible and invisible whitespace, there\u0026rsquo;s another category. Indistinguishable whitespace.\nThis category includes, for the most part, tabs and spaces.\nHere we indent with 4 spaces. Here we indent with a tab. Since they look the same, but are not the same character, they cause many issues.\nSignificance There are 2 ways for whitespace to be significant. It can be human-significant, meaning that it is significant for the reader; and it can be machine-significant, meaning that it matters to the computer.\nMost (if not all) whitespace complaints stem from disparity between those two concepts. From cases where whitespace is human-significant and not machine-significant, or vice versa.\nGoing forward, we\u0026rsquo;ll call situations where human- and machine-significance match \u0026ldquo;matched significance\u0026rdquo;, and cases where they do not \u0026ldquo;mismatched significance\u0026rdquo;.\nLet\u0026rsquo;s look at some examples.\nMatched Significance Plain Text This line has spaces it in. Lines are separated by line breaks. We can have multiple spaces. Or multiple line breaks. With the exception of invisible whitespace (trailing spaces or line breaks), there is no mismatch.\nPython 1 2 def f(): return 1 In Python, whitespace is significant for both the human and the machine in defining scopes. The second line is indented, marking it a part of the function defined on the first line.\nMismatched Significance Markdown 1 2 3 4 5 6 In Markdown, there\u0026#39;s some mismatch. The previous line ends with 2 significant spaces. This means that when rendered, it will remain a separate line, while the rest of the lines will get merged. This is not consistent across all variants, making it even worse. Since human readers cannot see the 2 trailing spaces, there\u0026rsquo;s a mismatch. We expect one output, but the computer gives us another.\nC 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 static OSStatus SSLVerifySignedServerKeyExchange(SSLContext *ctx, bool isRsa, SSLBuffer signedParams, uint8_t *signature, UInt16 signatureLen) { OSStatus err; ... if ((err = SSLHashSHA1.update(\u0026amp;hashCtx, \u0026amp;serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(\u0026amp;hashCtx, \u0026amp;signedParams)) != 0) goto fail; goto fail; if ((err = SSLHashSHA1.final(\u0026amp;hashCtx, \u0026amp;hashOut)) != 0) goto fail; ... fail: SSLFreeBuffer(\u0026amp;signedHashes); SSLFreeBuffer(\u0026amp;hashCtx); return err; } This is Apple\u0026rsquo;s goto-fail bug and it is one of my favourite examples of significant whitespace. In line 12 there\u0026rsquo;s a goto fail statement. Due to the indentation (whitespace!) it reads (to the human) as if it belongs in the same block as line 11. But since indentation is insignificant whitespace in C, the computer ignores it.\nTo be more explicit in C\u0026rsquo;s syntax, we\u0026rsquo;ll write it as follows:\n1 2 3 4 if ((err = SSLHashSHA1.update(\u0026amp;hashCtx, \u0026amp;signedParams)) != 0) { goto fail; } goto fail; Making it clear that it will always goto fail.\nI like this example as it is a significant security issue that was (at least in part) caused by whitespace.\nIndistinguishable Hell So we know whitespace is significant. Both to humans and to machines. Taking a look at any code-formatters we\u0026rsquo;ll also see that people like it that way. If I put these 2 different formatting options for a vote, I\u0026rsquo;m pretty sure which one will win:\n1 2 3 4 5 6 7 8 9 void f(int x) { if (x \u0026lt; 10) { printf(\u0026#34;%d is smaller than 10.\\n\u0026#34;, x); else if { printf(\u0026#34;%d is 10.\\n\u0026#34;, x); } else { printf(\u0026#34;%d is larger than 10.\\n\u0026#34;, x); } } 1 2 3 4 5 6 7 8 9 void f(int x) { if (x \u0026lt; 10) { printf(\u0026#34;%d is smaller than 10.\\n\u0026#34;, x); else if { printf(\u0026#34;%d is 10.\\n\u0026#34;, x); } else { printf(\u0026#34;%d is larger than 10.\\n\u0026#34;, x); } } After all, none of us really want to count matching braces.\nSo why does \u0026ldquo;significant whitespace\u0026rdquo; get so much hate?\nWell, consider the following:\n1 2 3 4 # Python def f(): print(\u0026#34;Indented using spaces.\u0026#34;) print(\u0026#34;Indented using tab.\u0026#34;) 1 2 3 # makefile target: echo \u0026#34;Fail!\u0026#34; Both of these examples look valid, but they aren\u0026rsquo;t. To the naked (human-) eye, they are indistinguishable from valid code. But they mismatch spaces and tabs. Two indistinguishable types of whitespace.\nThis is, as mentioned before, the cause of most of the issues people have with whitespace. They expect it to work, but it doesn\u0026rsquo;t. In addition to that, there\u0026rsquo;s no meaningful or straightforward way to detect it when you look at the code. It\u0026rsquo;s an invisible problem.\nHate The Right Things I am not going to tell you to stop hating significant whitespace. Whitespace hurt you, and that anger needs to be directed somewhere. I will ask you, though, to point it in the right direction.\nVisible, distinguishable whitespace, with matching human- and machine-significance, is a good thing. It helps you make sense of the code, and helps ensure that the computer makes the same kind of sense of it as well.\nInvisible, yet machine-significant whitespace is bad. It leads to surprising outcomes and confuses the human writing the text.\nVisible, yet machine-insignificant whitespace is also bad. It leads to surprising outcomes and is tricky to detect.\nIndistinguishable, yet significant whitespace is the worst. It leads to bugs, errors, and pain. Use whatever tool is available in your toolbox to fight it. Use linters, formatters, and if all else fails - in-editor \u0026ldquo;visible whitespace\u0026rdquo; features. Avoid writing systems that allow it. And remember - the problem with Python is not that indentation is significant, it\u0026rsquo;s that tabs are allowed.\n","permalink":"https://tamir.dev/posts/significant-whitespace/","summary":"All whitespace is significant.\nIt might not always matter to your computer, or compiler, or piece of code. But to you, a human reading the code, it is significant.\nI often here people complaining about significant whitespace. They say it makes no sense, that it makes working with the code harder. That whitespace, specifically in code, should not be significant. But the unavoidable truth is that whitespace is always significant, regardless of the language you use.","title":"Significant Whitespace"},{"content":"Preface I\u0026rsquo;ve been wanting to write this post for years now. I did the research and had a (bad) draft of this lying around for years (since mid 2018, it seems). But I never felt that it was quite good enough.\nBlogvent was not a huge success in terms of post-quantity1 for me. I had some health issues that stopped me when I was getting started, and I found it hard to jump back in. That said, I see it as a very successful endeavor. I finally got my website up, and I feel more comfortable about posting imperfect posts.\nSo here goes.\nMoving to LA In The Karate Kid (the 1984 one), Lucille LaRusso \u0026amp; her son move from Newark, New Jersey, to Reseda, Los Angeles, California. From that point on, the movie focuses almost entirely on her son, Daniel.\nThe movie opens with Lucille LaRusso \u0026amp; her son, Daniel, moving from New Jersey to California. This move sets up the entire movie, which later follows Daniel as he adapts to his new environment. Despite that, the movie spends almost no time explaining the move.\nThis makes sense, as Daniel is the protagonist, and the teen audience of the movie probably don\u0026rsquo;t really care about Lucille\u0026rsquo;s motivation. This means that we only get a few small glimpses into the reasoning behind the move.\nIn this post, we\u0026rsquo;ll dig into that move.\nRocket Computers We first hear about the LaRusso\u0026rsquo;s reason for relocating when they unpack their car at their new place. Daniel meets Freddy, and they talk (5 minutes into the movie):\n1 2 3 4 5 6 7 Freddy: Where you from? Daniel: New Jersey. Freddy: Wow! Whatcha doin\u0026#39; here? Daniel: My mom got a new job with a company working out here. Rocket computers. Flight of the future. Freddy: Never heard of it. Daniel: It\u0026#39;s up and coming. So now we know - they moved because Lucille got a fancy new job. For a long time, I was happy with this explanation. Sadly, it doesn\u0026rsquo;t hold up.\nManager Training You see, later in the movie (23 minutes into the movie), Daniel meets his mother for lunch—at the restaurant where she works. This is already suspicious—wasn\u0026rsquo;t she working with computers?\nAdditionally, she\u0026rsquo;s raving about the relative benefits of working there!\n1 2 3 4 5 6 7 Lucille: Guess what? I\u0026#39;m going to be trained as a manager. Isn\u0026#39;t that great? Daniel: Yeah. Lucille: They have this program: two nights a week. As soon as a spot opens, you\u0026#39;re in. And the benefits?! I\u0026#39;d never get them working in computers. They pay for EVERYTHING. Now, this made no sense to me. Sure, it\u0026rsquo;s the 80\u0026rsquo;s, but being a waitress or manager can\u0026rsquo;t be better than working in computers, can it? So I went ahead and did some research.\nCrunching Numbers I was lucky enough to chance upon the Bureau of Labor Statistics\u0026rsquo; Occupational Outlook Handbook of 1988. It gives the median earnings of 1986 for some potentially relevant professions:\nRestaurant Managers make $22,400 a year Computer Programmers make $27,000 Computer Operators make $16,500 Computer Systems Analysis make $32,800 While the pay for a waitress (according to weekly earnings in 1983 and weekly earnings in 1985) is around $8000 (weekly earning of $152 to $159 multiplied by 52 weeks a year).\nDepending on which computer-related profession Lucille had, she may make more money as a restaurant manager, but the waiter pay makes a transition entirely unreasonable.\nThis makes no sense.\nThe Missing Scene! This bugged me for years. Literally. But today, while looking at the The Karate Kid screenplay to make sure I get the quotes right, it finally made sense! While going over the script looking for \u0026ldquo;computer\u0026rdquo; references, I came upon a scene I did not recognize. A scene that never made it into the movie! In that scene, Lucille sheds light onto her changing jobs:\n1 2 3 4 5 6 Lucille: I got a job. Daniel: Yeah. I know. Rocket Computers. Flight to the future. Lucille: Crash landed in the present is more like it. They went bankrupt last Friday. Can you believe it? That\u0026rsquo;s the scene that I was missing! The reason behind her switching jobs!\nThis was supposed to happen right before Miyagi comes in to fix the faucet (I frantically scanned through the movie again to make sure I did not somehow forget the scene existing in the movie). I\u0026rsquo;m really happy having found this. It\u0026rsquo;s been bothering me since forever and it\u0026rsquo;s nice to know that it actually was thought through.\nIn addition to that, we get to learn that this is probably Lucille\u0026rsquo;s first computer job after taking night classes, which would explain why becoming a waitress (at least temporarily) made more sense than finding another computer job:\n1 2 3 Daniel: So after going to school and all those night classes \u0026#39;n stuff for computers, you\u0026#39;re going to be a waitress? In Conclusion The Karate Kid is a fantastic movie. It\u0026rsquo;s still one of my favourites and it holds up amazingly well. I am delighted that completing this post allowed me to solve the mystery of Lucille\u0026rsquo;s job change.\nI hoped you enjoyed it too!\nBlogvent Posts\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/posts/karate-kid/","summary":"Preface I\u0026rsquo;ve been wanting to write this post for years now. I did the research and had a (bad) draft of this lying around for years (since mid 2018, it seems). But I never felt that it was quite good enough.\nBlogvent was not a huge success in terms of post-quantity1 for me. I had some health issues that stopped me when I was getting started, and I found it hard to jump back in.","title":"The Karate Kid"},{"content":"This Sunday, during the monthly meeting of the Israeli WG21 National Body discussion forum1, we discussed a paper by Yehezkel Bernat. The paper discusses a specific issue with C++ ranges and iterators. It demonstrates it with the following code sample:\n1 2 3 4 5 6 7 8 9 10 11 #include \u0026lt;ranges\u0026gt; #include \u0026lt;iostream\u0026gt; namespace rv = std::views; int main() { for (auto i : rv::iota(0) | rv::filter([](auto i) { return i \u0026lt; 11; }) | rv::take(11)) std::cout \u0026lt;\u0026lt; i \u0026lt;\u0026lt; \u0026#39;\\n\u0026#39;; } The code in the example does the following:\nCreate an infinite2 range counting from 0 upwards (rv::iota(0)) Filters that to only return numbers smaller than 11 (rv::filter(...)) Takes the first 11 numbers - so 0 through 10 (rv::take(11)) Loops over those and prints them. But there is a problem.\nThe Problem Instead of printing all 11 numbers and terminating, it prints them, and then keeps running indefinitely3. This happens because after taking the 11th element, it also tries to increment towards the 12th. Since the range is not exhausted, iota keeps running through numbers. Those numbers keep get filtered out by filter as they are all 11 or larger, and so we never terminate.\nIf you come from other languages (like Python or Java), this should surprise you. In those other languages, similar code will work perfectly well.\nThis happens due to the design of iterators in C++.\nC++ Iterators C++ has many types of iterators. The simplest one being an input-iterator. For us to iterate over it, we need 2 operators4:\n*it, to get a value from the iterator; ++it, to increment the iterator. When iterating, we start with it pointing to the first element of our sequence. This means that we first use *it to get the current value, and only then use ++it to advance.\nThat\u0026rsquo;s why we had an issue in our above example - we read the 11th value, then tried to advance the iterator to an element that will never exist.\nAdditionally, we can\u0026rsquo;t skip the increment. If we do, a subsequent read from the iterator will repeat the current value.\nOther Languages The iterator design in C++ is significantly different from other languages. While C++ does read-then-increment, the common design in other languages is increment-then-read.\nPython uses __next__() to get the next value of an iterator, raising StopIteration if none exist.\nJava uses hasNext() to check if a value exists (and advance to it), then next() to get the value.\nIn both cases, there\u0026rsquo;s no reason to advance after we got the value we want. As a result - the issue won\u0026rsquo;t reproduce.\n1 2 3 4 5 6 7 8 9 from itertools import islice, count for i in islice( # This is the equivalent of rv::take filter( lambda x: x \u0026lt; 11, count(0) ), 11): print(i) Further Thoughts I don\u0026rsquo;t know the reason behind C++\u0026rsquo;s design. When looking at it, it feels like a derivative of the C-style loop,\n1 for (int i = 0; i \u0026lt; 10; ++i) { ... } Where the increment happens after each iteration. To me, this also seems consistent with the design of the range-based for loop in C++. And while it makes sense for iterating over pre-existing data, it feels a bit off to me when iterating calculated (or fetched) data. In those cases, Python and Java\u0026rsquo;s design feels more appropriate to me.\nAnother thing that comes to mind is Arno Schödl\u0026rsquo;s talk Why Iterators Got It All Wrong from CppNorth 2022. It discusses the design of iterators and how they mix up pointing to elements (begin() points to the first element) and borders (end() point after the last element). It seems to me that if begin() were to point before the first element (and therefore need to be incremented before being dereferenced) the issue would be resolved. Then again, I\u0026rsquo;m probably missing something.\nI recommend that you watch the talk and see what you think.\nQuite a mouthful, I know. And WG21 is the ISO C++ standard. You can see the agenda for said meeting here.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIn theory. Eventually the number will overflow.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAgain, in theory.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWe need more, but we\u0026rsquo;ll ignore them for simplicity.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/blogvent/2022-12-08/","summary":"This Sunday, during the monthly meeting of the Israeli WG21 National Body discussion forum1, we discussed a paper by Yehezkel Bernat. The paper discusses a specific issue with C++ ranges and iterators. It demonstrates it with the following code sample:\n1 2 3 4 5 6 7 8 9 10 11 #include \u0026lt;ranges\u0026gt; #include \u0026lt;iostream\u0026gt; namespace rv = std::views; int main() { for (auto i : rv::iota(0) | rv::filter([](auto i) { return i \u0026lt; 11; }) | rv::take(11)) std::cout \u0026lt;\u0026lt; i \u0026lt;\u0026lt; \u0026#39;\\n\u0026#39;; } The code in the example does the following:","title":"Thoughts on Iterators in Python and C++"},{"content":"I always knew image alt-text existed. I knew about it from learning HTML - it\u0026rsquo;s the text that shows when the image doesn\u0026rsquo;t load. And it\u0026rsquo;s also on XKCD as the extra pun (TIL that\u0026rsquo;s actually title= and not alt=). It\u0026rsquo;s only recently, when Twitter added image descriptions1 and my feed started talking about them, that I realized how important they are for accessibility.\nSince then, I\u0026rsquo;ve been making an effort to describe the images in all my tweets to the best of my ability. It\u0026rsquo;s not always easy (try describing control-flow graphs and explaining the differences between them,) but it is definitely worth it.\nYesterday I blogged for the first time in a long while. As I added images to the post, I realized that I don\u0026rsquo;t know how to properly describe images in Markdown. Until that day - I\u0026rsquo;ve never really done that.\nMarkdown Alt Text The Markdown2 image syntax consists of 3 parts:\n1 ![alt text](url \u0026#34;title\u0026#34;) That will (usually3) be rendered as:\n1 \u0026lt;img src=\u0026#34;url\u0026#34; alt=\u0026#34;alt text\u0026#34; title=\u0026#34;title\u0026#34;/\u0026gt; This is simple enough, and even allows us to write long-form alt text:\n1 2 3 4 5 6 ![This image doesn\u0026#39;t exist. I am only writing this to show that we can have really long alt-text. It can even span multiple lines as long as there are no blank lines in it.][long-alt-text] [long-alt-text]: my-image-url This also benefits us as we\u0026rsquo;re writing - we have a description of the images we use, in our text-only editors!\nAnd we should all use it to improve our posts and make them more accessible. I still need to go over my past posts and properly describe the images in them.\nFuture Ideas While learning a bit on this, I searched for \u0026ldquo;markdown accessibility\u0026rdquo;. This lead me an article about improving the accessibility of my markdown. It seems that there\u0026rsquo;s quite a lot I need to fix in my blog. So that\u0026rsquo;s one thing I need to figure out.\nGenerally, the things to fix fall into 2 categories: authoring and rendering. Rendering is purely a technical fix - I need to ensure my blog theme is accessible by changing the templates. The authoring part is a bit tricker. It\u0026rsquo;s too easy for me to miss alt-text here and there, or have multiple links with the same name. So for this - I think I\u0026rsquo;ll have to write a linter.\nFrom the Twitter docs, saved to the archive.org so that it doesn\u0026rsquo;t disappear:\nHow to add image descriptions How to write great image descriptions \u0026#160;\u0026#x21a9;\u0026#xfe0e; When talking about Markdown I\u0026rsquo;ll try to refer to CommonMark unless stated otherwise.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIn this blog, for example, I display the title using \u0026lt;figure\u0026gt; and \u0026lt;figcaption\u0026gt; tags.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/blogvent/2022-12-02/","summary":"I always knew image alt-text existed. I knew about it from learning HTML - it\u0026rsquo;s the text that shows when the image doesn\u0026rsquo;t load. And it\u0026rsquo;s also on XKCD as the extra pun (TIL that\u0026rsquo;s actually title= and not alt=). It\u0026rsquo;s only recently, when Twitter added image descriptions1 and my feed started talking about them, that I realized how important they are for accessibility.\nSince then, I\u0026rsquo;ve been making an effort to describe the images in all my tweets to the best of my ability.","title":"Markdown Images \u0026 Accessibility"},{"content":"When I started my professional career I was not doing software engineering, but rather reverse-engineering. Taking a compiled binary, disassembling it, and trying to understand what it does. My job was essentially reading code, with the stated goal in my job description was \u0026ldquo;vulnerability research\u0026rdquo;. Since then, I moved to the other side, and am now doing \u0026ldquo;forward engineering\u0026rdquo;.\nWhile it is very tempting to say that as programmers (or developers, or software engineers, or whichever term you\u0026rsquo;re comfortable with) we only write code, we know that this is not true. If fact, we spend a lot of our time reading code. Be it code reviews, reading docs, debugging, or finding that one line of code we need to stick in someone else\u0026rsquo;s face and say \u0026ldquo;See!?!?!?! I told you!!!\u0026rdquo;.\nReading code is a vital part of any code-related role. Be it forward- or reverse-engineering. That said, the perspective if very different. I see that difference when talking to my friends about code, some of them being developers and other reverse-engineers. But I also see it in how I approach my tasks. My approach to reading code is dramatically different between development and reverse-engineering tasks.\nPart of it is the difference in goals - as a software developer, I want the code to work well. I am not (usually) looking for bugs, because I really don\u0026rsquo;t wanna find them. As a reverse-engineer bugs are what we\u0026rsquo;re looking for. This leads to a different perspective. In addition to that, as a developer I tend to let my hubris control me, assuming that I can write bug-free code. That I can understand the entire system. When researching code for vulnerabilities, the assumptions are different. First and foremost - all systems have bugs. We just need to find them. And second - systems are big. Too big. This means that we need to focus, get to the interesting parts of an unknown system as fast as possible.\nThis, in my opinion, also leads to very different tools.\nThe developer\u0026rsquo;s tool of choice is a code editor. Be it vi, or emacs, or VSCode, or any other editor. We read the code as text, and process it as text. The editor might give as some colors or linking, but its still text. When we want to know whether 2 functions are similar, we read them both. When we want to know if a function has a complex flow - we read it.\na truncate in Go, as rendered by the Goland IDE The reverse-engineer\u0026rsquo;s tool of choice is often IDA (or Binary Ninja or Ghidra or Hopper or something similar). While those tools do allow reading the disassembled code as text, they also offer an additional view - a graph view.\nsame function, in IDAs graph view With a bit of training and getting used to it, the Graph view allows a reverse-engineer to quickly discern the flow of the function. With green and red arrows denoting the true and false branches of a condition, blue lines denoting unconditional jumps, and bold lines for backlinks (usually loops).\nWhile the graph does not convey all the information about a function, it still conveys a lot. It makes it easy to see if two functions are similar, or if a function is complicated.\nI like this ability, and I miss it dearly in my developer tooling. So I started playing around with graph-visualization of functions in Go, to see how it turns out. Using the least amount of code I could - taking advantage of Go\u0026rsquo;s SSA1 libraries, and graphing using GraphViz - I managed to create something that, while initial, I\u0026rsquo;m very happy with. The project is called go-overview-graph and renders source-code and graphs side-by-side. Here are some example graphs, because they are so pretty! I highly recommend going over the website and having a look yourself!\nSingle Static Assignment\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://tamir.dev/blogvent/2022-12-01/","summary":"When I started my professional career I was not doing software engineering, but rather reverse-engineering. Taking a compiled binary, disassembling it, and trying to understand what it does. My job was essentially reading code, with the stated goal in my job description was \u0026ldquo;vulnerability research\u0026rdquo;. Since then, I moved to the other side, and am now doing \u0026ldquo;forward engineering\u0026rdquo;.\nWhile it is very tempting to say that as programmers (or developers, or software engineers, or whichever term you\u0026rsquo;re comfortable with) we only write code, we know that this is not true.","title":"A Bit of Codeviz"},{"content":"So a bit of a last minute decision, but here goes!\nInspired by Ólafur Waage\u0026rsquo;s post, I decided to try and follow suit. After all - I\u0026rsquo;ve not blogged for about a year and a half, and that\u0026rsquo;s way too long.\nSo, blatantly copying Ólafur:\nRules One post every day in December. Each post has to be about tech/code in some way. Each post has to be longer than 4 tweets. (1120 characters) Posts don’t have to be revolutionary (this is a writing exercise, it’s ok to write a basic tutorial about if statements). Posts have to be written that day, you can’t stock up. Posts can be a continuation of Yesterday’s post (again, this is mainly a writing exercise). No edits or people reviewing your drafts (writing exercise, etc) ","permalink":"https://tamir.dev/posts/blogvent-calendar-2022/","summary":"So a bit of a last minute decision, but here goes!\nInspired by Ólafur Waage\u0026rsquo;s post, I decided to try and follow suit. After all - I\u0026rsquo;ve not blogged for about a year and a half, and that\u0026rsquo;s way too long.\nSo, blatantly copying Ólafur:\nRules One post every day in December. Each post has to be about tech/code in some way. Each post has to be longer than 4 tweets.","title":"Blogvent Calendar"},{"content":"TL;DR Graph memory-usage over time, correlate with logs, profit.\nOverconfidence Recently, I had to reduce the memory consumption of a Python process that became entirely unreasonable. Now, a while back I wrote about finding memory leaks in Python. I was pleased with myself and sure that with the knowledge I gained then, I can surely get this done!\nAnd oh, was I wrong\u0026hellip;\nHarsh Reality You see, both pympler and tracemalloc are wonderful tools. But like all tools, they have limitations. When you have a long-running (days) process with many (hundreds of millions) objects, the memory and performance costs of your tools add up quite significantly. Waiting for pympler to query all objects takes forever, and following references is completely impractical; viewing tracemalloc statistics is nice, but doesn\u0026rsquo;t help you narrow things down enough.\nSo, after 2 weeks of zero-to-minimal improvements (though I was sure I\u0026rsquo;m on the right track) I decided to try a different approach to understanding the memory usage of my code.\nTo The Rescue Enter memlog.py.\nmemlog is a simple, naive tool. It tracks the overall memory usage on a machine, and logs it (with a timestamp) to a CSV. That\u0026rsquo;s it. While the recorded data may include significant noise, running your code (\u0026amp; memlog) inside a container can reduce it significantly. Also, background memory noise tends to be insignificant when your process hogging all of your memory\u0026hellip;\nSo, I ran my process (with logs), ran memlog, and plotted a memory-over-time graph: And oh, oh no.\nInsight Looking at the graph, we can divide it into 3 parts:\nA near-instant rise at the beginning. This is by far the bulk of the memory-usage increase; A slow, gradual increase over the entire time-scale; A near-instant drop in memory-usage. Those parts are basically:\nLoading the data-set and various initialization; The bulk of the processing; Program termination. And for the past 2 weeks I\u0026rsquo;ve been busy reducing the memory-usage of\u0026hellip; the second part. Being absolutely sure it\u0026rsquo;s the most significant.\nSo yeah, that hurt. But only for a short time. For you see, with this newfound knowledge I could safely focus on the first few minutes of execution and disregard the rest for the time being.\nTrue. I\u0026rsquo;ll have to test the whole thing once I\u0026rsquo;m make any significant changes. Memory-usage might spike at a later point. Memory-optimization may cause performance degradation. But unless I reduce that uptick at the beginning I won\u0026rsquo;t get any significant improvements.\nProfit A week later, we managed to reduce memory-usage by 30% while reducing overall processing time by a similar percentage. We had to:\nAdd a de-duplicating weakref based cache; Add a pre-processing step; Make our code more cache-friendly by sorting our data; Remove a massively over-engineered control mechanism. But it was all made possible by focusing on the right part. Had I not plotted that memory graph, I could\u0026rsquo;ve easily spent another 2 weeks without any significant progress.\nOld \u0026amp; Wise So whatever you do, I highly suggest you graph your data. No need to be smart about it. Log it, graph it, correlate to your logs.\n","permalink":"https://tamir.dev/posts/more-memory-profiling-in-python/","summary":"TL;DR Graph memory-usage over time, correlate with logs, profit.\nOverconfidence Recently, I had to reduce the memory consumption of a Python process that became entirely unreasonable. Now, a while back I wrote about finding memory leaks in Python. I was pleased with myself and sure that with the knowledge I gained then, I can surely get this done!\nAnd oh, was I wrong\u0026hellip;\nHarsh Reality You see, both pympler and tracemalloc are wonderful tools.","title":"More Memory Profiling (in Python)"},{"content":"In writing our CI setup at Vdoo, we came across some interesting challenges. Having solved them and used the solutions for quite a while, we decided it is best to share, and hopefully save others some time and effort solving similar problems.\nOf course, there are alternative solutions to the challenges we have dealt with, and some solutions are probably superior to ours. We are happy to hear about such solutions and to improve our own.\nAs for the code presented in this post - it was extracted from our CI and cleaned up a bit. As such, it is missing some necessary boilerplate. It will not work as-is, and some work will be required to adapt it to your CI. That said, it should clearly lay out the solutions. (You can think of it as slide-ware.)\nLFS-Check All transitions can be bumpy. For us, the transition from storing binary files as regular git blobs to storing them using LFS was one such bumpy transition.\nWe made sure to include all the LFS-relevant files \u0026amp; patterns in a .gitattributes file, but ensuring everyone (including people who only occasionally work on the relevant repo) properly setup their environments for LFS took some time. In the mean-time, we kept getting files that should be in LFS committed and pushed as regular files in Merge-Requests.\nTo circumvent that, we set up a simple check at the start of our CI process to ensure all relevant files are indeed stored in LFS.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 lfs-check: only: refs: - merge_requests script: - git lfs install - git add --renormalize -u - | if ! git diff --cached --name-only --exit-code ; then echo echo echo \u0026#34;==============================\u0026#34; echo \u0026#34;# Please renormalize files #\u0026#34; echo \u0026#34;==============================\u0026#34; echo echo \u0026#34;git add -u --renormalize\u0026#34; echo \u0026#34;git commit --amend\u0026#34; exit 1 fi When people pushed the files the wrong way - the CI would fail with an informative error and instructions.\nRequired Commit Every so often a change is made to the code, rendering the code before the change unworkable or irrelevant. This can happen for many reasons. Here are some examples:\nA bug was fixed in the CI. This is all too common when forgetting to properly lock your dependencies (including recursive ones!) A very time-consuming update was made (re-training an ML model, anyone?) The change is significant and will make rebasing a pain A significant bug was fixed, making tests on the previous versions mostly irrelevant Once you introduce such a change to your code, you want people to know about it, and you want to stop wasting cycles on it.\nTo achieve this goal, we created a required-commit mechanism in our CI. For the CI to work, the required-commit must be an ancestor of the current commit. If it isn\u0026rsquo;t - the CI fails with a descriptive error \u0026amp; instructions for fixing the issue.\nOnce we have a new required-commit, we inform all developers in a dedicated Slack channel and update the CI to match. This ensures that even if a developer misses the notification on Slack, the CI will let them know what needs to be done.\nThe solution consists of a simple CI job:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 required-commit: only: refs: - merge_requests script: - apt-get update - | if ! git merge-base --is-ancestor ${REQUIRED_COMMIT:-HEAD} HEAD ; then echo echo echo \u0026#34;=============================\u0026#34; echo \u0026#34;# Rebase Required #\u0026#34; echo \u0026#34;=============================\u0026#34; echo echo \u0026#34;Your base commit is out of date.\u0026#34; echo \u0026#34;Please update to ${REQUIRED_COMMIT} or later.\u0026#34; echo \u0026#34;The easiest fix is to rebase-onto or merge-from origin/main.\u0026#34; echo echo exit 1 fi And a custom variable defined in the CI settings (see Create a custom variable in the UI):\nConditionally Building Job Docker Images Some of our code is deployed via Docker images. As such - we want our CI to build and test those images. Some tests require running a Docker container and communicating with it, but some tests (especially unit- and integration-tests) are easier to run inside the said containers. To accommodate the latter, we use our Docker images as the base images for the CI test jobs.\nThis is easy enough to do in the CI. In our case, however, building the Docker images takes a very long time. In trying to reduce this time, we split our build into two parts. The first - a long compilation phase, building some rarely-changing code; the second - installation of our fast-changing Python code \u0026amp; all relevant dependencies.\nNoticing the split between the fast-changing and rarely-changing parts of our build, we decided to split it in half, only building the first part when there\u0026rsquo;s an actual change to it.\nTo do that, however, we have to conditionally build the Docker image for the first half, and in the second half use either the preexisting first half or the newly built one.\nThe Solution - Build, Proxy, Promote Our solution uses a model consisting of multiple CI jobs handling different parts.\nBuild jobs - responsible for building docker images. Either conditionally (for the first part) or consistently (for the second part). Proxy jobs - responsible for handling the conditional nature of the build jobs, providing the next job with the relevant tag for the Docker images - either :latest or the current commit. Promote jobs - responsible for tagging the newly built images with :latest and pushing them. They run last. For our use-case, we used the following setup:\nConditional Build job to build the rarely-changing code Proxy job to yield the relevant tags Build job to build \u0026amp; install the fast-changing code Test job, to test the newly build code Promote job, pushing the newly build images as :latest if the tests passed To implement it, we created the following .yml configuration, representing the build-proxy-promote model, and used include:file to bring it into our .gitlab-ci.yml.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 # This file allows creating a prebuild-proxy-(build)-promote workflow with ease. # # The idea is that that in the prebuild step we build less-frequently-changed # docker images than in the build step. This allows us to significantly speed # up CI times. # # Setting Up # ========== # # A basic setup consists of the following: # # 1\\. Prebuild (extends .bpp:build) - build docker images if relevant files changed # 2\\. Proxy (extends .bpp:proxy) - allows the rest of the CI to know whether Prebuild created new images or not # 3\\. Build [Optional] (extends .bpp:build) - builds extra, more-frequently-changing images. # This is not a conditional step! # 4\\. Use (custom step) - here we actually use the images we created! # 5\\. Promote (extends .bpp:promote) - if required, pushes the newly built images to the project\u0026#39;s repository. # # These 5 steps should be in 5 different, consecutive stages for things to work. # The prebuild step, being conditional, should not have any other step requiring it. # All other steps (that need the prebuilt images) should require the proxy step instead, # and use the ${PROXY_TAG} to as a label to the relevant docker images. .bpp:build: variables: # The names of all the docker images we want to pull from our registry TO_PULL: \u0026#34;\u0026#34; # The tag to use for pulling the images. This will usually be ${PROXY_TAG} PULL_TAG: \u0026#34;\u0026#34; # The name of the image and path of the dockerfile for building docker images. # The root path for the dockers will be the root of the project # Format the variable as follows: # # \u0026gt;- # \u0026#34;some_name the/relevant/path/Dockerfile\u0026#34; # \u0026#34;some_other_name another/relevant/path/Dockerfile\u0026#34; # # Note that the quotes are significant! TO_BUILD: \u0026#34;\u0026#34; # The names of the images we want to push. # They will all be pushed with the ${CI_COMMIT_SHA} tag. TO_PUSH: \u0026#34;\u0026#34; script: - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY_IMAGE} - export DOCKER_BUILDKIT=1 # This cannot be in the `variables` field since users overwrite it. - | for IMAGE_NAME in ${TO_PULL} do echo \u0026#34;***********************************\u0026#34; echo \u0026#34;Pulling ${IMAGE_NAME}\u0026#34; echo \u0026#34;-----------------------------------\u0026#34; echo \u0026#34;docker pull ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG}\u0026#34; echo \u0026#34;DOCKER_BUILDKIT=1 docker tag \\ ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG} \\ ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest\u0026#34; echo \u0026#34;***********************************\u0026#34; docker pull ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG} docker tag \\ ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PULL_TAG} \\ ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest done - | eval \u0026#34;ARRAY=($TO_BUILD)\u0026#34; for ITEM in \u0026#34;${ARRAY[@]}\u0026#34; do MY_NAME=${ITEM% *} MY_PATH=${ITEM#* } echo \u0026#34;***********************************\u0026#34; echo \u0026#34;Building ${MY_NAME} from ${MY_PATH}\u0026#34; echo \u0026#34;-----------------------------------\u0026#34; echo \u0026#34;DOCKER_BUILDKIT=1 docker build \\ --build-arg BUILDKIT_INLINE_CACHE=1 \\ -t ${CI_REGISTRY_IMAGE}/${MY_NAME} \\ -t ${CI_REGISTRY_IMAGE}/${MY_NAME}:${CI_COMMIT_SHA} \\ -f ${MY_PATH} \\ --label \u0026#34;commit_sha=${CI_COMMIT_SHA}\u0026#34; \\ .\u0026#34; echo \u0026#34;***********************************\u0026#34; docker build \\ --build-arg BUILDKIT_INLINE_CACHE=1 \\ -t ${CI_REGISTRY_IMAGE}/${MY_NAME} \\ -t ${CI_REGISTRY_IMAGE}/${MY_NAME}:${CI_COMMIT_SHA} \\ -f ${MY_PATH} \\ --label \u0026#34;commit_sha=${CI_COMMIT_SHA}\u0026#34; \\ . done - | for IMAGE_NAME in $TO_PUSH do echo \u0026#34;***********************************\u0026#34; echo \u0026#34;Pushing ${IMAGE_NAME}\u0026#34; echo \u0026#34;-----------------------------------\u0026#34; echo \u0026#34;DOCKER_BUILDKIT=1 docker push ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${CI_COMMIT_SHA}\u0026#34; echo \u0026#34;***********************************\u0026#34; docker push ${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${CI_COMMIT_SHA} done .bpp:proxy: variables: # The names of jobs we want to proxy - if any of them succeeded, we proxy. BUILD_JOBS: \u0026#34;\u0026#34; script: - PROXY_TAG=latest - apt-get -qq update - apt-get -qq install jq # Get the successful jobs for the current pipeline - \u0026gt;- curl --header \u0026#34;PRIVATE-TOKEN:${GITLAB_TOKEN}\u0026#34; \u0026#34;${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs?scope[]=success\u0026#34; \u0026gt; jobs.json # Compare the job names from the pipeline with the provided job names - EXECUTED=$(comm -12 \u0026lt;(jq -r \u0026#39;.[].name\u0026#39; jobs.json | sort) \u0026lt;(echo ${BUILD_JOBS} | tr \u0026#39; \u0026#39; \u0026#39;\\n\u0026#39; | sort)) # If a build job was executed, we need to set the proxy tag to the current commit sha. - | if [ ! -z \u0026#34;$EXECUTED\u0026#34; ] then PROXY_TAG=${CI_COMMIT_SHA} fi - echo \u0026#34;PROXY_TAG=${PROXY_TAG}\u0026#34; \u0026gt;\u0026gt; deploy.env # Print out the proxy tag - for debug purposes - echo \u0026#34;PROXY_TAG=${PROXY_TAG}\u0026#34; artifacts: # To get the proxy tag, you need to get the artifacts from this job. # The proxy tag will be ${PROXY_TAG} reports: dotenv: deploy.env .bpp:promote: variables: # The names of the images you wish to promote. # They need to have been built \u0026amp; pushed by a previous build step. TO_PROMOTE: \u0026#34;\u0026#34; script: - | if [ \u0026#34;${PROXY_TAG}\u0026#34; = \u0026#34;latest\u0026#34; ]; then echo \u0026#34;Nothing to promote.\u0026#34; else echo \u0026#34;Promoting docker image.\u0026#34; docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY_IMAGE} for IMAGE in ${TO_PROMOTE} ; do docker pull ${CI_REGISTRY_IMAGE}/${IMAGE}:${CI_COMMIT_SHA} docker tag ${CI_REGISTRY_IMAGE}/${IMAGE}:${CI_COMMIT_SHA} ${CI_REGISTRY_IMAGE}/${IMAGE}:latest docker push ${CI_REGISTRY_IMAGE}/${IMAGE}:latest done fi ","permalink":"https://tamir.dev/posts/gitlab-ci-tricks/","summary":"In writing our CI setup at Vdoo, we came across some interesting challenges. Having solved them and used the solutions for quite a while, we decided it is best to share, and hopefully save others some time and effort solving similar problems.\nOf course, there are alternative solutions to the challenges we have dealt with, and some solutions are probably superior to ours. We are happy to hear about such solutions and to improve our own.","title":"GitLab CI Tricks"},{"content":"Last week I had to fix a memory leak in a Python program for the first time. A long running process started eating too much RAM (only ~20GB to much) and the friendly OOM Killer had to step in and terminate this. Since this kept happening, I had to go ahead and fix the issue.\nStep 1 - Reproduction As with every bug, before you can reliably fix it, you must reproduce it.\nNow, while I had a reliable reproduction (after all, the process had regular dates with the OOM Killer), 3 days isn\u0026rsquo;t the best cycle time when you wanna solve a bug. So into the code we go.\nThe main idea is to start with the main loop, and try to narrow down the code that is must run for the leak to manifest. The process involves some educated guesses (where are the likely memory and allocation hogs in your process? What parts are likely to leak? Do you have any code that requires cleanup?), waiting, frustration, and tools.\ntracemalloc While each developer and codebase have their own unique guesses and frustrations, good tooling applies more widely. For this part, I used Python\u0026rsquo;s tracemalloc module.\nAmong other things, tracemalloc allows tracking memory usage between 2 points in your code in a very low-overhead manner.\n1 2 3 4 5 tracemalloc.start() # Start the memory trace code_suspected_of_leak() current, peak = tracemalloc.get_traced_memory() # Get memory stats After running this code, peak will hold the peak-memory-usage during the trace period, and current will hold the difference from the start of the trace to the current state. You should expect current to be non-zero. But if it goes too high - your code is probably leaking.\nBy placing such traces around suspect pieces of our code, we can find which parts are leaking. Just remember - only do this with functions that are expected to retain no state. If a function mutates an external object, or is a member function, it is very to exhibit changes in memory usage.\nStep 2 - Triage Once we have a reproduction (that hopefully takes a relatively short amount of time), we want to find the leaking code. We can try and keep narrowing our measured code down until we find the relevant line, but the deeper we go, the harder it is to separate the leak from normal execution.\nSo at this point, we\u0026rsquo;d like to look into the allocated memory, and see which objects are there when they shouldn\u0026rsquo;t be.\npympler For inspecting the objects in a Python process, I recommend using pympler.\nPympler is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.\nWe\u0026rsquo;re going to use it to do 2 things.\nInspecting Allocated Objects First, we\u0026rsquo;re going to use pympler to show us which objects were allocated during our repro \u0026amp; are still allocated.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 from pympler import tracker, muppy, refbrowser from functools import lru_cache # Naive code to trigger a leak class Value: def __init__(self, value): self._value = value def __repr__(self): return f\u0026#34;Value({self._value})\u0026#34; @lru_cache(maxsize=100) def get_value(value): return Value(value) def code_suspected_of_leak(): for x in range(10): print(get_value(x)) # Measuring code def main(): tr = tracker.SummaryTracker() code_suspected_of_leak() tr.print_diff() Once we run this, we get a nice table showing us a summary of objects created and destroyed:\n1 2 3 4 5 6 7 8 9 10 11 types | # objects | total size ======================= | =========== | ============ list | 4892 | 500.59 KB str | 4887 | 341.45 KB int | 1053 | 28.79 KB dict | 13 | 1.90 KB __main__.Value | 10 | 640 B function (store_info) | 1 | 144 B cell | 2 | 112 B weakref | 1 | 88 B tuple | -8 | -680 B As you can see - there are quite a few primitive objects generated, and also some __main__.Value objects. In my experience, primitives are harder to track, as they lack meaning in the code. Your own types, however, are usually only used in certain parts of the codebase, making them easier to make sense of.\nNow that we see that we have 10 new Value objects, it is time to figure out who\u0026rsquo;s holding them in memory.\n1 2 3 4 5 6 7 def output_function(o): return str(type(o)) all_objects = muppy.get_objects() root = muppy.filter(all_objects, Value)[0] cb = refbrowser.ConsoleBrowser(root, maxdepth=2, str_func=output_function) cb.print_tree() This\u0026rsquo;ll print the following:\n1 2 3 \u0026lt;class \u0026#39;__main__.Value\u0026#39;\u0026gt;-+-\u0026lt;class \u0026#39;list\u0026#39;\u0026gt; +-\u0026lt;class \u0026#39;functools._lru_cache_wrapper\u0026#39;\u0026gt;-+-\u0026lt;class \u0026#39;list\u0026#39;\u0026gt; +-\u0026lt;class \u0026#39;dict\u0026#39;\u0026gt; Giving away the issue - the lru_cache is keeping our Value objects. Just as designed\u0026hellip;\nI know this looks like a bit of a contrived example, but the lru_cache keeping objects in memory was exactly the issue I had. It was just buried under far more code.\nStep 3 - Solution Currently, I use the ugliest solution I can imagine - functions decorated with lru_cache have a cache_clear() method, and I\u0026rsquo;m calling that at specific places in my code. It\u0026rsquo;s ugly, but it works.\nA cleaner solution would require dedicated caches \u0026amp; better cleanup mechanisms. You can read a relevant discussion here.\n","permalink":"https://tamir.dev/posts/finding-a-memory-leak-in-my-python-code/","summary":"Last week I had to fix a memory leak in a Python program for the first time. A long running process started eating too much RAM (only ~20GB to much) and the friendly OOM Killer had to step in and terminate this. Since this kept happening, I had to go ahead and fix the issue.\nStep 1 - Reproduction As with every bug, before you can reliably fix it, you must reproduce it.","title":"Finding a memory-leak in my Python code"},{"content":"This post is brought to you in the spirit of converting tweetstorms to blogposts. to the tweetstorm\nSurprise! 🎁 In Python, if property access raises AttributeError, and the class implemented getattr, it will get called with the property name. This results in some very cryptic errors.\nIf you run the following code (repl):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 class Thing: name = \u0026#34;Thing\u0026#34; class NameProvider: def __init__(self, name): self.name = name def get_name(self): return self.nam class ThingWrapper: def __init__(self, thing, name_provider): self.thing = thing self.name_provider = name_provider @property def name(self): return self.name_provider.get_name() def __getattr__(self, name): return getattr(self.thing, name) def main(): thing = Thing() name_provider = NameProvider(name=\u0026#34;Not Thing\u0026#34;) thing_wrapper = ThingWrapper(thing, name_provider) print(thing.name) print(thing_wrapper.name) if __name__ == \u0026#39;__main__\u0026#39;: main() You\u0026rsquo;ll get a surprising result:\n1 2 Thing Thing You might have expected \u0026quot;Not Thing\u0026quot; as the second line, or maybe an exception to be raised from NameProvider.get_name() due to the typo there (self.nam instead of self.name). But instead, we got the name attribute from our Thing instance.\nAnalysis 🔎 If you\u0026rsquo;ve every used __getattr__() you know that it is called when the named attribute was not found using other lookup mechanisms. That said, it might not be clear to you that this includes properties raising AttributeError exceptions. It definitely wasn\u0026rsquo;t clear to me.\nThat is, it was unclear to me despite being clearly stated in the documentation for getattr()\nobject.__getattr__(self, name) Called when the default attribute access fails with an AttributeError (either __getattribute__() raises an AttributeError because name is not an instance attribute or an attribute in the class tree for self; or __get__() of a name property raises AttributeError). This method should either return the (computed) attribute value or raise an AttributeError exception.\nBeside being surprising, there are 2 main issues here:\nAny code down the stack from the property can effectively change attribute lookup for the class by throwing an AttributeError. In the above example - a typo in NameProvider caused an attribute to be taken from Thing instead, against the programmer\u0026rsquo;s obvious intention. The exception is silenced. There is no way for the programmer to catch the exception outside the property getter. This makes the errors very hard to track down. This also means that whenever you add __getattr__() to a class, you\u0026rsquo;re silencing all AttributeError exceptions that were previously thrown from properties. Like anything in Python, you can hack around the issue. In this case - with fancy decorators!\nSolution? 🐍 Consider the following code (repl):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 class ExceptionCatcher: def __init__(self, f): self.f = f self.exception = None def __call__(self, *args, **kwargs): try: return self.f(*args, **kwargs) except Exception as e: self.exception = e raise else: self.exception = None def store_exception(f): return ExceptionCatcher(f) def load_exception(f): def _raise_property_exception(instance, name): try: class_attr = getattr(instance.__class__, name) if not isinstance(class_attr, property): return exception = class_attr.fget.exception except AttributeError: return if exception: raise exception def _wrapper(*args, **kwargs): _raise_property_exception(*args, **kwargs) return f(*args, **kwargs) return _wrapper class ThingWrapper: def __init__(self, thing, name_provider): self.thing = thing self.name_provider = name_provider @property @store_exception def name(self): return self.name_provider.get_name() @load_exception def __getattr__(self, name): return getattr(self.thing, name) If you run this version, you\u0026rsquo;ll get the following exception:\n1 AttributeError: \u0026#39;NameProvider\u0026#39; object has no attribute \u0026#39;nam\u0026#39; This matches our expectations far better.\nThis result is achieved in two steps. First, we store all the exceptions thrown from name() so that we can throw them again if needed. Then, before calling __getattr__(), we check if we got there due to a property raising an exception. If we did - we just re-raise that exception.\nThe rest is implementation details, and I probably missed something there (you might notice that I corrected a bug when converting the tweets to this post - in the previous version, I forget to reset the exception storage after successful property retrieval).\nWhile this solution works, and may be useful for detecting similar bugs, I would probably avoid using it in production code. Instead, I\u0026rsquo;d be happy to have some standard Python construct to provide this functionality.\n","permalink":"https://tamir.dev/posts/til-python-attribute-lookup-order-is-tricky/","summary":"This post is brought to you in the spirit of converting tweetstorms to blogposts. to the tweetstorm\nSurprise! 🎁 In Python, if property access raises AttributeError, and the class implemented getattr, it will get called with the property name. This results in some very cryptic errors.\nIf you run the following code (repl):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 class Thing: name = \u0026#34;Thing\u0026#34; class NameProvider: def __init__(self, name): self.","title":"TIL Python attribute lookup order is tricky"},{"content":"Today we set out to implement a feature I saw and liked in Kotlin - Extension Methods.\nYou can follow along with working code samples here, or get the code here\nExtension methods are a nice piece of syntactic-sugar that allow you to define free-functions and call them like instance methods. In Kotlin, it looks something like this:\n1 2 3 4 5 6 7 8 fun Square.draw() { drawSquare(this) } // ... val square = getSquare() square.draw() Now, since they are free, static functions, they follow the same rules. They are not part of the class, nor have access to private members. And they can only be called in a scope where they are visible. Adding them in your code does not affect other code. Additionally, true member functions, if they exist, take precedence over extension methods (this is especially important with generic extension methods).\nIn our code today, we\u0026rsquo;ll try to mimic the features of extension methods as closely as possible. We\u0026rsquo;ll use the following syntax:\n1 2 3 @extend(Square) def draw(square): draw_square(square) For extension methods, and the following implementation of Square in our code throughout:\n1 2 3 4 5 from dataclasses import dataclass @dataclass class Square: length: int Monkey Patching 🙈 Python is a very dynamic language. Among other things, it allows us to change the attributes of (non-builtin) types at run-time. This means that we can extend our Square class by adding a draw method to it at run-time.\n1 Square.draw = draw_square We\u0026rsquo;re now free to call square.draw(). Before we discuss the draw-backs, let\u0026rsquo;s implement it with the syntax we defined:\n1 2 3 4 5 6 7 8 def monkey_extend(cls): def _decorator(f): setattr(cls, f.__name__, f) return _decorator @monkey_extend(Square) def draw(square): draw_square(square) Let\u0026rsquo;s go over this. monkey_extend is a decorator with arguments. This is a common pattern where we use a decorator factory (monkey_extend) to create a new decorator (_decorator) as a closure, giving it access to the parameters passed to the factory (cls). Then, in the core of the decorator, we use setattr to do our monkey-patching.\nWhile this works, it has several issues:\nScope - once set, it can be used with any Square in any scope Precedence - it will override any existing Square.draw Dealing with precedence is easy (using hasattr to check for existing .draw) so we\u0026rsquo;ll focus on the scoping first.\nDynamic Attribute Lookup ✨ The first thing we know is that we need our new attribute to be there in some cases, and be gone in others - we need dynamic resolution. To do that, we\u0026rsquo;ll use __getattr__. In Python classes, __getattr__ is used in attribute lookup as a last resort, called when the other ways of looking up attributes came up empty. We\u0026rsquo;ll write our __getattr__ along the following lines:\n1 2 3 4 5 6 def my_getattr(obj, name): if not has_extension(obj, name): raise AttributeError() if not is_in_scope(name): raise AttributeError() return our_extension The first check, has_extension, is basically checking whether the name we got matches the name of our extension method. Nothing to elaborate yet. Scoping, once again, remains the trickier part.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 import functools import inspect from collections import ChainMap def scoped_extend(cls): def _decorator(f): def _getattr(obj, name): # (2) if name != f.__name__: raise AttributeError() # (3) frame = inspect.stack()[1].frame scope = ChainMap(frame.f_locals, frame.f_globals) if scope.get(f.__name__) == f: raise AttributeError() # (4) return functools.partial(f, obj) # (1) cls.__getattr__ = _getattr return f return _decorator This is a bit much, so we\u0026rsquo;ll go over it in detail.\nAs a basis, we used the same decorator-with-parameters pattern here. We have scoped_extend take the class we want to extend, then return _decorator to get the job done. But instead of setting the attribute we want to extend, we monkey-patch cls\u0026rsquo;s __getattr__ to our implementation (See (1)). This will override any existing implementation of __getattr__, but we\u0026rsquo;ll get to that later. For now, we\u0026rsquo;ll focus on our implementation of __getattr__.\nIn (2) we implemented has_extnesion - we simply compare the name we got to the name of our extension method. Then, in (3), comes some Python magic. Python allows us to inspect the running program, to see where we were called from and what variables were in scope in that code. To do that, we use the inspect module. We use inspect.stack() to get the call-stack for the current execution, then access the second frame ([1]) to get our caller. This will be where getattr(obj, name) is invoked or obj.name is used. We use .frame to get the execution frame, and .f_locals and f_globals to get the local and global variables available in that scope. They are equivalent to calling globals() or locals() in the relevant frame.\nWith the scope at hand, we perform a lookup to see whether the extension method we defined is in that scope. To make sure we have our extension method, we get it by name, then ensure that it is truly our method.\nFinally, in (4), when we know our method should be active, we bind it to the instance of the extended class and return it.\nBetter Scoping While our scope retrieval code works, it\u0026rsquo;s better to put it in a function rather than use it inline:\n1 2 3 def _is_in_scope(name, value): frame = inspect.stack()[2].frame return ChainMap(frame.f_locals, frame.f_globals).get(name) == value But, oh, we have to increment the stack index to 2 since we\u0026rsquo;re deeper in the callstack. This is risky. Instead, we\u0026rsquo;ll use the following trick to get the frame:\n1 2 3 4 5 6 7 8 9 def _get_first_external_stack_frame(): for frameinfo in inspect.stack(): if frameinfo.filename == __file__: continue return frameinfo.frame def _is_in_scope(name, value): frame = _get_first_external_stack_frame() return ChainMap(frame.f_locals, frame.f_globals).get(name) == value Instead of counting the frames in our code, changing them with every change - we\u0026rsquo;ll use the module system. We know that all of our scaffolding is in the same module, but the usage is not. This allows us to easily traverse the stack until we find code that does not belong in our module. That is our calling code.\nSince you\u0026rsquo;re probably wondering - yes. You need to change _get_first_external_stack_frame() if you want to put it in a different module. Implementing it is left as an exercise to the reader.\nPreserving __getattr__ As mentioned before, our current implementation overrides any existing __getattr__ function for the class. Lucky for us, fixing it is easy:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 def no_override_extend(cls): def _decorator(f): def _default(_obj, _name): raise AttributeError() original_getattr = getattr(cls, \u0026#39;__getattr__\u0026#39;, _default) def _getattr(obj, name): with suppress(AttributeError): return original_getattr(obj, name) if name != f.__name__: raise AttributeError() if not _is_in_scope(f): raise AttributeError() return functools.partial(f, obj) cls.__getattr__ = _getattr return f return _decorator In (1) we get the original __getattr__ method, to be stored for later usage. We use the _default function to avoid an extra if later. In (2) we use the saved __getattr__, making sure that we only proceed to our code if it raised an AttributeError exception.\nInterlude 🐍 With no_override_extend we have our first \u0026ldquo;to-spec\u0026rdquo; implementation of extension methods. We have both scoping and precedence down. It is time to celebrate and rest. But our quest is not done yet.\nWhile our code works well for a proof-of-concept, there are still significant usability issues with it. Since the extension methods we create have nice and clean names, it is likely that we\u0026rsquo;ll want to use those names for other things. Unfortunately, once we do that, we\u0026rsquo;ll override the existing extension methods and they will no longer work:\n1 2 3 4 5 6 7 8 9 10 @extend(Square) def draw(square): draw_square(square) def draw(): print(\u0026#34;Drawing is awesome!\u0026#34;) # ... square.draw() # This will fail, as `draw` has been replaced in this scope. Indirection 🔀 The Fundemental Theorem of Software Engineering (FTSE) says that any problem can be solved by adding another level of indirection. Let\u0026rsquo;s see how this applies to our problem.\nAs mentioned in the interlude, our main issue is that of naming. Our extension method is bound to a name, and that name can be overriden in the scope that defines it. If that happens, we lose our extension method. To solve that, we\u0026rsquo;ll add another level of indirection - a scope that can safely hold our extension methods and protect them from being overriden. If you read our previous post you might recall that classes are wonderful for scopes. So we\u0026rsquo;ll use a class.\nOur new syntax will look like this:\n1 2 3 4 @extension class ExtensionMethods(Square): def draw(self): draw_square(self) While we\u0026rsquo;re still using a decorator, you may notice that it takes no parameters. Instead, we use the extended type as the base type for our extension class. This allows us to write the extensions like any other subclass, with standard Python syntax, and then use the decorator to install the extensions in it.\nSince we\u0026rsquo;ve already gone over the principles behind the construction of the decorator, let\u0026rsquo;s jump straight to the code and focus on the differences from the previous version:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 def extension(scope_cls): def _default(_obj, _name): raise AttributeError() # (1) cls = scope_cls.__base__ original_getattr = getattr(cls, \u0026#39;__getattr__\u0026#39;, _default) def _getattr(obj, name): with suppress(AttributeError): return original_getattr(obj, name) # (2) if not hasattr(scope_cls, name): raise AttributeError() # (3) if not _is_in_scope(scope_cls): raise AttributeError() # (4) f = getattr(scope_cls, name) return functools.partial(f, obj) cls.__getattr__ = _getattr return scope_cls First, you can see that there is no nested decorator - only the main one. And, as we mentioned before, we use inheritance to indicate which type we\u0026rsquo;re extending. So in (1) we access the base-class of our extension class to get the class we\u0026rsquo;re extending. Then, in (2) we check whether the requested attribute exists in our extension class. As you can see, the changes are pretty simple and straight-forward. In (3) we make the most important change - we check for the extension class in the scope, not the extension methods. This is the core of this change! And lastly, in (4), we get the required attribute from out extension class.\nAnd with that, we\u0026rsquo;re done.\nFinal Words I hope you enjoyed this article. Regardless of that, I hope you never use it in production code.\n","permalink":"https://tamir.dev/posts/snake-eyes-extension-methods/","summary":"Today we set out to implement a feature I saw and liked in Kotlin - Extension Methods.\nYou can follow along with working code samples here, or get the code here\nExtension methods are a nice piece of syntactic-sugar that allow you to define free-functions and call them like instance methods. In Kotlin, it looks something like this:\n1 2 3 4 5 6 7 8 fun Square.draw() { drawSquare(this) } // .","title":"Snake Eyes: Extension Methods"},{"content":"One of my pet peeves is taking concepts from other languages and \u0026ldquo;translating\u0026rdquo; them to Python. Not because it makes good code, but because it\u0026rsquo;s a challenge and it makes me happy.\nThis time, I\u0026rsquo;ve gone after two simple concepts - nested code blocks and IIFE. Both serve similar purposes, and both are missing from Python.\nIn C++, blocks are often used to limits the lifetime of objects and keep them out of our way when we\u0026rsquo;re done with them. In Python, lifetime is usually less of a concern (as we replace RAII and destructors with context-managers), but having variable names out of our way is desirable.\nIIFE offers us a bit more in terms of functionality, as it both creates a scope for our operations, and allows us to get a value from that scope. This is useful both for simpler flow control, and for easily initializing const-qualified variables.\nPython does not have any of those constructs. There is no way to create a nested code-block in Python (adding another level of indentation would just have it complain about unexpected-indent), and while lambdas exist, they only allow for a single expression, making them mostly irrelevant for IIFE. On the other hand, Python offers us two wonderful constructs that can be used virtually everywhere - classes and functions.\nClasses \u0026amp; Nested Blocks 🧱🧱🧱 In Python, both classes and functions can be nested. You can define a class inside a class, a function in a function, a class in a function or a function in a class. It is all the same. What\u0026rsquo;s more - you can have flow-control in both function (well, obviously) and class bodies (meta-programming much?). Additionally, both nested functions and nested classes create new variable scopes, keeping their insides inside, and are also closures, capable of capturing values from their enclosing scopes. As such, they are the perfect tools for our language-bending shenanigans.\nFirst, nested code blocks. I offer you the following solution:\n1 2 3 4 5 6 7 8 9 10 def f(x): print(\u0026#39;Classes are great for creating blocks.\u0026#39;) class _: y = x * 2 print(f\u0026#39;y = {y}\u0026#39;) print(\u0026#39;y is not defined here.\u0026#39;) y f(21) By defining a class, we create a new scope. Inside it, we can do whatever we want, knowing that the code will get execute inline and in order, and the results will not leak into the enclosing scope.\nThat said - there are some caveats. First, the class remains in scope, and so do all the variables defined in it. They cannot be garbage collected until the function terminates. You can verify it yourself by trying to access _.y in the above example. To remedy that, we need to get rid of the class, or at least its contents. There are many ways to achieve it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 # Replace the class with a bool @bool class _: x = 1 print(x) # Replace the class with None def empty(*args): return None @empty class _: x = 1 print(x) # Use a metaclass to delete all the variables inside the class class BlockMeta(type): def __new__(cls, name, bases, dict_): return super().__new__(cls, name, bases, {}) class Block(metaclass=BlockMeta): pass class _(Block): x = 1 print(x) I am personally torn between the meta-class approach, as it is explicit and clear, and the @bool approach, as it requires to additional boilerplate.\nThe second issue with classes as blocks is that while they can be nested freely, a nested class cannot access the variables of its nesting class, rendering block-nesting moot. I do not have a solution for that at present.\nFunctions \u0026amp; IIFE 🐍🐍🐍 With a solution for nested blocks in hand, it is time to get proper IIFE in Python. For that, we\u0026rsquo;ll naturally be using function. Along with those, we\u0026rsquo;ll use a function\u0026rsquo;s best friend - the decorator!\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 def iife(f): return f() def describe_number(n): @iife def message(): if n \u0026lt; 0: return f\u0026#39;{n} is smaller than 0\u0026#39; elif n \u0026gt; 0: return f\u0026#39;{n} is larger than 0\u0026#39; return f\u0026#39;{n} is 0\u0026#39; print(message) describe_number(-1) describe_number(0) describe_number(1) Using the decorator, we immediately call the function, binding the function\u0026rsquo;s name to the return value instead of the function itself. A function returning None (or without a return statement) will just bind the name to None.\nWhile this looks a bit more messy, it can also double as a solution for nested blocks. And unlike the class solution - it can be freely nested.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 def iife(f): return f() def block(f): f() def f(x): print(\u0026#39;Functions are great for creating blocks.\u0026#39;) @block def _(): my_x = x + 1 @iife def y(): return my_x * 2 @block def _(): print(f\u0026#39;y = {y}\u0026#39;) print(\u0026#39;y is not defined here.\u0026#39;) y f(20) That\u0026rsquo;s it for today. I hope you had some fun.\n","permalink":"https://tamir.dev/posts/snake-eyes-scopes-and-iife/","summary":"One of my pet peeves is taking concepts from other languages and \u0026ldquo;translating\u0026rdquo; them to Python. Not because it makes good code, but because it\u0026rsquo;s a challenge and it makes me happy.\nThis time, I\u0026rsquo;ve gone after two simple concepts - nested code blocks and IIFE. Both serve similar purposes, and both are missing from Python.\nIn C++, blocks are often used to limits the lifetime of objects and keep them out of our way when we\u0026rsquo;re done with them.","title":"Snake Eyes: Scopes and IIFE"},{"content":"After multiple attempts at finding a funny narrative that holds for the entire article and failing miserably, I decided to go with the technical parts alone. Enough of my colleagues found it interesting, so I guess it will hold up without the jokes.\nPython gives us multiple ways to check that the objects we pass to functions are of the types we expect. Each method has its advantages and disadvantages.\nJust not caring The first option is naturally to not care about types - just write your code, and hope for the best. This is a viable method, and is often employed. It is especially fitting in short snippets or scripts you don\u0026rsquo;t expect to maintain much. It just works with no overhead whatsoever.\n1 2 3 def poke(duck): duck.quack() duck.walk() Inheritance Another option, as common in OOP languages, is to use inheritance. We can define an Anas class, and expect all of its derivatives to be sufficiently duck-like.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 class Anas: def quack(self): pass def walk(self): pass class Duck(Anas): def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walks like duck.\u0026#39;) class Mallard(Anas): def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walks like duck.\u0026#39;) def poke(duck): assert isinstance(duck, Anas) Interfaces While inheritance kinda gets the job done, a robotic duck is definitely not of the genus Anas, while it does implement all the characteristics we care about. So instead of hierarchical inheritance, we can use interfaces.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 from abc import ABC, abstractmethod class IDuck(ABC): @abstractmethod def quack(self): pass @abstractmethod def walk(self): pass class Duck(IDuck): def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walks like duck.\u0026#39;) class RoboticDuck(IDuck): def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walks like duck.\u0026#39;) def poke(duck): assert isinstance(duck, IDuck) Great. And if we don\u0026rsquo;t control the types, we can always write adapters.\nThe Duck Test But this is Python. We can do better.\nAs we know, Python uses duck-typing. So we should be able to use the Duck Test for types. In our example, every object implementing quack() and walk() is a duck. That\u0026rsquo;s easy enough to check.\n1 2 3 4 5 6 7 8 9 10 def is_a_duck(duck): for attr in (\u0026#39;quack\u0026#39;, \u0026#39;walk\u0026#39;): if not hasattr(duck, attr): return False return True def poke(duck): assert is_a_duck(duck) duck.quack() duck.walk() This works. But we list the isinstance(...) call. We can surely do better.\nMetaclasses \u0026amp; Subclass Hooks Metaclasses are wonderful constructs. As their name may suggest, they take part in the construction of classes. They even allow us to set hooks into basic Python mechanisms, like isinstance(...), using __subclasshook__.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 from abc import ABC def is_a_duck(duck): for attr in (\u0026#39;quack\u0026#39;, \u0026#39;walk\u0026#39;): if not hasattr(duck, attr): return False return True class DuckChecker(ABC): @classmethod def __subclasshook__(cls, C): if cls is not DuckChecker: return NotImplemented return is_a_duck(C) def poke(duck): assert isinstance(duck, DuckChecker) duck.quack() duck.walk() And we\u0026rsquo;re back in business. That said, is_a_duck is still a stringly-typed mess, and gonna be very painful to maintain.\nWouldn\u0026rsquo;t it be nice if we could just use our IDuck interface to check for duck-ness?\nAbstract Methods, Again! Lucky for us - we can!\nAmong other things, the ABC parent class enumerates all @abstractmethods and stores them in the __abstractmethods__ member variable. This means that we can easily enumerate them in our subclass hook and check for them.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 from abc import ABC, abstractmethod class IDuck(ABC): @abstractmethod def quack(self): pass @abstractmethod def walk(self): pass @classmethod def __subclasshook__(cls, C): if cls is not IDuck: return NotImplemented for attr in cls.__abstractmethods__: if not hasattr(C, attr): return False return True class Duck: def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walks like a duck.\u0026#39;) def poke(duck): assert isinstance(duck, IDuck) duck.quack() duck.walk() poke(Duck()) # Quack! # Walks like a duck. Awesome. Next step - separating the interface from the checking logic.\nProtocols Reading through Python documentation and nomenclature, you might have seen the term \u0026ldquo;protocol\u0026rdquo; here and there. It is Python\u0026rsquo;s way to call duck-typed interfaces. So you could say we just created a \u0026ldquo;protocol checker\u0026rdquo;. Now, we can separate it into a base-class.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 from abc import ABC, abstractmethod class Protocol(ABC): _is_protocol = True @classmethod def __subclasshook__(cls, C): if not cls._is_protocol: return NotImplemented for attr in cls.__abstractmethods__: if not hasattr(C, attr): return False return True class IDuck(Protocol): @abstractmethod def quack(self): pass @abstractmethod def walk(self): pass And that\u0026rsquo;s it. That little _is_protocol flag is there for good reason. Usually, we\u0026rsquo;d check protocol-ness using isinstance(...). In this case, however, we\u0026rsquo;re hooking into that mechanism and that would lead to infinite recursion.\nWe can now use our Protocol base-class freely to create new protocols as we need them, with friendly interface-like syntax. We\u0026rsquo;re almost done.\nThis Dog is a Duck In some cases, the protocol checks may not be what we want. The obvious reasons coming to mind are:\nWe can\u0026rsquo;t really check the desired semantics using the protocol trick. We want to wreck havoc. For those cases (well, mostly for the first one) the ABC base class provides another trick. Instead of defining __subclasshook__ to check the interface, we can simple register classes as valid, \u0026ldquo;virtual subclasses\u0026rdquo;.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 from abc import ABC class IDuck(ABC): pass class Duck: def quack(self): print(\u0026#39;Quack!\u0026#39;) def walk(self): print(\u0026#39;Walk like a duck.\u0026#39;) IDuck.register(Duck) def poke(duck): assert isinstance(duck, IDuck) duck.quack() duck.walk() Remember that this method puts all the pressure on the programmer. Writing IDuck.register(Dog) is the equivalent of saying \u0026ldquo;I vouch for this dog to be a duck\u0026rdquo;. It might pass inspection, but won\u0026rsquo;t necessarily yield the desired results.\nSummary In this article we covered multiple ways of checking the \u0026ldquo;duck-ness\u0026rdquo; of objects. From belonging to the Anas genus, to just placing a sticker on their head saying \u0026ldquo;Duck!\u0026rdquo;. Some of those methods are more useful or applicable than others, but I still think it worthwhile to be familiar with all of them. Additionally, there are many topics not covered here, like static type checking.\nFurther Reading The metaclass techniques demonstrated here are simplified versions of code from the abc and typing modules. I highly recommend going through those modules and their respective docs, at least at a glance, to extend your knowledge and cover up any holes left by my hand-wavy explanation.\n","permalink":"https://tamir.dev/posts/recognizing-ducks/","summary":"After multiple attempts at finding a funny narrative that holds for the entire article and failing miserably, I decided to go with the technical parts alone. Enough of my colleagues found it interesting, so I guess it will hold up without the jokes.\nPython gives us multiple ways to check that the objects we pass to functions are of the types we expect. Each method has its advantages and disadvantages.","title":"Recognizing Ducks"},{"content":"Internationalization is a difficult problem is software-engineering. Usually that statement would refer to the technical aspects of providing a good user experience for your customers. Today, however, I am referring to the social aspects.\nJust as it has become common-place to have your users spread across the globe, so it has with developers. It is not uncommon to work with teams in another country, or even to have a specific team-member working from a remote location. Being able to cooperate with developers across the globe is a great enabler. But as the case is with - well - everything, it has some obvious - and some less obvious - pitfalls to avoid.\nWhere (When?) Are My Colleagues? First and foremost - know the time-zones your colleagues reside in.\nYou already know that some of your teammates tend to arrive a bit early or a bit late. You know it, and you adjust to it. You won\u0026rsquo;t make a colleague who starts 2 hours before you stay for a meeting when you know they should be picking up their kids from kindergarten, right?\nWell, all you need to do now is adjust to having teammates arriving at work 10 hours before you.\nTime For Your 12AM Daily Standup Yes, that\u0026rsquo;s AM. The one that\u0026rsquo;s confusingly set at midnight.\nNow imagine that you\u0026rsquo;re in a UTC-07:00 time-zone (Pacific Daylight Time) and the rest of your team is in a UTC+03:00 time-zone (Israel Daylight Time). Your daily-standup is at 10AM, which sounds reasonable. But that\u0026rsquo;s in Israel. Sadly, this translates to 12AM for you. But you can accommodate that, can\u0026rsquo;t you?\nLuckily, in the real-world company HQs tend to be in the USA, not Israel. So you\u0026rsquo;re safe. You can hold your meetings at any time you want, and they\u0026rsquo;ll have to adjust. You can have them wake up early, or stay up late, or just work very awkward shifts. They\u0026rsquo;ll accommodate. But do you really want to make your teammates do that?\nGet It Done By EOD Today No.\nIt\u0026rsquo;s not that I don\u0026rsquo;t want to, it simply isn\u0026rsquo;t possible.\nIf you have a large enough time-difference (say, 6 hours difference over a 9 hours work-day) your day starts when your colleagues\u0026rsquo; day ends (or later!)\nNo matter how hard they work, they cannot get it done by EOD if EOD has already passed. You can talk about getting things done \u0026ldquo;by tomorrow\u0026rdquo;, but not EOD. It causes un-needed tension as the need to explain this comes up over and over during planning meetings.\nPlanned Downtime: 9PM-6AM Great. You\u0026rsquo;ve just killed of a day\u0026rsquo;s work for your distant colleagues. 9PM San-Francisco is 7AM in Tel-Aviv. Making it a 7AM-4PM downtime for them.\nI\u0026rsquo;m not saying you can\u0026rsquo;t do that, but you should try and keep that in mind.\nAnd Then, There\u0026rsquo;s Israel And on the seventh day God ended His work which He had done, and He rested on the seventh day from all His work which He had done.\nThen God blessed the seventh day and sanctified it, because in it He rested from all His work which God had created and made.\nGenesis 2:2-3\nAnd so in Israel we rest on the 7th day, and add the 6th in for good measure.\nWe work Sunday through Thursday, and rest on Friday and Saturday. And this is annoying.\nIt is annoying when we need support on Sunday and have to wait.\nIt is annoying when we travel, and are never sure if we should report Friday as overtime, or Sunday as a day off.\nAnd it is annoying to be expected to work on Friday evening just because it\u0026rsquo;s a working-day for you. You wouldn\u0026rsquo;t want to work Saturday nights, would you? I didn\u0026rsquo;t think so.\nAlso, you should expect email send on Fridays and late Thursdays to be ignored until Sunday.\nIn Conclusion This has been a bit of a rant, and a bit of advice as well. I\u0026rsquo;m not sure which dominated the tone.\nThis might seem like a minor issue, but it can cause a great deal of pain when ignored.\nBut in the end, the rules are simple. All you need to do is be aware, and be considerate. You can do it for the people you physically see everyday, and you can do it for your remote colleagues.\nTry and set your long-distance meeting at times suitable for both ends. And when that isn\u0026rsquo;t possible - make sure you make an effort as well. Waking up an hour earlier every now and then goes a long way.\n","permalink":"https://tamir.dev/posts/working-across-timezones/","summary":"Internationalization is a difficult problem is software-engineering. Usually that statement would refer to the technical aspects of providing a good user experience for your customers. Today, however, I am referring to the social aspects.\nJust as it has become common-place to have your users spread across the globe, so it has with developers. It is not uncommon to work with teams in another country, or even to have a specific team-member working from a remote location.","title":"Working Across Time-Zones"},{"content":"In the few years since Rust came out, I\u0026rsquo;ve frequently found myself explaining what an amazing language Rust is. Laying out the ways in which it is going to change the systems-programming world, and what a pleasure Rust is to code in.\nA few months back, I finally got around to trying it out. It was horrid. The darned borrow-checker just wouldn\u0026rsquo;t let me do anything. I couldn\u0026rsquo;t get anything meaningful to compile. Despite years of programming in various languages I felt completely incompetent to handle it. I resolved to drop Rust and focus on other things.\nDespite my initial disappointment and discouragement, I still could not get Rust out of my head. Its premise is too promising to put down after one go. Besides, I cannot be beaten by a programming language! And yet, it can wait. It is too much work for now. Some day, maybe in a few years, I\u0026rsquo;ll write Rust.\nUsing Rust, however, is a completely different experience. Rust tools are absolutely amazing. As a Windows user through-and-through I got used to open-source tools (especially command line tools) not being supported on Windows. (No, cygwin is not a valid solution.) I don\u0026rsquo;t blame the devs - they work on Linux. Even if they have the time to spend on the Windows port, they don\u0026rsquo;t necessarily have a Windows machine to test it on. And yet - I am used to being disappointed. That is why, when I first heard of rg(an amazing grep replacement) and fd(an amazing find replacement) I knew that they would not work on Windows. But, being my optimistic self - I checked. And a good thing I did that.\nTo install Rust tools, the easiest way is to install the Rust toolset and compile them. A daunting task in every other language, yet a breeze in Rust.\nHead over to rustup.rs and install Rust (A single command-line on Linux, a single executable on Windows) cargo install ripgrep fd-find That\u0026rsquo;s it. Really. Now use the tools. This was when I realized how amazing Rust really is. Even if you ignore the language completely - it\u0026rsquo;s tooling and package management is unparalleled. Having published Python packages in the past, I was amazed at the simple publishing and installation mechanisms. Having used C and C++ I was simply amazed at a systems-programming with package management. So while still somewhat scared of the borrow-checker, I decided that my next CLI tool will be written in Rust. The easy-as-pie deployment bought be over completely.\nSome months after that, I finally found myself in need of a new CLI tool. I was faced with a continuous log I wanted to filter for duplicates. sort -u sorts, so it cannot work on streams. Of course, there is probably some sed or awk magic I can use, but I want something simple. Besides, a tool that filters out duplicates seems like the perfect beginner project for getting hands-on with Rust. So I went ahead and creates uq. After finishing it, I published it on crates.io. cargo install uq on a second machine, and it worked. Both Windows and Linux. A friend tried it, and it simply worked! I never had such a pleasant deployment experience. It is practically easier to install from source then send a compiled binary! And it works cross-platform out of the box.\nA short while later I wanted to group some log entries by a regex. I looked around and could not find a simple way to do it. So, once again, I turned to Rust. Some borrow-checker-wrestling later and the first version of groupby was complete. Yay!\nA short time later I had one of the best open-source experiences I\u0026rsquo;ve ever had. Someone started using groupby, looked at my terrible code, and posted this issue:\nHello\nI find this little program will be useful for many things I do (I usually do something like that with combination of sed, sort, …). I also looked into the code. Do I guess right that you\u0026rsquo;re still learning Rust? Could I provide few little tips?\nI glimpsed at least one unwrap that can be triggered by the user input (giving a too large group ID), which will result in ugly error message instead of nice useful one. Do you choose BTreeMap/BTreeSet for some specific reason? Is it the order of elements? If not, HashMap and HashSet are likely to be faster. Both variants (unique vs all) look very similar and differ only in the inner data type and the method used to push/insert. I think this could be done with just one piece of code that is generic over the type, and adding your own trait that implements the adding for either. Would you like me to show you such code?\nHaving someone more experienced in Rust come in and help me improve my very naïve code was great. And it was my first time getting a \u0026ldquo;this is great, may I help you?\u0026rdquo; comment and not a \u0026ldquo;this is great, I want this as well\u0026rdquo; one.\nFor the time being, I keep spending more time wrestling the borrow-checker than writing actual code in Rust. But I am (almost) sure it is due to lack of experience. On the plus-side, I\u0026rsquo;m becoming better at detecting lifetime issues in other languages as well.\nSo, for anyone who hasn\u0026rsquo;t done it yet, I highly recommend using Rust-based tools. Just for the amazing experience of things working (and compiling!) out of the box. Later, if you choose to actually code in it, be sure to brace yourself for a somewhat bumpy ride. Friends tell me that after a time Rust becomes easier, speeding up their development. I\u0026rsquo;m not there yet, but I\u0026rsquo;m working on it.\n","permalink":"https://tamir.dev/posts/adventures-in-rust/","summary":"In the few years since Rust came out, I\u0026rsquo;ve frequently found myself explaining what an amazing language Rust is. Laying out the ways in which it is going to change the systems-programming world, and what a pleasure Rust is to code in.\nA few months back, I finally got around to trying it out. It was horrid. The darned borrow-checker just wouldn\u0026rsquo;t let me do anything. I couldn\u0026rsquo;t get anything meaningful to compile.","title":"Adventures in Rust"},{"content":"Recently I\u0026rsquo;ve been helping \u0026amp; tutoring some true code beginners. Not someone new to a language, but completely new to programming.\nI\u0026rsquo;ve done a lot of training in the past. Both beginner and advanced training, both programming and reverse-engineering. But as green as my previous students have been, they have always had some prior knowledge, some experience with code. In at least one programming language.\nUsually the training is about teaching language features, special tricks, best practices, and getting the trainees familiar with the new patterns. The trainees send out probes, looking for familiar things, and I just fill them in at the right time. They know what they are looking for, or can be easily guided towards the right thing. When people are completely green, they don\u0026rsquo;t.\nThis is a very new experience for me, and it got me thinking a lot about programming and the ways we approach code. The patterns we seek to find or form. The amazing number of things that we do without even thinking as experience programmers. Each of those, no matter how simple, needs to be broken up and explained to new-comers. They have no previous knowledge to build upon for this.\nFrom my current experience, it seems the understanding the meaning of syntax, and understanding forward-flowing programs is easy enough. Conditionals are a non-issue. The first road-block comes with loops. Especially writing loops. Where do I put the return statement? Where do I define my variables? Trying to explain those things, and give simple rules, I came to some useful realizations of useful patterns, and some painful truths about our use of jargon.\nLet\u0026rsquo;s go ahead and see the patterns.\nFind Loops 1 2 3 4 5 6 7 8 9 public static int indexOf(String[] haystack, String needle) { for (int i = 0; i \u0026lt; haystack.length; ++i) { if (needle.equals(haystack[i])) { return i; } } return -1; } In those loops we iterate over the array, looking for an item that fulfills our condition. Once we find it, we immediately return that value. There is no need to declare any variables.\nCount Loops 1 2 3 4 5 6 7 8 9 10 public static int countOf(String[] haystack, String needle) { int count = 0; for (int i = 0; i \u0026lt; haystack.length; ++i) { if (needle.equals(haystack[i])) { count++; } } return count; } In those loops we iterate over the array, looking for items that fulfill our condition. Whenever we find one, we increment the count. Once we exhaust the iteration, we return the counter. The counter and return statement are outside the loop.\nAction Loops 1 2 3 4 5 public static void printAll(String[] haystack) { for (int i = 0; i \u0026lt; haystack.length; ++i) { System.out.println(haystack[i]); } } In those loops we iterate over the array, and perform an action on each and every item. There are no variables and no return statements.\nNow, those simple loops can do quite a lot, and can be expanded and composed to do more. And I find that they help beginners. But did you spot my error? I used the word \u0026ldquo;iterate\u0026rdquo;.\nVocabulary While the meaning of \u0026ldquo;iterate\u0026rdquo; is clear to existing programmers, and looking at the loops you can easily tell that we are iterating or looping over haystack, it is not clear for beginners. Moreover, the words themselves sound weird. This is critical, and becomes more pronounced as you try and loop in slightly more advanced ways\n1 2 3 4 int i = 0; for (Node current = myList.head; current != null; current = current.getNext(), ++i) { // ... } Here, we are looping (or iterating) over myList, but we don\u0026rsquo;t change anything about it, or even access it directly. We do change i (which is no longer our counter) and current which is a node. This makes the code and language quite dissonant. \u0026ldquo;We iterate over myList while maintaining an index\u0026rdquo; is a true statement, but not an immediate translation from the code. The language forces to go in a very roundabout manner. This is true for many languages. But now, consider slightly more modern syntax:\n1 2 for i, node in enumerate(myList): # ... 1 2 3 for i, node := range myList { // ... } 1 2 3 for (i, node) in myList.enumerate() { // ... } In all of those, the situation is far clearer. We can see that we\u0026rsquo;re looping over myList, and it is clear that we have both a node and an index. While this difference might be minor for experienced programmers, it is a world of difference for newcomers.\nLearning to code is not just learning a programming language. Not just learning to think in a specific way. It is learning your own language again. You know English? Well, now you need to learn programming-English. You know Hebrew? Learn programming-Hebrew. We keep changing the meaning of existing words, and expect people to follow and understand them. It is hard. The least we can do is try and minimize the difference between the code we read (programming languages - Java, C, C++, Python, Go, Rust\u0026hellip;) and the code we speak (well, I guess English is a programming language as well?).\n","permalink":"https://tamir.dev/posts/types-of-loops/","summary":"Recently I\u0026rsquo;ve been helping \u0026amp; tutoring some true code beginners. Not someone new to a language, but completely new to programming.\nI\u0026rsquo;ve done a lot of training in the past. Both beginner and advanced training, both programming and reverse-engineering. But as green as my previous students have been, they have always had some prior knowledge, some experience with code. In at least one programming language.\nUsually the training is about teaching language features, special tricks, best practices, and getting the trainees familiar with the new patterns.","title":"Types of Loops"},{"content":"First, an apology. The first part of this post was published on May 26. It is now September. I had most of the code for this part done by then. But finalizing the code took some more effort. Once that was done, explaining took a while. There were quite a few things I had to learn myself first. So now, months later, I present this humble offering to the Gods of C++ and template meta-programming.\nGeneralizing In Part 1 we created our State or SelfReturning class (provided below for reference). It works, but as you can see - required modifications whenever we change the function arguments or return types.\nCompilation\n1 2 3 4 5 6 7 8 9 10 11 struct SelfReturning { using RetType = std::pair\u0026lt;SelfReturning, const Context\u0026gt;; using FuncType = RetType(*)(const Context\u0026amp;, Event); SelfReturning(FuncType func) : _func{func} {}; RetType operator()(const Context\u0026amp; ctx, Event evt) { return _func(ctx, evt); } FuncType _func; }; using State = SelfReturning; The first thing we want to do is get rid of this requirement. First, function arguments. Compilation\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 template \u0026lt;class... Args\u0026gt; struct SelfReturning { using RetType = std::pair\u0026lt;SelfReturning, const Context\u0026gt;; using FuncType = RetType(*)(Args... args); // (1) SelfReturning(FuncType func) : _func{func} {}; RetType operator()(Args... args) { // (2) return _func(std::forward\u0026lt;Args\u0026gt;(args)...); } FuncType _func; }; using State = SelfReturning\u0026lt;const Context\u0026amp;, Event\u0026gt;; Here we use variadic templates and perfect forwarding to forward all the function arguments directly to the target function. You can see that in (1) and (2) we use Args... and not the common Args\u0026amp;\u0026amp;.... This is because the types are defined by the class template and are not deduced on the function call.\nWith this behind us, we address the return type. Here we come to another recursive issue. While the return type std::pair\u0026lt;SelfReturning, const Context\u0026gt; depends on our SelfReturning type, SelfReturning itself depends on the return type. This means that just passing in the return type will not work (much like our original return-type issue). To solve it, we use a template-template parameter.\nCompilation\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 template \u0026lt;template \u0026lt;class T\u0026gt; class Base, class... Args\u0026gt; // (1) struct SelfReturning { using RetType = Base\u0026lt;SelfReturning\u0026gt;; //(2) using FuncType = RetType(*)(Args... args); SelfReturning(FuncType func) : _func{ func } {} RetType operator() (Args... args) { return _func(std::forward\u0026lt;Args\u0026gt;(args)...); } FuncType _func; }; template \u0026lt;class T\u0026gt; using PairWithCtx = std::pair\u0026lt;T, const Context\u0026gt;; // (3) using State = SelfReturning\u0026lt;PairWithCtx, const Context\u0026amp;, Event\u0026gt;; In (1), we pass in the template for the return type. In (2), we instantiate the type with our SelfReturning class. As we\u0026rsquo;ve seen before, C++ allows this type of recursion, so we\u0026rsquo;re safe. In (3) we define our return-type template to be a pair with a const Context as the second member. Done.\nBut what if we want to only return the SelfReturning class? For that, we define a new template - an identity template.\n1 2 3 4 5 6 7 template \u0026lt;class T\u0026gt; struct identity { using type = T; }; template \u0026lt;class T\u0026gt; using identity_t = typename identity\u0026lt;T\u0026gt;::type; We define the identity struct to hold a type, and use the identity_t alias to access the type directly. This looks a bit odd, but C++ does not allow us to alias the template parameter directly. When isntatiating the identity_t template with a type, we get the safe type again. Using that, we can return SelfReturning directly.\n1 using State = SelfReturning\u0026lt;identity_t\u0026gt;; Personally, though, I hate having to write down the trivial cases explicitly. So let\u0026rsquo;s use some dirty tricks.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 template \u0026lt;template \u0026lt;class T\u0026gt; class Base = identity_t, class... Args\u0026gt; // (1) struct SelfReturning { using RetType = Base\u0026lt;SelfReturning\u0026gt;; using FuncType = RetType(*)(Args... args); SelfReturning(FuncType func) : _func{ func } {} RetType operator() (Args... args) { return _func(std::forward\u0026lt;Args\u0026gt;(args)...); } FuncType _func; template \u0026lt;class... AltArgs\u0026gt; using WithArgs = SelfReturning\u0026lt;Base, AltArgs...\u0026gt;; // (2) }; In (1) we simply add identity_t as the default argument for Base. This lets us write the most trivial case (return SelfReturning, take no arguments) as SelfReturning\u0026lt;\u0026gt;. Nice. However, if we put anything into this set of template arguments, it will override identity_t. That\u0026rsquo;s what the code at (2) is for. We set WithArgs to be SelfReturning with whatever Base parameter it already has, thus only accepting template parameters for the arguments. Now we can write all of the following with ease. Compilation\n1 2 3 4 5 6 7 using Trivial = SelfReturning\u0026lt;\u0026gt;; using InPair = SelfReturning\u0026lt;PairWithCtx\u0026gt;; using TrivialWithArgs = SelfReturning\u0026lt;\u0026gt;::WithArgs\u0026lt;Context\u0026amp;, Event\u0026gt;; using InPairWithArgs = SelfReturning\u0026lt;PairWithCtx\u0026gt;::WithArgs\u0026lt;const Context\u0026amp;, Event\u0026gt;; // Or alternatively using InPairWithArgs2 = SelfReturning\u0026lt;PairWithCtx, const Context\u0026amp;, Event\u0026gt;; In Part 1 I promised generalizing the SelfReturning class and getting some compile time guarantees. We\u0026rsquo;ve accomplished our generalization goal, so it\u0026rsquo;s time to get some safety in place.\nIncreasing Safety While our use of the switch statement to discern different events is nice and concise, it is also somewhat error prone. It is easy to miss a case (though that can be prevented using compiler errors) or accidentally mistake one event for another. The latter is especially true if we want to pass information along with our event notification. One easy way to avoid those mistakes is to resolve the choice using function overloading instead of switch statements. Consider the following\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 #include \u0026lt;cstdio\u0026gt; // (1) enum class EventType {A, B}; void Switch(EventType evt) { switch(evt) { case EventType::A: puts(\u0026#34;A\u0026#34;); return; case EventType::B: puts(\u0026#34;B\u0026#34;); return; } } // (2) struct EventA {}; struct EventB {}; void Overload(EventA) { puts(\u0026#34;A\u0026#34;); } void Overload(EventB) { puts(\u0026#34;B\u0026#34;); } In (1) we use a switch to discern the event type. It is easy to forget a return or a break. If we passed more data along, the signature for Switch would likely change to void Switch(EventType evt, void* data). That\u0026rsquo;s definitely bad. In (2), we cannot mistake the types, and data can easily be passed inside the event structs. Sadly, the events are not different types, and C++ does not allow for heterogeneous containers. Or does it?\nEnter C++17\u0026rsquo;s ✨std::variant✨.\nWhat is std::variant, you ask? Well, it is a union. A safe union! Safe meaning that you can only get a value from it if it really is there. No more type confusion; no more casting void pointers. But how do we get the values out of std::variant? Using std::visit, of course! Compilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 #include \u0026lt;variant\u0026gt; #include \u0026lt;vector\u0026gt; #include \u0026lt;cstdio\u0026gt; struct EventA {}; // (1) struct EventB {}; struct EventHandler { // (2) void operator() (EventA) { puts(\u0026#34;A\u0026#34;); } void operator() (EventB) { puts(\u0026#34;B\u0026#34;); } }; using event_t = std::variant\u0026lt;EventA, EventB\u0026gt;; int main() { std::vector\u0026lt;event_t\u0026gt; events = {EventA{}, EventB{}}; for (auto\u0026amp; event : events) { std::visit(EventHandler{}, event); // (3) } return 0; } In (1) we define our new event types. This time they are different types, not just different values. In (2) we define our event handler. All we need is an function overload for every possible type, and a struct with multiple operator() methods is an easy way to do it. Now all that is left to do is call std::visit with our handler and an event. If we forget to handle an event type - the code does not compile! This way, we know that we always handle all event types, and never mix them up.\nNow, if you liked the previous part, you probably don\u0026rsquo;t like writing a different handler class for every function. It completely ruins the locality of the code. But, we are using C++17, aren\u0026rsquo;t we? Compilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 #include \u0026lt;variant\u0026gt; #include \u0026lt;vector\u0026gt; #include \u0026lt;cstdio\u0026gt; template\u0026lt;class... Ts\u0026gt; struct overloaded : Ts... { using Ts::operator()...; }; // (1) template\u0026lt;class... Ts\u0026gt; overloaded(Ts...) -\u0026gt; overloaded\u0026lt;Ts...\u0026gt;; struct EventA {}; struct EventB {}; using event_t = std::variant\u0026lt;EventA, EventB\u0026gt;; int main() { std::vector\u0026lt;event_t\u0026gt; events = {EventA{}, EventB{}}; for (auto\u0026amp; event : events) { std::visit(overloaded { // (2) [] (EventA) { puts(\u0026#34;A\u0026#34;); }, [] (EventB) { puts(\u0026#34;B\u0026#34;); } }, event); } return 0; } If you\u0026rsquo;re not familiar with C++17, there may be a lot to take in here. In (1) we define a class that takes multiple lambdas and overloads them. In (2) we instantiate that class to inline our event handling functions. The full explanation to this code is a bit long, so I wrote another post to explain it.\nApplied to the state-machine, it will look like this: Compilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 #include \u0026lt;tuple\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;cstdlib\u0026gt; #include \u0026lt;variant\u0026gt; template\u0026lt;class... Ts\u0026gt; struct overloaded : Ts... { using Ts::operator()...; }; template\u0026lt;class... Ts\u0026gt; overloaded(Ts...) -\u0026gt; overloaded\u0026lt;Ts...\u0026gt;; struct EventA {}; struct EventB {}; using Event = std::variant\u0026lt;EventA, EventB\u0026gt;; struct Context { Context Inc() const { return Context{counter + 1}; } int counter = 0; }; template \u0026lt;class T\u0026gt; struct identity { using type = T; }; template \u0026lt;class T\u0026gt; using identity_t = typename identity\u0026lt;T\u0026gt;::type; template \u0026lt;template \u0026lt;class T\u0026gt; class Base = identity_t, class... Args\u0026gt; struct SelfReturning { using RetType = Base\u0026lt;SelfReturning\u0026gt;; using FuncType = RetType(*)(Args... args); SelfReturning(FuncType func) : _func{ func } {} RetType operator() (Args... args) { return _func(std::forward\u0026lt;Args\u0026gt;(args)...); } FuncType _func; template \u0026lt;class... AltArgs\u0026gt; using WithArgs = SelfReturning\u0026lt;Base, AltArgs...\u0026gt;; }; template \u0026lt;class T\u0026gt; using PairWithCtx = std::pair\u0026lt;T, const Context\u0026gt;; using State = SelfReturning\u0026lt;PairWithCtx\u0026gt;::WithArgs\u0026lt;const Context\u0026amp;, Event\u0026gt;; State::RetType A(const Context\u0026amp;, Event); State::RetType B(const Context\u0026amp;, Event); State::RetType A(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State A, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA) { return make_pair(A, ctx); }, [\u0026amp;] (EventB) { return make_pair(B, ctx.Inc()); } }, evt); } State::RetType B(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State B, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA) { return make_pair(A, ctx.Inc()); }, [\u0026amp;] (EventB) { return make_pair(B, ctx); } }, evt); } int main() { State state = A; Context ctx{}; Event events[] = {EventB{}, EventA{}, EventB{}, EventA{}, }; for (auto evt : events) { std::tie(state, ctx) = state(ctx, evt); } return 0; } As you can see, the change is minimal.\nPassing In Data With that, it is time to address an issue I completely neglected in Part 1. Passing in data.\nOur current state-machine model is based on the idea that the events themselves are the only information the states need. This is naive. In many real-life scenarios, events carry data with them. Now, with std::variant, we can puts data into the different event types. All we need to do is add data-members to our event structs. We define our new, data-carrying events as follows:\n1 2 3 4 5 6 struct EventA { const char* msg{nullptr}; }; struct EventB { int number{0}; }; Nothing else needs to change. And now, in the state functions, we can easily access the event data:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 State::RetType A(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State A, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA e) { if (e.msg != nullptr) { printf(\u0026#34;A message = \\\u0026#34;%s\\\u0026#34;\u0026#34;, e.msg); } else { puts(\u0026#34;A message = nullptr\u0026#34;); } return make_pair(A, ctx); }, [\u0026amp;] (EventB) { return make_pair(B, ctx.Inc()); } }, evt); } State::RetType B(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State B, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA e) { return make_pair(A, ctx.Inc()); }, [\u0026amp;] (EventB e) { printf(\u0026#34;B number = %d\\n\u0026#34;, e.number); return make_pair(B, ctx); } }, evt); } Et voilà.\nPutting everything together now, we get the following code:\nCompilation,Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 #include \u0026lt;tuple\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;cstdlib\u0026gt; #include \u0026lt;variant\u0026gt; template\u0026lt;class... Ts\u0026gt; struct overloaded : Ts... { using Ts::operator()...; }; template\u0026lt;class... Ts\u0026gt; overloaded(Ts...) -\u0026gt; overloaded\u0026lt;Ts...\u0026gt;; struct EventA { const char* msg{nullptr}; }; struct EventB { int number{0}; }; using Event = std::variant\u0026lt;EventA, EventB\u0026gt;; struct Context { Context Inc() const { return Context{counter + 1}; } int counter = 0; }; template \u0026lt;class T\u0026gt; struct identity { using type = T; }; template \u0026lt;class T\u0026gt; using identity_t = typename identity\u0026lt;T\u0026gt;::type; template \u0026lt;template \u0026lt;class T\u0026gt; class Base = identity_t, class... Args\u0026gt; struct SelfReturning { using RetType = Base\u0026lt;SelfReturning\u0026gt;; using FuncType = RetType(*)(Args... args); SelfReturning(FuncType func) : _func{ func } {} RetType operator() (Args... args) { return _func(std::forward\u0026lt;Args\u0026gt;(args)...); } FuncType _func; template \u0026lt;class... AltArgs\u0026gt; using WithArgs = SelfReturning\u0026lt;Base, AltArgs...\u0026gt;; }; template \u0026lt;class T\u0026gt; using PairWithCtx = std::pair\u0026lt;T, const Context\u0026gt;; using State = SelfReturning\u0026lt;PairWithCtx\u0026gt;::WithArgs\u0026lt;const Context\u0026amp;, Event\u0026gt;; State::RetType A(const Context\u0026amp;, Event); State::RetType B(const Context\u0026amp;, Event); State::RetType A(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State A, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA e) { if (e.msg != nullptr) { printf(\u0026#34;A message = \\\u0026#34;%s\\\u0026#34;\u0026#34;, e.msg); } else { puts(\u0026#34;A message = nullptr\u0026#34;); } return make_pair(A, ctx); }, [\u0026amp;] (EventB) { return make_pair(B, ctx.Inc()); } }, evt); } State::RetType B(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State B, counter = %d\\n\u0026#34;, ctx.counter); return std::visit(overloaded{ [\u0026amp;] (EventA e) { return make_pair(A, ctx.Inc()); }, [\u0026amp;] (EventB e) { printf(\u0026#34;B number = %d\\n\u0026#34;, e.number); return make_pair(B, ctx); } }, evt); } int main() { State state = A; Context ctx{}; Event events[] = {EventB{}, EventA{}, EventB{}, EventB{10}, EventA{}, EventA{\u0026#34;Hello, world!\u0026#34;}}; for (auto evt : events) { std::tie(state, ctx) = state(ctx, evt); } return 0; } Summary As promised, we have used some dark template magic to achieve:\nA nice generalization of SelfReturning, allowing customization of both return types and argument types; Better compile-time safety by replacing the switch statement with overload resolution; Passing data along with the events. Hopefully, a lot of fun along the way. ","permalink":"https://tamir.dev/posts/a-functional-style-state-machine-in-cpp-part-2/","summary":"First, an apology. The first part of this post was published on May 26. It is now September. I had most of the code for this part done by then. But finalizing the code took some more effort. Once that was done, explaining took a while. There were quite a few things I had to learn myself first. So now, months later, I present this humble offering to the Gods of C++ and template meta-programming.","title":"A Functional-Style State Machine in C++, Part 2"},{"content":"C++17 has granted us with std::variant. Simply put, it is a type-safe union. To access the value it stores, you can either request a specific type (using std::get or something similar) or \u0026ldquo;visit\u0026rdquo; the variant, automatically handling only the data-type that is actually there. Visiting is done using std::visit, and is fairly straight forward.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 #include \u0026lt;variant\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;vector\u0026gt; using var_t = std::variant\u0026lt;int, const char*\u0026gt;; // (1) struct Print { // (2) void operator() (int i) { printf(\u0026#34;%d\\n\u0026#34;, i); } void operator () (const char* str) { puts(str); } }; int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; // (3) for (auto\u0026amp; v : vars) { std::visit(Print{}, v); // (4) } return 0; } In (1) we define our variant type. In (2) we define a class with an overloaded operator(). This is needed for the call to std::visit. In (3) we define a vector of variants. In (4) we visit each variant. We pass in an instance of Print, and overload resolution ensures that the correct overload will be called for every type. But this example forces us to write and name an object for the overloaded operator(). We can do better. In fact, the example for std::visit on cppreference already does. Here is an example derived from it:\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 #include \u0026lt;variant\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;vector\u0026gt; template\u0026lt;class... Ts\u0026gt; struct overloaded : Ts... { using Ts::operator()...; }; // (1) template\u0026lt;class... Ts\u0026gt; overloaded(Ts...) -\u0026gt; overloaded\u0026lt;Ts...\u0026gt;; // (2) using var_t = std::variant\u0026lt;int, const char*\u0026gt;; int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; for (auto\u0026amp; v : vars) { std::visit(overloaded { // (3) [](int i) { printf(\u0026#34;%d\\n\u0026#34;, i); }, [](const char* str) { puts(str); } }, v); } return 0; } This is certainly more compact, and we removed the Print struct. But how does it work? You can see a class-template (1), lambdas passed in as arguments for the construction (3), and something with an arrow and some more template magic (2). Let\u0026rsquo;s build it step by step.\nFirst, we want to break the print functions out of Print and compose them later.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 struct PrintInt { //(1) void operator() (int i) { printf(\u0026#34;%d\\n\u0026#34;, i); } }; struct PrintCString { // (2) void operator () (const char* str) { puts(str); } }; struct Print : PrintInt, PrintCString { // (3) using PrintInt::operator(); using PrintCString::operator(); }; In (1) and (2), we define the same operators as before, but in separate structs. In (3), we are inherit from both of those structs, then explicitly use their operator(). This results in exactly the same results as before. Next, we convert Print into a class template. I\u0026rsquo;ll jump ahead and convert it directly to a variadic template.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 template \u0026lt;class... Ts\u0026gt; // (1) struct Print : Ts... { using Ts::operator()...; }; int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; for (auto\u0026amp; v : vars) { std::visit(Print\u0026lt;PrintCString, PrintInt\u0026gt;{}, v); // (2) } return 0; } In (1) we define the template. We take an arbitrary number of classes, inherit from them, and use their operator(). In (2) we instantiate the Print class-template with PrintCString and PrintInt to get their functionality. Next, we want to use lambdas to do the same. This is possible because lambdas are not functions; they are objects implementing operator().\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; auto PrintInt = [](int i) { printf(\u0026#34;%d\\n\u0026#34;, i); }; // (1) auto PrintCString = [](const char* str) { puts(str); }; for (auto\u0026amp; v : vars) { std::visit( Print\u0026lt;decltype(PrintCString), decltype(PrintInt)\u0026gt;{PrintCString, PrintInt}, // (2) v); } return 0; } In (1) we define the lambdas we need. In (2) we instantiate the template with our lambdas. This is ugly. Since lambdas have unique types, we need to define them before using them as template parameters (deducing their types using decltype). Then, we need to pass the lambdas as arguments for aggregate initialization as lambdas have a delete default constructor. We are close, but not quite there yet. The \u0026lt;decltype(PrintCString), decltype(PrintInt)\u0026gt; part is really ugly, and causes repetition. But it is needed as ctors cannot do type-deduction. So in proper C++ style, we will create a function to circumvent that.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 template \u0026lt;class... Ts\u0026gt; // (1) auto MakePrint(Ts... ts) { return Print\u0026lt;Ts...\u0026gt;{ts...}; } int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; for (auto\u0026amp; v : vars) { std::visit( MakePrint( // (2) [](const char* str) { puts(str); }, [](int i) { printf(\u0026#34;%d\\n\u0026#34;, i); } ), v); } return 0; } In (1) we define our helper function, to perform type deduction and forward it to the ctor. In (2) we take advantage of our newly found type-deduction to define the lambdas inline. But this is C++17, and we can do better.\nC++17 added user-defined deduction guides. Those allow us to instruct the compiler to perform the same actions as our helper function, but without adding another function. Using a suitable deduction guide, the code is as follows.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 #include \u0026lt;variant\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;vector\u0026gt; using var_t = std::variant\u0026lt;int, const char*\u0026gt;; template \u0026lt;class... Ts\u0026gt; struct Print : Ts... { using Ts::operator()...; }; template \u0026lt;class...Ts\u0026gt; Print(Ts...) -\u0026gt; Print\u0026lt;Ts...\u0026gt;; // (1) int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!\u0026#34;}; for (auto\u0026amp; v : vars) { std::visit( Print{ // (2) [](const char* str) { puts(str); }, [](int i) { printf(\u0026#34;%d\\n\u0026#34;, i); } }, v); } return 0; } In (1) we define a deduction guide which acts as our previous helper function, and in (2) we use the constructor instead of a helper function. Done.\nNow we have fully recreated the original example. As Print is no longer indicative of the template-class\u0026rsquo; behavior, overloaded is probably a better name.\n1 2 template\u0026lt;class... Ts\u0026gt; struct overloaded : Ts... { using Ts::operator()...; }; template\u0026lt;class... Ts\u0026gt; overloaded(Ts...) -\u0026gt; overloaded\u0026lt;Ts...\u0026gt;; ","permalink":"https://tamir.dev/posts/that-overloaded-trick-overloading-lambdas-in-cpp17/","summary":"C++17 has granted us with std::variant. Simply put, it is a type-safe union. To access the value it stores, you can either request a specific type (using std::get or something similar) or \u0026ldquo;visit\u0026rdquo; the variant, automatically handling only the data-type that is actually there. Visiting is done using std::visit, and is fairly straight forward.\nCompilation, Execution\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 #include \u0026lt;variant\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;vector\u0026gt; using var_t = std::variant\u0026lt;int, const char*\u0026gt;; // (1) struct Print { // (2) void operator() (int i) { printf(\u0026#34;%d\\n\u0026#34;, i); } void operator () (const char* str) { puts(str); } }; int main() { std::vector\u0026lt;var_t\u0026gt; vars = {1, 2, \u0026#34;Hello, World!","title":"That `overloaded` Trick: Overloading Lambdas in C++17"},{"content":"If you go to any of your colleagues now and ask them, \u0026ldquo;can a function in C++ return itself?\u0026rdquo; they will probably give you the wrong answer. Now ask them what the return type of the function is going to be. Here, let me help you:\n1 2 3 using SelfReturning = SelfReturning (*)(); SelfReturning A() { return A; } Great! But it doesn\u0026rsquo;t compile. and neither does\n1 auto A() { return A; } It turns out that C++\u0026rsquo;s type-system does not allow for recursive types. This is annoying. There is no reason why a function should not be able to return itself. It is even more annoying given that object methods can return the objects that hold them:\n1 2 3 struct A { A operator()() { return A(); } }; This code works. And for obvious reasons. Object methods are not a part of the object. They do not affect the object size or its construction. They are just a syntactic utility. There is no type-system recursion going on here.\nWith functions there is obvious type-recursion. But if you look at the work the compiler actually has to do - it seems absurd. A function is never constructed, it just is. It is not allocated. The function\u0026rsquo;s signature changes nothing about the type.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 void* A() { return reinterpret_cast\u0026lt;void*\u0026gt;(A); // Same as C\u0026#39;s `(void*)A` } int main() { auto a = A; while (true) { // Cast back to function pointer a = reinterpret_cast\u0026lt;void*(*)()\u0026gt;(A()); } return 0; } See? No missing information. The compiler has all the knowledge it needs, but the type-system still prevents us from writing our code (or, in this case, from writing it in a type-safe manner). We can do better.\nWe already know that objects can be used to break type recursion. Let\u0026rsquo;s see if we can use them here without creating so much boiler-plate code:\n1 2 3 4 5 6 7 8 9 struct SelfReturning { using FuncType = SelfReturning(*)(); // (1) SelfReturning(FuncType func) : _func{func} {} // (2) SelfReturning operator() () { return _func(); } // (3) private: FuncType _func; }; The answer is yes. We can. Just substitute this class for the failed type definition of the first example and everything works as advertised. But how does it work? To break the type-recursion, we create a proxy object. Its sole purpose is to hold a function pointer and call it. Line (1) defines the function type that we expect to hold. Note that there is no direct recursion there. (2) is the constructor, taking the function pointer and storing it. (3) is where we forward the call to the function pointer. Note that here, too there is no type recursion as the type of the class is distinct from the type of its operator() function. As a bonus, this compiles identical to the reinterpret_cast\u0026lt;void*\u0026gt; version in both Clang and GCC when using -O3 (see here and here), and at the same time maintaining type-safety. Zero-cost abstraction at work.\nBut why is that interesting? What are the use-cases?\nWell, during the last few months, I\u0026rsquo;ve routinely consumed one programming-related talk per day. I find it a great way to expand my knowledge, and far easier to do than reading an article every day.\nLast week, while working on some minor state-machine, I came across Declarative Thinking, Declarative Practice by Kevlin Henney. Upon seeing this slide:\nI thought - bare functions instead of the State design pattern? I have to try that! So I went ahead and wrote my code, iterating through the steps described above. At a quick glance, the functor solution may seem satisfying. But in effect functors, unlike functions, have different types and cannot be assigned to the same variable. To bridge the gap, we use an abstract base-class and polymorphism. Once we do that, we are forced to use pointers to hold the states. I use std::unique_ptr as I don\u0026rsquo;t want to manage the memory myself.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 #include \u0026lt;memory\u0026gt; struct IState { virtual std::unique_ptr\u0026lt;IState\u0026gt; operator()() = 0; virtual ~IState() {}; }; struct A : public IState { std::unique_ptr\u0026lt;IState\u0026gt; operator()(); }; struct B : public IState { std::unique_ptr\u0026lt;IState\u0026gt; operator()(); }; std::unique_ptr\u0026lt;IState\u0026gt; A::operator()() { return std::make_unique\u0026lt;B\u0026gt;(); } std::unique_ptr\u0026lt;IState\u0026gt; B::operator()() { return std::make_unique\u0026lt;A\u0026gt;(); } int main() { std::unique_ptr\u0026lt;IState\u0026gt; state = std::make_unique\u0026lt;A\u0026gt;(); while (true) { state = (*state)(); } return 0; } The proxy-object trick, however, has no such overhead. We know that we are using objects, but the code does not show it. The compiled version is far simpler as well (see here and here).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 struct State { using FuncType = State(*)(); State(FuncType func) : _func{func} {}; State operator()() { return _func(); } FuncType _func; }; State A(); State B(); State A() { return B; } State B() { return A; } int main() { State state = A; while (true) { state = state(); } return 0; } Enhancing it a bit, to handle events and operate on a context, we still maintain very simple, straight-forward code. For the purpose of this example, abort() and printf() are used instead of throw std::runtime_error and std::cout because the compiled output is easier to read. See compilation here and execution here.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 #include \u0026lt;cstdio\u0026gt; #include \u0026lt;cstdlib\u0026gt; enum class Event{ A, B, }; struct Context { int counter = 0; }; struct State { using FuncType = State(*)(Context\u0026amp;, Event); State(FuncType func) : _func{func} {}; State operator()(Context\u0026amp; ctx, Event evt) { return _func(ctx, evt); } FuncType _func; }; State A(Context\u0026amp;, Event); State B(Context\u0026amp;, Event); State A(Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State A, counter = %d\\n\u0026#34;, ctx.counter); ++ctx.counter; switch (evt) { case Event::A : return A; case Event::B : return B; default: abort(); } } State B(Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State B, counter = %d\\n\u0026#34;, ctx.counter); ++ctx.counter; switch (evt) { case Event::A : return A; case Event::B : return B; default: abort(); } } int main() { State state = A; Context ctx{}; Event events[] = {Event::B, Event::A, Event::B, Event::A, }; for (auto evt : events) { state = state(ctx, evt); } return 0; } For those keen on functional programming, we can even pass in a const reference to the context, and return a new context along with the new state. Compilation, execution.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 #include \u0026lt;tuple\u0026gt; #include \u0026lt;cstdio\u0026gt; #include \u0026lt;cstdlib\u0026gt; enum class Event{ A, B, }; struct Context { Context Inc() const { return Context{counter + 1}; } int counter = 0; }; struct State { using RetType = std::pair\u0026lt;State, const Context\u0026gt;; using FuncType = RetType(*)(const Context\u0026amp;, Event); State(FuncType func) : _func{func} {}; RetType operator()(Context\u0026amp; ctx, Event evt) { return _func(ctx, evt); } FuncType _func; }; State::RetType A(const Context\u0026amp;, Event); State::RetType B(const Context\u0026amp;, Event); State::RetType A(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State A, counter = %d\\n\u0026#34;, ctx.counter); switch (evt) { case Event::A : return {A, ctx}; case Event::B : return {B, ctx.Inc()}; default: abort(); } } State::RetType B(const Context\u0026amp; ctx, Event evt) { printf(\u0026#34;State B, counter = %d\\n\u0026#34;, ctx.counter); switch (evt) { case Event::A : return {A, ctx.Inc()}; case Event::B : return {B, ctx}; default: abort(); } } int main() { State state = A; Context ctx{}; Event events[] = {Event::B, Event::A, Event::B, Event::A, }; for (auto evt : events) { std::tie(state, ctx) = state(ctx, evt); } return 0; } And that\u0026rsquo;s it. We have a state machine based on pure-, bare-functions, in C++. It has a nice, simple look to it and as we\u0026rsquo;ve seen, compiles into far simpler code than the alternatives. On the way, we\u0026rsquo;ve also learned a bit about C++\u0026rsquo;s type system and how to use objects to overcome its limitations. In the next post (soon to be populated now online!) I will show some exciting (read: never use in production) dark template magic to both generalize the State object and to get some compile-time guarantees. Stay tuned!\n","permalink":"https://tamir.dev/posts/a-functional-style-state-machine-in-cpp/","summary":"If you go to any of your colleagues now and ask them, \u0026ldquo;can a function in C++ return itself?\u0026rdquo; they will probably give you the wrong answer. Now ask them what the return type of the function is going to be. Here, let me help you:\n1 2 3 using SelfReturning = SelfReturning (*)(); SelfReturning A() { return A; } Great! But it doesn\u0026rsquo;t compile. and neither does\n1 auto A() { return A; } It turns out that C++\u0026rsquo;s type-system does not allow for recursive types.","title":"A Functional-Style State Machine in C++"},{"content":"TL;DR Windows\u0026rsquo; shell sucks so people write tools. Tools are fun to use.\nLinux\u0026rsquo;s shell is amazing so people write terrible bash scripts and makefiles.\nBe sensible. Use Python. Use C.\nThe rest of the post is me ranting, letting off some steam. Have fun, and don\u0026rsquo;t take it too seriously.\n🐢 A Story of Shells 🐢 1 2 3 my_var=$(application arg0 arg1) for /f %%i in (\u0026#39;application arg0 arg1\u0026#39;) do set my_var=%%i 🪟 Windows 🪟 I\u0026rsquo;ve been a Windows user for a very long time. In fact, it has always been my main OS. And I absolutely love it. That is not to say, though, that there aren\u0026rsquo;t issues with that. One such thing is the CLI.\nFor years, the only thing we had on Windows was cmd.exe and .bat files. And they are ghastly. They are pretty straight-forward for very simple things, but try using them for anything more advanced (like setting a variable to the output of a command) and you\u0026rsquo;ll quickly realize that this is not the tool for you. You need a real programming language. So you\u0026rsquo;ll use Python, or C, or something. It will take you a bit longer to write the tool you needed, but in the process it will become an actual tool and not an annoying script.\nIn recent years there has been some serious improvements - PowerShell came along, cmder made the shell look a bit better, and I have some Linux tools (grep, awk, xargs\u0026hellip;) running in my Windows shell. And yet, people tend to use more general purpose languages on Windows.\n🐧 Penguins 🐧 On Linux, however, things have always been good with the shell. It has awesome terminals, and a huge amount of utilities that can be chained (piped) to unleash powers beyond imagination. And unlike Windows, where you need to use funny-looking APIs to get information, Linux just gives you everything in handy text files.\nIn fact, Linux is optimized for the shell in ways that make me wince. It is actually easier to parse system information (like, say, a process list) in the shell than in C code. Because instead of proper APIs, we have text files. Text files. The kernel takes binary data, formats it into strings, and the user-mode code can then scanf the code back into binary data. Amazing!\nBut enough of that.\nThe Linux shell is truly remarkable. You can do pretty much anything in a bash script. And people do. Now, there\u0026rsquo;s a bit of Linux philosophy that I really like. \u0026ldquo;Do one thing and do it well.\u0026rdquo; find finds things, grep greps, xargs xargs, and awk awks. So far - so good. But what about bash itself? While each of the shell utilities does one thing and does it well, bash, and especially bash scripts, do not. They allow you to quickly implement advanced behaviours by hacking together multiple commands. You write more and more and more, and everything just works. And then, it doesn\u0026rsquo;t. And you need to fix it. If you\u0026rsquo;re lucky, and the code is well documented you might get away with that. But more often than not, it won\u0026rsquo;t be. And the wonderful \u0026ldquo;let\u0026rsquo;s take strings and pass them around\u0026rdquo; programming style might come back to bite you.\nBut, again, bash is fantastic for quick hacks. And works fairly well in general when you\u0026rsquo;re not taking input parameters, and when it\u0026rsquo;s small enough. I don\u0026rsquo;t like it, but it works.\n🔥 Make Your Own Hell 🔥 The real issue is make.\nmake is a beast spawned in the deepest dungeons of hell (and I am awfully sorry if I offended any such beast by the comparison.) Makefiles give you the benefits of never leaving your shell / code-editor while you work. You just create a makefile, define your targets, and you\u0026rsquo;re good to go. In truth - that is fantastic. You can even include bash scripts in your makefiles to do custom steps. Or generate makefiles. Or use automake. Or anything else that might make the task of writing makefiles easier to do and harder to maintain. But you write, and you specify, and it just works! Most of the time.\n** Dramatic Pause **\nYou might have noticed I don\u0026rsquo;t like make. That\u0026rsquo;s true. This post is mainly me complaining about make and blowing off some steam. Now, I appreciate make. I\u0026rsquo;ve used it to build things that I never would\u0026rsquo;ve managed on Windows. It\u0026rsquo;s an extremely powerful tool. But, you see, as a Windows user, I naturally enjoy IDEs. One such wonderful creation is Visual Studio. As far as C/C++ development tools go it is unparalleled. \u0026ldquo;But wait!\u0026rdquo; say my Linux friends, \u0026ldquo;VS forces you to create foul \u0026lsquo;projects\u0026rsquo; using their fiendish \u0026lsquo;GUI\u0026rsquo;! In Linux, we just write makefiles!\u0026rdquo; That is true, and it is quite annoying with small projects (where I usually end up having a .bat file to trigger the build with all the relevant flags,) but it is godsend for anything more complex. You have GUI, and Projects, and Solutions and whatnot. It\u0026rsquo;s great. A bit slow to define, but oh so easy to use!\nAnd now, after I lost most of my Linux-oriented readers, I can get to the point. Linux has a super-powerful shell, so people use it to make a super-powerful mess. Windows has a super-useless shell, so people don\u0026rsquo;t use it. Instead - they build tools! User-friendly(-ish) tools with GUI. And they use general purpose programming languages. And that\u0026rsquo;s good.\n🍕 Takeaway 🍕 Should you stop using bash? Or cripple it? Or keep Windows\u0026rsquo; shell down? Hell no!\nUse whatever tools you deem fit. But use them wisely. Keep shell-scripting to small automation tasks, and try to use more maintainable programming languages when you write something larger or more complex. And do write new tools. Tools are fun.\n","permalink":"https://tamir.dev/posts/the-windows-cli-sucks-and-thats-good/","summary":"TL;DR Windows\u0026rsquo; shell sucks so people write tools. Tools are fun to use.\nLinux\u0026rsquo;s shell is amazing so people write terrible bash scripts and makefiles.\nBe sensible. Use Python. Use C.\nThe rest of the post is me ranting, letting off some steam. Have fun, and don\u0026rsquo;t take it too seriously.\n🐢 A Story of Shells 🐢 1 2 3 my_var=$(application arg0 arg1) for /f %%i in (\u0026#39;application arg0 arg1\u0026#39;) do set my_var=%%i 🪟 Windows 🪟 I\u0026rsquo;ve been a Windows user for a very long time.","title":"The Windows CLI sucks, and that's good."},{"content":"As git users, we know that we should \u0026ldquo;commit early, commit often.\u0026rdquo; While this is a wonderful thing to do, it does mean that from time to time we make a mistake and need to fix a commit. Maybe we forget to git add a new file, or missed a typo. So we go ahead and git commit --amend. Problem solved. Great.\nBut personally, I hate it.\nFor one thing, amending commits hides history. Once you amend, that past state is gone before you can properly test the new one. True, you can also restore it via git reflog, but no-one really likes using that. It should be a last resort.\nFor another thing, amending is very limited. Say I am writing some C code. I write my first module, add it and commit.\n1 2 git add FirstModule.h git commit -m \u0026#34;Added FirstModule\u0026#34; I write my second module, and add it as well.\n1 2 git add SecondModule.h SecondModule.c git commit -m \u0026#34;Added SecondModule\u0026#34; And now, after adding that second commit, I realize that I forgot to commit FirstModule.c. git commit --amend to the rescue? Not really. I now have to resort to the black, frightening voodoo magic called git rebase.\nFirst, we commit the forgotten module\n1 2 git add FirstModule.c git commit -m \u0026#34;Added FirstModule.c, forgotten eariler.\u0026#34; And then rebase - git rebase -i HEAD~3\n1 2 3 pick 1db8687 Added FirstModule pick 336941b Added SecondModule pick 7884909 Added FirstModule.c, forgotten eariler. Change to\n1 2 3 pick 1db8687 Added FirstModule fixup 7884909 Added FirstModule.c, forgotten eariler. pick 336941b Added SecondModule Save \u0026amp; Quit, and we\u0026rsquo;re done.\n1 2 3 4 5 6 7 * 1946e37d105ffebcbd91bb958f8a2fce6160c761 (HEAD -\u0026gt; master) Added SecondModule | create mode 100644 SecondModule.c | create mode 100644 SecondModule.h * 8ffbb9f2915e060a6c4771e13f5a82442743724c Added FirstModule | create mode 100644 FirstModule.c | create mode 100644 FirstModule.h * 815e7bab6ee1fa5bf1df10f5705919b48cbe214c First Commit Not that hard, is it?\nBut still, moving between amending and rebasing can be cumbersome. Especially as most of the time there is no real need to rebase and it\u0026rsquo;s easy to forget the process. Enter git commit --fixup (or --squash) and git rebase -i --autosquash.\nThese commands save us the work of reordering the commits and changing from pick to fixup or squash. Making our rebasing work a lot easier.\nI like defining the following aliases:\n1 2 3 4 5 [alias] ri = rebase -i --autosquash mri = rebase -i fix = commit --fixup squ = commit --squash Using those aliases, the rebasing we did earlier would work as follows:\n1 2 3 git add FirstModule.c git fix HEAD~1 git ri HEAD~3 We\u0026rsquo;d get the following rebase automatically\n1 2 3 pick 1db8687 Added FirstModule fixup 50a3650 fixup! Added FirstModule pick 336941b Added SecondModule Exit the editor, and be done with it.\nWe can use fix as many times as we want (just go ahead and git fix HEAD -a) before the rebase. Our log may look funny\n1 2 3 4 5 6 7 * fe0c2a0 (HEAD -\u0026gt; master) fixup! fixup! fixup! fixup! Added SecondModule * a53cd32 fixup! fixup! fixup! Added SecondModule * 9c19f2d fixup! fixup! Added SecondModule * b758a53 fixup! Added SecondModule * 902d65e Added SecondModule * 67f1260 Added FirstModule * 815e7ba First Commit But the rebase doesn\u0026rsquo;t care\n1 2 3 4 5 pick 902d65e Added SecondModule fixup b758a53 fixup! Added SecondModule fixup 9c19f2d fixup! fixup! Added SecondModule fixup a53cd32 fixup! fixup! fixup! Added SecondModule fixup fe0c2a0 fixup! fixup! fixup! fixup! Added SecondModule Conclusion Stop using git commit --amend and start using git fix (git commit --fixup) instead. It is a no-fear, low-overhead alternative, and it far more flexible. Here are the aliases again, in case you want them:\n1 2 3 4 5 [alias] ri = rebase -i --autosquash mri = rebase -i fix = commit --fixup squ = commit --squash ","permalink":"https://tamir.dev/posts/dont-amend-fix/","summary":"As git users, we know that we should \u0026ldquo;commit early, commit often.\u0026rdquo; While this is a wonderful thing to do, it does mean that from time to time we make a mistake and need to fix a commit. Maybe we forget to git add a new file, or missed a typo. So we go ahead and git commit --amend. Problem solved. Great.\nBut personally, I hate it.\nFor one thing, amending commits hides history.","title":"Don't Amend, Fix"}]