Arnaud Gourlay's blog

Pinch Points - a Rust game with crabs

2026-09-06T00:00:00+00:00
Pinch Points</a> is a kid-friendly routing game for 1-6 players, built in Rust with Bevy.
In this game, players compete to save the most crabs by routing them to their respective castles.
Check out what it looks like!
A nice home menu </figcaption> </figure>
The main versus mode, with multiplayer support </figcaption> </figure>
And 100 puzzles to solve </figcaption> </figure>
You can install it from:
prebuilt binaries</a></li>
crates.io</code> with cargo install --locked pinch-points</code></li> </ul> The rest of the post is not the usual technical deep-dive article found on this blog. Rather, it is my experience using AI to create a video game and what I learned in the process.Motivation</a> </h2> The game is based on the mechanics of ChuChuRocket!</code> developed by Sonic Team and released in 1999 on the Dreamcast. In this game, up to four players compete to save mice from cats by guiding them into their rockets by placing arrows on a board. An iconic game cover for the Dreamcast </figcaption> </figure> The main game mode </figcaption> </figure> I was very fond of this game as a teenager; the gameplay is fast and so simple that everyone gets it and has fun. As a programmer, I always thought that I could try to create something with similar gameplay while adding a few twists to modernize it and spice things up. First, the theme: no more mice and cats in space. Given how much I like Rust, I wanted to make it about crabs! I had a clear idea for the theme and the changes I wanted to make. This led me to think that creating this game could be a well-scoped task to gain experience with AI code generation. After iterating on a specification document, I handed it over to Claude Code</code> with Opus 5 and experienced the magic of generative coding firsthand. AI iterations</a> </h2> The first iteration took around an hour and was, retrospectively, by far the biggest productivity gain. All the boilerplate required to bootstrap the project and wire things together. The basic structure of the game was there; at first I was very satisfied. At this point, however, the game was far from being done, because my specifications were not precise enough. What followed was a long and somewhat tedious process of going back and forth to implement certain things exactly as I wanted. As expected, I did not know exactly what I wanted until I started playing the game myself! Not only were there a lot of bugs to fix, but new changes would often break things elsewhere in the game. While the coding agent was excellent at performing localized tasks, it was simply not good at preserving the global consistency of the game! For this reason, I started putting constant pressure on the agent to improve code quality. An easy first step is to look at code coverage to guide the generation of missing unit tests. This helps prevent regressions but it is not enough to improve code quality. After checking the code superficially, I would find myself repeatedly asking for: splitting large non-cohesive files</li> splitting large structs</li> introducing missing enums to replace fragile if/else cascades</li> adding debug assertions</li> </ul> The quality would slowly increase but it required regular attention. End result</a> </h2> To be honest, I have mixed feelings about this experience. On one hand, I am pretty satisfied with the game, I find it fun, and I've received some good feedback. On the other hand, I am not satisfied with the creative process and that is not because AI is bad. Quite the opposite: in many coding tasks, I found it surprisingly capable. My main issues are related to ownership and mastery. Although I precisely specified every aspect of the game through many playtests, I still feel like I do not own this game. What I do have is a 50 KLOC codebase whose internals I don't fully understand — unlike my other side projects, which I have crafted with care. You got the thing you asked for, but because you didn't build it yourself, you don't feel that it is truly yours. As the saying goes, "Be careful what you wish for!" In any case, the game exists now, so you might as well enjoy it.


The transmission
2026-05-03T00:00:00+00:00
Tractors, beige boxes and what's next</em></p>
My father was very fond of tractors.</p>
He would never miss an occasion to stop for any tractor we came across on holidays, at a fair or an agricultural museum.</p>
His eyes would light up and he would start commenting on the particular models in front of us.</p>
As a kid I understood to some degree that tractors were special to my father, but it took me much longer to really understand why.</p>
The father</a>
</h2>
My father was born in the late 1940s in a small village deep in the French countryside.
Farming was the main activity for most people in such villages.</p>
Having access to a tractor was a big deal at the time. The mechanization of work marked the arrival of modernity.</p>
One day your parents are working outside with horses, the next day the animals could be sold off because this new loud machine had arrived to get things done much faster.</p>
My father told stories of driving Massey-Ferguson tractors as a kid and I remember being quite jealous at the time.
I'd only get to pretend to drive at those agricultural fairs, standing on the clutch and gripping the massive wheel.</p>

    


    
    
    Massey-Ferguson 35 Deluxe. 
 Photo: Acroterion</a>, CC BY-SA 4.0</a></p>
</figcaption>
    
</figure>
Comfortable around machines, he enrolled in a trade school in the 1960s where he eventually discovered the world of analog electronics.</p>
He graduated and landed a job repairing large industrial and medical equipment.
Think large expensive objects that customers are definitely not happy to see standing still.</p>
Troubleshooting complex machines all day, he ended up fixing everything at home too.
I remember the scenes: him under the car, the TV cracked open with its tube exposed in the living room, the washing machine sitting drumless on the kitchen floor.</p>
The boy</a>
</h2>
When I was around 13 years old, my father brought home a large beige box from Hewlett-Packard.</p>

    


    
    
    Tractor for millennials. 
 Photo: Thomas Schanz, CC BY-SA 4.0</a></p>
</figcaption>
    
</figure>
That Pentium II running at 233 MHz was rather high-end for the time.</p>
In the pre-Internet era, I used it mostly for gaming and homework thanks to educational programs like Encarta.</p>
The first time I looked inside the box was when my father upgraded to a larger hard drive.
I realized a computer was just the sum of its parts.</p>
The family computer got bricked a few times after I messed with DLLs and the Windows registry. To get me off it, my father gave me an old computer he had salvaged from the street.</p>
I got it to work after re-installing Windows 95 — child's play.</p>
Computer technology was moving very fast back then. People would leave entire computers on the curb because the new ones were so much better.</p>
My father had noticed my interest early on, so he kept bringing home abandoned desktops to keep me busy.
From those, I would salvage the best parts into my Frankenstein build, swapping components and operating systems.</p>
At some point, I had a plastic bag full of AMD Athlon CPUs that no one seemed to want.</p>
Then the Internet arrived. I got a new computer and turned my Frankenstein build into a multimedia player.</p>
In high school I was that guy burning CDs and fixing my friends' computers.</p>
Gaming led to tinkering with hardware, and from there to programming.</p>
The son</a>
</h2>
A few years back, I took my son to a technology museum.</p>
I showed him a beige box under glass and explained how cool it was back in the day.</p>
He was unimpressed and ran off to see the racing cars section.</p>
I can't really blame him, the exhibit was underwhelming.</p>
But at that very moment, I caught myself looking at it the way my father had looked at those tractors.</p>
This was the transmission: curiosity passed from one generation to the next.</p>
Beyond the technology itself, it was about tools you could take apart.</p>
But what are the tractors and beige boxes for this generation?</p>

    


    
    
    My son building. 
 Photo: me</p>
</figcaption>
    
</figure>
My father's passion for repair was a product of his era. That attitude is mostly gone.
Modern appliances aren't built to be opened, and when something breaks, you call someone or replace it.</p>
And yet, my son binges educational shows, loves all kinds of building toys and asks unfiltered questions about how the world works.</p>
The curiosity is there. Now I have to figure out what to bring home.</p>


New Key-Value Store in Rust at Qdrant
2025-02-10T00:00:00+00:00
I have been working at Qdrant</a> for a while now, where I get the opportunity to work on very interesting problems.</p>
This article is an external publication</a> regarding my latest contributions to a new key-value storage engine specialized for our needs.</p>
The storage is not yet available as a standalone crate for the moment; in the meantime, there are a few pointers to get started reading the code for the curious.</p>

the Gridstore code internal crate</a></li>
usage for payload storage</a></li>
usage for sparse vector storage</a></li>
</ul>


Playing guitar tablatures in Rust
2024-07-14T00:00:00+00:00
If you ever tried to learn guitar, chances are you are familiar with guitar tablatures.</p>
It is a simple way to visualize music for guitar, using ASCII characters to represent strings and frets as an alternative to sheet music.</p>
For instance, here are the first four measures of the song "Smoke on the Water" by Deep Purple:</p>
e|-----------------|-----------------|-----------------|-----------------|</span></span>
B|-----------------|-----------------|-----------------|-----------------|</span></span>
G|-----3---5-------|---3---6-5-------|-----3---5-----3-|-----------------|</span></span>
D|-5---3---5-----5-|---3---6-5-------|-5---3---5-----3-|---5-------------|</span></span>
A|-5-------------5-|-----------------|-5---------------|---5-------------|</span></span>
E|-----------------|-----------------|-----------------|-----------------| <- top</span></span></code></pre>
This song is played in standard tuning (EADGBe), which is conveyed by the letters on the left indicating the tuning for each string, while the numbers indicate where to put your fingers on the fretboard.</p>
Beyond the text representation, the de facto standard is the format used by the Guitar Pro</a> software to render and synthesize sound for the tablature.</p>
Those binary files have the .gp3</code>, .gp4</code>, .gp5</code> or .gp6</code> extension depending on the software version used to produce them, and can be easily found on the internet on websites such as Ultimate Guitar</a>.</p>
Although the software to play the tablature is proprietary, some versions of the file format are well documented, and there are even open-source projects that can read them.</p>
The best OSS tablature player is probably TuxGuitar</a> which is very feature-rich and a fantastic tool to learn guitar.</p>
Since TuxGuitar</code> appears to be unmaintained and Java-based, I thought it would be a fun challenge to write my own tablature player in Rust.</p>
Introducing Ruxguitar</a>
</h1>
I named my project Ruxguitar</code>, a portmanteau of Rust</code> and Guitar</code>.</p>
The project is still in its early stages, but I believe it is now functional enough to be officially announced to the world - aka this very blog post!</p>
Rather than describing what the project does, have a look at the following video which shows the tablature player in action with a song a bit more complex:</p>

  
  Your browser does not support the video tag.
</video>
You can obviously find the source code on GitHub</a> with pre-built binaries</a> available for Linux, macOS and Windows.</p>
Feel free to go play with it and come back when you want to read the rest of this post which will detail various aspects of the internals.</p>
Parsing the tablature</a>
</h1>
The first step in building a tablature player is to parse the binary tablature file.</p>
During my research, I found a specification of the .gp4</code> file format on dguitar</a> to get started.</p>
The structure of the file is roughly the following:</p>

file version to know which version of the file format is used</li>
information about the piece (e.g. title, subtitle, artist, album, etc.)</li>
lyrics</li>
number of measures and number of tracks</li>
measures per track in the following order:

measure 1/track 1</li>
measure 1/track 2</li>
...</li>
measure 1/track m</li>
measure 2/track 1</li>
measure 2/track 2</li>
...</li>
measure 2/track m</li>
...</li>
measure n/track 1</li>
measure n/track 2</li>
...</li>
measure n/track m</li>
</ul>
</li>
within each measure, we find the number of beats to read</li>
within each beat, we find the beat duration and the number of notes to read</li>
within each note, we find the string, the fret, the duration, the effect, etc.</li>
</ol>
I decided to use the nom</a> crate to parse the tablature because I am familiar</a> with it for parsing binary format.</p>
Here is a quick peek at the code driving the parser so you get an idea of how it looks:</p>
pub</span> fn</span> parse_gp_data</span>(</span>file_data</span>:</span> &</span>[</span>u8</span>]</span>)</span> -></span> Result</span><</span>Song</span>,</span> RuxError</span>></span> {</span></span>
    let</span> (</span>rest</span>,</span> base_song</span>)</span> =</span> flat_map</span>(</span>parse_gp_version</span>,</span> |</span>version</span>|</span> {</span></span>
        map</span>(</span></span>
            tuple</span>(</span>(</span></span>
                parse_info</span>(</span>version</span>)</span>,</span>                                     </span></span>
                cond</span>(</span>version</span> <</span> GpVersion</span>::</span>GP5</span>,</span> parse_bool</span>)</span>,</span>              </span></span>
                cond</span>(</span>version</span> >=</span> GpVersion</span>::</span>GP4</span>,</span> parse_lyrics</span>)</span>,</span>           </span></span>
                cond</span>(</span>version</span> >=</span> GpVersion</span>::</span>GP5_10</span>,</span> take</span>(</span>19</span>usize</span>)</span>)</span>,</span>       </span></span>
                cond</span>(</span>version</span> >=</span> GpVersion</span>::</span>GP5</span>,</span> parse_page_setup</span>)</span>,</span>       </span></span>
                cond</span>(</span>version</span> >=</span> GpVersion</span>::</span>GP5</span>,</span> parse_int_sized_string</span>)</span>,</span> </span></span>
                parse_int</span>,</span>                                               </span></span>
                cond</span>(</span>version</span> ></span> GpVersion</span>::</span>GP5</span>,</span> parse_bool</span>)</span>,</span>              </span></span>
                parse_signed_byte</span>,</span>                                       </span></span>
                cond</span>(</span>version</span> ></span> GpVersion</span>::</span>GP3</span>,</span> parse_int</span>)</span>,</span>               </span></span>
                parse_midi_channels</span>,</span>                                     </span></span>
            )</span>)</span>,</span></span>
            move</span> |</span>(</span></span>
                song_info</span>,</span></span>
                triplet_feel</span>,</span></span>
                lyrics</span>,</span></span>
                _master_effect</span>,</span></span>
                page_setup</span>,</span></span>
                tempo_name</span>,</span></span>
                tempo</span>,</span></span>
                hide_tempo</span>,</span></span>
                key_signature</span>,</span></span>
                octave</span>,</span></span>
                midi_channels</span>,</span></span>
            )</span>|</span> {</span></span>
                //</span> init base song</span></span>
                let</span> tempo</span> =</span> Tempo</span>::</span>new</span>(</span>tempo</span>,</span> tempo_name</span>)</span>;</span></span>
                Song</span> {</span></span>
                    version</span>,</span></span>
                    song_info</span>,</span></span>
                    triplet_feel</span>,</span></span>
                    lyrics</span>,</span></span>
                    page_setup</span>,</span></span>
                    tempo</span>,</span></span>
                    hide_tempo</span>,</span></span>
                    key_signature</span>,</span></span>
                    octave</span>,</span></span>
                    midi_channels</span>,</span></span>
                    measure_headers</span>:</span> vec!</span>[</span>]</span>,</span></span>
                    tracks</span>:</span> vec!</span>[</span>]</span>,</span></span>
                }</span></span>
            }</span>,</span></span>
        )</span></span>
    }</span>)</span>(</span>file_data</span>)</span></span>
    .</span>map_err</span>(</span>|</span>_err</span>|</span> {</span></span>
        log</span>::</span>error!</span>(</span>"</span>Failed to parse GP data</span>"</span>)</span>;</span></span>
        RuxError</span>::</span>ParsingError</span>(</span>"</span>Failed to parse GP data</span>"</span>.</span>to_string</span>(</span>)</span>)</span></span>
    }</span>)</span>?</span>;</span></span>
    //</span> parse tracks & measures</span></span>
    ...</span></span></code></pre>
The heavy lifting required to parse the tracks and the measures is done in a different function that will be skipped for brevity.</p>
At some point it became quite tedious to handle the various versions of the file format and I decided to focus on the .gp5</code> version which is widely used.</p>
To be honest, this part was rather challenging because the file format is quite complex and the documentation is not always clear.</p>
Luckily I was able to inspect the parsers from TuxGuitar</code> and the guitarpro</a> crate to get a better understanding of the file format.</p>
To ensure correctness, I have written a few unit tests for specific tablature files to check that the parser is working as expected.</p>
This approach is useful to get started but not very scalable, so I am also validating some high level invariants of the resulting Song</code> structure over a directory containing several hundred tablatures.</p>
I found a few bugs in the parser this way, and I am confident that the parser is working as expected.</p>
Building a UI</a>
</h1>
At this point, we have an in-memory representation of the tablature but no way to display it.</p>
Not only should the user be able to see the tablature but also be able to interact with it.</p>
I really wanted to use a native GUI library to ensure that the application would look and feel like a native application on all platforms.</p>
The state</a> of GUI libraries in Rust required me to do some research.</p>
I needed a truly event-based library to handle the synchronization during playback while also being able to draw the tablature in a custom way with some kind of canvas abstraction.</p>
Based on those requirements, I decided to give Iced</a> a spin as it checked all the boxes.</p>
Spoiler alert: I am very happy with my choice so I did not try other libraries.</p>
Iced</a>
</h2>
The Iced</code> library is very well written but could benefit from a bit more documentation.</p>
I recommend reading the source code of the examples</a> to get a better understanding of how to use the library.</p>
I started with the text editor</a> example which I slowly adapted to my needs.</p>
At some point, I ran into a bug in version 0.12.0</code> which forced me to upgrade to the 0.13.0</code> version which was not released yet.</p>
This means I had to use the main</code> branch of the Iced</code> repository which was a bit scary but it worked out fine.</p>
All the breakages I encountered were due to the Iced</code> library being in active development and I am very grateful to the maintainers for their hard work.</p>
The library is architected around messages and subscriptions that trigger the update of the UI.</p>
For instance, those are my messages:</p>
#</span>[</span>derive</span>(</span>Debug</span>,</span> Clone</span>)</span>]</span></span>
pub</span> enum</span> Message</span> {</span></span>
    OpenFile</span>,</span> //</span> open file dialog</span></span>
    FileOpened</span>(</span>Result</span><</span>(</span>Vec</span><</span>u8</span>></span>,</span> String</span>)</span>,</span> PickerError</span>></span>)</span>,</span> //</span> file content & file name</span></span>
    TrackSelected</span>(</span>TrackSelection</span>)</span>,</span> //</span> track selection</span></span>
    FocusMeasure</span>(</span>usize</span>)</span>,</span> //</span> used when clicking on measure in tablature</span></span>
    FocusTick</span>(</span>usize</span>)</span>,</span> //</span> focus on a specific tick in the tablature</span></span>
    PlayPause</span>,</span> //</span> toggle play/pause</span></span>
    StopPlayer</span>,</span> //</span> stop playback</span></span>
    ToggleSolo</span>,</span> //</span> toggle solo mode</span></span>
}</span></span></code></pre>
And this is the simplified application entry point:</p>
impl</span> RuxApplication</span> {</span></span>
    pub</span> fn</span> start</span>(</span>args</span>:</span> ApplicationArgs</span>)</span> -></span> iced</span>::</span>Result</span> {</span></span>
        iced</span>::</span>application</span>(</span></span>
            RuxApplication</span>::</span>title</span>,</span></span>
            RuxApplication</span>::</span>update</span>,</span></span>
            RuxApplication</span>::</span>view</span>,</span></span>
        )</span></span>
        .</span>subscription</span>(</span>RuxApplication</span>::</span>subscription</span>)</span></span>
        .</span>theme</span>(</span>RuxApplication</span>::</span>theme</span>)</span></span>
        .</span>font</span>(</span>ICONS_FONT</span>)</span></span>
        .</span>centered</span>(</span>)</span></span>
        .</span>antialiasing</span>(</span>true</span>)</span></span>
        .</span>run</span>(</span>)</span></span>
    }</span></span>
}</span></span></code></pre>
The application is built around functions that are orchestrated by the Iced</code> engine appropriately.</p>
The update</code> function has the signature Fn(&mut State, Message) -> C</code> where:</p>

State</code> is the application state that can be modified (here RuxApplication</code>)</li>
Message</code> is the message to process</li>
C</code> is an output Task</code> potentially producing a new Message</code></li>
</ul>
The view</code> function has the signature Fn(&'a State) -> Widget</code> and renders a Widget</code> based on the current &State</code>.</p>
Drawing the tablature</a>
</h2>
I started by crafting the code which carefully draws a single measure on an Iced::Canvas</code>.</p>
This means:</p>

drawing each string</li>
for each beat, drawing the notes on the strings and potential beat effect (e.g. palm mute)</li>
for each note, adding the potential note effect (e.g. slide, hammer-on, bend)</li>
annotate the measure with additional information (e.g. measure number, tempo, part annotation, chord)</li>
</ul>
It required a bit of trial and error to get the offsets right but I am happy with the result.</p>

    


    
    
</figure>
Once I have a collection of canvas measures, they are assembled in a responsive grid layout to display the whole tablature using the wrap</code> widget from the iced-aw</a> crate.</p>

    


    
    
</figure>
The measures can have different lengths depending on the number of beats which makes silent measures very small and crazy guitar solo measures long.</p>
Making sounds</a>
</h1>
To recap, we have an in-memory representation of a tablature and we have the UI, now we need to make some sounds!</p>
What we want is a way to turn each note, for each beat, for each measure, for each track into a specific sound at the right</strong> time.</p>
This can be achieved using a MIDI synthesizer which is software that can produce sounds based on MIDI events.</p>
Synthesizing MIDI events</a>
</h2>
There are different kinds of MIDI events but the most important for us are the NoteOn</code> and NoteOff</code> ones.</p>

Note On: Indicates that a note is being pressed. It includes the note number (pitch) and velocity (how hard the note is pressed).</li>
Note Off: Indicates that a note is being released.</li>
</ul>
For each note in the tablature, we can generate a pair of MIDI events annotated with:</p>

the timestamp, also referred to as a tick, at which they should be executed.</li>
the track to which they belong.</li>
</ul>
pub</span> enum</span> MidiEventType</span> {</span></span>
    NoteOn</span>(</span>i32</span>,</span> i32</span>,</span> i16</span>)</span>,</span>  //</span> midi channel, note, velocity</span></span>
    NoteOff</span>(</span>i32</span>,</span> i32</span>)</span>,</span>      //</span> midi channel, note</span></span>
    ...</span></span>
}</span></span>
</span>
pub</span> struct</span> MidiEvent</span> {</span></span>
    pub</span> tick</span>:</span> usize</span>,</span></span>
    pub</span> event</span>:</span> MidiEventType</span>,</span></span>
    pub</span> track</span>:</span> usize</span>,</span></span>
}</span></span></code></pre>
All those events are pushed into a single array sorted by the event tick.</p>
This approach provides an efficient way to find the next events to play at any given time via binary search later on.</p>
Those MidiEvents</code> can be transformed into audio waves using a synthesizer before being sent to the audio output.</p>
For the synthesizer I settled on the rustysynth</a> crate which provides a neat MIDI synthesizer.</p>
Here is a simplified version of the code to play a MIDI event:</p>
let</span> synthesizer_settings</span> =</span> SynthesizerSettings</span>::</span>new</span>(</span>SAMPLE_RATE</span> as</span> i32</span>)</span>;</span></span>
let</span> mut</span> synthesizer</span> =</span> Synthesizer</span>::</span>new</span>(</span>&</span>sound_font</span>,</span> &</span>synthesizer_settings</span>)</span>;</span></span>
</span>
let</span> midi_event</span> =</span> //</span> find next event to play</span></span>
match</span> midi_event</span>.</span>event </span>{</span></span>
    MidiEventType</span>::</span>NoteOn</span>(</span>channel</span>,</span> key</span>,</span> velocity</span>)</span> =></span> {</span></span>
        synthesizer</span>.</span>note_on</span>(</span>channel</span>,</span> key</span>,</span> velocity</span> as</span> i32</span>)</span>;</span></span>
    }</span></span>
    MidiEventType</span>::</span>NoteOff</span>(</span>channel</span>,</span> key</span>)</span> =></span> {</span></span>
        synthesizer</span>.</span>note_off</span>(</span>channel</span>,</span> key</span>)</span>;</span></span>
    }</span></span>
    ...</span></span>
}</span>                    </span></span></code></pre>
It is important to note that a synthesizer requires a soundfont file to produce sound.</p>
For the sake of simplicity, I included the TimGM6mb.sf2</code> soundfont file in the binary at compile time.</p>
const</span> TIMIDITY_SOUND_FONT</span>:</span> &</span>[</span>u8</span>]</span> =</span> include_bytes!</span>(</span>"</span>../../resources/TimGM6mb.sf2</span>"</span>)</span>;</span></span></code></pre>
The trade-off is that the binary is a bit larger but the user does not have to worry about finding the soundfont file.</p>
However, it is possible to provide a larger soundfont file to get better sound quality using the --sound-font-file</code> command line argument.</p>
For instance, I like to use FluidR3_GM.sf2</code> which is present on most systems and easy to find online (here</a> or there</a>).</p>
./ruxguitar</span> -</span>-sound-font-file</span> /usr/share/sounds/sf2/FluidR3_GM.sf2</span></span></code></pre>Audio loop</a>
</h2>
The audio output stream is managed by a dedicated thread which will produce sound at a regular interval.</p>
I picked the cpal</a> crate which is a cross-platform audio library.</p>
Here is a simplified version of the code to set up an audio loop:</p>
let</span> host</span> =</span> cpal</span>::</span>default_host</span>(</span>)</span>;</span></span>
let</span> device</span> =</span> host</span>.</span>default_output_device</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
</span>
let</span> config</span> =</span> device</span>.</span>default_output_config</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
let</span> stream_config</span>:</span> cpal</span>::</span>StreamConfig</span> =</span> config</span>.</span>into</span>(</span>)</span>;</span></span>
</span>
let</span> channels_count</span> =</span> stream_config</span>.</span>channels </span>as</span> usize</span>;</span></span>
assert_eq!</span>(</span>channels_count</span>,</span> 2</span>)</span>;</span></span>
</span>
//</span> 4410 samples at 44100 Hz is 0.1 second</span></span>
let</span> mono_sample_count</span> =</span> 4410</span>;</span></span>
</span>
let</span> mut</span> left</span>:</span> Vec</span><</span>f32</span>></span> =</span> vec!</span>[</span>0_</span>f32</span>;</span> mono_sample_count</span>]</span>;</span></span>
let</span> mut</span> right</span>:</span> Vec</span><</span>f32</span>></span> =</span> vec!</span>[</span>0_</span>f32</span>;</span> mono_sample_count</span>]</span>;</span></span>
</span>
//</span> build audio loop</span></span>
let</span> stream</span> =</span> device</span>.</span>build_output_stream</span>(</span></span>
    &</span>stream_config</span>,</span></span>
    move</span> |</span>output</span>:</span> &</span>mut</span> [</span>f32</span>]</span>,</span> _</span>:</span> &</span>cpal</span>::</span>OutputCallbackInfo</span>|</span> {</span></span>
        let</span> midi_events</span> =</span> //</span> find events to play</span></span>
        for</span> event</span> in</span> midi_events</span> {</span></span>
            //</span> synthesize events</span></span>
            synthesizer</span>.</span>process</span>(</span>event</span>)</span></span>
        }</span></span>
</span>
        //</span> Split buffer in two channels (left and right)</span></span>
        let</span> channel_len</span> =</span> output</span>.</span>len</span>(</span>)</span> /</span> channels_count</span>;</span></span>
</span>
        //</span> Render the waveform.</span></span>
        synthesizer</span>.</span>render</span>(</span>&</span>mut</span> left</span>[</span>..</span>channel_len</span>]</span>,</span> &</span>mut</span> right</span>[</span>..</span>channel_len</span>]</span>)</span>;</span></span>
        </span></span>
        //</span> Interleave the left and right channels into the output buffer.</span></span>
        for</span> (</span>i</span>,</span> (</span>l</span>,</span> r</span>)</span>)</span> in</span> left</span>.</span>iter</span>(</span>)</span>.</span>zip</span>(</span>right</span>.</span>iter</span>(</span>)</span>)</span>.</span>take</span>(</span>channel_len</span>)</span>.</span>enumerate</span>(</span>)</span> {</span></span>
            output</span>[</span>i</span> *</span> 2</span>]</span> =</span> *</span>l</span>;</span></span>
            output</span>[</span>i</span> *</span> 2</span> +</span> 1</span>]</span> =</span> *</span>r</span>;</span></span>
        }</span></span>
    }</span></span>
)</span></span>
//</span> Start the stream.</span></span>
let</span> stream</span> =</span> stream</span>.</span>unwrap</span>(</span>)</span>;</span></span>
stream</span>.</span>play</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span></code></pre>
For each run of the audio loop, it is possible to compute the next window of time to process by taking into account:</p>

the current timestamp of the audio player</li>
the tempo of the current measure</li>
how much time has passed since the previous interval</li>
</ul>
const</span> QUARTER_TIME</span>:</span> i32</span> =</span> 960</span>;</span> //</span> 1 quarter note = 960 ticks</span></span>
</span>
fn</span> tick_increase</span>(</span>tempo_bpm</span>:</span> i32</span>,</span> elapsed_seconds</span>:</span> f64</span>)</span> -></span> usize</span> {</span></span>
    let</span> tempo_bps</span> =</span> tempo_bpm</span> as</span> f64</span> /</span> 60</span>.</span>0</span>;</span></span>
    let</span> bump</span> =</span> QUARTER_TIME</span> as</span> f64</span> *</span> tempo_bps</span> *</span> elapsed_seconds</span>;</span></span>
    bump</span> as</span> usize</span></span>
}</span></span></code></pre>
Using the resulting tick increase, we can efficiently query our array of MIDI events to find the next events to play using binary search.</p>
let</span> tick_increase</span> =</span> tick_increase</span>(</span>tempo</span>,</span> elapsed_seconds</span>)</span>;</span></span>
let</span> next_tick</span> =</span> self</span>.</span>current_tick </span>+</span> tick_increase</span>;</span></span>
//</span> assume we already have a cursor for the start of the events (a.k.a. the index of the last event played)</span></span>
let</span> start_index</span> =</span> self</span>.</span>current_cursor</span>;</span></span>
let</span> end_index</span> =</span> match</span> sorted_events</span>[</span>start_index</span>..</span>]</span>.</span>binary_search_by_key</span>(</span>start_index</span>,</span> |</span>event</span>|</span> event</span>.</span>tick</span>)</span></span>
{</span></span>
    Ok</span>(</span>next_position</span>)</span> =></span> start_index</span> +</span> next_position</span>,</span></span>
    Err</span>(</span>next_position</span>)</span> =></span> {</span></span>
        if</span> next_position</span> ==</span> 0</span> {</span></span>
            //</span> no matching elements</span></span>
            return</span> Some</span>(</span>&</span>[</span>]</span>)</span>;</span></span>
        }</span></span>
        //</span> return slice until the last event</span></span>
        start_index</span> +</span> next_position</span> -</span> 1</span></span>
    }</span></span>
}</span>;</span></span>
//</span> return slice of events to play</span></span>
return</span> Some</span>(</span>&</span>self</span>.</span>sorted_events</span>[</span>start_index</span>..=</span>end_index</span>]</span>)</span></span></code></pre>
Now that we have the audio loop running, we can focus on the integration between the audio player and the UI.</p>
Putting it all together</a>
</h1>
Having a perfect integration is crucial to providing a smooth user experience:</p>


when clicking on the "Play" button, the tablature cursor should start moving, the notes should be highlighted as they are played.</p>
</li>

when a measure is clicked, the player should jump to the corresponding position in the tablature and the correct notes should be played.</p>
</li>

when clicking on a different track, the whole tablature should be updated to show the new track and the sound should be updated accordingly.</p>
</li>

when clicking on the "Solo" button, all other tracks should be muted.</p>
</li>

when clicking on the "Stop" button, the tablature cursor should be reset to the beginning and the sound should stop.</p>
</li>
</ul>
I think you get the idea.</p>
The critical bridge between the audio player and the UI is implemented using the iced::Subscription</code> mechanism.</p>
Subscriptions are a way to listen to external events and publish them as messages to the application.</p>
For instance, here is how the application reacts to pressing the space bar to toggle the playback:</p>
let</span> keyboard_subscription</span> =</span> keyboard</span>::</span>on_key_press</span>(</span>|</span>key</span>,</span> _modifiers</span>|</span> match</span> key</span>.</span>as_ref</span>(</span>)</span> {</span></span>
    keyboard</span>::</span>Key</span>::</span>Named</span>(</span>Space</span>)</span> =></span> Some</span>(</span>Message</span>::</span>PlayPause</span>)</span>,</span></span>
    _</span> =></span> None</span>,</span></span>
}</span>)</span>;</span></span></code></pre>
The update function does not care whether the message has been triggered by the keyboard or by clicking on the "Play" button.</p>
Using a similar mechanism, the audio player can send messages to the application to update the UI based on the current playback position.</p>
The application holds the receiving end of a tokio::sync::watch</a> channel containing the current timestamp which is published by the audio thread.</p>
fn</span> audio_player_beat_subscription</span>(</span>&</span>self</span>)</span> -></span> impl</span> Stream</span><</span>Item</span> =</span> Message</span>></span> {</span></span>
    let</span> beat_receiver</span> =</span> self</span>.</span>beat_receiver</span>.</span>clone</span>(</span>)</span>;</span></span>
    stream</span>::</span>channel</span>(</span>1</span>,</span> move</span> |</span>mut</span> output</span>|</span> async</span> move</span> {</span></span>
        let</span> mut</span> receiver</span> =</span> beat_receiver</span>.</span>lock</span>(</span>)</span>.</span>await</span>;</span></span>
        loop</span> {</span></span>
            //</span> get tick from audio player</span></span>
            let</span> tick</span> =</span> *</span>receiver</span>.</span>borrow_and_update</span>(</span>)</span>;</span></span>
            //</span> publish to UI</span></span>
            output</span></span>
                .</span>send</span>(</span>Message</span>::</span>FocusTick</span>(</span>tick</span>)</span>)</span></span>
                .</span>await</span></span>
                .</span>expect</span>(</span>"</span>send failed</span>"</span>)</span>;</span></span>
            //</span> wait for next beat</span></span>
            receiver</span>.</span>changed</span>(</span>)</span>.</span>await</span>.</span>expect</span>(</span>"</span>receiver failed</span>"</span>)</span>;</span></span>
        }</span></span>
    }</span>)</span></span>
}</span></span>
...</span></span>
//</span> setup subscription</span></span>
Subscription</span>::</span>run_with_id</span>(</span>"</span>audio-player-beat</span>"</span>,</span> audio_player_beat_subscription</span>)</span>)</span>;</span></span></code></pre>
The tablature processes the FocusTick</code> message to update the current measure position and highlight the notes.</p>

    


    
    
</figure>
A lot</strong> of details were required to maintain the illusion that everything is properly synchronized with the user's actions.</p>
Future work</a>
</h1>
The current version of Ruxguitar</code> is pretty much an MVP to get the project started.</p>
It is nowhere near TuxGuitar</code> in terms of features and usability.</p>
Here are a few ideas for the future:</p>

support more file formats (currently only .gp5</code> is supported)</li>
display more information about the tablature (e.g. rhythm, time signature, key signature etc.)</li>
support repeating measures</li>
support slowing down and speeding up the playback</li>
</ul>
Conclusion</a>
</h1>
I have been working on Ruxguitar</code> over the past year and I am very happy with the result.</p>
Not only did I learn a lot on the way but I also actually built a complex piece of software that appears to work.</p>
Working on such a large project alone required a lot of discipline because there were many times when I felt like giving up, running into cryptic bugs or being stuck on a feature for weeks.</p>
It would have been impossible to build Ruxguitar</code> without TuxGuitar</code> as a reference implementation and I am very grateful for the work done by the TuxGuitar</code> team over the years.</p>
After so much work on this project, it's about time I get back to playing guitar instead of writing software for it!</p>


Rust Meetup Linz: Qdrant - a vector search engine in Rust
2023-05-12T00:00:00+00:00
I had the privilege and pleasure to give a talk at the Rust Meetup Linz</a> about my work as a full time contributor on the Qdrant</a> vector search engine.</p>
I wanted the talk to be rather accessible given that vector search engines have recently been gaining popularity.</p>
The presentation is structured roughly in two parts:</p>
First the focus is on the use cases for the technology to really understand where it shines.</p>
The second part is about peeking under the hood to take a closer look at the internals, and to see why Rust is such a good fit for the job.</p>
The slides are available here</a> and the recording here</a>.</p>

</iframe>


Follow up on cracking ZIP archives in Rust
2023-04-03T00:00:00+00:00
In a previous article</a>, we explored how to build - step by step - a CLI in Rust to brute-force protected ZIP archives.</p>
The outcome of this research was the creation of the project zip-password-finder</a>.</p>
Since the last publication, the project has been slowly evolving to mitigate the shortcomings that were highlighted.</p>
In this follow-up entry we will go over the improvements made and the latest performance numbers.</p>
A short recap</a>
</h2>
The project zip-password-finder</code> supports two different modes to find the password: either from a dictionary file or by generating candidates from a charset.</p>
It uses a channel-based architecture to distribute candidate passwords to a set of workers who are responsible for testing them in parallel.</p>

    


    
    
</figure>
It is important to mention that ZIP archives can be encrypted using either ZipCrypto or AES</a>.</p>
In practice, ZipCrypto can be attacked</a> and is much cheaper to brute force.</p>
As it can still be found in the wild, most notably produced by Windows machines, zip-password-finder</code> handles this format transparently for the user.</p>
However, we won't give it much thought in this article given its issues and our desire to focus on the more difficult AES case.</p>
Test drive</a>
</h2>
Here is the CLI we are going to work with.</p>
./zip-password-finder</span> -</span>h</span></span>
Find</span> the</span> password</span> of</span> protected</span> ZIP</span> files</span></span>
</span>
Usage:</span> zip-password-finder</span> [OPTIONS</span>] --inputFile </span><</span>inputFile</span>></span></span>
</span>
Options:</span></span>
  -i,</span> -</span>-inputFile</span> <</span>inputFil</span>e</span>></span></span>
          path</span> to</span> zip</span> input</span> file</span></span>
  -w,</span> -</span>-workers</span> <</span>worker</span>s</span>></span></span>
          number</span> of</span> workers</span></span>
  -p,</span> -</span>-passwordDictionary</span> <</span>passwordDictionar</span>y</span>></span></span>
          path</span> to</span> a</span> password</span> dictionary</span> file</span></span>
  -c,</span> -</span>-charset</span> <</span>charse</span>t</span>></span></span>
          charset</span> to</span> use</span> to</span> generate</span> password</span> [default:</span> medium]</span> [possible</span> values:</span> basic,</span> easy,</span> medium,</span> hard]</span></span>
      --minPasswordLen</span> <</span>minPasswordLe</span>n</span>></span></span>
          minimum</span> password</span> length</span> [default:</span> 1]</span></span>
      --maxPasswordLen</span> <</span>maxPasswordLe</span>n</span>></span></span>
          maximum</span> password</span> length</span> [default:</span> 10]</span></span>
  -h,</span> -</span>-help</span></span>
          Print</span> help</span> information</span></span>
  -V,</span> -</span>-version</span></span>
          Print</span> version</span> information</span></span></code></pre>
We can give it a try by creating an encrypted zip archive; let's call it secret.zip</code> with the four-character password "ab12"</code>.</p>
zipinfo</span> secret.zip</span> </span></span>
Archive:</span>  secret.zip</span></span>
Zip</span> file</span> size:</span> 209</span> bytes,</span> number</span> of</span> entries:</span> 1</span></span>
-rw-rw-r--</span>  6.3</span> unx</span>       13</span> Bx</span> u099</span> 23-Mar-27</span> 20:17</span> secret.txt</span></span>
1</span> file,</span> 13</span> bytes</span> uncompressed,</span> 21</span> bytes</span> compressed:</span>  -</span>61.5%</span></span></code></pre>
Given that the password is pretty short, we can ask zip-password-finder</code> to generate all possible passwords from the charset formed by the lowercase letters, uppercase letters, and digits (called medium).</p>
time</span> ./zip-password-finder -i test-encrypted.zip --charset medium</span></span></code></pre>
It returns with the right answer shortly before the one-minute mark.</p>
time</span> ./zip-password-finder-0.2 -i </span>~</span>/Workspace/blog-data/find-password-zip/secret.zip -c medium</span></span>
Using</span> 4</span> workers</span> to</span> test</span> passwords</span></span>
Generating</span> passwords</span> with</span> length</span> from</span> 1</span> to</span> 10</span> for</span> charset</span> with</span> length</span> 62</span></span>
[</span>'</span>a</span>'</span>, </span>'</span>b</span>'</span>, </span>'</span>c</span>'</span>, </span>'</span>d</span>'</span>, </span>'</span>e</span>'</span>, </span>'</span>f</span>'</span>, </span>'</span>g</span>'</span>, </span>'</span>h</span>'</span>, </span>'</span>i</span>'</span>, </span>'</span>j</span>'</span>, </span>'</span>k</span>'</span>, </span>'</span>l</span>'</span>, </span>'</span>m</span>'</span>, </span>'</span>n</span>'</span>, </span>'</span>o</span>'</span>, </span>'</span>p</span>'</span>, </span>'</span>q</span>'</span>, </span>'</span>r</span>'</span>, </span>'</span>s</span>'</span>, </span>'</span>t</span>'</span>, </span>'</span>u</span>'</span>, </span>'</span>v</span>'</span>, </span>'</span>w</span>'</span>, </span>'</span>x</span>'</span>, </span>'</span>y</span>'</span>, </span>'</span>z</span>'</span>, </span>'</span>A</span>'</span>, </span>'</span>B</span>'</span>, </span>'</span>C</span>'</span>, </span>'</span>D</span>'</span>, </span>'</span>E</span>'</span>, </span>'</span>F</span>'</span>, </span>'</span>G</span>'</span>, </span>'</span>H</span>'</span>, </span>'</span>I</span>'</span>, </span>'</span>J</span>'</span>, </span>'</span>K</span>'</span>, </span>'</span>L</span>'</span>, </span>'</span>M</span>'</span>, </span>'</span>N</span>'</span>, </span>'</span>O</span>'</span>, </span>'</span>P</span>'</span>, </span>'</span>Q</span>'</span>, </span>'</span>R</span>'</span>, </span>'</span>S</span>'</span>, </span>'</span>T</span>'</span>, </span>'</span>U</span>'</span>, </span>'</span>V</span>'</span>, </span>'</span>W</span>'</span>, </span>'</span>X</span>'</span>, </span>'</span>Y</span>'</span>, </span>'</span>Z</span>'</span>, </span>'</span>0</span>'</span>, </span>'</span>1</span>'</span>, </span>'</span>2</span>'</span>, </span>'</span>3</span>'</span>, </span>'</span>4</span>'</span>, </span>'</span>5</span>'</span>, </span>'</span>6</span>'</span>, </span>'</span>7</span>'</span>, </span>'</span>8</span>'</span>, </span>'</span>9</span>'</span>]</span></span>
Starting</span> search</span> space</span> for</span> password</span> length</span> 1</span> (62</span> possibilities</span>) </span></span>
Starting</span> search</span> space</span> for</span> password</span> length</span> 2</span> (3844</span> possibilities</span>) (</span>61</span> passwords</span> generated</span> so</span> far</span>)</span></span>
Starting</span> search</span> space</span> for</span> password</span> length</span> 3</span> (238328</span> possibilities</span>) (</span>3905</span> passwords</span> generated</span> so</span> far</span>)</span></span>
Starting</span> search</span> space</span> for</span> password</span> length</span> 4</span> (14776336</span> possibilities</span>) (</span>242233</span> passwords</span> generated</span> so</span> far</span>)</span></span>
Password</span> found</span> '</span>ab12</span>'</span></span>
</span>
real</span> 0m54,262s</span></span>
user</span> 3m40,620s</span></span>
sys</span>  0m2,431s</span></span></code></pre>
Finding a 4-letter password on such a small charset was a rather easy task for a modern CPU.</p>
There are only (26 lower + 26 upper + 10 digit)^4 = 14.776.336</code> candidates to test.</p>
However, a learning from the previous article is that scalability suffers as the number of workers increases.</p>
Improving on this limitation is critical for being able to work against real-world passwords.</p>
We will reuse this test file for the benchmarks in the rest of the article.</p>
Independent workers</a>
</h2>
Previous measurements clearly showed that performance did not improve linearly with the number of workers.</p>
We previously attributed the flat line between 4 and 8 workers to the Hyper-threading on my CPU, which does not bode well for this type of workload.</p>

    


    
    
</figure>
The most logical hypothesis is that there is a source of contention in the current architecture as the number of workers increases.</p>
It seems straightforward to believe that workers are somehow competing and synchronizing with each other to poll candidates from the channel.</p>
Picking an easy architecture helped us get started at the beginning, but it could very well be a limiting factor to get faster.</p>
An interesting insight is that workers do not need to share the same source of passwords.</p>
Each worker could agree upfront to work on a subset of the generated passwords.</p>
Given n workers, each worker generates and tests 1/n</code> of the passwords with an offset.</p>
For instance, with 3 workers on a lowercase charset:</p>

first worker : "a", "d", "g", "j", ...</li>
second worker: "b", "e", "h", "k", ...</li>
third worker : "c", "f", "i", "l", ...</li>
</ul>
This way, we remove all coordination at runtime, so the workers can make progress in isolation.</p>

    


    
    
</figure>
This pattern can be elegantly integrated by refactoring the dictionary reader and password generator to be proper Iterator</code>s.</p>
let</span> mut</span> passwords_iter</span>:</span> Box</span><</span>dyn</span> Iterator</span><</span>Item</span> =</span> String</span>></span>></span> =</span> match</span> strategy</span> {</span></span>
    Strategy</span>::</span>GenPasswords</span> {</span></span>
        charset</span>,</span></span>
        min_password_len</span>,</span></span>
        max_password_len</span>,</span></span>
    }</span> =></span> {</span></span>
        let</span> iterator</span> =</span> password_generator_iter</span>(</span>&</span>charset</span>,</span> min_password_len</span>,</span> max_password_len</span>)</span>;</span></span>
        Box</span>::</span>new</span>(</span>iterator</span>)</span></span>
    }</span>,</span></span>
    Strategy</span>::</span>PasswordFile</span>(</span>dictionary_path</span>)</span> =></span> {</span></span>
        let</span> iterator</span> =</span> password_dictionary_reader_iter</span>(</span>&</span>dictionary_path</span>)</span>;</span></span>
        Box</span>::</span>new</span>(</span>iterator</span>)</span></span>
    }</span></span>
}</span>;</span></span>
</span>
//</span> filtering iterator for worker `worker_index`</span></span>
passwords_iter</span> =</span> if</span> worker_count</span> ></span> 1</span> {</span></span>
    let</span> filtered</span> =</span> passwords_iter</span>.</span>enumerate</span>(</span>)</span>.</span>filter_map</span>(</span>move</span> |</span>(</span>i</span>,</span> candidate</span>)</span>|</span> {</span></span>
        if</span> i</span> %</span> worker_count</span> ==</span> worker_index</span> -</span> 1</span> {</span></span>
            Some</span>(</span>candidate</span>)</span></span>
        }</span> else</span> {</span></span>
            None</span></span>
        }</span></span>
    }</span>)</span></span>
    Box</span>::</span>new</span>(</span>filtered</span>)</span></span>
}</span> else</span> {</span></span>
    passwords_iter</span></span>
}</span></span>
</span>
//</span> processing loop</span></span>
for</span> password</span> in</span> passwords_iter</span> {</span></span>
    ...</span>        </span></span>
}</span>   </span></span></code></pre>
Benchmarking both architectures on the secret.zip</code> archive with various numbers of workers yields the following result.</p>

    


    
    
</figure>
The performance improved slightly, but the shape of the curve did not change, meaning the scalability profile did not get any better.</p>
Switching to independent workers is beneficial, but it did not have the desired effect.</p>
There is more research to be done to improve scalability.</p>
Cheaper iteration</a>
</h2>
One obvious strategy to go faster is to make testing a candidate as cheap as possible.</p>
The first version of the project was rather naive in that regard.</p>
let</span> mut</span> archive</span> =</span> zip</span>::</span>ZipArchive</span>::</span>new</span>(</span>file</span>)</span>.</span>expect</span>(</span>"</span>Archive validated before-hand</span>"</span>)</span>;</span></span>
loop</span> {</span></span>
    match</span> receive_password</span>.</span>recv</span>(</span>)</span> {</span></span>
        Err</span>(</span>_</span>)</span> =></span> break</span>,</span> //</span> channel disconnected, stop thread</span></span>
        Ok</span>(</span>password</span>)</span> =></span> {</span></span>
            let</span> res</span> =</span> archive</span>.</span>by_index_decrypt</span>(</span>0</span>,</span> password</span>.</span>as_bytes</span>(</span>)</span>)</span>;</span> //</span> decrypt first file in archive</span></span>
            match</span> res</span> {</span></span>
                Err</span>(</span>e</span>)</span> =></span> panic!</span>(</span>"</span>Unexpected error </span>{</span>:?</span>}</span>"</span>,</span> e</span>)</span>,</span></span>
                Ok</span>(</span>Err</span>(</span>_</span>)</span>)</span> =></span> (</span>)</span>,</span> //</span> invalid password - continue</span></span>
                Ok</span>(</span>Ok</span>(</span>_</span>)</span>)</span> =></span> {</span></span>
                    println!</span>(</span>"</span>Password found:</span>{</span>}</span>"</span>,</span> password</span>)</span>;</span></span>
                    break</span>;</span> //</span> stop thread</span></span>
                }</span></span>
            }</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>
For each candidate, the function archive.by_index_decrypt(0, password)</code> is called; hopefully it does not do too much.</p>
This function belongs to the zip-rs</code> project and performs the following actions:</p>

Find file data in the archive based on file index used (here 0).</li>
Verify that the file is encrypted.</li>
Load raw content of the file.</li>
Prepare AES decoder (get salt & verification key).</li>
Compute key for candidate and salt.</li>
Compare derived key to verification key.</li>
</ol>
Well, that is a lot</strong> of repetition!</p>
Ideally, we just want to perform the 5th and 6th steps for each candidate, the rest can be pre-computed.</p>
To do this, I had to fork the zip-rs</code> crate to expose some internals.</p>
On the AesReader</code>:</p>
///</span> Read the AES header bytes and returns the key and salt.</span></span>
///</span></span>
///</span> # Returns</span></span>
///</span></span>
///</span> the key and the salt</span></span>
pub</span> fn</span> get_key_and_salt</span>(</span>mut</span> self</span>)</span> -></span> io</span>::</span>Result</span><</span>(</span>Vec</span><</span>u8</span>></span>,</span> Vec</span><</span>u8</span>></span>)</span>></span> {</span></span>
    let</span> salt_length</span> =</span> self</span>.</span>aes_mode</span>.</span>salt_length</span>(</span>)</span>;</span></span>
</span>
    let</span> mut</span> salt</span> =</span> vec!</span>[</span>0</span>;</span> salt_length</span>]</span>;</span></span>
    self</span>.</span>reader</span>.</span>read_exact</span>(</span>&</span>mut</span> salt</span>)</span>?</span>;</span></span>
</span>
    //</span> next are 2 bytes used for password verification</span></span>
    let</span> mut</span> pwd_verification_value</span> =</span> vec!</span>[</span>0</span>;</span> PWD_VERIFY_LENGTH</span>]</span>;</span></span>
    self</span>.</span>reader</span>.</span>read_exact</span>(</span>&</span>mut</span> pwd_verification_value</span>)</span>?</span>;</span></span>
    Ok</span>(</span>(</span>pwd_verification_value</span>,</span> salt</span>)</span>)</span></span>
}</span></span></code></pre>
And on the ZipArchiveReader</code>:</p>
///</span> Returns key and salt</span></span>
pub</span> fn</span> get_aes_key_and_salt</span>(</span></span>
    &</span>mut</span> self</span>,</span></span>
    file_number</span>:</span> usize</span>,</span></span>
)</span> -></span> ZipResult</span><</span>Option</span><</span>(</span>AesMode</span>,</span> Vec</span><</span>u8</span>></span>,</span> Vec</span><</span>u8</span>></span>)</span>></span>></span> {</span></span>
    let</span> data</span> =</span> self</span></span>
        .</span>shared</span></span>
        .</span>files</span></span>
        .</span>get</span>(</span>file_number</span>)</span></span>
        .</span>ok_or</span>(</span>ZipError</span>::</span>FileNotFound</span>)</span>?</span>;</span></span>
</span>
    if</span> !</span>data</span>.</span>encrypted </span>{</span></span>
        return</span> Err</span>(</span>ZipError</span>::</span>UnsupportedArchive</span>(</span>ZipError</span>::</span>PASSWORD_REQUIRED</span>)</span>)</span>;</span></span>
    }</span></span>
    let</span> limit_reader</span> =</span> find_content</span>(</span>data</span>,</span> &</span>mut</span> self</span>.</span>reader</span>)</span>?</span>;</span></span>
    match</span> data</span>.</span>aes_mode </span>{</span></span>
        None</span> =></span> Ok</span>(</span>None</span>)</span>,</span></span>
        Some</span>(</span>(</span>aes_mode</span>,</span> _</span>)</span>)</span> =></span> {</span></span>
            let</span> (</span>key</span>,</span> salt</span>)</span> =</span> AesReader</span>::</span>new</span>(</span>limit_reader</span>,</span> aes_mode</span>,</span> data</span>.</span>compressed_size</span>)</span></span>
                .</span>get_key_and_salt</span>(</span>)</span> //</span> NEW</span></span>
                .</span>expect</span>(</span>"</span>AES reader failed</span>"</span>)</span>;</span></span>
            Ok</span>(</span>Some</span>(</span>(</span>aes_mode</span>,</span> key</span>,</span> salt</span>)</span>)</span>)</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>
Luckily, Cargo makes it very low-friction to work with a forked crate!</p>
zip</span> =</span> {</span> git</span> =</span> "</span>https://github.com/agourlay/zip.git</span>"</span>,</span> branch</span> =</span> "</span>zip-password-finder</span>"</span> }</span></span></code></pre>
Thanks to this insight, it is possible to pre-compute most of the AES information, removing a lot of work from the processing loop.</p>
//</span> AES info bindings initialized once</span></span>
let</span> mut</span> derived_key_len</span> =</span> ...</span></span>
let</span> mut</span> derived_key</span>:</span> Vec</span><</span>u8</span>></span> =</span> ...</span></span>
let</span> mut</span> salt</span>:</span> Vec</span><</span>u8</span>></span> =</span> ...</span></span>
let</span> mut</span> key</span>:</span> Vec</span><</span>u8</span>></span> =</span> ...</span></span>
</span>
//</span> processing loop</span></span>
for</span> password</span> in</span> passwords_iter</span> {</span></span>
    let</span> password_bytes</span> =</span> password</span>.</span>as_bytes</span>(</span>)</span>;</span></span>
    let</span> mut</span> potential_match</span> =</span> true</span>;</span></span>
    //</span> process AES KEY</span></span>
    //</span> use PBKDF2 with HMAC-Sha1 to derive the key</span></span>
    pbkdf2</span>::</span>pbkdf2</span>::</span><</span>Hmac</span><</span>Sha1</span>></span>></span>(</span>password_bytes</span>,</span> &</span>salt</span>,</span> 1000</span>,</span> &</span>mut</span> derived_key</span>)</span>.</span>expect</span>(</span>"</span>PBKDF2 should not fail</span>"</span>)</span>;</span></span>
    let</span> pwd_verify</span> =</span> &</span>derived_key</span>[</span>derived_key_len</span> -</span> 2</span>..</span>]</span>;</span></span>
    //</span> the last 2 bytes should equal the password verification value</span></span>
    potential_match</span> =</span> key</span> ==</span> pwd_verify</span>;</span></span>
    </span></span>
    if</span> potential_match</span> {</span></span>
        //</span> try to decode archive to eliminate collision</span></span>
        let</span> res</span> =</span> archive</span>.</span>by_index_decrypt</span>(</span>0</span>,</span> password</span>.</span>as_bytes</span>(</span>)</span>)</span>;</span> //</span> decrypt first file in archive</span></span>
        ...</span></span>
    }</span></span>
}</span></span></code></pre>
Benchmarking the difference:</p>
 hyperfine</span> -</span>-runs</span> 2</span> \</span></span>
 -</span>-warmup</span> 1</span> \</span></span>
 -</span>n</span> before</span> "</span>./zip-password-finder-before -i secret.zip -c medium</span>"</span> \</span></span>
 -</span>n</span> after</span>  "</span>./zip-password-finder-after  -i secret.zip -c medium</span>"</span></span></code></pre>Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

before</code></td> 58.977 ± 0.512</td> 58.615</td> 59.339</td> 1.08 ± 0.01</td></tr>
after</code></td> 54.476 ± 0.037</td> 54.450</td> 54.503</td> 1.00</td></tr>
</tbody></table>
A solid 8% improvement is something I gladly take.</p>
Performance over time</a>
</h2>
Outside of the two previously mentioned improvements, the runtime improved slowly over time.</p>
Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

0.1</code></td> 62.317 ± 0.224</td> 62.159</td> 62.475</td> 1.21 ± 0.01</td></tr>
0.2</code></td> 58.823 ± 0.013</td> 58.814</td> 58.833</td> 1.14 ± 0.01</td></tr>
0.3</code></td> 59.423 ± 0.047</td> 59.390</td> 59.456</td> 1.15 ± 0.01</td></tr>
0.4</code></td> 55.562 ± 0.034</td> 55.538</td> 55.586</td> 1.08 ± 0.01</td></tr>
0.5</code></td> 56.877 ± 0.362</td> 56.621</td> 57.132</td> 1.10 ± 0.01</td></tr>
0.5.1</code></td> 54.230 ± 0.424</td> 53.930</td> 54.530</td> 1.05 ± 0.01</td></tr>
0.6</code></td> 51.619 ± 0.412</td> 51.327</td> 51.911</td> 1.00</td></tr>
0.6.1</code></td> 52.421 ± 0.166</td> 52.303</td> 52.538</td> 1.02 ± 0.01</td></tr>
0.6.2</code></td> 52.149 ± 0.155</td> 52.039</td> 52.259</td> 1.01 ± 0.01</td></tr>
0.6.3</code></td> 52.073 ± 0.193</td> 51.937</td> 52.209</td> 1.01 ± 0.01</td></tr>
</tbody></table>
However, the performance optimizations are hitting a wall; a single bottleneck is now largely dominating the CPU usage.</p>
Each worker is busy doing the following: (full flamegraph</a> available).</p>

    


    
    
</figure>
Computing the SHA1 for each candidate takes up to 90% of the total CPU time; improving this implementation would have a massive</strong> impact on the runtime!</p>
It is interesting to note that the function is called sha1::compress::soft::compress</code> - now that's a weird name.</p>
Let's take a look under the hood</a>.</p>
cpufeatures</span>::</span>new!</span>(</span>shani_cpuid</span>,</span> "</span>sha</span>"</span>,</span> "</span>sse2</span>"</span>,</span> "</span>ssse3</span>"</span>,</span> "</span>sse4.1</span>"</span>)</span>;</span></span>
</span>
pub</span> fn</span> compress</span>(</span>state</span>:</span> &</span>mut</span> [</span>u32</span>;</span> 5</span>]</span>,</span> blocks</span>:</span> &</span>[</span>[</span>u8</span>;</span> 64</span>]</span>]</span>)</span> {</span></span>
    if</span> shani_cpuid</span>::</span>get</span>(</span>)</span> {</span></span>
        unsafe</span> {</span></span>
            digest_blocks</span>(</span>state</span>,</span> blocks</span>)</span>;</span></span>
        }</span></span>
    }</span> else</span> {</span></span>
        super</span>::</span>soft</span>::</span>compress</span>(</span>state</span>,</span> blocks</span>)</span>;</span></span>
    }</span></span>
}</span></span>
</span>
#</span>[</span>target_feature</span>(</span>enable </span>=</span> "</span>sha,sse2,ssse3,sse4.1</span>"</span>)</span>]</span></span>
unsafe</span> fn</span> digest_blocks</span>(</span>state</span>:</span> &</span>mut</span> [</span>u32</span>;</span> 5</span>]</span>,</span> blocks</span>:</span> &</span>[</span>[</span>u8</span>;</span> 64</span>]</span>]</span>)</span> {</span></span>
   ...</span></span>
}</span>    </span></span></code></pre>
This code is picking an implementation at runtime based on the capabilities of the CPU.</p>
The branching depends on the sha</code>, sse2</code>, ssse3</code> and sse4.1</code> instruction sets.</p>
Here, soft</code> stands for software</code>, meaning not using hardware acceleration.</p>
But why is my CPU not eligible for the optimized SIMD implementation?</p>
lscpu</span></span>
Architecture:</span>            x86_64</span></span>
  CPU</span> op-mode</span>(</span>s</span>)</span>:</span>        32-bit,</span> 64-bit</span></span>
  Address</span> sizes:</span>         39</span> bits</span> physical,</span> 48</span> bits</span> virtual</span></span>
  Byte</span> Order:</span>            Little</span> Endian</span></span>
CPU(s</span>):                  8</span></span>
  On-line</span> CPU</span>(</span>s</span>)</span> list:</span>   0-7</span></span>
Vendor</span> ID:</span>               GenuineIntel</span></span>
  Model</span> name:</span>            Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i7-10610U</span> CPU</span> @</span> 1.80GHz</span></span>
.</span>.</span>.</span></span>
 Flags:</span>               ...sse</span> sse2</span> ssse3</span> sse4_1</span> sse4_2...</span></span></code></pre>
Sadly, I have all the necessary sse</code> instructions but missing the sha</code> one.</p>
This means we need to make sure to use a CPU with SHA instructions</a> to get the best performance.</p>
Optimizing build</a>
</h2>
Given that I do not have access to adequate hardware, all I can do for the time being is tweak my build in order to squeeze out one last performance gain.</p>
The first idea is to enable Link-time Optimization (LTO), which is an optimization taking place at compile time.</p>
Be aware that it can make the compilation much slower!</p>
There are two possible values for it:</p>
[profile.release]</span></span>
lto = "thin" or "fat"</span></span></code></pre>
A second idea is to build a binary with the best instructions for the current CPU.</p>
RUSTFLAGS</span>=</span>"</span>-C target-cpu=native</span>"</span> cargo</span> build</span> -</span>-release</span></span></code></pre>
Benchmarking those configurations yields the following results:</p>
Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

native</code></td> 52.456 ± 2.987</td> 50.344</td> 54.568</td> 1.00</td></tr>
master</code></td> 57.439 ± 1.135</td> 56.637</td> 58.241</td> 1.09 ± 0.07</td></tr>
native-lto</code></td> 59.161 ± 0.437</td> 58.852</td> 59.470</td> 1.13 ± 0.06</td></tr>
lto-thin</code></td> 61.677 ± 0.153</td> 61.569</td> 61.786</td> 1.18 ± 0.07</td></tr>
lto-fat</code></td> 67.777 ± 0.277</td> 67.582</td> 67.973</td> 1.29 ± 0.07</td></tr>
</tbody></table>
Well, we better not enable LTO but there is a case to be made for a native build.</p>
This performance related section was a bit messy, but we learned a few important things:</p>

Do not blindly enable LTO without validating that it is beneficial.</li>
If you are super serious about performance, you would probably benefit from producing native binaries for your hardware.</li>
Some libraries have alternate implementations depending on the hardware capabilities.</li>
</ul>
State of the art</a>
</h2>
To appreciate the performance of zip-password-finder</code>, I thought it would be a good idea to compare it to the state of the art of ZIP cracking.</p>
It seems serious</strong> people are using the Hashcat</a> project, which is apparently the "World's fastest password cracker"!</p>

    


    
    
</figure>
sudo</span> apt-get</span> install</span> hashcat</span></span></code></pre>
This tool runs on GPUs; in my case, I had to install the OpenCL Runtime for Intel Core</a>.</p>
It supports a very large list of algorithms, including the one we need here, WinZip</code>.</p>
In order to use Hashcat</code>, we first need to extract the underlying hash of our zip file into a format that is understood by Hashcat</code>.</p>
To that end, we will use zip2john</code> from the John the Ripper</a> project.</p>
Installing it manually</a> with:</p>
git</span> clone</span> https://github.com/openwall/john</span> -</span>b</span> bleeding-jumbo</span> john</span></span>
cd</span> john/src/</span></span>
./configure</span></span>
make</span> -</span>s</span> clean</span> &&</span> make</span> -</span>sj4</span></span></code></pre>
and then running it</p>
cd</span> /Workspace/john/run</span></span>
./zip2john</span> ~/zip-files/secret.zip</span></span></code></pre>
yields the text</p>
secret.zip/secret.txt:$zip2$*0*1*0*abf015b7750c67f9*f8fe*d*c97f48465ca6a99031db50f79a*d9e156b830df03899575*$/zip2$:secret.txt:secret.zip:/home/agourlay/zip-files/secret.zip</span></span></code></pre>
The output needs to be cleaned up and saved into a file, hash.txt</code> here.</p>
$zip2$*0*1*0*abf015b7750c67f9*f8fe*d*c97f48465ca6a99031db50f79a*d9e156b830df03899575*$/zip2$</span></span></code></pre>
After a rather tedious setup, it is finally time to crack our hash!</p>
We need to instruct hashcat</code> to:</p>

perform a brute force attack</li>
target the hash.txt</code></li>
use an algorithm for ZIP archive hashes</li>
generate passwords for a charset with lower case, upper case and digits (up to 4 chars)</li>
</ul>
Luckily for us, the help prompt is truly excellent!</p>
Here are the interesting excerpts:</p>
For the attack mode:</p>
-</span> [</span> Attack</span> Modes</span> ]</span> -</span></span>
</span>
  #</span> | Mode</span></span>
 =</span>=</span>=</span>+</span>=</span>=</span>=</span>=</span>=</span>=</span></span>
  0</span> |</span> Straight</span></span>
  1</span> |</span> Combination</span></span>
  3</span> |</span> Brute-force</span></span>
  6</span> |</span> Hybrid</span> Wordlist</span> +</span> Mask</span></span>
  7</span> |</span> Hybrid</span> Mask</span> +</span> Wordlist</span></span>
  9</span> |</span> Association</span></span></code></pre>
For the hash type:</p>
-</span> [</span> Hash</span> modes</span> ]</span> -</span></span>
      #</span> | Name                                                | Category</span></span>
  13600</span> |</span> WinZip</span>                                              |</span> Archive</span></span></code></pre>
For the charset:</p>
  ?</span> |</span> Charset</span></span>
 =</span>=</span>=</span>+</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span></span>
  l</span> |</span> abcdefghijklmnopqrstuvwxyz</span> [a-z</span>]</span></span>
  u</span> |</span> ABCDEFGHIJKLMNOPQRSTUVWXYZ</span> [A-Z</span>]</span></span>
  d</span> |</span> 0123456789</span>                 [0-9</span>]</span></span>
  h</span> |</span> 0123456789abcdef</span>           [0-9a-f</span>]</span></span>
  H</span> |</span> 0123456789ABCDEF</span>           [0-9A-F</span>]</span></span>
  s</span> |</span>  !</span>"</span>#$%&'()*+,-./:;<=>?@[\]^_</span>`</span>{|}~</span></span>
  a</span> |</span> ?l?u?d?s</span></span>
  b</span> |</span> 0x00</span> -</span> 0xff</span></span></code></pre>
According to the documentation</a>, it seems we need to set up a four-character mask that will be explored incrementally.</p>
After a bit of trial and error, it gives the following magic incantation:</p>
hashcat</span> -</span>a</span> 3</span> hash.txt</span> -</span>m</span> 13600</span> -</span>1</span> ?l?u?d</span> ?1?1?1?1</span> -</span>-increment</span></span></code></pre>
Let's go!</p>
time</span> hashcat -a 3 hash.txt -m 13600 -1 </span>?</span>l</span>?</span>u</span>?</span>d </span>?</span>1</span>?</span>1</span>?</span>1</span>?</span>1 --increment</span></span>
hashcat</span> (v6.2.5</span>) starting</span></span>
</span>
OpenCL</span> API</span> (OpenCL</span> 3.0</span> ) - Platform</span> #</span>1 [Intel(R) Corporation]</span></span>
=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span>=</span></span>
*</span> Device</span> #</span>1: Intel(R) UHD Graphics [0x9b41], 12640/25384 MB (2047 MB allocatable), 24MCU</span></span>
</span>
Minimum</span> password</span> length</span> supported</span> by</span> kernel:</span> 0</span></span>
Maximum</span> password</span> length</span> supported</span> by</span> kernel:</span> 256</span></span>
</span>
Hashes:</span> 1</span> digests</span>;</span> 1</span> unique</span> digests,</span> 1</span> unique</span> salts</span></span>
Bitmaps:</span> 16</span> bits,</span> 65536</span> entries,</span> 0x0000ffff</span> mask,</span> 262144</span> bytes,</span> 5/13</span> rotates</span></span>
</span>
Optimizers</span> applied:</span></span>
*</span> Zero-Byte</span></span>
*</span> Single-Hash</span></span>
*</span> Single-Salt</span></span>
*</span> Brute-Force</span></span>
*</span> Slow-Hash-SIMD-LOOP</span></span>
</span>
Watchdog:</span> Hardware</span> monitoring</span> interface</span> not</span> found</span> on</span> your</span> system.</span></span>
Watchdog:</span> Temperature</span> abort</span> trigger</span> disabled.</span></span>
</span>
Host</span> memory</span> required</span> for</span> this</span> attack:</span> 1475</span> MB</span></span>
</span>
The</span> wordlist</span> or</span> mask</span> that</span> you</span> are</span> using</span> is</span> too</span> small.</span></span>
This</span> means</span> that</span> hashcat</span> cannot</span> use</span> the</span> full</span> parallel</span> power</span> of</span> your</span> device</span>(</span>s</span>)</span>.</span></span>
Unless</span> you</span> supply</span> more</span> work,</span> your</span> cracking</span> speed</span> will</span> drop.</span></span>
For</span> tips</span> on</span> supplying</span> more</span> work,</span> see:</span> https://hashcat.net/faq/morework</span></span>
</span>
Approaching</span> final</span> keyspace</span> -</span> workload</span> adjusted.</span>           </span></span>
</span>
Session..........:</span> hashcat</span>                                </span></span>
Status...........:</span> Exhausted</span></span>
Hash.Mode........:</span> 13600</span> (WinZip</span>)</span></span>
Hash.Target......:</span> $</span>zip2</span>$</span>*</span>0</span>*</span>1</span>*</span>0</span>*</span>abf015b7750c67f9</span>*</span>f8fe</span>*</span>d</span>*</span>c97f48465ca6a9.../zip2</span>$</span></span>
Time.Started.....:</span> Sat</span> Apr</span>  1</span> 15:53:49</span> 2023</span> (0</span> secs</span>)</span></span>
Time.Estimated...:</span> Sat</span> Apr</span>  1</span> 15:53:49</span> 2023</span> (0</span> secs</span>)</span></span>
Kernel.Feature...:</span> Pure</span> Kernel</span></span>
Guess.Mask.......:</span> ?1</span> [1</span>]</span></span>
Guess.Charset....:</span> -</span>1</span> ?l?u?d,</span> -</span>2</span> Undefined,</span> -</span>3</span> Undefined,</span> -</span>4</span> Undefined</span> </span></span>
Guess.Queue......:</span> 1/4</span> (25.00%</span>)</span></span>
Speed.#1.........:</span>      117</span> H/s</span> (6.27ms</span>) @ Accel:32 Loops:999 Thr:8 Vec:1</span></span>
Recovered........:</span> 0/1</span> (0.00%</span>) Digests</span></span>
Progress.........:</span> 62/62</span> (100.00%</span>)</span></span>
Rejected.........:</span> 0/62</span> (0.00%</span>)</span></span>
Restore.Point....:</span> 1/1</span> (100.00%</span>)</span></span>
Restore.Sub.#1...:</span> Salt:0</span> Amplifier:61-62</span> Iteration:0-999</span></span>
Candidate.Engine.:</span> Device</span> Generator</span></span>
Candidates.#1....:</span> X</span> -</span>></span> X</span></span>
</span>
The</span> wordlist</span> or</span> mask</span> that</span> you</span> are</span> using</span> is</span> too</span> small.</span></span>
This</span> means</span> that</span> hashcat</span> cannot</span> use</span> the</span> full</span> parallel</span> power</span> of</span> your</span> device</span>(</span>s</span>)</span>.</span></span>
Unless</span> you</span> supply</span> more</span> work,</span> your</span> cracking</span> speed</span> will</span> drop.</span></span>
For</span> tips</span> on</span> supplying</span> more</span> work,</span> see:</span> https://hashcat.net/faq/morework</span></span>
</span>
Approaching</span> final</span> keyspace</span> -</span> workload</span> adjusted.</span>           </span></span>
</span>
Session..........:</span> hashcat</span>                                </span></span>
Status...........:</span> Exhausted</span></span>
Hash.Mode........:</span> 13600</span> (WinZip</span>)</span></span>
Hash.Target......:</span> $</span>zip2</span>$</span>*</span>0</span>*</span>1</span>*</span>0</span>*</span>abf015b7750c67f9</span>*</span>f8fe</span>*</span>d</span>*</span>c97f48465ca6a9.../zip2</span>$</span></span>
Time.Started.....:</span> Sat</span> Apr</span>  1</span> 15:53:52</span> 2023</span> (0</span> secs</span>)</span></span>
Time.Estimated...:</span> Sat</span> Apr</span>  1</span> 15:53:52</span> 2023</span> (0</span> secs</span>)</span></span>
Kernel.Feature...:</span> Pure</span> Kernel</span></span>
Guess.Mask.......:</span> ?1?1</span> [2</span>]</span></span>
Guess.Charset....:</span> -</span>1</span> ?l?u?d,</span> -</span>2</span> Undefined,</span> -</span>3</span> Undefined,</span> -</span>4</span> Undefined</span> </span></span>
Guess.Queue......:</span> 2/4</span> (50.00%</span>)</span></span>
Speed.#1.........:</span>     7321</span> H/s</span> (6.27ms</span>) @ Accel:16 Loops:999 Thr:16 Vec:1</span></span>
Recovered........:</span> 0/1</span> (0.00%</span>) Digests</span></span>
Progress.........:</span> 3844/3844</span> (100.00%</span>)</span></span>
Rejected.........:</span> 0/3844</span> (0.00%</span>)</span></span>
Restore.Point....:</span> 62/62</span> (100.00%</span>)</span></span>
Restore.Sub.#1...:</span> Salt:0</span> Amplifier:61-62</span> Iteration:0-999</span></span>
Candidate.Engine.:</span> Device</span> Generator</span></span>
Candidates.#1....:</span> Xa</span> -</span>></span> XQ</span></span>
</span>
The</span> wordlist</span> or</span> mask</span> that</span> you</span> are</span> using</span> is</span> too</span> small.</span></span>
This</span> means</span> that</span> hashcat</span> cannot</span> use</span> the</span> full</span> parallel</span> power</span> of</span> your</span> device</span>(</span>s</span>)</span>.</span></span>
Unless</span> you</span> supply</span> more</span> work,</span> your</span> cracking</span> speed</span> will</span> drop.</span></span>
For</span> tips</span> on</span> supplying</span> more</span> work,</span> see:</span> https://hashcat.net/faq/morework</span></span>
</span>
Approaching</span> final</span> keyspace</span> -</span> workload</span> adjusted.</span>           </span></span>
</span>
Session..........:</span> hashcat</span>                                </span></span>
Status...........:</span> Exhausted</span></span>
Hash.Mode........:</span> 13600</span> (WinZip</span>)</span></span>
Hash.Target......:</span> $</span>zip2</span>$</span>*</span>0</span>*</span>1</span>*</span>0</span>*</span>abf015b7750c67f9</span>*</span>f8fe</span>*</span>d</span>*</span>c97f48465ca6a9.../zip2</span>$</span></span>
Time.Started.....:</span> Sat</span> Apr</span>  1</span> 15:53:55</span> 2023</span> (6</span> secs</span>)</span></span>
Time.Estimated...:</span> Sat</span> Apr</span>  1</span> 15:54:01</span> 2023</span> (0</span> secs</span>)</span></span>
Kernel.Feature...:</span> Pure</span> Kernel</span></span>
Guess.Mask.......:</span> ?1?1?1</span> [3</span>]</span></span>
Guess.Charset....:</span> -</span>1</span> ?l?u?d,</span> -</span>2</span> Undefined,</span> -</span>3</span> Undefined,</span> -</span>4</span> Undefined</span> </span></span>
Guess.Queue......:</span> 3/4</span> (75.00%</span>)</span></span>
Speed.#1.........:</span>    40570</span> H/s</span> (8.17ms</span>) @ Accel:16 Loops:999 Thr:16 Vec:1</span></span>
Recovered........:</span> 0/1</span> (0.00%</span>) Digests</span></span>
Progress.........:</span> 238328/238328</span> (100.00%</span>)</span></span>
Rejected.........:</span> 0/238328</span> (0.00%</span>)</span></span>
Restore.Point....:</span> 3844/3844</span> (100.00%</span>)</span></span>
Restore.Sub.#1...:</span> Salt:0</span> Amplifier:61-62</span> Iteration:0-999</span></span>
Candidate.Engine.:</span> Device</span> Generator</span></span>
Candidates.#1....:</span> XFz</span> -</span>></span> XQz</span></span>
</span>
$</span>zip2</span>$</span>*</span>0</span>*</span>1</span>*</span>0</span>*</span>abf015b7750c67f9</span>*</span>f8fe</span>*</span>d</span>*</span>c97f48465ca6a99031db50f79a</span>*</span>d9e156b830df03899575</span>*</span>$/zip2$:ab12</span></span>
                                                          </span></span>
Session..........:</span> hashcat</span></span>
Status...........:</span> Cracked</span></span>
Hash.Mode........:</span> 13600</span> (WinZip</span>)</span></span>
Hash.Target......:</span> $</span>zip2</span>$</span>*</span>0</span>*</span>1</span>*</span>0</span>*</span>abf015b7750c67f9</span>*</span>f8fe</span>*</span>d</span>*</span>c97f48465ca6a9.../zip2</span>$</span></span>
Time.Started.....:</span> Sat</span> Apr</span>  1</span> 15:54:04</span> 2023</span> (1</span> sec</span>)</span></span>
Time.Estimated...:</span> Sat</span> Apr</span>  1</span> 15:54:05</span> 2023</span> (0</span> secs</span>)</span></span>
Kernel.Feature...:</span> Pure</span> Kernel</span></span>
Guess.Mask.......:</span> ?1?1?1?1</span> [4</span>]</span></span>
Guess.Charset....:</span> -</span>1</span> ?l?u?d,</span> -</span>2</span> Undefined,</span> -</span>3</span> Undefined,</span> -</span>4</span> Undefined</span> </span></span>
Guess.Queue......:</span> 4/4</span> (100.00%</span>)</span></span>
Speed.#1.........:</span>    49139</span> H/s</span> (101.71ms</span>) @ Accel:16 Loops:999 Thr:16 Vec:1</span></span>
Recovered........:</span> 1/1</span> (100.00%</span>) Digests</span></span>
Progress.........:</span> 36864/14776336</span> (0.25%</span>)</span></span>
Rejected.........:</span> 0/36864</span> (0.00%</span>)</span></span>
Restore.Point....:</span> 0/238328</span> (0.00%</span>)</span></span>
Restore.Sub.#1...:</span> Salt:0</span> Amplifier:5-6</span> Iteration:0-999</span></span>
Candidate.Engine.:</span> Device</span> Generator</span></span>
Candidates.#1....:</span> aari</span> -</span>></span> arIS</span></span>
</span>
Started:</span> Sat</span> Apr</span>  1</span> 15:53:42</span> 2023</span></span>
Stopped:</span> Sat</span> Apr</span>  1</span> 15:54:05</span> 2023</span></span>
</span>
real</span> 0m23,617s</span></span>
user</span> 0m6,331s</span></span>
sys</span>  0m10,747s</span></span></code></pre>
That's a lot of logs; each block represents the outcome of the exploration of length-value space.</p>
In any case, it found the password in 23 seconds, two times faster than zip-password-finder</code>!</p>
At peak speed, it processed 49139 H/s</code> which is roughly 10 times faster than zip-password-finder</code> on my configuration.</p>
It does so while still complaining about not being able to fully use the system.</p>
For this reason, I expect the difference between the two tools to be much larger with a larger password.</p>
Especially if you have a real graphics card, the throughput will likely be expressed in MH</code>.</p>
Hashcat is a fantastic tool with great performance; it was however not easy to set up, so it may be reserved for advanced users.</p>
Future work</a>
</h2>
The re-architecting has not solved the scaling issues; it would be interesting to dig deeper and benchmark the code on a machine with a much larger number of cores.</p>
Talking about testing different hardware, it would be great to have results using a CPU with the sha</code> instruction to benefit from proper hardware acceleration.</p>
Being able to speed up the default implementation of the sha1</code> crate would also yield significant gains.</p>
A current issue is the accumulation of technical debt; by contributing the AES info extraction helpers upstream to zip-rs</code>, we would avoid having to maintain a fork forever.</p>
Finally, Hashcat has made a great impression, and I would like to explore it for further inspiration.</p>
Conclusion</a>
</h2>
This article has explained a few performance optimizations that helped make zip-password-finder</code> a bit faster over time.</p>
However, it has also highlighted that the recent efforts have entered a zone of diminishing returns due to a clear bottleneck.</p>
It seems cracking hashes requires dedicated hardware: be it CPUs with particular instructions or even better GPUs.</p>
This has been made somewhat clear by comparing our Rust program to the great Hashcat.</p>
In general, expert tooling tends to perform very well but suffers from an accessibility issue.</p>
The project zip-password-finder</code> is very easy to use but has a long way to go to be competitive.</p>


Support for thread dump in hprof-slurp 0.5.0
2022-10-17T00:00:00+00:00
This short article is an announcement for the 0.5.0</a> release of the hprof-slurp</a> project.</p>
The goal of this project is to build a JVM heap dump analyzer written in Rust specialized for very large heap dumps.</p>
This release introduces the support for reporting the traces of the threads found in the heap dump.</p>
Meaning, users can now see which part of their application was running at the time of the dump!</p>
It is achieved by stitching together the stacktrace and stackframe data into a coherent output at the end of the parsing phase.</p>
It may slightly increase the memory consumption at runtime due to the additional data captured.</p>
There is still room for improvement as important pieces of information are missing, such as the name of the thread and its Java Thread.State</a>.</p>
However, the addition of this feature is a net improvement over what was previously possible and will hopefully help users during their investigations.</p>
All non-empty threads' stacktraces are reported and their format should be compatible with existing tools, such as the Analyze stacktrace</code> feature in IntelliJ IDEA for convenience.</p>
Example</a>
</h2>
I will use the pets.bin</code> dump file introduced in a previous benchmarking article</a>.</p>
This is a 34Gb heap dump taken while hammering a running instance of Spring's REST petclinic</a>.</p>
The thread dump feature is on by default, the analysis of pets.bin</code> outputs a total of 224 stacktraces.</p>
Below is the longest I could find for fun.</p>
Thread 222</span></span>
  at jdk.internal.misc.Unsafe.park (Unsafe.java:native method)</span></span>
  at java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:252)</span></span>
  at java.util.concurrent.SynchronousQueue$TransferQueue.transfer (SynchronousQueue.java:704)</span></span>
  at java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:903)</span></span>
  at com.zaxxer.hikari.util.ConcurrentBag.borrow (ConcurrentBag.java:151)</span></span>
  at com.zaxxer.hikari.pool.HikariPool.getConnection (HikariPool.java:180)</span></span>
  at com.zaxxer.hikari.pool.HikariPool.getConnection (HikariPool.java:162)</span></span>
  at com.zaxxer.hikari.HikariDataSource.getConnection (HikariDataSource.java:128)</span></span>
  at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection (DatasourceConnectionProviderImpl.java:122)</span></span>
  at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection (NonContextualJdbcConnectionAccess.java:38)</span></span>
  at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded (LogicalConnectionManagedImpl.java:108)</span></span>
  at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection (LogicalConnectionManagedImpl.java:138)</span></span>
  at org.hibernate.internal.SessionImpl.connection (SessionImpl.java:520)</span></span>
  at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction (HibernateJpaDialect.java:152)</span></span>
  at org.springframework.orm.jpa.JpaTransactionManager.doBegin (JpaTransactionManager.java:421)</span></span>
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction (AbstractPlatformTransactionManager.java:400)</span></span>
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction (AbstractPlatformTransactionManager.java:373)</span></span>
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary (TransactionAspectSupport.java:595)</span></span>
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction (TransactionAspectSupport.java:382)</span></span>
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke (TransactionInterceptor.java:119)</span></span>
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:186)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed (CglibAopProxy.java:753)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept (CglibAopProxy.java:698)</span></span>
  at org.springframework.samples.petclinic.service.ClinicServiceImpl$$EnhancerBySpringCGLIB$$a06977e9.findAllVisits (<generated>:unknown line number)</span></span>
  at org.springframework.samples.petclinic.rest.controller.VisitRestController.listVisits (VisitRestController.java:60)</span></span>
  at org.springframework.samples.petclinic.rest.controller.VisitRestController$$FastClassBySpringCGLIB$$8641bcb0.invoke (<generated>:unknown line number)</span></span>
  at org.springframework.cglib.proxy.MethodProxy.invoke (MethodProxy.java:218)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint (CglibAopProxy.java:783)</span></span>
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:163)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed (CglibAopProxy.java:753)</span></span>
  at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke (MethodValidationInterceptor.java:123)</span></span>
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:186)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed (CglibAopProxy.java:753)</span></span>
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept (CglibAopProxy.java:698)</span></span>
  at org.springframework.samples.petclinic.rest.controller.VisitRestController$$EnhancerBySpringCGLIB$$2bf2b676.listVisits (<generated>:unknown line number)</span></span>
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:native method)</span></span>
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)</span></span>
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)</span></span>
  at java.lang.reflect.Method.invoke (Method.java:568)</span></span>
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke (InvocableHandlerMethod.java:205)</span></span>
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest (InvocableHandlerMethod.java:150)</span></span>
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle (ServletInvocableHandlerMethod.java:117)</span></span>
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod (RequestMappingHandlerAdapter.java:895)</span></span>
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal (RequestMappingHandlerAdapter.java:808)</span></span>
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle (AbstractHandlerMethodAdapter.java:87)</span></span>
  at org.springframework.web.servlet.DispatcherServlet.doDispatch (DispatcherServlet.java:1067)</span></span>
  at org.springframework.web.servlet.DispatcherServlet.doService (DispatcherServlet.java:963)</span></span>
  at org.springframework.web.servlet.FrameworkServlet.processRequest (FrameworkServlet.java:1006)</span></span>
  at org.springframework.web.servlet.FrameworkServlet.doGet (FrameworkServlet.java:898)</span></span>
  at javax.servlet.http.HttpServlet.service (HttpServlet.java:655)</span></span>
  at org.springframework.web.servlet.FrameworkServlet.service (FrameworkServlet.java:883)</span></span>
  at javax.servlet.http.HttpServlet.service (HttpServlet.java:764)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:227)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.apache.tomcat.websocket.server.WsFilter.doFilter (WsFilter.java:53)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:327)</span></span>
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke (FilterSecurityInterceptor.java:115)</span></span>
  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter (FilterSecurityInterceptor.java:81)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter (ExceptionTranslationFilter.java:122)</span></span>
  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter (ExceptionTranslationFilter.java:116)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.session.SessionManagementFilter.doFilter (SessionManagementFilter.java:126)</span></span>
  at org.springframework.security.web.session.SessionManagementFilter.doFilter (SessionManagementFilter.java:81)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter (AnonymousAuthenticationFilter.java:109)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter (SecurityContextHolderAwareRequestFilter.java:149)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter (RequestCacheAwareFilter.java:63)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter (LogoutFilter.java:103)</span></span>
  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter (LogoutFilter.java:89)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter (HeaderWriterFilter.java:90)</span></span>
  at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal (HeaderWriterFilter.java:75)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter (SecurityContextPersistenceFilter.java:110)</span></span>
  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter (SecurityContextPersistenceFilter.java:80)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal (WebAsyncManagerIntegrationFilter.java:55)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:336)</span></span>
  at org.springframework.security.web.FilterChainProxy.doFilterInternal (FilterChainProxy.java:211)</span></span>
  at org.springframework.security.web.FilterChainProxy.doFilter (FilterChainProxy.java:183)</span></span>
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate (DelegatingFilterProxy.java:354)</span></span>
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter (DelegatingFilterProxy.java:267)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal (RequestContextFilter.java:100)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.springframework.web.filter.FormContentFilter.doFilterInternal (FormContentFilter.java:93)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal (WebMvcMetricsFilter.java:96)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal (CharacterEncodingFilter.java:201)</span></span>
  at org.springframework.web.filter.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:117)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:189)</span></span>
  at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:162)</span></span>
  at org.apache.catalina.core.StandardWrapperValve.invoke (StandardWrapperValve.java:197)</span></span>
  at org.apache.catalina.core.StandardContextValve.invoke (StandardContextValve.java:97)</span></span>
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke (AuthenticatorBase.java:540)</span></span>
  at org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java:135)</span></span>
  at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:92)</span></span>
  at org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.java:78)</span></span>
  at org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java:357)</span></span>
  at org.apache.coyote.http11.Http11Processor.service (Http11Processor.java:382)</span></span>
  at org.apache.coyote.AbstractProcessorLight.process (AbstractProcessorLight.java:65)</span></span>
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process (AbstractProtocol.java:895)</span></span>
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun (NioEndpoint.java:1732)</span></span>
  at org.apache.tomcat.util.net.SocketProcessorBase.run (SocketProcessorBase.java:49)</span></span>
  at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1191)</span></span>
  at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:659)</span></span>
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run (TaskThread.java:61)</span></span>
  at java.lang.Thread.run (Thread.java:833)</span></span></code></pre>


Brute forcing protected ZIP archives in Rust
2022-10-03T00:00:00+00:00
This article explains how to brute force the password of protected ZIP archives using Rust. It is primarily targeted at beginner Rust developers, but it will surely be interesting to a wider audience.</p>
The full code with better error handling and proper command line arguments (CLI) is available at zip-password-finder</a>.</p>
Context</a>
</h2>
A while back, I found myself in possession of a ZIP archive containing family data that I could not access.
The archive was protected by a password which no one knew.</p>
After a short investigation, I found several tools advertised as being capable of recovering passwords for various types of compressed archives.
However, most of them looked rather fishy or required a license, which made me rather skeptical.</p>
It is at that point that I reached the conclusion that building such a tool myself would be a good learning opportunity.</p>
I might be able to recover the data and it could be the subject of a blog article!</p>
Architecture</a>
</h2>
We can picture our future program as having two main parts.</p>
The first part is about generating candidate passwords. This could be done by either loading passwords from a dictionary or by simply generating all possible words ourselves.</p>
The second part is then responsible for testing those candidates against the archive, one by one until the password is found or no more candidates are available.</p>
Testing millions of passwords using a brute force strategy sounds like something that is CPU bound. Therefore, we should structure our program so that the second part can scale nicely with the number of cores on the host machine.</p>
A common approach for this type of problem is to use a set of workers to process tasks from a shared queue.</p>

    


    
    
</figure>
In practice, we will orchestrate the password generator and workers as threads communicating via a channel.</p>
Channels are a type of unidirectional asynchronous communication between threads that allows information to flow between the Sender</code> and the Receiver</code>.</p>
There are several flavors of channels. In our case, we need to support a single producer and multiple consumers, a.k.a. SPMC (but MPMC will do as well).</p>
Dictionary</a>
</h2>
In order to keep the scope small for the beginning, we will start by retrieving passwords from an existing dictionary file.</p>
The following file Passwords/xato-net-10-million-passwords.txt</a> found on GitHub looks like a perfect candidate.</p>
Let's fetch it and have a peek at its content:</p>
wget</span> https://raw.githubusercontent.com/danielmiessler/SecLists/master/Passwords/xato-net-10-million-passwords.txt</span></span></code></pre>ls</span> -</span>lah</span> xato-net-10-million-passwords.txt</span> |</span> awk</span> -</span>F</span> "</span> "</span> {</span>'</span>print $5</span>'</span>}</span> </span></span>
47M</span></span></code></pre>wc</span> -</span>l</span> xato-net-10-million-passwords.txt</span> </span></span>
5189454</span> xato-net-10-million-passwords.txt</span></span></code></pre>head</span> -</span>10</span> xato-net-10-million-passwords.txt</span> </span></span>
123456</span></span>
password</span></span>
12345678</span></span>
qwerty</span></span>
123456789</span></span>
12345</span></span>
1234</span></span>
111111</span></span>
1234567</span></span>
dragon</span></span></code></pre>
It actually contains only 5.189.454 passwords but that should be more than enough for our testing use case. We can start getting our hands dirty.</p>
cargo</span> new</span> zip-password-finder</span></span></code></pre>
Our program will have a thread reading the content of the password dictionary and pushing each line into a shared channel with the workers.</p>
There will be millions of password candidates. We need something very fast.</p>
We will use the crossbeam-channel</a> crate for our channels because they are far superior to the channels available in std::sync::mpsc</code> in every way.</p>
Those better channels might even make it to the standard library in the future</a>.</p>
cd</span> zip-password-finder/</span></span>
cargo</span> add</span> crossbeam-channel</span></span></code></pre>
Here is our reader:</p>
use</span> crossbeam_channel</span>::</span>Sender</span>;</span></span>
use</span> std</span>::</span>fs</span>::</span>File</span>;</span></span>
use</span> std</span>::</span>io</span>::</span>{</span>BufRead</span>,</span> BufReader</span>}</span>;</span></span>
use</span> std</span>::</span>path</span>::</span>PathBuf</span>;</span></span>
use</span> std</span>::</span>thread</span>;</span></span>
use</span> std</span>::</span>thread</span>::</span>JoinHandle</span>;</span></span>
</span>
pub</span> fn</span> start_password_reader</span>(</span>file_path</span>:</span> PathBuf</span>,</span> send_password</span>:</span> Sender</span><</span>String</span>></span>)</span> -></span> JoinHandle</span><</span>(</span>)</span>></span> {</span></span>
    thread</span>::</span>Builder</span>::</span>new</span>(</span>)</span></span>
        .</span>name</span>(</span>"</span>password-reader</span>"</span>.</span>to_string</span>(</span>)</span>)</span></span>
        .</span>spawn</span>(</span>move</span> |</span>|</span> {</span></span>
            let</span> file</span> =</span> File</span>::</span>open</span>(</span>file_path</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
            let</span> reader</span> =</span> BufReader</span>::</span>new</span>(</span>file</span>)</span>;</span></span>
            for</span> line</span> in</span> reader</span>.</span>lines</span>(</span>)</span> {</span></span>
                match</span> send_password</span>.</span>send</span>(</span>line</span>.</span>unwrap</span>(</span>)</span>)</span> {</span></span>
                    Ok</span>(</span>_</span>)</span> =></span> {</span>}</span></span>
                    Err</span>(</span>_</span>)</span> =></span> break</span>,</span> //</span> channel disconnected, stop thread</span></span>
                }</span></span>
            }</span></span>
        }</span>)</span></span>
        .</span>unwrap</span>(</span>)</span></span>
}</span></span></code></pre>
There are a few interesting things to note here:</p>

It is a good practice to give names to threads for better error reporting and performance observability.</li>
The function returns a JoinHandle</code> for the caller, our main thread, to wait on.</li>
The entire file is read and pushed into the channel. This might be a problem for larger files.</li>
</ul>
Password checker worker</a>
</h2>
We will be using the zip-rs</a> crate for testing the password candidates pulled from the channel.</p>
cargo</span> add</span> zip</span></span></code></pre>
It supports both ZipCrypto</a> and AES</a> encrypted zip archives.</p>
This is the first version of the worker thread:</p>
use</span> crossbeam_channel</span>::</span>Sender</span>;</span></span>
use</span> std</span>::</span>fs</span>::</span>File</span>;</span></span>
use</span> std</span>::</span>io</span>::</span>{</span>BufRead</span>,</span> BufReader</span>}</span>;</span></span>
use</span> std</span>::</span>path</span>::</span>PathBuf</span>;</span></span>
use</span> std</span>::</span>thread</span>;</span></span>
use</span> std</span>::</span>thread</span>::</span>JoinHandle</span>;</span></span>
</span>
pub</span> fn</span> password_checker</span>(</span></span>
    index</span>:</span> usize</span>,</span></span>
    file_path</span>:</span> &</span>Path</span>,</span></span>
    receive_password</span>:</span> Receiver</span><</span>String</span>></span>,</span></span>
)</span> -></span> JoinHandle</span><</span>(</span>)</span>></span> {</span></span>
    let</span> file</span> =</span> fs</span>::</span>File</span>::</span>open</span>(</span>file_path</span>)</span>.</span>expect</span>(</span>"</span>File should exist</span>"</span>)</span>;</span></span>
    thread</span>::</span>Builder</span>::</span>new</span>(</span>)</span></span>
        .</span>name</span>(</span>format!</span>(</span>"</span>worker-</span>{</span>}</span>"</span>,</span> index</span>)</span>)</span></span>
        .</span>spawn</span>(</span>move</span> |</span>|</span> {</span></span>
            let</span> mut</span> archive</span> =</span> zip</span>::</span>ZipArchive</span>::</span>new</span>(</span>file</span>)</span>.</span>expect</span>(</span>"</span>Archive validated before-hand</span>"</span>)</span>;</span></span>
            loop</span> {</span></span>
                match</span> receive_password</span>.</span>recv</span>(</span>)</span> {</span></span>
                    Err</span>(</span>_</span>)</span> =></span> break</span>,</span> //</span> channel disconnected, stop thread</span></span>
                    Ok</span>(</span>password</span>)</span> =></span> {</span></span>
                        let</span> res</span> =</span> archive</span>.</span>by_index_decrypt</span>(</span>0</span>,</span> password</span>.</span>as_bytes</span>(</span>)</span>)</span>;</span> //</span> decrypt first file in archive</span></span>
                        match</span> res</span> {</span></span>
                            Err</span>(</span>e</span>)</span> =></span> panic!</span>(</span>"</span>Unexpected error </span>{</span>:?</span>}</span>"</span>,</span> e</span>)</span>,</span></span>
                            Ok</span>(</span>Err</span>(</span>_</span>)</span>)</span> =></span> (</span>)</span>,</span> //</span> invalid password - continue</span></span>
                            Ok</span>(</span>Ok</span>(</span>_</span>)</span>)</span> =></span> {</span></span>
                                println!</span>(</span>"</span>Password found:</span>{</span>}</span>"</span>,</span> password</span>)</span>;</span></span>
                                break</span>;</span> //</span> stop thread</span></span>
                            }</span></span>
                        }</span></span>
                    }</span></span>
                }</span></span>
            }</span></span>
        }</span>)</span></span>
        .</span>unwrap</span>(</span>)</span></span>
}</span></span></code></pre>
This straightforward approach is unfortunately not enough as it yields false positive passwords from time to time.</p>
The Rustdoc for by_index_decrypt</code> actually warns about it.</p>


  
  
  
    
      Warning</strong>
  </p>
  
    This function sometimes accepts wrong passwords.
This is because the ZIP spec only allows us to check for a 1/256 chance that the password is correct.
There are many passwords out there that will also pass the validity checks we are able to perform.
This is a weakness of the ZipCrypto algorithm, due to its fairly primitive approach to cryptography.</p>

  </div>
  
</blockquote>

We can work around those collisions by performing a full read of the archive with the password to make sure it is actually correct.</p>
 use crossbeam_channel::{Receiver, Sender};</span></span>
 use std::fs::File;</span></span>
-</span>use std::io::{BufRead, BufReader};</span></span>
+</span>use std::io::{BufRead, BufReader, Read};</span></span>
 use std::path::{Path, PathBuf};</span></span>
 use std::thread::JoinHandle;</span></span>
 use std::{fs, thread};</span></span>
@@</span> -38,10 +38,14</span> @@</span> pub fn password_checker(</span></span>
        let res = archive.by_index_decrypt(0, password.as_bytes()); // decrypt first file in archive</span></span>
        match res {</span></span>
            Err(e) => panic!("Unexpected error {:?}", e),</span></span>
            Ok(Err(_)) => (), // invalid password - continue</span></span>
-</span>           Ok(Ok(_)) => {</span></span>
-</span>               println!("Password found:{}", password);</span></span>
-</span>               break; // stop thread</span></span>
+</span>           Ok(Err(_)) => (), // invalid password</span></span>
+</span>           Ok(Ok(mut zip)) => {</span></span>
+</span>               // Validate password by reading the zip file to make sure it is not merely a hash collision.</span></span>
+</span>               let mut buffer = Vec::with_capacity(zip.size() as usize);</span></span>
+</span>               match zip.read_to_end(&mut buffer) {</span></span>
+</span>                   Err(_) => (), // password collision - continue</span></span>
+</span>                   Ok(_) => {</span></span>
+</span>                       println!("Password found:{}", password);</span></span>
+</span>                       break; // stop thread</span></span>
+</span>                   }</span></span>
+</span>               }</span></span>
            }</span></span>
        }</span></span>
    }</span></span></code></pre>
Again, a few interesting things to note here:</p>

Each worker thread gets a unique name.</li>
The worker is looping</code> until the password is found or the channel is closed.</li>
Each worker opens the Zip file in memory to avoid extra coordination.</li>
The password found is printed to the console.</li>
</ul>
Putting it together</a>
</h2>
We can now wire things up together nicely and get one step closer to our goal.</p>
pub</span> fn</span> password_finder</span>(</span>zip_path</span>:</span> &</span>str</span>,</span> password_list_path</span>:</span> &</span>str</span>,</span> workers</span>:</span> usize</span>)</span> {</span></span>
    let</span> zip_file_path</span> =</span> Path</span>::</span>new</span>(</span>zip_path</span>)</span>;</span></span>
    let</span> password_list_file_path</span> =</span> Path</span>::</span>new</span>(</span>password_list_path</span>)</span>.</span>to_path_buf</span>(</span>)</span>;</span></span>
</span>
    //</span> channel with backpressure</span></span>
    let</span> (</span>send_password</span>,</span> receive_password</span>)</span>:</span> (</span>Sender</span><</span>String</span>></span>,</span> Receiver</span><</span>String</span>></span>)</span> =</span></span>
        crossbeam_channel</span>::</span>bounded</span>(</span>workers</span> *</span> 10_000</span>)</span>;</span></span>
</span>
    //</span> thread handle for password reader</span></span>
    let</span> password_gen_handle</span> =</span> start_password_reader</span>(</span>password_list_file_path</span>,</span> send_password</span>)</span>;</span></span>
</span>
    //</span> save all workers handles</span></span>
    let</span> mut</span> worker_handles</span> =</span> Vec</span>::</span>with_capacity</span>(</span>workers</span>)</span>;</span></span>
    for</span> i</span> in</span> 1</span>..=</span>workers</span> {</span></span>
        let</span> join_handle</span> =</span> password_checker</span>(</span>i</span>,</span> zip_file_path</span>,</span> receive_password</span>.</span>clone</span>(</span>)</span>)</span>;</span></span>
        worker_handles</span>.</span>push</span>(</span>join_handle</span>)</span>;</span></span>
    }</span></span>
</span>
    //</span> wait for workers to finish</span></span>
    for</span> h</span> in</span> worker_handles</span> {</span></span>
        h</span>.</span>join</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
    }</span></span>
</span>
    //</span> wait for the end of the file</span></span>
    password_gen_handle</span>.</span>join</span>(</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
}</span></span></code></pre>
The first important point here is the use of a bounded channel between the password generator and the workers.</p>
A bounded channel will block the sender once its capacity is reached. This is handy to implement back-pressure.</p>
In our case, the producer is much faster than the consumers, so we are at risk of loading the entire dictionary inside the channel, which is not great for memory usage.</p>
To avoid slowing down the consumers, the capacity is sized according to the number of workers.</p>
Secondly, we are saving all JoinHandle</code> to await the termination of all threads.</p>
And finally, the main function!</p>
fn</span> main</span>(</span>)</span> {</span></span>
    let</span> zip_path</span> =</span> env</span>::</span>args</span>(</span>)</span>.</span>nth</span>(</span>1</span>)</span>.</span>unwrap</span>(</span>)</span>;</span></span>
    let</span> dictionary_path</span> =</span> "</span>/opt/xato-net-10-million-passwords.txt</span>"</span>;</span></span>
    let</span> workers</span> =</span> 3</span>;</span></span>
    password_finder</span>(</span>&</span>zip_path</span>,</span> dictionary_path</span>,</span> workers</span>)</span>;</span></span>
}</span></span></code></pre>
As a command line argument our program accepts the path to an encrypted Zip file, which will be brute forced using a dictionary with 3 workers.</p>
The dictionary path is hard-coded for the time being.</p>
For testing purposes, we need an archive encrypted using a password from the dictionary.</p>
I am picking the entry number 1.000.000, vaanes</code>, to create our encrypted test zip.</p>
sed</span> -</span>ne</span> 1000000p</span> xato-net-10-million-passwords.txt</span></span>
vaanes</span></span></code></pre>
Here are the details of the test archive created using this password.</p>
zipinfo</span> encrypted-test-dict.zip</span></span>
</span>
Archive:</span>  encrypted-test-dict.zip</span></span>
Zip</span> file</span> size:</span> 202</span> bytes,</span> number</span> of</span> entries:</span> 1</span></span>
-rw-rw-r--</span>  6.3</span> unx</span>        9</span> Bx</span> u099</span> 22-Sep-24</span> 12:19</span> test-xato</span></span>
1</span> file,</span> 9</span> bytes</span> uncompressed,</span> 16</span> bytes</span> compressed:</span>  -</span>77.8%</span></span></code></pre>
It is pretty small and uses compression as a storage method.</p>
Let's run our program!</p>
cd</span> /target/release</span></span>
time</span> ./zip-password-finder encrypted-test-dict.zip</span></span></code></pre>
Checking on the CPU usage with htop</code> yields an expected result.</p>

    


    
    
</figure>
Three worker threads are going at full speed, and one thread is reading the passwords at a leisurely pace.</p>
After 4 minutes, the program prints on the terminal.</p>
Password found:vaanes</span></span></code></pre>
But the program does not terminate right away; it takes more than 20 minutes to finish!</p>
real</span>    23m36,203s</span></span>
user</span>    72m29,567s</span></span>
sys</span>     0m41,570s</span></span></code></pre>
First the good news: the program found the password!</p>
It processed 1 million passwords in around four minutes using 3 workers which gives us:</p>
1.000.000 passwords tested / (4 * 60 seconds) / 3 workers =</span></span>
1389 passwords tested per second per worker assuming linear scaling</span></span></code></pre>
This result provides us with an idea of what can be achieved with this approach.</p>
Secondly, the bad news: our program does not terminate properly, which is a big issue.</p>
Graceful shutdown</a>
</h2>
Looking at the code more closely, we can see that we terminate our threads only under specific conditions.</p>
For the password dictionary thread:</p>

If the channel has no consumers attached.</li>
If we reach the end of the dictionary.</li>
</ul>
And for the worker threads:</p>

If the channel is empty and has no producer attached.</li>
If we find the password.</li>
</ul>
This explains why our program kept on running previously; we only stopped the thread that found the password!</p>
The program went through the rest of the dictionary using only two workers.</p>
There are different ways to solve this problem. We will try to come up with an explicit</strong> shutdown sequence that does not rely on timeout nor on the channel being disconnected.</p>
We need a way for the worker thread that finds the password to be able to inform other threads that they can shut down.</p>
First, let's have worker threads communicate the password to the main thread instead of simply printing it out in the terminal.</p>
This design is much better and will enable unit testing later on!</p>
As a follow-up, once the main thread receives the password, it will notify all threads to stop using a thread-safe signal.</p>
 use std::path::{Path, PathBuf};</span></span>
+</span>use std::sync::atomic::{AtomicBool, Ordering};</span></span>
+</span>use std::sync::Arc;</span></span>
 use std::thread::JoinHandle;</span></span>
 use std::{env, fs, thread};</span></span>
 </span></span>
-</span>pub fn start_password_reader(file_path: PathBuf, send_password: Sender<String>) -> JoinHandle<()> {</span></span>
+</span>pub fn start_password_reader(</span></span>
+</span>    file_path: PathBuf,</span></span>
+</span>    send_password: Sender<String>,</span></span>
+</span>    stop_signal: Arc<AtomicBool>,</span></span>
+</span>) -> JoinHandle<()> {</span></span>
     thread::Builder::new()</span></span>
         .name("password-reader".to_string())</span></span>
         .spawn(move || {</span></span>
             let file = File::open(file_path).unwrap();</span></span>
             let reader = BufReader::new(file);</span></span>
             for line in reader.lines() {</span></span>
-</span>                match send_password.send(line.unwrap()) {</span></span>
-</span>                    Ok(_) => {}</span></span>
-</span>                    Err(_) => break, // channel disconnected, stop thread</span></span>
+</span>                if stop_signal.load(Ordering::Relaxed) {</span></span>
+</span>                    break;</span></span>
+</span>                } else {</span></span>
+</span>                    match send_password.send(line.unwrap()) {</span></span>
+</span>                        Ok(_) => {}</span></span>
+</span>                        Err(_) => break, // channel disconnected, stop thread</span></span>
+</span>                    }</span></span>
                 }</span></span>
             }</span></span>
         })</span></span>
@@</span> -25,13 +35,15</span> @@</span> pub fn password_checker(</span></span>
     index: usize,</span></span>
     file_path: &Path,</span></span>
     receive_password: Receiver<String>,</span></span>
+</span>    stop_signal: Arc<AtomicBool>,</span></span>
+</span>    send_password_found: Sender<String>,</span></span>
 ) -> JoinHandle<()> {</span></span>
     let file = fs::File::open(file_path).expect("File should exist");</span></span>
     thread::Builder::new()</span></span>
         .name(format!("worker-{}", index))</span></span>
         .spawn(move || {</span></span>
             let mut archive = zip::ZipArchive::new(file).expect("Archive validated before-hand");</span></span>
-</span>            loop {</span></span>
+</span>            while !stop_signal.load(Ordering::Relaxed) {</span></span>
                 match receive_password.recv() {</span></span>
                     Err(_) => break, // channel disconnected, stop thread</span></span>
                     Ok(password) => {</span></span>
@@</span> -44,7 +56,12</span> @@</span> pub fn password_checker(</span></span>
                                 let mut buffer = Vec::with_capacity(zip.size() as usize);</span></span>
                                 match zip.read_to_end(&mut buffer) {</span></span>
                                     Err(_) => (), // password collision - continue</span></span>
-</span>                                    Ok(_) => {</span></span>
-</span>                                        println!("Password found:{}", password);</span></span>
-</span>                                        break; // stop thread</span></span>
+</span>                                    Ok(_) => {</span></span>
+</span>                                        // Send password and continue processing while waiting for signal</span></span>
+</span>                                        send_password_found</span></span>
+</span>                                            .send(password)</span></span>
+</span>                                            .expect("Send found password should not fail");</span></span>
+</span>                                    }</span></span>
                                 }</span></span>
                             }</span></span>
                         }</span></span>
@@</span> -55,7 +72,7</span> @@</span> pub fn password_checker(</span></span>
         .unwrap()</span></span>
 }</span></span></code></pre>
The threads can exit their looping state by probing a stop signal materialized by an Arc<AtomicBool></code> which is safe and cheap to operate.</p>
Even the worker thread which finds the password will simply wait for the signal, this way we avoid having to handle the special case where there is only a single worker.</p>
And for the wiring and actual graceful shutdown.</p>
-</span>pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize) {</span></span>
+</span>pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize) -> Option<String> {</span></span>
     let zip_file_path = Path::new(zip_path);</span></span>
     let password_list_file_path = Path::new(password_list_path).to_path_buf();</span></span>
 </span></span>
@@</span> -63,28 +80,59</span> @@</span> pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize)</span></span>
     let (send_password, receive_password): (Sender<String>, Receiver<String>) =</span></span>
         crossbeam_channel::bounded(workers * 10_000);</span></span>
 </span></span>
+</span>    let (send_found_password, receive_found_password): (Sender<String>, Receiver<String>) =</span></span>
+</span>        crossbeam_channel::bounded(1);</span></span>
+</span></span>
+</span>    // stop signals to shut down threads</span></span>
+</span>    let stop_workers_signal = Arc::new(AtomicBool::new(false));</span></span>
+</span>    let stop_gen_signal = Arc::new(AtomicBool::new(false));</span></span>
+</span></span>
     // thread handle for password reader</span></span>
-</span>    let password_gen_handle = start_password_reader(password_list_file_path, send_password);</span></span>
+</span>    let password_gen_handle = start_password_reader(</span></span>
+</span>        password_list_file_path,</span></span>
+</span>        send_password,</span></span>
+</span>        stop_gen_signal.clone(),</span></span>
+</span>    );</span></span>
 </span></span>
     // save all workers handles</span></span>
     let mut worker_handles = Vec::with_capacity(workers);</span></span>
     for i in 1..=workers {</span></span>
-</span>        let join_handle = password_checker(i, zip_file_path, receive_password.clone());</span></span>
+</span>        let join_handle = password_checker(</span></span>
+</span>            i,</span></span>
+</span>            zip_file_path,</span></span>
+</span>            receive_password.clone(),</span></span>
+</span>            stop_workers_signal.clone(),</span></span>
+</span>            send_found_password.clone(),</span></span>
+</span>        );</span></span>
         worker_handles.push(join_handle);</span></span>
     }</span></span>
 </span></span>
-</span>    // wait for workers to finish</span></span>
-</span>    for h in worker_handles {</span></span>
-</span>        h.join().unwrap();</span></span>
-</span>    }</span></span>
+</span>    // drop reference in `main` so that it disappears completely with workers for a clean shutdown</span></span>
+</span>    drop(send_found_password);</span></span>
 </span></span>
-</span>    // wait for the end of the file</span></span>
-</span>    password_gen_handle.join().unwrap();</span></span>
+</span>    match receive_found_password.recv() {</span></span>
+</span>        Ok(password_found) => {</span></span>
+</span>            // stop generating values first to avoid deadlock on channel</span></span>
+</span>            stop_gen_signal.store(true, Ordering::Relaxed);</span></span>
+</span>            password_gen_handle.join().unwrap();</span></span>
+</span>            // stop workers</span></span>
+</span>            stop_workers_signal.store(true, Ordering::Relaxed);</span></span>
+</span>            for h in worker_handles {</span></span>
+</span>                h.join().unwrap();</span></span>
+</span>            }</span></span>
+</span>            Some(password_found)</span></span>
+</span>        }</span></span>
+</span>        Err(_) => None,</span></span>
+</span>    }</span></span>
 }</span></span>
 </span></span>
 fn main() {</span></span>
     let zip_path = env::args().nth(1).unwrap();</span></span>
     let dictionary_path = "/home/agourlay/Workspace/blog-data/find-password-zip/xato-net-10-million-passwords.txt";</span></span>
     let workers = 3;</span></span>
-</span>    password_finder(&zip_path, dictionary_path, workers);</span></span>
+</span></span>
+</span>    match password_finder(&zip_path, dictionary_path, workers) {</span></span>
+</span>        Some(password_found) => println!("Password found:{}", password_found),</span></span>
+</span>        None => println!("No password found :("),</span></span>
+</span>    }</span></span>
 }</span></span>
</span></code></pre>
The shutdown procedure must be carefully handled to avoid a deadlock of our threads.</p>
Using channels can make our threads block forever if the channel is not disconnected:</p>

A consuming thread blocks on an empty channel.</li>
A producing thread blocks on a full bounded channel.</li>
</ul>
This makes the task more difficult given that signals are efficient only if the threads are able to run their inner loop.</p>
Given that we are using a bounded channel, it makes sense to shut down the producer first.</p>
In addition, it is important to manually drop send_found_password</code> to shut down properly when no password is found:</p>

The password generator thread exits at the end of the dictionary.</li>
All workers exit when the channel is disconnected.</li>
All workers drop their clone of Arc<send_found_password></code>.</li>
Deadlock occurs: receive_found_password.recv()</code> won't be triggered unless all producers are gone but</strong> the main thread still holds the initial Arc<send_found_password></code>.</li>
</ol>
This gives us a valid shutdown sequence:</p>

Drop the initial Arc<send_found_password></code>.</li>
Wait for the found password from a worker.</li>
Toggle shutdown signal password generator.</li>
Join the password generator handle.</li>
Toggle shutdown signal for workers.</li>
Join all the worker handles.</li>
</ol>
Finding a correct sequence can be a bit subtle, so it is better to test it.</p>
An alternative solution would be to use the timeout-flavored API recv_timeout</code> and send_timeout</code> which enable you to block only for a specific duration.</p>
I have decided to not use those as they are not strictly required, and it forces us to better understand our shutdown sequence :)</p>
Let's give it a try!</p>
cd</span> /target/release</span></span>
time</span> ./zip-password-finder encrypted-test-dict.zip</span></span>
Password</span> found:vaanes</span></span>
</span>
real</span>  4m23,968s</span></span>
user</span>  13m30,767s</span></span>
sys</span>   0m7,130s</span></span></code></pre>
This is much better; the application shuts down as soon as the password is found!</p>
But something important is still missing. As a user, we have no feedback while the program is waiting.</p>
It would be great to see how much progress the program is making and have an idea of the estimated run time.</p>
Progress bar</a>
</h2>
The easiest way to visualize the progress of our program is to set up a progress bar.</p>
One of the best crates for this is Indicatif</a>.</p>
cargo</span> add</span> indicatif</span></span></code></pre>
The progress bar needs to be configured according to our use case.</p>
</span>
     let zip_file_path = Path::new(zip_path);</span></span>
     let password_list_file_path = Path::new(password_list_path).to_path_buf();</span></span>
 </span></span>
+</span>    // Setup progress bar</span></span>
+</span>    let progress_bar = ProgressBar::new(0);</span></span>
+</span>    let progress_style = ProgressStyle::default_bar()</span></span>
+</span>        .template("[{elapsed_precise}] {wide_bar} {pos}/{len} throughput:{per_sec} (eta:{eta})")</span></span>
+</span>        .expect("Failed to create progress style");</span></span>
+</span>    progress_bar.set_style(progress_style);</span></span>
+</span></span>
+</span>    // Refresh terminal 2 times per second to avoid flickering effect</span></span>
+</span>    let draw_target = ProgressDrawTarget::stdout_with_hz(2);</span></span>
+</span>    progress_bar.set_draw_target(draw_target);</span></span>
+</span></span>
+</span>    // Size progress bar according to dictionary size</span></span>
+</span>    let file =</span></span>
+</span>        BufReader::new(File::open(password_list_file_path.clone()).expect("Unable to open file"));</span></span>
+</span>    let total_password_count = file.lines().count();</span></span>
+</span>    progress_bar.set_length(total_password_count as u64);</span></span>
+</span></span>
     // MPMC channel with backpressure</span></span>
     let (send_password, receive_password): (Sender<String>, Receiver<String>) =</span></span>
         crossbeam_channel::bounded(workers * 10_000);</span></span>
@@</span> -103,6 +122,7</span> @@</span> pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize)</span></span>
             receive_password.clone(),</span></span>
             stop_workers_signal.clone(),</span></span>
             send_found_password.clone(),</span></span>
+</span>            progress_bar.clone(),</span></span>
         );</span></span>
         worker_handles.push(join_handle);</span></span>
     }</span></span>
@@</span> -120,6 +140,7</span> @@</span> pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize)</span></span>
             for h in worker_handles {</span></span>
                 h.join().unwrap();</span></span>
             }</span></span>
+</span>            progress_bar.finish_and_clear();</span></span>
             Some(password_found)</span></span>
         }</span></span>
         Err(_) => None,</span></span></code></pre>
The most important points are:</p>

We set up a template to display some useful information regarding processing speed and ETA.</li>
The progress bar refresh rate is fixed to avoid flickering due to the high frequency of updates.</li>
The progress bar length is sized according to the length of the dictionary.</li>
</ul>
The integration is rather straightforward on the workers' side; simply increment the progress bar after each candidate.</p>
@@</span> -37,6 +38,7</span> @@</span> pub fn password_checker(</span></span>
     receive_password: Receiver<String>,</span></span>
     stop_signal: Arc<AtomicBool>,</span></span>
     send_password_found: Sender<String>,</span></span>
+</span>    progress_bar: ProgressBar,</span></span>
 ) -> JoinHandle<()> {</span></span>
     let file = fs::File::open(file_path).expect("File should exist");</span></span>
     thread::Builder::new()</span></span>
@@</span> -65,6 +67,7</span> @@</span> pub fn password_checker(</span></span>
                                 }</span></span>
                             }</span></span>
                         }</span></span>
+</span>                        progress_bar.inc(1);</span></span>
                     }</span></span>
                 }</span></span>
             }</span></span>
</span></code></pre>
Running this new version displays our new fancy progress bar in action!</p>

    


    
    
</figure>
The processing speed and ETA confirm our previous observations.</p>
But can we go any faster?</p>
Scaling up</a>
</h2>
We previously found that the program can test around 1389 passwords per second per worker when used with 3 workers.</p>
What about automatically sizing the number of workers according to the host?</p>
 use std::sync::atomic::{AtomicBool, Ordering};</span></span>
 use std::sync::Arc;</span></span>
-</span>use std::thread::JoinHandle;</span></span>
+</span>use std::thread::{available_parallelism, JoinHandle};</span></span>
 use std::{env, fs, thread};</span></span>
+</span>use std::cmp::max;</span></span>
 </span></span>
 pub fn start_password_reader(</span></span>
     file_path: PathBuf,</span></span>
@@</span> -151,7 +152,8</span> @@</span> fn main() {</span></span>
     let zip_path = env::args().nth(1).unwrap();</span></span>
     let dictionary_path = "/opt/xato-net-10-million-passwords.txt";</span></span>
-</span>    let workers = 3;</span></span>
+</span>    let num_cores = available_parallelism().unwrap().get();</span></span>
+</span>    let workers = max(1,  num_cores - 1);</span></span></code></pre>
With this change, we are using num_cores - 1</code> worker threads and one dictionary reader thread.</p>
On my 8-core machine, it yields the following usage.</p>

    


    
    
</figure>
This is a nice CPU utilization to keep you warm during the winter.</p>
Unfortunately, it does not seem to be much faster.</p>
cd</span> /target/release</span></span>
time</span> ./zip-password-finder encrypted-test-dict.zip</span></span>
Password</span> found:vaanes</span></span>
</span>
real</span>  3m57,826s</span></span>
user</span>  27m52,591s</span></span>
sys</span>   0m9,621s</span></span></code></pre>
That is barely 30 seconds faster than the 3 workers version!</p>
It seems our program is suffering from scalability issues.</p>
To investigate this, let's start by making the number of workers configurable through the CLI.</p>
 fn main() {</span></span>
-</span>    let zip_path = env::args().nth(1).unwrap();</span></span>
+</span>    let mut args_iter = env::args().skip(1);</span></span>
+</span>    let zip_path = args_iter.next().unwrap();</span></span>
     let dictionary_path = "/opt/xato-net-10-million-passwords.txt";</span></span>
-</span>    let num_cores = available_parallelism().unwrap().get();</span></span>
-</span>    let workers = max(1,  num_cores - 1);</span></span>
+</span>    let workers: usize = match args_iter.next() {</span></span>
+</span>        None => {</span></span>
+</span>            let num_cores = available_parallelism().unwrap().get();</span></span>
+</span>            max(1, num_cores - 1)</span></span>
+</span>        }</span></span>
+</span>        Some(count) => count.parse().unwrap(),</span></span>
+</span>    };</span></span></code></pre>
Then let's reach for our favorite CLI benchmarking tool, the fantastic hyperfine</a>.</p>
Using the following magic incantation, we will get reliable measurements for runs with different worker count.</p>
hyperfine</span> -</span>-runs</span> 2</span> \</span></span>
 -</span>-warmup</span> 1</span> \</span></span>
 -</span>-export-markdown</span> workers.md</span> \</span></span>
 -</span>-export-json</span> workers.json</span> \</span></span>
 -</span>n</span> 1</span> "</span>./zip-password-finder encrypted-test-dict.zip 1</span>"</span> \</span></span>
 -</span>n</span> 2</span> "</span>./zip-password-finder encrypted-test-dict.zip 2</span>"</span> \</span></span>
 -</span>n</span> 3</span> "</span>./zip-password-finder encrypted-test-dict.zip 3</span>"</span> \</span></span>
 -</span>n</span> 4</span> "</span>./zip-password-finder encrypted-test-dict.zip 4</span>"</span> \</span></span>
 -</span>n</span> 5</span> "</span>./zip-password-finder encrypted-test-dict.zip 5</span>"</span> \</span></span>
 -</span>n</span> 6</span> "</span>./zip-password-finder encrypted-test-dict.zip 6</span>"</span> \</span></span>
 -</span>n</span> 7</span> "</span>./zip-password-finder encrypted-test-dict.zip 7</span>"</span> \</span></span>
 -</span>n</span> 8</span> "</span>./zip-password-finder encrypted-test-dict.zip 8</span>"</span> </span></span></code></pre>
After the runs, the workers.md</code> file contains the following table:</p>
Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

1</code></td> 559.522 ± 7.091</td> 554.509</td> 564.536</td> 2.51 ± 0.03</td></tr>
2</code></td> 338.590 ± 0.102</td> 338.518</td> 338.662</td> 1.52 ± 0.00</td></tr>
3</code></td> 264.297 ± 0.650</td> 263.838</td> 264.757</td> 1.18 ± 0.00</td></tr>
4</code></td> 223.284 ± 0.047</td> 223.251</td> 223.317</td> 1.00</td></tr>
5</code></td> 224.617 ± 0.082</td> 224.559</td> 224.675</td> 1.01 ± 0.00</td></tr>
6</code></td> 226.690 ± 0.421</td> 226.392</td> 226.987</td> 1.02 ± 0.00</td></tr>
7</code></td> 232.129 ± 0.009</td> 232.122</td> 232.135</td> 1.04 ± 0.00</td></tr>
8</code></td> 234.367 ± 0.118</td> 234.284</td> 234.450</td> 1.05 ± 0.00</td></tr>
</tbody></table>
It looks quite bad. Let's plot it so that visual people can be disappointed as well.</p>

    


    
    
</figure>
One would expect performance to grow linearly, but that is absolutely not the case here.</p>
Not only is the speedup not linear, but the program only gets faster up to 4 workers at approximately 4500 passwords/sec.</p>
Sounds like a good time to mention Amdahl's law</a>, which predicts the theoretical speedup when using multiple processors depending on the portion of the work being parallel.</p>

    


    
    
</figure>
Our workload is supposed</strong> to be very parallelisable, each worker processing candidate passwords in isolation at full speed. We should be getting more impressive speedups.</p>
The simplest explanation is that our program is not as parallel as we thought. There must be some form of hidden coordination preventing our program from scaling properly.</p>
Hyper-threading</a>
</h2>
However, I find it very suspicious that the best performance occurs at 4 workers. This is a nice power of 2.</p>
I am running the benchmarks on a laptop running Linux on an Intel i7-10610U</a> CPU.</p>
Looking at the specification, we can see the following.</p>
lscpu</span></span>
.</span>.</span>.</span></span>
Vendor</span> ID:</span>               GenuineIntel</span></span>
  Model</span> name:</span>            Intel</span>(</span>R</span>)</span> Core</span>(</span>TM</span>)</span> i7-10610U</span> CPU</span> @</span> 1.80GHz</span></span>
    CPU</span> family:</span>          6</span></span>
    Model:</span>               142</span></span>
    Thread(s</span>) per core:  2</span></span>
    Core(s</span>) per socket:  4</span></span></code></pre>
Instead of 8 cores, there are 4 physical cores and 2 logical threads per core!</p>
In my case this is Intel's Hyper-threading Technology</a>, which enables cores to have two lanes of execution. The core can switch between those while waiting for data to be fetched from main memory or caches.</p>
Outside the various marketing performance claims from Intel, it seems that Hyper-threading</code> is not adapted to all types of workload.</p>
Given our very CPU-heavy workload, in which we need the full attention of each core on a single worker. It makes sense to not rely on it and only use the number of physical cores to size the number of workers.</p>
cargo</span> add</span> num_cpus</span></span></code></pre>
The previous benchmark has also shown that the best performance occurs where the number of workers is equal to the number of cores, meaning that the reader thread is very cheap in comparison.</p>
     let workers: usize = match args_iter.next() {</span></span>
-</span>        None => {</span></span>
-</span>            let num_cores = available_parallelism().unwrap().get();</span></span>
-</span>            max(1, num_cores - 1)</span></span>
-</span>        }</span></span>
+</span>        None => num_cpus::get_physical(),</span></span>
         Some(count) => count.parse().unwrap(),</span></span>
     };</span></span>
</span></code></pre>
Hopefully, those workers will be allocated to different cores even if the host supports Hyper-threading</code> for maximum efficiency.</p>
Going faster</a>
</h2>
Even limiting ourselves to physical cores, the scaling from one to four cores was not entirely linear.</p>
For a quick sanity check, we can generate a flamegraph to see where the CPU time is spent.</p>

    


    
    
</figure>
The various threads are nicely visible (full flamegraph</a> available).</p>
It seems the reader thread spends most of its time snoozing, blocked when the channel is full, meaning that it is not a bottleneck.</p>
We can zoom in on one of the workers.</p>

    


    
    
</figure>
It is basically spending most of its time in sha1::compress::soft::compress</code> in a deep call stack from zip-rs</code>.</p>
There is nothing obvious to optimize from our worker code given the decryption API we have to work with.</p>
We can focus on decreasing potential synchronization issues between our worker threads. When can they possibly interfere with each other?</p>
Looking at the code closely, we can see that for every password candidate the workers are accessing:</p>

the receive channel handle</li>
the shutdown signal</li>
the progress bar</li>
</ul>
A first hypothesis is that those actions increase the cost of processing a single candidate and introduce some form of coordination between threads as they are accessing shared memory.</p>
A naive approach to reducing coordination is to give more to the workers to process in isolation by, for instance, batching up a bunch of candidates together in the channel.</p>
What would the optimal batch size be? Let's implement batching and make the batch size configurable in place of the "workers" argument for a quick experiment.</p>
First batching the reader thread:</p>
 pub fn start_password_reader(</span></span>
     file_path: PathBuf,</span></span>
-</span>    send_password: Sender<String>,</span></span>
+</span>    send_password: Sender<Vec<String>>,</span></span>
+</span>    batch_size: usize,</span></span>
     stop_signal: Arc<AtomicBool>,</span></span>
 ) -> JoinHandle<()> {</span></span>
     thread::Builder::new()</span></span>
@@</span> -19,13 +20,18</span> @@</span> pub fn start_password_reader(</span></span>
         .spawn(move || {</span></span>
             let file = File::open(file_path).unwrap();</span></span>
             let reader = BufReader::new(file);</span></span>
+</span>            let mut batch = Vec::with_capacity(batch_size);</span></span>
             for line in reader.lines() {</span></span>
-</span>                if stop_signal.load(Ordering::Relaxed) {</span></span>
-</span>                    break;</span></span>
-</span>                } else {</span></span>
-</span>                    match send_password.send(line.unwrap()) {</span></span>
-</span>                        Ok(_) => {}</span></span>
-</span>                        Err(_) => break, // channel disconnected, stop thread</span></span>
+</span>                batch.push(line.unwrap());</span></span>
+</span>                // push once the batch is full</span></span>
+</span>                if batch.len() == batch_size {</span></span>
+</span>                    if stop_signal.load(Ordering::Relaxed) {</span></span>
+</span>                        break;</span></span>
+</span>                    } else {</span></span>
+</span>                        match send_password.send(batch.clone()) {</span></span>
+</span>                            Ok(_) => batch.clear(),</span></span>
+</span>                            Err(_) => break, // channel disconnected, stop thread</span></span>
+</span>                        }</span></span>
                     }</span></span></code></pre>
Then the worker thread:</p>
 pub fn password_checker(</span></span>
     index: usize,</span></span>
     file_path: &Path,</span></span>
-</span>    receive_password: Receiver<String>,</span></span>
+</span>    receive_password: Receiver<Vec<String>>,</span></span>
     stop_signal: Arc<AtomicBool>,</span></span>
     send_password_found: Sender<String>,</span></span>
     progress_bar: ProgressBar,</span></span>
@@</span> -49,26 +55,30</span> @@</span> pub fn password_checker(</span></span>
             while !stop_signal.load(Ordering::Relaxed) {</span></span>
                 match receive_password.recv() {</span></span>
                     Err(_) => break, // channel disconnected, stop thread</span></span>
-</span>                    Ok(password) => {</span></span>
-</span>                        let res = archive.by_index_decrypt(0, password.as_bytes());</span></span>
-</span>                        match res {</span></span>
-</span>                            Err(e) => panic!("Unexpected error {:?}", e),</span></span>
-</span>                            Ok(Err(_)) => (), // invalid password - continue</span></span>
-</span>                            Ok(Ok(mut zip)) => {</span></span>
-</span>                                // Validate password by reading the zip file to make sure it is not merely a hash collision.</span></span>
-</span>                                let mut buffer = Vec::with_capacity(zip.size() as usize);</span></span>
-</span>                                match zip.read_to_end(&mut buffer) {</span></span>
-</span>                                    Err(_) => (), // password collision - continue</span></span>
-</span>                                    Ok(_) => {</span></span>
-</span>                                        // Send password and continue processing while waiting for signal</span></span>
-</span>                                        send_password_found</span></span>
-</span>                                            .send(password)</span></span>
-</span>                                            .expect("Send found password should not fail");</span></span>
+</span>                    Ok(passwords) => {</span></span>
+</span>                        let passwords_len = passwords.len() as u64;</span></span>
+</span>                        // process batch</span></span>
+</span>                        for password in passwords {</span></span>
+</span>                            let res = archive.by_index_decrypt(0, password.as_bytes());</span></span>
+</span>                            match res {</span></span>
+</span>                                Err(e) => panic!("Unexpected error {:?}", e),</span></span>
+</span>                                Ok(Err(_)) => (), // invalid password - continue</span></span>
+</span>                                Ok(Ok(mut zip)) => {</span></span>
+</span>                                    // Validate password by reading the zip file to make sure it is not merely a hash collision.</span></span>
+</span>                                    let mut buffer = Vec::with_capacity(zip.size() as usize);</span></span>
+</span>                                    match zip.read_to_end(&mut buffer) {</span></span>
+</span>                                        Err(_) => (), // password collision - continue</span></span>
+</span>                                        Ok(_) => {</span></span>
+</span>                                            // Send password and continue processing while waiting for signal</span></span>
+</span>                                            send_password_found</span></span>
+</span>                                                .send(password)</span></span>
+</span>                                                .expect("Send found password should not fail");</span></span>
+</span>                                        }</span></span>
                                     }</span></span>
                                 }</span></span>
                             }</span></span>
                         }</span></span>
-</span>                        progress_bar.inc(1);</span></span>
+</span>                        progress_bar.inc(passwords_len);</span></span>
                     }</span></span>
                 }</span></span>
             }</span></span></code></pre>
And a bit of wiring up:</p>
-</span>pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize) -> Option<String> {</span></span>
+</span>pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize, batch_size: usize) -> Option<String> {</span></span>
     let zip_file_path = Path::new(zip_path);</span></span>
     let password_list_file_path = Path::new(password_list_path).to_path_buf();</span></span>
 </span></span>
@@</span> -97,7 +107,7</span> @@</span> pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize)</span></span>
     progress_bar.set_length(total_password_count as u64);</span></span>
 </span></span>
     // MPMC channel with backpressure</span></span>
-</span>    let (send_password, receive_password): (Sender<String>, Receiver<String>) =</span></span>
+</span>    let (send_password, receive_password): (Sender<Vec<String>>, Receiver<Vec<String>>) =</span></span>
         crossbeam_channel::bounded(workers * 10_000);</span></span>
 </span></span>
     let (send_found_password, receive_found_password): (Sender<String>, Receiver<String>) =</span></span>
@@</span> -111,6 +121,7</span> @@</span> pub fn password_finder(zip_path: &str, password_list_path: &str, workers: usize)</span></span>
     let password_gen_handle = start_password_reader(</span></span>
         password_list_file_path,</span></span>
         send_password,</span></span>
+</span>        batch_size,</span></span>
         stop_gen_signal.clone(),</span></span>
     );</span></span>
 </span></span>
@@</span> -153,12 +164,11</span> @@</span> fn main() {</span></span>
     let zip_path = args_iter.next().unwrap();</span></span>
     let dictionary_path =</span></span>
         "/home/agourlay/Workspace/blog-data/find-password-zip/xato-net-10-million-passwords.txt";</span></span>
-</span>    let workers: usize = match args_iter.next() {</span></span>
-</span>        None => num_cpus::get_physical(),</span></span>
-</span>        Some(count) => count.parse().unwrap(),</span></span>
-</span>    };</span></span>
+</span>    let batch_size: usize = args_iter.next().unwrap().parse().unwrap();</span></span>
+</span></span>
+</span>    let workers: usize = num_cpus::get_physical();</span></span>
 </span></span>
-</span>    match password_finder(&zip_path, dictionary_path, workers) {</span></span>
+</span>    match password_finder(&zip_path, dictionary_path, workers, batch_size) {</span></span>
         Some(password_found) => println!("Password found:{}", password_found),</span></span>
         None => println!("No password found :("),</span></span>
     }</span></span></code></pre>
Using Hyperfine again, we will compare different batch sizes.</p>
hyperfine</span> -</span>-runs</span> 2</span> \</span></span>
 -</span>-warmup</span> 1</span> \</span></span>
 -</span>-export-markdown</span> batch.md</span> \</span></span>
 -</span>-export-json</span> batch.json</span> \</span></span>
 -</span>n</span> 10</span> "</span>./zip-password-finder encrypted-test-dict.zip 10</span>"</span> \</span></span>
 -</span>n</span> 50</span> "</span>./zip-password-finder encrypted-test-dict.zip 50</span>"</span> \</span></span>
 -</span>n</span> 100</span> "</span>./zip-password-finder encrypted-test-dict.zip 100</span>"</span> \</span></span>
 -</span>n</span> 500</span> "</span>./zip-password-finder encrypted-test-dict.zip 500</span>"</span> \</span></span>
 -</span>n</span> 1000</span> "</span>./zip-password-finder encrypted-test-dict.zip 1000</span>"</span> \</span></span>
 -</span>n</span> 5000</span> "</span>./zip-password-finder encrypted-test-dict.zip 5000</span>"</span> \</span></span>
 -</span>n</span> 10000</span> "</span>./zip-password-finder encrypted-test-dict.zip 10000</span>"</span> </span></span></code></pre>
It seems 1000</code> yields the best results.</p>
Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

10</code></td> 234.756 ± 0.292</td> 234.550</td> 234.963</td> 1.09 ± 0.00</td></tr>
50</code></td> 226.616 ± 9.372</td> 219.989</td> 233.243</td> 1.05 ± 0.04</td></tr>
100</code></td> 216.498 ± 1.364</td> 215.534</td> 217.462</td> 1.00 ± 0.01</td></tr>
500</code></td> 217.122 ± 1.992</td> 215.714</td> 218.531</td> 1.00 ± 0.01</td></tr>
1000</code></td> 216.080 ± 0.066</td> 216.033</td> 216.127</td> 1.00</td></tr>
5000</code></td> 218.929 ± 0.114</td> 218.848</td> 219.009</td> 1.01 ± 0.00</td></tr>
10000</code></td> 222.792 ± 0.213</td> 222.642</td> 222.943</td> 1.03 ± 0.00</td></tr>
</tbody></table>
However, the version without batching for 4 cores used to run in 224 seconds. That's only an 8-second improvement. I am not sure if it is worth the added complexity.</p>
We started from a wild guess regarding the source of the synchronization, so we should not be surprised by the disappointing results.</p>
We are stopping the performance investigation here in order to save some content for the future :)</p>
Password generator</a>
</h2>
Our dictionary approach was great to get us started, but in real life, it is rather improbable to find the password you are targeting within a dictionary.</p>
Well, at least it did not work for me.</p>
A password generator is present in the final code available at zip-password-finder</a>, but I won't go into the implementation details because the code is rather ugly and does not add much at this point.</p>
What we need is to generate all the possible passwords based on a set of constraints.</p>
First, the minimum and maximum length to generate. For instance, between 3 and 7 characters.</p>
Second, the charset to use. It could be only letters, with uppercase, maybe with digits and punctuation.</p>
We can compute the number of passwords possible based on those constraints.</p>
let</span> mut</span> total_password_count</span> =</span> 0</span>;</span></span>
for</span> i</span> in</span> min_password_len</span>..=</span>max_password_len</span> {</span></span>
    total_password_count</span> +=</span> charset_len</span>.</span>pow</span>(</span>i</span> as</span> u32</span>)</span></span>
}</span></span></code></pre>
Here is an example to convince yourself that this grows very fast.</p>
Given a charset with lowercase letters, upper case letters and digits. Forming a set of 26 + 26 + 10 = 62</code> elements.</p>
A simple 7-character password using elements from this charset yields 62^7 = 3.521.614.606.208</code> combinations.</p>
62^7 passwords / 4500 passwords per sec / 60 / 60 / 24 / 365 =~ 25 years</code></p>
A naive brute force strategy requires much better performance to be tractable.</p>
Future work</a>
</h2>
I still have not found the password for the archive that spawned this experiment and article.</p>
However, given the current throughput, I'd have to let the program run for much longer than I am comfortable with.</p>
Fixing the scalability issue is the most important task to make this application run efficiently on machines with a higher core count.</p>
Then, improving the nominal performance per worker by investigating possible gains in the zip-rs</code> crate or a different approach to get a multiplying effect.</p>
I will make sure to document my progress in a follow-up article.</p>
Conclusion</a>
</h2>
In this article we have built a simple tool in Rust to brute force the password of protected ZIP archives.</p>
It can process around 4500 passwords/sec on 4-core machines and has issues scaling up which makes it impractical for non-trivial passwords.</p>
To get there we have learned how to apply the pool of workers</code> pattern and how to shut it down gracefully.</p>
We also learned how to measure performance in a systematic way and discussed scalability expectations.</p>
At a more general level, we confirmed that long passwords are very efficient against brute force attacks.</p>
This fact has been very well described in this xkcd</a> comic.</p>

    


    
    
</figure>
And that's all for today, I hope you learned a thing or two on the way - thank you for making it to the end!</p>
If you want to play with the application, the full code with better error handling & proper CLI is available at zip-password-finder</a>.</p>


  
  
  
    
      Update</strong>
  </p>
  
    The article submission on reddit/r/rust</a> contains excellent comments.</p>

  </div>
  
</blockquote>


A performance retrospective using Rust (part 3)
2022-08-08T00:00:00+00:00
This article is the third part of a performance retrospective regarding the hprof-slurp</a> project.</p>
It is highly recommended to start with the first part</a> to get a good grasp of the context.</p>
In this edition, we will focus on an arguably minor performance issue that will teach us a bit more about Rust along the way.</p>
A short disclaimer: this article has been written using Rust 1.62.1 and its conclusions might not be valid in the future.</p>
Try trait</a>
</h2>
As usual, our investigation is prompted by a strange artefact in a flamegraph.</p>
Below - in purple - you can see core::ops::try_trait::Try>::branch</code> represents 8.5% of the work in the parser thread (full flamegraph</a>).</p>

    


    
    
</figure>
You should know by now that the parser thread is the bottleneck for the whole application. It needs to be as fast as possible.</p>
Therefore, every bit of computation happening on it must be accounted for.</p>
In this case, the slowness seemed to be caused by the Try</code> trait, which does not appear in the code base explicitly.</p>
Our best chance is to simply look at the Rust doc for the branch</a> method.</p>

    


    
    
</figure>
Alright, so this function is a 'nightly-only experimental' API and is called every time the operator ?</code> is used on something that implements the Try</code> trait.</p>
The only possible implementations when using stable Rust are found in the standard library.</p>

    


    
    
</figure>
In our case the culprit must be the Result</code> type which is used to track the errors occurring during the parsing phase.</p>
After some digging, I found an existing issue</a> on GitHub appropriately named Reduced performance when using question mark operator instead of try!</code> where several users are reporting similar experiences.</p>
The issue seems to be a bit stale, so we are going to use a workaround this time.</p>
Workaround</a>
</h2>
If the question mark operator is having an impact on the performance, the most logical step is to not use it.</p>
The previous flamegraph points us to the parse_hprof_record</code> function where we will start by replacing ?</code> with combinators on Result</code>.</p>
The first change uses Result::and_then</code> to chain two results:</p>
     pub fn parse_hprof_record(&mut self) -> impl FnMut(&[u8]) -> IResult<&[u8], Record> + '_ {</span></span>
         |i| {</span></span>
             if self.heap_dump_remaining_len == 0 {</span></span>
-</span>                let (r1, tag) = parse_u8(i)?;</span></span>
-</span>                if self.debug_mode {</span></span>
-</span>                    println!("Found record tag:{} remaining bytes:{}", tag, i.len());</span></span>
-</span>                }</span></span>
-</span>                match tag {</span></span>
-</span>                    TAG_STRING => parse_utf8_string(r1),</span></span>
-</span>                    TAG_LOAD_CLASS => parse_load_class(r1),</span></span>
-</span>                    TAG_UNLOAD_CLASS => parse_unload_class(r1),</span></span>
-</span>                    TAG_STACK_FRAME => parse_stack_frame(r1),</span></span>
-</span>                    TAG_STACK_TRACE => parse_stack_trace(r1),</span></span>
-</span>                    TAG_ALLOC_SITES => parse_allocation_sites(r1),</span></span>
-</span>                    TAG_HEAP_SUMMARY => parse_heap_summary(r1),</span></span>
-</span>                    TAG_START_THREAD => parse_start_thread(r1),</span></span>
-</span>                    TAG_END_THREAD => parse_end_thread(r1),</span></span>
-</span>                    TAG_CONTROL_SETTING => parse_control_settings(r1),</span></span>
-</span>                    TAG_CPU_SAMPLES => parse_cpu_samples(r1),</span></span>
-</span>                    TAG_HEAP_DUMP_END => parse_heap_dump_end(r1),</span></span>
-</span>                    TAG_HEAP_DUMP | TAG_HEAP_DUMP_SEGMENT => {</span></span>
-</span>                        map(parse_header_record, |hr| {</span></span>
-</span>                            // record expected GC segments length</span></span>
-</span>                            self.heap_dump_remaining_len = hr.length;</span></span>
-</span>                            HeapDumpStart { length: hr.length }</span></span>
-</span>                        })(r1)</span></span>
+</span>                parse_u8(i).and_then(|(r1, tag)| {</span></span>
+</span>                    if self.debug_mode {</span></span>
+</span>                        println!("Found record tag:{} remaining bytes:{}", tag, i.len());</span></span>
+</span>                    }</span></span>
+</span>                    match tag {</span></span>
+</span>                        TAG_STRING => parse_utf8_string(r1),</span></span>
+</span>                        TAG_LOAD_CLASS => parse_load_class(r1),</span></span>
+</span>                        TAG_UNLOAD_CLASS => parse_unload_class(r1),</span></span>
+</span>                        TAG_STACK_FRAME => parse_stack_frame(r1),</span></span>
+</span>                        TAG_STACK_TRACE => parse_stack_trace(r1),</span></span>
+</span>                        TAG_ALLOC_SITES => parse_allocation_sites(r1),</span></span>
+</span>                        TAG_HEAP_SUMMARY => parse_heap_summary(r1),</span></span>
+</span>                        TAG_START_THREAD => parse_start_thread(r1),</span></span>
+</span>                        TAG_END_THREAD => parse_end_thread(r1),</span></span>
+</span>                        TAG_CONTROL_SETTING => parse_control_settings(r1),</span></span>
+</span>                        TAG_CPU_SAMPLES => parse_cpu_samples(r1),</span></span>
+</span>                        TAG_HEAP_DUMP_END => parse_heap_dump_end(r1),</span></span>
+</span>                        TAG_HEAP_DUMP | TAG_HEAP_DUMP_SEGMENT => {</span></span>
+</span>                            map(parse_header_record, |hr| {</span></span>
+</span>                                // record expected GC segments length</span></span>
+</span>                                self.heap_dump_remaining_len = hr.length;</span></span>
+</span>                                HeapDumpStart { length: hr.length }</span></span>
+</span>                            })(r1)</span></span>
+</span>                        }</span></span>
+</span>                        x => panic!("{}", format!("unhandled record tag {}", x)),</span></span>
                     }</span></span>
-</span>                    x => panic!("{}", format!("unhandled record tag {}", x)),</span></span>
-</span>                }                     </span></span>
+</span>                })</span></span></code></pre>
And a second change using Result::map</code>:</p>
             } else {</span></span>
                 // GC record mode</span></span>
-</span>                let (r1, gc_sub) = parse_gc_record(i)?;</span></span>
-</span>                let gc_sub_len = i.len() - r1.len();</span></span>
-</span>                self.heap_dump_remaining_len -= gc_sub_len as u32;</span></span>
-</span>                Ok((r1, GcSegment(gc_sub)))</span></span>
+</span>                parse_gc_record(i).map(|(r1, gc_sub)| {</span></span>
+</span>                    let gc_sub_len = i.len() - r1.len();</span></span>
+</span>                    self.heap_dump_remaining_len -= gc_sub_len as u32;</span></span>
+</span>                    (r1, GcSegment(gc_sub))</span></span>
+</span>                })</span></span>
             }</span></span>
         }</span></span>
     }</span></span>
</span></code></pre>
After applying those two changes, we can indeed see that core::ops::try_trait::Try>::branch</code> now represents only 0.5% of time in a different call stack.</p>

    


    
    
</figure>
Most of it now comes from nom::combinator::flat_map</code> and when zooming in we can also find a tiny contribution from nom::combinator::map</code>.</p>

    


    
    
</figure>
The previous occurrence of core::ops::try_trait::Try>::branch</code> has been replaced by Result::map</code> which informs us that our second change is executed more often.</p>
Observing flamegraphs is nice but does it translate into runtime performance gains?</p>
hyperfine</span> -</span>-runs</span> 10</span> \</span></span>
 -</span>-warmup</span> 5</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>n</span> with-?</span> "</span>./hprof-slurp-with-? -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> with-combinators</span> "</span>./hprof-slurp-combinators -i pets.bin</span>"</span></span></code></pre>Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

with-?</code></td> 36.614 ± 0.138</td> 36.373</td> 36.835</td> 1.04 ± 0.02</td></tr>
with-combinators</code></td> 35.062 ± 0.766</td> 34.202</td> 35.969</td> 1.00</td></tr>
</tbody></table>
Using the question mark operator is 4% slower than the combinators version on this run.</p>
That is not a massive improvement, but I think it is a reasonable trade-off given the change applied and the application's context.</p>
Going deeper</a>
</h2>
We can go deeper by applying the same tricks to nom</code> to see if we can scrape some additional gains.</p>
Let's clone the repository and apply the following changes.</p>
First on the flat_map</code> combinator.</p>
pub fn flat_map<I, O1, O2, E: ParseError<I>, F, G, H>(</span></span>
  mut parser: F,</span></span>
  mut applied_parser: G,</span></span>
) -> impl FnMut(I) -> IResult<I, O2, E></span></span>
where</span></span>
  F: Parser<I, O1, E>,</span></span>
  G: FnMut(O1) -> H,</span></span>
  H: Parser<I, O2, E>,</span></span>
{</span></span>
   move |input: I| {</span></span>
-</span>    let (input, o1) = parser.parse(input)?;</span></span>
-</span>    applied_parser(o1).parse(input)</span></span>
+</span>    parser</span></span>
+</span>      .parse(input)</span></span>
+</span>      .and_then(|(input, o1)| applied_parser(o1).parse(input))</span></span>
   }</span></span>
}</span></span></code></pre>
And then for the map</code> combinator.</p>
pub fn map<I, O1, O2, E, F, G>(mut parser: F, mut f: G) -> impl FnMut(I) -> IResult<I, O2, E></span></span>
where</span></span>
  F: Parser<I, O1, E>,</span></span>
  G: FnMut(O1) -> O2,</span></span>
{</span></span>
-</span>  move |input: I| {</span></span>
-</span>    let (input, o1) = parser.parse(input)?;</span></span>
-</span>    Ok((input, f(o1)))</span></span>
-</span>  }</span></span>
+</span>  move |input: I| parser.parse(input).map(|(input, o1)| (input, f(o1)))</span></span>
</span>
}</span></span></code></pre>
In order to use our local patched version of nom</code>, we simply need to tell Cargo about it.</p>
 [dependencies]</span></span>
-</span>nom = "7.1.1"</span></span>
+</span>nom = { path = "../nom" }</span></span></code></pre>
This is a pretty neat feature from Cargo which enables quick experiments!</p>
After validating that the flamegraph is now free from any traces of branch</code> (trust me on this one, there are enough screenshots in this article), we can run a final benchmark.</p>
hyperfine</span> -</span>-runs</span> 10</span> \</span></span>
 -</span>-warmup</span> 5</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>n</span> with-?</span> "</span>./hprof-slurp-with-? -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> with-combinators</span> "</span>./hprof-slurp-combinators -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> with-nom-combinators</span> "</span>./hprof-slurp-combinators -i pets.bin</span>"</span></span></code></pre>
I re-ran the benchmark several times and the result is that the gain of the nom</code> patch is more often 0% than 1%.</p>
It is more or less what we could expect from optimizing a chunk of code accounting for 0.5% of the CPU time.</p>
This means it is not worth it to make a PR to upstream the performance patch as we do not have enough data to back it up with.</p>
An important aspect of performance work is that diminishing returns can occur quickly. Therefore, knowing when to stop digging in the same spot is essential.</p>
Conclusion</a>
</h2>
In this article, we have learned that the question mark operator can have an impact on the runtime performance of a program.</p>
How much of an impact, however, depends on the exact workload of the program and where the operator is used.</p>
It seems reasonable to believe that this behaviour should not be an issue for the majority of Rust programs out there where the question mark operator is not used to shortcut very hot code.</p>
As always, it is good to benchmark instead of making assumptions, which could lead to making the code less readable without concrete gains.</p>
The next article in this series</a> will go through another interesting optimization encountered while making hprof-slurp</code> faster.</p>


A performance retrospective using Rust (part 2)
2022-07-23T00:00:00+00:00
This article is the second part of a performance retrospective regarding the hprof-slurp</a> project. It is highly recommended to start with the first part</a> to get a good grasp of the context.</p>
We left off after a presentation of the project and a detailed description of the benchmarking methodology used to measure performance gains.</p>
In this article we will focus on a specific issue that has been plaguing the 0.3.x series.</p>
Avoiding memcopy</a>
</h2>
As the program got faster, I witnessed the memcpy</code> instructions slowly creeping up in the flamegraphs to the point where they accounted for most of the computation.</p>
Below - in purple - you can see the _memcpy_avx_unaligned_erms</code> representing most of the work in the parser thread (full flamegraph</a>).</p>

    


    
    
</figure>
At first, I thought it was an inherent cost of parsing large files, but as it grew to become the largest bottleneck in hprof-slurp</code>, I decided to investigate the issue to - at the very least - understand it.</p>
I found the answer to this mystery in the fantastic Rust performance book</a>.</p>

  
  Rust types that are larger than 128 bytes are copied with memcpy rather than inline code.
Shrinking these types to 128 bytes or less can make the code faster by avoiding memcpy calls and reducing memory traffic.</p>
</div>
  
  

    — Rust performance book</p>

  </div>
  
</blockquote>
Therefore, within my code I needed to find the large types that are used very often on the parser's path.</p>
Let's generate the type sizes with:</p>
RUSTFLAGS</span>=</span>-</span>Z</span>p</span>r</span>i</span>n</span>t</span>-</span>t</span>y</span>p</span>e</span>-</span>s</span>i</span>z</span>e</span>s</span> cargo</span> +nightly</span> build</span> -</span>-release</span></span></code></pre>
In my opinion, it can be a bit cumbersome to navigate the output of -Zprint-type-sizes</code>.</p>
One alternative recommended by the Rust performance book is to use DHAT’s “copy profiling” mode, which works great as well.</p>
I will stick to -Zprint-type-sizes</code> for the remainder of the article as it helps understand the memory layout on the way.</p>
Looking at the types I own in the output section, we can find the following</p>
print-type-size</span> type:</span> `</span>parser::record::Record</span>`</span>:</span> 136</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     discriminant:</span> 2</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>GcSegment</span>`</span>:</span> 134</span> bytes</span></span>
print-type-size</span>         padding:</span> 6</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.0</span>`</span>:</span> 128</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>AllocationSites</span>`</span>:</span> 62</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.flags</span>`</span>:</span> 2</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.cutoff_ratio</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_live_bytes</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_live_instances</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.number_of_sites</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_bytes_allocated</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_instances_allocated</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.allocation_sites</span>`</span>:</span> 24</span> bytes</span></span>
.</span>.</span>.</span> more</span> variants</span> with</span> decreasing</span> size</span></span></code></pre>
The parser::record::Record</code> is an enum which represents all possible top-level records found in an hprof</code> file.</p>
The largest variant is GcSegment</code>, with 134 bytes, which is larger than the 128 bytes mentioned above.</p>
The corresponding Rust code looks like the following:</p>
pub</span> enum</span> Record</span> {</span></span>
  GcSegment</span>(</span>GcRecord</span>)</span>,</span></span>
  AllocationSites</span> {</span></span>
    flags</span>:</span> u16</span>,</span></span>
    cutoff_ratio</span>:</span> u32</span>,</span></span>
    total_live_bytes</span>:</span> u32</span>,</span></span>
    total_live_instances</span>:</span> u32</span>,</span></span>
    total_bytes_allocated</span>:</span> u64</span>,</span></span>
    total_instances_allocated</span>:</span> u64</span>,</span></span>
    number_of_sites</span>:</span> u32</span>,</span></span>
    allocation_sites</span>:</span> Vec</span><</span>AllocationSite</span>></span>,</span></span>
  }</span>,</span></span>
  ...</span> more</span> variants</span></span>
}</span>  </span></span></code></pre>
The parser::gc_record::GcRecord</code> is itself an enum as well. It models all possible elements found on the JVM's heap.</p>
Let's find in the output the size of a GcRecord</code>.</p>
print-type-size</span> type:</span> `</span>parser::gc_record::GcRecord</span>`</span>:</span> 128</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     discriminant:</span> 1</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>ClassDump</span>`</span>:</span> 127</span> bytes</span></span>
print-type-size</span>         padding:</span> 1</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.constant_pool_size</span>`</span>:</span> 2</span> bytes,</span> alignment:</span> 2</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.stack_trace_serial_number</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.instance_size</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_object_id</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.super_class_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_loader_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.signers_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.protection_domain_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.const_fields</span>`</span>:</span> 24</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.static_fields</span>`</span>:</span> 24</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.instance_fields</span>`</span>:</span> 24</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>ObjectArrayDump</span>`</span>:</span> 55</span> bytes</span></span>
print-type-size</span>         padding:</span> 3</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.stack_trace_serial_number</span>`</span>:</span> 4</span> bytes,</span> alignment:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.number_of_elements</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.object_id</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.array_class_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.elements</span>`</span>:</span> 24</span> bytes</span></span>
.</span>.</span>.</span> more</span> variant</span> with</span> decreasing</span> size</span></span></code></pre>
For the following definition:</p>
pub</span> enum</span> GcRecord</span> {</span></span>
    ClassDump</span> {</span></span>
        class_object_id</span>:</span> u64</span>,</span></span>
        stack_trace_serial_number</span>:</span> u32</span>,</span></span>
        super_class_object_id</span>:</span> u64</span>,</span></span>
        class_loader_object_id</span>:</span> u64</span>,</span></span>
        signers_object_id</span>:</span> u64</span>,</span></span>
        protection_domain_object_id</span>:</span> u64</span>,</span></span>
        instance_size</span>:</span> u32</span>,</span></span>
        constant_pool_size</span>:</span> u16</span>,</span></span>
        const_fields</span>:</span> Vec</span><</span>(</span>ConstFieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
        static_fields</span>:</span> Vec</span><</span>(</span>FieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
        instance_fields</span>:</span> Vec</span><</span>FieldInfo</span>></span>,</span></span>
    }</span>,</span></span>
    ObjectArrayDump</span> {</span></span>
        object_id</span>:</span> u64</span>,</span></span>
        stack_trace_serial_number</span>:</span> u32</span>,</span></span>
        number_of_elements</span>:</span> u32</span>,</span></span>
        array_class_id</span>:</span> u64</span>,</span></span>
        elements</span>:</span> Vec</span><</span>u64</span>></span>,</span></span>
    }</span>,</span></span>
    ...</span> more</span> variants</span></span>
}</span></span></code></pre>
At this point, we can see that there is logically a direct mapping between the fields of the structs and their actual sizes.</p>
Moreover, there is a critical observation to make regarding enums.</p>
Enums are sized according to the largest variant</strong>.</p>
An enum is sized with its largest variant plus some memory for an internal discriminant field.</p>
This means the 134 bytes for a GcSegment</code> are being allocated for every parser::record::Record</code> variant.</p>
This quickly adds up when a program is crunching millions of records.</p>
The common wisdom for decreasing the stack memory footprint is to use heap allocation, often referred to as boxing.</p>
Let's try boxing the content of GcSegment</code>.</p>
GcSegment</span>(</span>Box</span><</span>GcRecord</span>></span>)</span>,</span></span></code></pre>
This change gives the following type sizes (after applying the necessary modifications to handle the new Box type at the call sites).</p>
print-type-size</span> type:</span> `</span>parser::record::Record</span>`</span>:</span> 64</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     discriminant:</span> 2</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>AllocationSites</span>`</span>:</span> 62</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.flags</span>`</span>:</span> 2</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.cutoff_ratio</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_live_bytes</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_live_instances</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.number_of_sites</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_bytes_allocated</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.total_instances_allocated</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.allocation_sites</span>`</span>:</span> 24</span> bytes</span></span>
.</span>.</span>.</span> more</span> variant</span> with</span> decreasing</span> size</span></span>
print-type-size</span>     variant</span> `</span>GcSegment</span>`</span>:</span> 14</span> bytes</span></span>
print-type-size</span>         padding:</span> 6</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.0</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span></code></pre>
Looking good!</p>
The type parser::record::Record</code> went from 136 bytes to 64 bytes because all variants are now sized according to the next largest variant, AllocationSites</code> at 62 bytes.</p>
The memcpy</code> should not be a problem anymore.</p>
hyperfine</span> -</span>-runs</span> 3</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>n</span> 0.3.3</span> "</span>./hprof-slurp-0.3.3 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> box</span> "</span>./hprof-slurp -i pets.bin</span>"</span></span></code></pre>Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

0.3.3</code></td> 77.442 ± 2.537</td> 74.513</td> 78.968</td> 1.00</td></tr>
box</code></td> 270.785 ± 0.907</td> 270.031</td> 271.792</td> 3.50 ± 0.12</td></tr>
</tbody></table>
Ouch! It is over three times slower!</p>
Let's have a look at the flamegraph to understand where the CPU spends its time.</p>

    


    
    
</figure>
In purple, you can see the cost of Box::new</code> in the parser thread (full flamegraph</a> available).</p>
Welp, we did fix the memcpy</code> issue, but the application is terribly slow.</p>
Boxing is, unfortunately, not some kind of magic sauce that will fix all our problems.</p>
Performing heap allocation has a higher cost than stack allocation, and we are now performing it for each instance found in the JVM heap dump.</p>
Reasonable boxing</a>
</h2>
In the previous section, we have seen that GcSegment</code> cannot be boxed efficiently because it appears too frequently during the execution of the application.</p>
We also have seen that the largest variant of GcSegment</code> is ClassDump</code>, clocking in at 127 bytes.</p>
Based on the enumeration sizing discovery, we know that the 127 bytes for a ClassDump</code> are being allocated for every GcSegment</code> record.</p>
However, the ClassDump</code> is a relatively rare element compared to the actual instance dumps. An application has a few hundred classes but millions of instances.</p>
It is unfortunate to systematically pay the full price for something that is infrequent.</p>
The intuition is that we should be moving the boxing cost to the ClassDump</code> only.</p>
Let's try to box the individual fields in ClassDump</code> instead.</p>
pub</span> enum</span> GcRecord</span> {</span></span>
    ClassDump</span> {</span></span>
        class_object_id</span>:</span> u64</span>,</span></span>
        stack_trace_serial_number</span>:</span> u32</span>,</span></span>
        super_class_object_id</span>:</span> u64</span>,</span></span>
        class_loader_object_id</span>:</span> u64</span>,</span></span>
        signers_object_id</span>:</span> u64</span>,</span></span>
        protection_domain_object_id</span>:</span> u64</span>,</span></span>
        instance_size</span>:</span> u32</span>,</span></span>
        constant_pool_size</span>:</span> u16</span>,</span></span>
        const_fields</span>:</span> Vec</span><</span>(</span>ConstFieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
        static_fields</span>:</span> Vec</span><</span>(</span>FieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
        instance_fields</span>:</span> Vec</span><</span>FieldInfo</span>></span>,</span></span>
    }</span></span>
    ...</span></span>
}</span>   </span></span></code></pre>
The Vec</code> type contains three words: a length, a capacity, and a pointer. That is 3 * 8 = 24 bytes on a 64-bit architecture.</p>
Boxing each vector would save 16 bytes per field.</p>
pub</span> enum</span> GcRecord</span> {</span></span>
    ClassDump</span> {</span></span>
        class_object_id</span>:</span> u64</span>,</span></span>
        stack_trace_serial_number</span>:</span> u32</span>,</span></span>
        super_class_object_id</span>:</span> u64</span>,</span></span>
        class_loader_object_id</span>:</span> u64</span>,</span></span>
        signers_object_id</span>:</span> u64</span>,</span></span>
        protection_domain_object_id</span>:</span> u64</span>,</span></span>
        instance_size</span>:</span> u32</span>,</span></span>
        constant_pool_size</span>:</span> u16</span>,</span></span>
        const_fields</span>:</span> Box</span><</span>Vec</span><</span>(</span>ConstFieldInfo</span>,</span> FieldValue</span>)</span>></span>></span>,</span></span>
        static_fields</span>:</span> Box</span><</span>Vec</span><</span>(</span>FieldInfo</span>,</span> FieldValue</span>)</span>></span>></span>,</span></span>
        instance_fields</span>:</span> Box</span><</span>Vec</span><</span>FieldInfo</span>></span>></span>,</span></span>
    }</span></span>
    ...</span></span>
}</span>    </span></span></code></pre>
Fetching the new type's size yields the following:</p>
print-type-size</span> type:</span> `</span>parser::gc_record::GcRecord</span>`</span>:</span> 80</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     discriminant:</span> 1</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>ClassDump</span>`</span>:</span> 79</span> bytes</span></span>
print-type-size</span>         padding:</span> 1</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.constant_pool_size</span>`</span>:</span> 2</span> bytes,</span> alignment:</span> 2</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.stack_trace_serial_number</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.instance_size</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_object_id</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.super_class_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_loader_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.signers_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.protection_domain_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.const_fields</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.static_fields</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.instance_fields</span>`</span>:</span> 8</span> bytes</span></span></code></pre>
This is exactly what we were expecting; one word per boxed field.</p>
But we have learned the hard way not to celebrate prematurely before checking how it pans out in a benchmark.</p>
hyperfine</span> -</span>-runs</span> 3</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>n</span> 0.3.3</span> "</span>./hprof-slurp-0.3.3 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> boxes</span> "</span>./hprof-slurp -i pets.bin</span>"</span></span></code></pre>Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

0.3.3</code></td> 76.691 ± 3.326</td> 73.080</td> 79.629</td> 1.12 ± 0.05</td></tr>
boxes</code></td> 68.625 ± 1.262</td> 67.202</td> 69.605</td> 1.00</td></tr>
</tbody></table>
It runs 11% faster. It's not bad for such a small change!</p>
But let's try to go faster by reducing the size of the ClassDump</code> variant even more.</p>
pub</span> struct</span> ClassDumpFields</span> {</span></span>
    const_fields</span>:</span> Vec</span><</span>(</span>ConstFieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
    static_fields</span>:</span> Vec</span><</span>(</span>FieldInfo</span>,</span> FieldValue</span>)</span>></span>,</span></span>
    instance_fields</span>:</span> Vec</span><</span>FieldInfo</span>></span>,</span></span>
}</span></span>
</span>
pub</span> enum</span> GcRecord</span> {</span></span>
    ClassDump</span> {</span></span>
        class_object_id</span>:</span> u64</span>,</span></span>
        stack_trace_serial_number</span>:</span> u32</span>,</span></span>
        super_class_object_id</span>:</span> u64</span>,</span></span>
        class_loader_object_id</span>:</span> u64</span>,</span></span>
        signers_object_id</span>:</span> u64</span>,</span></span>
        protection_domain_object_id</span>:</span> u64</span>,</span></span>
        instance_size</span>:</span> u32</span>,</span></span>
        constant_pool_size</span>:</span> u16</span>,</span></span>
        fields</span>:</span> Box</span><</span>ClassDumpFields</span>></span>,</span></span>
    }</span></span>
    ...</span></span>
}</span>    </span></span></code></pre>
Which has the following type size:</p>
print-type-size</span> type:</span> `</span>parser::gc_record::ClassDumpFields</span>`</span>:</span> 72</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     field</span> `</span>.const_fields</span>`</span>:</span> 24</span> bytes</span></span>
print-type-size</span>     field</span> `</span>.static_fields</span>`</span>:</span> 24</span> bytes</span></span>
print-type-size</span>     field</span> `</span>.instance_fields</span>`</span>:</span> 24</span> bytes</span></span>
.</span>.</span>.</span></span>
print-type-size</span> type:</span> `</span>parser::gc_record::GcRecord</span>`</span>:</span> 64</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>     discriminant:</span> 1</span> bytes</span></span>
print-type-size</span>     variant</span> `</span>ClassDump</span>`</span>:</span> 63</span> bytes</span></span>
print-type-size</span>         padding:</span> 1</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.constant_pool_size</span>`</span>:</span> 2</span> bytes,</span> alignment:</span> 2</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.stack_trace_serial_number</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.instance_size</span>`</span>:</span> 4</span> bytes</span></span>
print-type-size</span>         padding:</span> 4</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_object_id</span>`</span>:</span> 8</span> bytes,</span> alignment:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.super_class_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.class_loader_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.signers_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.protection_domain_object_id</span>`</span>:</span> 8</span> bytes</span></span>
print-type-size</span>         field</span> `</span>.fields</span>`</span>:</span> 8</span> bytes</span></span></code></pre>
We shaved off 16 more bytes.</p>
hyperfine</span> -</span>-runs</span> 3</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>n</span> 0.3.3</span> "</span>./hprof-slurp-0.3.3 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> boxes</span> "</span>./hprof-slurp-boxes -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> box-struct</span> "</span>./hprof-slurp -i pets.bin</span>"</span></span></code></pre>Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

0.3.3</code></td> 75.638 ± 4.154</td> 70.900</td> 78.660</td> 1.15 ± 0.06</td></tr>
boxes</code></td> 68.621 ± 0.399</td> 68.194</td> 68.986</td> 1.04 ± 0.01</td></tr>
box-struct</code></td> 65.978 ± 0.185</td> 65.764</td> 66.086</td> 1.00</td></tr>
</tbody></table>
This last change gave us an extra 4%, which is quite nice.</p>
To go further, one would need to remove unnecessary fields from the ClassDump</code> variant.</p>
Clippy</a>
</h2>
Before wrapping up this article, it is mandatory to note that Clippy has a lint for this class of problems, which I, of course</strong> discovered only much later.</p>
It is called large_enum_variant</a> and kicks in when the difference between the largest variant and the smallest variant is larger than 200 bytes.</p>
The documentation warns us about something very important that we discovered as well.</p>


  
  
  
    
      Warning</strong>
  </p>
  
    This lint obviously cannot take the distribution of variants in your running program into account. It is possible that the smaller variants make up less than 1% of all instances, in which case the overhead is negligible and the boxing is counter-productive. Always measure the change this lint suggests.</p>

  </div>
  
</blockquote>

Let's give this lint a try by creating a clippy.toml</code> file in the project and configuring this lint to be more aggressive with:</p>
enum-variant-size-threshold</span> =</span> 100</span> #</span> instead of the default value 200</span></span></code></pre>
Running cargo clippy</code> yields two entries:</p>
warning:</span> large</span> size</span> difference</span> between</span> variants</span></span>
  --</span>></span> src/parser/record.rs:97:5</span></span>
   |</span></span>
97</span> |</span>     GcSegment(GcRecord</span>),</span></span>
   |</span>     ^^^^^^^^^^^^^^^^^^^</span> this</span> variant</span> is</span> 128</span> bytes</span></span>
   |</span></span>
note:</span> and</span> the</span> second-largest</span> variant</span> is</span> 58</span> bytes:</span></span>
  --</span>></span> src/parser/record.rs:55:5</span></span>
   |</span></span>
55</span> |</span> /</span>     AllocationSites</span> {</span></span>
56</span> |</span> |</span>         flags:</span> u16,</span></span>
57</span> |</span> |</span>         cutoff_ratio:</span> u32,</span></span>
58</span> |</span> |</span>         total_live_bytes:</span> u32,</span></span>
.</span>.</span>.</span>  |</span></span>
63</span> |</span> |</span>         allocation_sites:</span> Vec</span><</span>AllocationSit</span>e</span>></span>,</span></span>
64</span> |</span> |</span>     },</span></span>
   |</span> |</span>_____^</span></span>
   =</span> help:</span> for</span> further</span> information</span> visit</span> https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant</span></span>
help:</span> consider</span> boxing</span> the</span> large</span> fields</span> to</span> reduce</span> the</span> total</span> size</span> of</span> the</span> enum</span></span>
   |</span></span>
97</span> |</span>     GcSegment(Box</span><GcRecord></span>),</span></span>
   |</span>  </span></span></code></pre>
We tried this already and it did not work. Clippy tries its best, but remember, it cannot know about the distribution of variants through a static analysis.</p>
And the second entry:</p>
warning:</span> large</span> size</span> difference</span> between</span> variants</span></span>
   --</span>></span> src/parser/gc_record.rs:127:5</span></span>
    |</span></span>
127</span> |</span> /</span>     ClassDump</span> {</span></span>
128</span> |</span> |</span>         class_object_id:</span> u64,</span></span>
129</span> |</span> |</span>         stack_trace_serial_number:</span> u32,</span></span>
130</span> |</span> |</span>         super_class_object_id:</span> u64,</span></span>
.</span>.</span>.</span>   |</span></span>
138</span> |</span> |</span>         instance_fields:</span> Vec</span><</span>FieldInf</span>o</span>></span>,</span></span>
139</span> |</span> |</span>     },</span></span>
    |</span> |</span>_____^</span> this</span> variant</span> is</span> 122</span> bytes</span></span>
    |</span></span>
    =</span> note:</span> `</span>#</span>[warn(clippy::large_enum_variant)]</span>`</span> on</span> by</span> default</span></span>
note:</span> and</span> the</span> second-largest</span> variant</span> is</span> 49</span> bytes:</span></span>
   --</span>></span> src/parser/gc_record.rs:120:5</span></span>
    |</span></span>
120</span> |</span> /</span>     PrimitiveArrayDump</span> {</span></span>
121</span> |</span> |</span>         object_id:</span> u64,</span></span>
122</span> |</span> |</span>         stack_trace_serial_number:</span> u32,</span></span>
123</span> |</span> |</span>         number_of_elements:</span> u32,</span></span>
124</span> |</span> |</span>         element_type:</span> FieldType,</span></span>
125</span> |</span> |</span>         array_value:</span> ArrayValue,</span></span>
126</span> |</span> |</span>     },</span></span>
    |</span> |</span>_____^</span></span>
    =</span> help:</span> for</span> further</span> information</span> visit</span> https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant</span></span>
help:</span> consider</span> boxing</span> the</span> large</span> fields</span> to</span> reduce</span> the</span> total</span> size</span> of</span> the</span> enum</span></span>
    |</span></span>
138</span> |</span>         instance_fields:</span> Box</span><</span>Vec</span><</span>FieldInf</span>o</span>>></span>,</span></span>
    |</span>                          ~~~~~~~~~~~~~~~~~~~</span></span></code></pre>
Which is the lead we ended up exploring as well.</p>
Sometimes the best way to learn is to rediscover what everyone already knows for yourself.</p>
Clippy is truly a fantastic tool and one of my personal favorites in the whole Rust ecosystem. Depending on your context, it may make sense to configure custom values for certain lints to get the most out of them.</p>
Conclusion</a>
</h2>
This article has presented a bottleneck due to excessive memcopy</code> which was slowing down hprof-slurp</code> prior to the 0.4.0 release.</p>
The final fix took only a few lines of code but had a strong</a> positive effect on performance.</p>
In order to get there, we had to understand a few details regarding the way Rust sizes types internally, more specifically, enumerations.</p>
As a rule of thumb, it is recommended to avoid single outsized variants when creating enumerations as it impacts the memory usage of all variants.</p>
One also must be careful when trying to reduce the stack memory pressure via boxing. Allocating on the heap is not magic, and its cost must be understood in the context of the application.</p>
The next article in this series</a> will go through another interesting optimization encountered while making hprof-slurp</code> faster.</p>


  
  
  
    
      Update</strong>
  </p>
  
    The article submission on reddit/r/rust</a> contains excellent comments.</p>

  </div>
  
</blockquote>


A performance retrospective using Rust (part 1)
2022-07-11T00:00:00+00:00
This is the first article in a series regarding making a simple JVM heap dump analyzer in Rust faster over time.</p>
Although it is primarily targeted at intermediate Rust developers, it will surely be relevant for engineers interested in performance in general.</p>
Performance optimization is highly context-dependent so it makes sense to spend some time explaining what problem we are trying to solve.</p>
Context</a>
</h2>
In a previous life, I found myself needing to analyze large JVM heap dumps. For a sense of scale, "large" stands for more than 50Gb in that context.</p>
The JVM ecosystem is pretty rich in terms of tooling regarding heap dump analysis. Most JVM developers are familiar with VisualVM</a> and EclipseMat</a>.</p>
Those tools are truly excellent; they offer a large panel of features</a> to drill down on the content of heap dumps to help you pinpoint your issue very precisely.</p>
However, they tend to be extremely memory hungry and slow when analyzing large files, which is forcing users to spin up expensive beefy instances from a cloud provider to get the job done.</p>
The tool I needed was fairly specific, my main concern was to get a quick overview of large heap dumps using a regular developer machine to decide if it actually makes sense to investigate further using the aforementioned workflow.</p>
The goal is not to replace the existing tooling but to optimize the investigation workflow and this led me to create the hprof-slurp</a> project.</p>
Hprof-slurp</a>
</h2>
The project is a CLI written in Rust which processes dump files in a streaming fashion.</p>
It trades off features for speed by performing only a single pass without storing intermediary results on the host, which reduces the depth of the analysis possible.</p>
The project is named after the hprof</a> format which is used by the JDK to encode heap dumps.</p>
The 3 main features available are:</p>

report top k</code> allocated classes.</li>
report the count of instances per class.</li>
report the largest instance size per class.</li>
</ul>
Let's have a look at an example of output processing: a tiny heap dump</a> which is used as part of the integration test pipeline.</p>
du</span> -</span>BM</span> hprof-64.bin</span> </span></span>
2M</span> hprof-64.bin</span></span></code></pre>
Let's process that 2Mb heap dump!</p>
./hprof-slurp</span> -</span>i</span> "</span>hprof-64.bin</span>"</span></span></code></pre>
Which instantaneously yields those two tables</p>
Top 20 allocated classes:</span></span>
</span>
Total size | Instances |     Largest | Class name                                  </span></span>
------------------------------------------------------------------------------------</span></span>
   1.99MiB |       436 |   634.78KiB | int[]</span></span>
 197.11KiB |      1991 |    16.02KiB | char[]</span></span>
  85.25KiB |       443 |     8.02KiB | byte[]</span></span>
  47.38KiB |      1516 |  32.00bytes | java/lang/String</span></span>
  45.42KiB |       560 |     8.02KiB | java/lang/Object[]</span></span>
  15.26KiB |       126 | 124.00bytes | java/lang/reflect/Field</span></span>
  14.77KiB |       378 |  40.00bytes | java/util/LinkedList$Node</span></span>
   9.94KiB |       212 |  48.00bytes | java/util/HashMap$Node</span></span>
   8.91KiB |       190 |  48.00bytes | java/util/LinkedList</span></span>
   8.42KiB |        98 |  88.00bytes | java/lang/ref/SoftReference</span></span>
   6.05KiB |       258 |  24.00bytes | java/lang/Integer</span></span>
   5.91KiB |        18 |     2.02KiB | java/util/HashMap$Node[]</span></span>
   5.86KiB |       150 |  40.00bytes | java/lang/StringBuilder</span></span>
   5.44KiB |       116 |  48.00bytes | java/util/Hashtable$Entry</span></span>
   5.05KiB |        38 | 136.00bytes | sun/util/locale/LocaleObjectCache$CacheEntry</span></span>
   5.00KiB |        40 | 128.00bytes | java/lang/ref/Finalizer</span></span>
   3.50KiB |        32 | 112.00bytes | java/net/URL</span></span>
   3.42KiB |        73 |  48.00bytes | java/io/File</span></span>
   3.17KiB |        12 | 776.00bytes | java/util/Hashtable$Entry[]</span></span>
   3.13KiB |        56 | 144.00bytes | java/lang/String[]</span></span>
</span>
Top 20 largest instances:</span></span>
</span>
 Total size | Instances |     Largest | Class name                                   </span></span>
--------------------------------------------------------------------------------------</span></span>
    1.99MiB |       436 |   634.78KiB | int[]</span></span>
  197.11KiB |      1991 |    16.02KiB | char[]</span></span>
   85.25KiB |       443 |     8.02KiB | byte[]</span></span>
   45.42KiB |       560 |     8.02KiB | java/lang/Object[]</span></span>
    5.91KiB |        18 |     2.02KiB | java/util/HashMap$Node[]</span></span>
    2.05KiB |         2 |     2.02KiB | java/lang/invoke/MethodHandle[]</span></span>
    2.02KiB |         1 |     2.02KiB | java/lang/Integer[]</span></span>
    3.17KiB |        12 | 776.00bytes | java/util/Hashtable$Entry[]</span></span>
462.00bytes |         1 | 462.00bytes | sun/misc/Launcher$AppClassLoader</span></span>
454.00bytes |         1 | 454.00bytes | sun/misc/Launcher$ExtClassLoader</span></span>
680.00bytes |         2 | 340.00bytes | simple/Producer</span></span>
680.00bytes |         2 | 340.00bytes | simple/Consumer</span></span>
    2.30KiB |         7 | 336.00bytes | java/util/jar/JarFile$JarFileEntry</span></span>
334.00bytes |         1 | 334.00bytes | java/lang/ref/Finalizer$FinalizerThread</span></span>
332.00bytes |         1 | 332.00bytes | java/lang/ref/Reference$ReferenceHandler</span></span>
    1.01KiB |         9 | 312.00bytes | java/lang/reflect/Field[]</span></span>
    1.48KiB |         7 | 272.00bytes | java/util/concurrent/ConcurrentHashMap$Node[]</span></span>
236.00bytes |         1 | 236.00bytes | sun/net/www/protocol/file/FileURLConnection</span></span>
440.00bytes |         2 | 220.00bytes | java/io/ExpiringCache$1</span></span>
432.00bytes |         2 | 216.00bytes | java/lang/NoSuchMethodError</span></span></code></pre>
And that's all there is to it.</p>
Architecture</a>
</h2>
As mentioned previously, the CLI is written in Rust and works in a synchronous multithreaded fashion.</p>
Here is a simplified architecture diagram for the current version (0.4.7).</p>

    


    
    
</figure>
The various threads are wired up together via channels to form a processing pipeline where all stages run in parallel (if the host has enough cores).</p>
The file reader thread proactively loads chunks of 64Mb from the input file to not starve the rest of the pipeline while waiting for IO. Loading too many chunks in advance has a direct impact on the memory usage, so I settled on 3 chunks based on experiments.</p>
Those chunks are then sent over to the streaming parser which was the most challenging work of the project because it must handle incomplete inputs.
A single chunk can contain millions of records, and a chunk is of course not aligned on the actual boundary of the hprof</code> records.</p>
Therefore, the parser tries to make sense of the incoming binary data while carefully managing its inner buffer.</p>
More concretely, it has been written by following the full specification of the hprof format</a> found in the heap dumper code of the JDK.</p>
The parser itself is written with the nom</a> library, which I found a real pleasure to work with due to its support for parsing incomplete data.</p>
As the content of the file is streamed through the parser, the class information is extracted and forwarded to the statistics recorder thread which keeps track of the instance counts in order to later compute the heavy hitters.</p>
Spoiler alert: as of version 0.4.7, the performance bottleneck is the parsing thread.</p>
Speeding up other stages of the pipeline would not yield any performance improvements.</p>
Test data</a>
</h2>
We are about to do some performance testing, which means we need some meaningful test data.</p>
For the sake of transparency or reproducibility, I will show you exactly how to generate a similar heap dump.</p>
To avoid completely synthetic data, we will be using Spring's REST petclinic</a>.</p>
It is a reasonable choice because it is relatively well known, and it uses an in-memory database by default which naturally amplifies the memory usage.</p>
In order to not waste too much time growing the heap, we will simply use the Java Epsilon GC</a> which does not clean up anything.</p>
This is the change to apply if you want to try this at home.</p>
git diff</span></span>
diff --git a/pom.xml b/pom.xml</span></span>
index b936a37..7621eb9 100644</span></span>
---</span> a/pom.xml</span></span>
+++</span> b/pom.xml</span></span>
@@</span> -165,6 +165,15</span> @@</span></span>
             <plugin></span></span>
                 <groupId>org.springframework.boot</groupId></span></span>
                 <artifactId>spring-boot-maven-plugin</artifactId></span></span>
+</span>                <configuration></span></span>
+</span>                    <executable>true</executable></span></span>
+</span>                    <jvmArguments></span></span>
+</span>                        -Xmx32g</span></span>
+</span>                        -XX:+UnlockExperimentalVMOptions</span></span>
+</span>                        -XX:+UseEpsilonGC</span></span>
+</span>                    </jvmArguments></span></span>
+</span>                </configuration></span></span>
                 <executions></span></span>
                     <execution></span></span>
</span></code></pre>
You could also add -XX:+HeapDumpOnOutOfMemoryError</code> if you prefer to produce a heap dump whenever an OutOfMemoryError</code> is thrown instead of performing the dump manually.</p>
In any case we can start the application using Maven.</p>
./mvnw</span> spring-boot:run</span></span></code></pre>
In order to increase the memory pressure, we will gently hammer one of the REST endpoints using Hey</a>.</p>
hey</span> -</span>m</span> POST</span> \</span></span>
 -</span>D</span> ./payload-owner.json</span> \</span></span>
 -</span>T</span> application/json</span> \</span></span>
 -</span>A</span> application/json</span> \</span></span>
 -</span>z</span> 3m</span> \</span></span>
 -</span>c</span> 50</span> \</span></span>
 http://127.0.0.1:9966/petclinic/api/owners</span></span></code></pre>
With the following payload-owner.json</code></p>
{</span></span>
  "</span>address</span>"</span>:</span> "</span>110 W. Liberty St.</span>"</span>,</span></span>
  "</span>city</span>"</span>:</span> "</span>Madison</span>"</span>,</span></span>
  "</span>firstName</span>"</span>:</span> "</span>George</span>"</span>,</span></span>
  "</span>lastName</span>"</span>:</span> "</span>Franklin</span>"</span>,</span></span>
  "</span>telephone</span>"</span>:</span> "</span>6085551023</span>"</span></span>
}</span></span></code></pre>
Once the desired heap size is reached, we can generate a heap dump with jmap</code>.</p>
jmap</span> -</span>dump:format=b,file=pets.bin</span> <</span>pi</span>d</span>></span></span></code></pre>
Which gives us a decent 34Gb heap dump file to play with.</p>
du</span> -</span>BM</span> pets.bin</span></span>
34486M</span> pets.bin</span></span></code></pre>
Let's see what hprof-slurp</code> has to say about it.</p>
./hprof-slurp</span> -</span>i</span> "</span>pets.bin</span>"</span></span></code></pre>Top 20 allocated classes:</span></span>
</span>
Total size | Instances |     Largest | Class name                                                                                 </span></span>
-----------------------------------------------------------------------------------------------------------------------------------</span></span>
   4.02GiB |  90184199 |   544.20KiB | byte[]</span></span>
   2.13GiB |  55748013 |   109.80KiB | java/lang/Object[]</span></span>
   1.72GiB |  51264762 |  36.00bytes | java/lang/String</span></span>
   1.57GiB |  14807226 | 114.00bytes | org/hibernate/validator/internal/engine/path/NodeImpl</span></span>
   1.18GiB |   8536353 | 148.00bytes | java/nio/HeapByteBuffer</span></span>
   1.03GiB |  18074217 |     3.75MiB | int[]</span></span>
1022.73MiB |   5765654 | 186.00bytes | java/lang/reflect/Method</span></span>
1021.18MiB |  26769597 |  40.00bytes | java/util/ArrayList</span></span>
 896.73MiB |  21370207 |  44.00bytes | java/lang/StringBuilder</span></span>
 838.65MiB |   7594346 |    16.02KiB | java/util/HashMap$Node[]</span></span>
 744.79MiB |   5276798 | 148.00bytes | java/nio/StringCharBuffer</span></span>
 743.57MiB |   5340350 | 146.00bytes | java/util/LinkedHashMap</span></span>
 631.70MiB |   6899814 |  96.00bytes | java/util/regex/Matcher</span></span>
 606.64MiB |   6490868 |  98.00bytes | org/hibernate/validator/internal/engine/constraintvalidation/ConstraintValidatorContextImpl</span></span>
 605.70MiB |    610435 |   128.02KiB | java/util/concurrent/ConcurrentHashMap$Node[]</span></span>
 584.97MiB |  14604425 |  42.00bytes | org/hibernate/validator/internal/engine/path/PathImpl</span></span>
 556.55MiB |  14589693 |  40.00bytes | java/util/ArrayList$Itr</span></span>
 520.24MiB |   6818920 |  80.00bytes | java/util/HashMap</span></span>
 425.14MiB |   8330358 |    32.02KiB | char[]</span></span>
 424.89MiB |   9281863 |  48.00bytes | java/util/HashMap$Node</span></span>
</span>
Top 20 largest instances:</span></span>
</span>
Total size | Instances |   Largest | Class name                                                        </span></span>
--------------------------------------------------------------------------------------------------------</span></span>
   1.03GiB |  18074217 |   3.75MiB | int[]</span></span>
   4.02GiB |  90184199 | 544.20KiB | byte[]</span></span>
 605.70MiB |    610435 | 128.02KiB | java/util/concurrent/ConcurrentHashMap$Node[]</span></span>
   2.13GiB |  55748013 | 109.80KiB | java/lang/Object[]</span></span>
 425.14MiB |   8330358 |  32.02KiB | char[]</span></span>
  13.70MiB |    211087 |  24.02KiB | long[]</span></span>
   5.81MiB |      3456 |  16.07KiB | jdk/internal/org/objectweb/asm/SymbolTable$Entry[]</span></span>
 314.02KiB |       159 |  16.07KiB | org/springframework/asm/SymbolTable$Entry[]</span></span>
 838.65MiB |   7594346 |  16.02KiB | java/util/HashMap$Node[]</span></span>
  81.18MiB |   2093066 |   9.05KiB | java/lang/String[]</span></span>
 432.84KiB |        54 |   8.02KiB | java/nio/ByteBuffer[]</span></span>
  15.83KiB |         2 |   7.91KiB | java/util/Locale[]</span></span>
  13.94MiB |    405627 |   4.67KiB | java/lang/Object[]</span></span>
  30.21KiB |        12 |   4.02KiB | net/bytebuddy/jar/asm/SymbolTable$Entry[]</span></span>
  47.46MiB |    214730 |   3.01KiB | java/lang/reflect/Method[]</span></span>
   4.23KiB |         6 |   2.97KiB | [[B[]</span></span>
   2.84KiB |         1 |   2.84KiB | java/lang/Character$UnicodeBlock[]</span></span>
   2.67KiB |         1 |   2.67KiB | jdk/internal/math/FDBigInteger[]</span></span>
   4.28KiB |         2 |   2.14KiB | sun/security/util/KnownOIDs[]</span></span>
   2.11KiB |         1 |   2.11KiB | org/springframework/boot/web/embedded/tomcat/TomcatEmbeddedContext</span></span></code></pre>
This output looks about right for an application using an in-memory database. We find the usual suspects regarding various arrays, HashMap nodes and Strings.</p>
Performance over time</a>
</h2>
It is often interesting to track the performance of a piece of software over time to be able to attribute gains to precise changes.</p>
Sometimes a single line change can have a tremendous effect, and sometimes a complete change of architecture is required to remove a bottleneck.</p>
Using the benchmarking tool hyperfine</a> we are able to measure accurately the execution time of our CLI in a blackbox fashion.</p>
For reference, I will be running the benchmarks on a laptop running Linux on an Intel i7-10610U</a> CPU.</p>
In this comparison analysis, we are interested in the relative speedup between releases and not the absolute durations, which are mostly a function of the dump size given a stable throughput.</p>
After downloading all the versions of hprof-slurp</code> into the same directory, we can compare how they handle the pets.bin</code> heap dump using the following hyperfine</code> magic incantation.</p>
hyperfine</span> -</span>-runs</span> 3</span> \</span></span>
 -</span>-export-markdown</span> hprof.md</span> \</span></span>
 -</span>-export-json</span> hprof.json</span> \</span></span>
 -</span>n</span> 0.1.0</span> "</span>./hprof-slurp-0.1.0 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.2.0</span> "</span>./hprof-slurp-0.2.0 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.2.1</span> "</span>./hprof-slurp-0.2.1 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.2.2</span> "</span>./hprof-slurp-0.2.2 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.3.0</span> "</span>./hprof-slurp-0.3.0 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.3.1</span> "</span>./hprof-slurp-0.3.1 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.3.2</span> "</span>./hprof-slurp-0.3.2 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.3.3</span> "</span>./hprof-slurp-0.3.3 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.0</span> "</span>./hprof-slurp-0.4.0 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.1</span> "</span>./hprof-slurp-0.4.1 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.2</span> "</span>./hprof-slurp-0.4.2 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.3</span> "</span>./hprof-slurp-0.4.3 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.4</span> "</span>./hprof-slurp-0.4.4 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.5</span> "</span>./hprof-slurp-0.4.5 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.6</span> "</span>./hprof-slurp-0.4.6 -i pets.bin</span>"</span> \</span></span>
 -</span>n</span> 0.4.7</span> "</span>./hprof-slurp-0.4.7 -i pets.bin</span>"</span></span></code></pre>
Which - after a long time - yields the following Markdown table in hprof.md</code>.</p>
Command</th> Mean [s]</th> Min [s]</th> Max [s]</th> Relative</th></tr></thead>

0.1.0</code></td> 131.227 ± 4.072</td> 126.660</td> 134.475</td> 3.83 ± 0.12</td></tr>
0.2.0</code></td> 119.450 ± 0.244</td> 119.206</td> 119.694</td> 3.49 ± 0.03</td></tr>
0.2.1</code></td> 119.038 ± 0.513</td> 118.452</td> 119.405</td> 3.47 ± 0.03</td></tr>
0.2.2</code></td> 92.710 ± 0.893</td> 91.685</td> 93.324</td> 2.71 ± 0.03</td></tr>
0.3.0</code></td> 84.097 ± 0.159</td> 83.953</td> 84.268</td> 2.45 ± 0.02</td></tr>
0.3.1</code></td> 81.579 ± 0.073</td> 81.497</td> 81.639</td> 2.38 ± 0.02</td></tr>
0.3.2</code></td> 81.255 ± 0.211</td> 81.028</td> 81.445</td> 2.37 ± 0.02</td></tr>
0.3.3</code></td> 77.121 ± 0.228</td> 76.937</td> 77.376</td> 2.25 ± 0.02</td></tr>
0.4.0</code></td> 67.911 ± 0.979</td> 66.826</td> 68.728</td> 1.98 ± 0.03</td></tr>
0.4.1</code></td> 65.925 ± 0.344</td> 65.645</td> 66.309</td> 1.92 ± 0.02</td></tr>
0.4.2</code></td> 65.779 ± 0.240</td> 65.602</td> 66.053</td> 1.92 ± 0.02</td></tr>
0.4.3</code></td> 63.398 ± 0.104</td> 63.291</td> 63.499</td> 1.85 ± 0.02</td></tr>
0.4.4</code></td> 35.473 ± 0.047</td> 35.444</td> 35.527</td> 1.04 ± 0.01</td></tr>
0.4.5</code></td> 34.695 ± 0.302</td> 34.440</td> 35.028</td> 1.01 ± 0.01</td></tr>
0.4.6</code></td> 34.285 ± 0.048</td> 34.254</td> 34.340</td> 1.00 ± 0.01</td></tr>
0.4.7</code></td> 34.264 ± 0.289</td> 34.000</td> 34.574</td> 1.00</td></tr>
</tbody></table>
And using the hprof.json</code> file and some python scripts</a> we generate the following whisker graph.</p>

    


    
    
</figure>
Some observations:</p>

the throughput increased by more than 70%</li>
0.1.0 has a large variance (might be a warmup issue)</li>
starting from 0.4.5 the iterative improvement is minimal</li>
</ul>
In any case, we reached a solid 1Gb/s throughput in the latest version given the size of our test file.</p>
Let's have a look at the peak memory consumption for 0.4.7 using Heaptrack</a>.</p>
heaptrack</span> ./hprof-slurp-0.4.7</span> -</span>i</span> pets.bin</span></span></code></pre>
This opens up the UI directly after the run if it is installed.</p>

    


    
    
</figure>
The memory usage is pretty stable; it seems we have been able to stream the whole 34 Gb file within 500Mb.</p>
Most of it is actually due to the various internal buffers which are used to communicate between the threads.
Those are pre-allocated and can grow to a certain size before being shrunk to release the memory.</p>
One could expect a stable similar memory usage for much larger dumps unless those contain large arrays of primitives or objects which need to be parsed.</p>
Future work</a>
</h2>
Obviously, I am still interested in making hprof-slurp</code> faster — hopefully not at the expense of code readability.</p>
Outside the performance concern, I believe the output could still be improved to be more useful.</p>
First, I would like to validate the precision of the statistics reported by comparing the output of hprof-slurp</code> against the existing tools. My gut feeling is that the numbers are not too far off and serve as a pretty good proxy.</p>
Moreover, additional extractions could be performed as long as it happens in a single pass, such as rendering the complete thread stack traces at the moment of the heap dump.</p>
Feel free to reach out</a> if you have suggestions.</p>
Conclusion</a>
</h2>
This article has shown that hprof-slurp</a> has gotten faster over time and that it is now processing heap dumps at around 1Gb/s.</p>
It fulfills the initial goal, which was to offer a quick overview of large heap dumps using a regular developer machine.</p>
The next articles in this series</a> will go through a selection of the most interesting optimizations that had the largest impact on performance.</p>
In the meantime, you can try to reproduce those results or, even better, analyze larger real-world heap dumps!</p>
Arnaud Gourlay's blog

Pinch Points - a Rust game with crabs

The transmission

New Key-Value Store in Rust at Qdrant

Playing guitar tablatures in Rust

Rust Meetup Linz: Qdrant - a vector search engine in Rust

Follow up on cracking ZIP archives in Rust

Support for thread dump in hprof-slurp 0.5.0

Brute forcing protected ZIP archives in Rust

Graceful shutdown</a> </h2> Looking at the code more closely, we can see that we terminate our threads only under specific conditions.</p> For the password dictionary thread:</p> If the channel has no consumers attached.</li>

A performance retrospective using Rust (part 3)

A performance retrospective using Rust (part 2)

A performance retrospective using Rust (part 1)

Graceful shutdown
