Guru Das Srinagesh

Nand2Tetris - Project 8 (VM Translator Part 2)

2025-11-08T21:19:59-08:00

In Project 8 of Nand2Tetris, we get to finish the implementation of the VM Translator that was begun in Project 7. Having gotten the hang of translating arithmetic and logical VM commands from there, we now focus on translating the remaining VM commands that deal with control flow and function call flow - specifically, label, goto, if-goto, function, call and return.

This completes the implementation of the VM Translator, i.e. the Jack language compiler's backend – but let's not get ahead of ourselves. This project was an order of magnitude more involved to put together than Project 7 because it's really three tasks in one, in decreasing order of difficulty:

Translate the control flow VM commands.
Translate the function call flow VM commands.
Two parts:
- (a) Handle both a single .vm file as input as well as a directory containing .vm files.
- (b) When translating the multiple .vm files together from a directory, add "bootstrapping" code as preamble.

As always, the code is here: guru-das-s/nand2tetris. In hindsight, the generated assembly "phrases" could indeed be better optimized and not so verbose, but that's a task for another day.

Table of Contents

Preparatory refactoring
Task 1: Emit code for the control flow VM commands
- The label and goto commands
- The if-goto command
Task 2: Emit code for the function-related VM commands
Task 3: Emit bootstrapping code
Reflections
Test script
Postscript

Preparatory refactoring

While scoping out the changes required to support the control flow statements, I realized that thus far, the first argument to a VM command was geared to support only a Segment ¹ and not a named label or a function name. This was remedied with the help of the todo!() macro to get things compiled and moving.

Task 1: Emit code for the control flow VM commands

The `label` and `goto` commands

The codegen for these two commands are trivial and are almost natively supported by the asm syntax without the need for any extra scaffolding.

pub const LABEL: &str = r#"// LABEL
(XYZ)
"#;

pub const GOTO: &str = r#"// GOTO
@XYZ
0;JMP
"#;

The `if-goto` command

The syntax of this command is:

gt // Or any other logical comparison operator
if-goto LABEL

In hindsight, I implemented this command in a very roundabout fashion:

pub const IF_GOTO: &str = r#"// IF_GOTO
// first, get results of prev bool op
@SP
M=M-1
A=M
D=M
@DONTJUMP.XYZ
D;JEQ
@LABL
0;JMP
(DONTJUMP.XYZ)
"#;

That is, if the result of the previous boolean operation is false, jump over the part where we do the unconditional jump to the target label; else, take the jump. This can be simplified to "take the jump if the result is true":

...
@LABL
D;JNE
"#;
...

Changes to phrases that occur within a function

The codegen for the control flow VM commands, and phrases that use phrase-local labels, are both affected by their occurrence within a function body.

The symbol names need to be made unique; otherwise, the assembler will not resolve them to the intended addresses ². This is made by prefixing label symbol names and phrase-local labels by the function's name.

pub const LABEL_IN_FUNC: &str = r#"// LABEL_IN_FUN
(FUNC$XYZ)
"#;

pub const GOTO_IN_FUNC: &str = r#"// GOTO_IN_FUN
@FUNC$XYZ
0;JMP
"#;

The `function` command

The syntax of this command is:

function functionName nVars

with the following semantics:

Here starts the declaration of a function that has name functionName and nVars local variables

The codegen for this command followed as two parts: to first emit a label containing the function name, and then the emission of as many local variables as is specified. This forms the "function frame" on the stack.

let mut function_phrase = phrases::FUNCTION.replace("FUNC", func_name);
function_phrase += &phrases::FUNCTION_LOCAL_VAR.repeat(num_local_vars as usize);
Ok(function_phrase)

The phrases are listed in phrases.rs.

The `call` command

The syntax of this command is:

// <numArgs> arguments to functionName have been pushed to stack
call functionName numArgs

Before we call a function, we must first know where to return to after executing that function. This necessarily has to be the first instruction following the call. How do we determine this address and use this in the emitted asm?

This may seem like a circular problem in asm land because at this point in the (to-be-translated) asm code...

VM land             asm land
-------             --------
call multiply       <to-be-emitted call code>
                    <to-be-emitted call code>
                    <to-be-emitted call code>
                    ...
                    <to-be-emitted call code>

pop temp 0          ADDRESS_TO_JUMP_TO_AFTER_CALL: <pop code follows>

...how do we know the ADDRESS_TO_JUMP_TO_AFTER_CALL if we haven't finished writing the implementation of call yet?

The solution is to preemptively add a "return label" to the end of the call asm, say, @CALLER$ret and then save that label name to the stack as the first step. Remember, we can easily use a symbol that is defined only later because the assembler will trivially take care of it in its first pass where it resolves symbol names to addresses.

And since we can call any (CALLEE) function more than once from a given (CALLER) function, we need to make the "return label" unique, so we need to tack on a monotonically-increasing number to it: @CALLER$ret.XYZ (we will replace the XYZ with the number in code).

The rest of the steps are to:

save the current context (registers) to the stack
modify the ARG pointer-register to point to the start of the numArgs number of function arguments
that were pushed to the stack prior to the call
modify the LCL pointer-register to point to the top of the stack for the callee function's use.
actually jump to the callee function's code and start executing it

The implementation of call is listed in phrases.rs.

The `return` command

The syntax of this command is:

return // no arguments

This trivial-seeming command is the yin to the yang of call. It does a ton of heavy lifting from its vantage point in the callee's code:

Replacing the caller-pushed function arguments with the function's return value
Invalidating the callee's function frame in the stack
Restoring the (register) context saved on the stack by the caller, and finally,
Jumping to the "return label", the instruction right after the call.

The implementation of return is listed in phrases.rs.

Task 3: Emit bootstrapping code

When translating a single file, the corresponding .tst test script provided in the project src directory sets up the stack pointer to its value in the standard mapping (256) and takes care of setting up any arguments as necessary. When translating multiple files, however, we need to add this bootstrapping code first thing before any other translations can take place.

The bootstrapping code sets SP to 256 and then calls Sys.init.

The implementation of the bootstrap is listed in phrases.rs.

Reflections

Any sufficiently advanced technology is indistinguishable from magic

- Arthur C. Clarke

There are broadly three parts to a magic trick: the pledge, the turn and the prestige:

the pledge forms the setup of the trick, the mise en place. The audience is shown something ordinary: a rabbit sitting on a table.
the turn generates great surprise: the rabbit disappears!
the prestige is the climax of the trick and provides resolution: the rabbit reappears from a hat. The audience marvels at the trick and wonders how it was achieved.

Allowing for some poetic licence, this basic structure may be applied to our understanding of the implementation of a function call.

function as the pledge: We know that multiply() multiplies two numbers together.
call as the turn: Execution jumps from the caller to the callee and the multiplication takes place!
return as the prestige: We magically return to wherever we left with the result of the multiplication ready and waiting for us in the caller's context.

Only, we now know how this was achieved.

Test script

Following the example of the test scripts added in the previous two projects, I added one for Project 8 too.

$ ./projects/8/test.sh
   Compiling nand2tetris v0.1.0 (/home/gurus/projects/nand2tetris)
    Finished `release` profile [optimized] target(s) in 2.84s
     Running `target/release/vmt -i projects/8/ProgramFlow/BasicLoop/BasicLoop.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -i projects/8/ProgramFlow/FibonacciSeries/FibonacciSeries.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -i projects/8/FunctionCalls/SimpleFunction/SimpleFunction.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -i projects/8/FunctionCalls/NestedCall/`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -i projects/8/FunctionCalls/FibonacciElement/`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -i projects/8/FunctionCalls/StaticsTest/`
End of script - Comparison ended successfully

Postscript

The code was complete in August and I had begun work on this post shortly thereafter. Life intervened (many times) as I tried to make progress on finishing this post, and I could only publish this blog post now. No AI tools or LLMs were used in writing this post.

An example to refresh one's memory: the segment in the push local 5 VM command is local. ↩
Described here in more detail. ↩

TIL: Firefox supports back and forward touchpad gestures by default

2025-02-22T23:02:00-08:00

Firefox supports the two-finger left and right swipe actions on the touchpad by default on Ubuntu.

Chrome does not.

I have now adopted Firefox as my default browser.

Nand2Tetris - Project 7 (VM Translator Part 1)

2025-01-01T09:58:42-08:00

Project 7 of Nand2Tetris is about translating virtual machine ¹ (VM) commands for the Hack computer platform to a series of Hack assembly language (asm) instructions. The main focus of this project is determining the sequence of assembly language instructions corresponding to each VM command, given that the programming model of the computer is that of a stack machine. Here, VM commands are roughly equivalent to bytecode. The virtual machine translator that emerges from this project shall form the backend of the compiler that shall be built upon it.

Continuing my Rust journey from Project 6, this blogpost describes in detail my solution to Project 7 in Rust.

The source code is here: guru-das-s/nand2tetris

Table of Contents

Problem description
VM Translator – Code architecture
Development and testing workflow
Interesting problems
What I learnt about Rust programming
Test script
Postscript

Problem description

What does a VM file even look like? Here's SimpleAdd.vm:

// Pushes and adds two constants.

push constant 7
push constant 8
add

This is the "bytecode" that the Jack language compiler (which will be introduced later!) will compile to - the first of a two-step process of converting high-level Jack language code to Hack asm code.

The goal of this project is to translate the above VM instructions to asm instructions that effect the intended operations: push and add in the above example. A stack area is assigned for us in the RAM of the Hack machine to be used in the asm instructions. We are responsible for growing and shrinking the stack.

There are code segments other than constant, e.g. BasicTest.vm:

...
pop temp 6
push local 0
push that 5
add
push argument 1
sub
...

These segments all map to various designated sections of RAM. The local segment starts from the RAM address stored in the LCL RAM address which, as we know from the Assembler design from Project 6, is a synonym of RAM[1]. The N in local N specifies the offset that must be applied to the base address stored in LCL, i.e. RAM[LCL + N]. That's just local, though – other segments must be interpreted differently as specified in the design instructions.

There are also operations other than add such as sub in the above example, all of which operate on the stack. Being a binary operator needing two operands, add adds the two top-most entries present on the stack. A unary operator such as not only works on the top-most entry on the stack.

VM Translator – Code architecture

The basic idea animating the design of the VM Translator is twofold:

having stock "phrases" of asm code corresponding to each combination of arithmetic operation, segment, and VM command that are crafted with placeholder-stubs in place of the actual values involved in the computation of the offsets required, and
replacing those stubs with the actual correct values derived from parsing the vm commands.

The code is, thus, organized into five modules:

#	Module	Function
1	`main.rs`	Handles command line args, calls `parser` and `asmwriter`
2	`spec.rs`	Defines basic enums and structs to represent parsed VM commands
3	`phrases.rs`	Stubbed-out asm code fragments for every operation and code segment
4	`parser.rs`	Parses a `.vm` file and returns a `Vec` of `VmCommand`s
5	`asmwriter.rs`	Writes the asm code for each `VmCommand` in the `Vec` to the output file

Development and testing workflow

The first step was to figure out the "phrases", i.e. the sequence of Hack asm instructions, corresponding to a given vm command such as push constant 7 and add. But where does one start?

One starts by observing what is actually happening in the memory space when those commands are being run. The stock VMEmulator.sh provides a powerful and convenient GUI readout of the contents of the stack, RAM memory, the various code segments of the Hack programming model and the ability to step through each VM command. Every sample vm program provided has a corresponding *VME.tst that runs it on the VM Emulator.

When the behaviour of the command under scrutiny is thus understood, the next logical step is to actually write down – by hand – the asm instructions that make those commands happen, and then run them on the stock CPUEmulator.sh that we encountered in Project 4. This requires simulating the test setup conditions that the *VME.tst script uses.

For example: While working on the first test (SimpleAdd.vm), I ran SimpleAddVME.tst on the VM Emulator and observed the (rather simple) effects on the Stack Pointer and the Stack. I then inspected the VM test script and recreated what it was doing by hand-coding its setup phase in asm – viz., initializing the Stack Pointer to a small enough value that would be visible in the CPUEmulator without having to scroll down. With this setup, I was able to write the spectacularly-named handcode/stuff.asm (hehe) that helped me hash out the asm instructions to push a few arbitrarily-chosen constants and implement the binary operator add and unary operator not for them, testing everything on the CPUEmulator. I followed a similar approach for the other tests and operations.

The next step was to generalize the phrases from the hand-coded asm files by stubbing out the arbitrary values with placeholder text "XYZ" and code segment names with "SEG" that would be substituted with the values from the parsed .vm files by the VM Translator being developed.

After hooking up the parser logic and asm writer logic in the VM Translator, the final step was to run the *.tst script for each test that would run the asm output file generated by the VM Translator on the CPUEmulator and return success or failure. An intermediate step was also to compare the asm output file with the one generated by the stock VMEmulator.

Interesting problems

Labels must be unique for conditional operator phrases

This problem was encountered while working on the second test, StackTest.vm. Fresh from having figured out the basic approach of the overall project in the first test, SimpleAdd.vm, I plugged in the phrases for other arithmetic and conditional operators, including Equals, Lesser Than and Greater Than, only to eventually find that the assembler was not translating the labels within those phrases in the expected manner.

To illustrate the problem: the hand-coded asm implementation of Equals is as follows:

// Implement Eq
// ------------

// Get first parameter and store it in D
@SP
M=M-1
A=M
D=M
// D has first parameter now. Get second parameter next
@SP
M=M-1
// Second parameter is in RAM[SP], store it in A
// after dereferencing RAM[SP] which is a pointer
A=M
// M has second parameter now.
D=D-M
@SP
A=M
M=-1
@ISEQUAL
D;JEQ
// We will get here only if
// D is not zero, i.e. the two numbers
// are dissimilar.
@SP
A=M
M=!M
(ISEQUAL)

Notice the definition and use of the label ISEQUAL. The expectation is that this label will resolve to the ROM address of the instruction right after (ISEQUAL).

The phrases for the other two conditional operators mentioned above also follow a similar structure, each using a label to conditionally skip over the result.

The problem occurs when the same phrase is plugged in multiple times in the same file as the result of multiple occurrences of a conditional operator in the .vm file being translated. When this happens, the label (ISEQUAL) is defined multiple times and the assembler resolves it to the ROM address of the instruction after the very last occurrence of the label. This leads to incorrect jump addresses for all the other instances of the operator that come before the last occurrence.

It is thus evident that the label in the phrases for conditional operators have to be made unique so that the assembler can resolve the labels separately. One way to make this happen is to append a monotonically increasing variable to the label used in the phrases. This implies the use of a local static variable (in C terms). Example: instead of (ISEQUAL), use (ISEQUAL.0), (ISEQUAL.1), (ISEQUAL.2), et c.

I implemented local static variables in Rust using OnceLock²:

static E: OnceLock<Mutex<u16>> = OnceLock::new();
...
let e = E.get_or_init(|| Mutex::new(0));
...
let mut eq = e.lock().unwrap();
...
*eq += 1;
...

Correspondingly, the label in the phrase was amended to (ISEQUAL.XYZ) with the above code replacing XYZ with the static variable eq³.

To increment SP, or not to increment SP?

The project instructions caution against forgetting to increment the Stack Pointer (SP). The SP is to be incremented only for the arithmetic and logic operations phrases, and not for the phrases that set each code segment's offset and calculate its address.

Implementing the `pop` operation

Implementing the pop segment i operation was initially difficult to do without the use of extra scratch registers – my first attempt utilized two. But something seemed fishy as running BasicTestVME.tst through the VMEmulator did not show those two registers being touched at all:

// Pop local 2 begins now
// ........................
// Step 1: Get value to pop
@SP
M=M-1
A=M
D=M
// Store it in first scratch register
// Only R13, R14 and R15 are free to use
// in my view, the TEMP segment taking up
// R5 - R12 for itself.
// The BasicTestVME.tst does not seem to be
// using R13 - R15 in my test run, or it
// must be cleverly clearing them to not
// give away this insight.
...
// Now, triumphantly do *(M[R14]) = M[R13]
// (R14 is a pointer)
@R13
D=M
@R14
A=M
M=D

It wasn't until later that it dawned on me that I could just use the stack instead.

// Pop local 2 begins now
// ........................
// Step 1: Get data to pop
@SP
M=M-1
A=M
D=M
// Step 2: Push this data to stack
@SP
M=M+1
A=M
M=D
// Now last two elements in stack
// are the same, with SP pointing
// to the last NON-EMPTY element.
// Set SP to the second-to-last spot.
@SP
M=M-1
// Step 3: Calculate address to pop to
@LCL
D=M
@2
D=D+A
// Set SP to this value
@SP
A=M
M=D
// Increment SP
@SP
M=M+1
// Now last two elements of stack are
// (N-1) ----- address
// (N)   ----- data     <SP points here>
// Now, finish it off
@SP
A=M
D=M
@SP
M=M-1
A=M
A=M
M=D
// Now SP points to right place.

This, then, translated to a two-part phrase for the operation: a preamble and the actual operation.

Implementing the `static` code segment

The static i code segment needs to be translated to a variable @FILE.i where FILE is the filename of the VM file being translated. Because of the good level of code encapsulation in the project, it was possible to implement it like this:

// XYZ will be handled by VmCommand::code_segment_i(), and
// FILE will be handled by Asmwriter::write().
pub const STATIC: &str = r#"// Push Static XYZ
@FILE.XYZ
A=M
D=A
"#;

This is the only point in the project (so far!) that the replacement logic spills over to an unrelated module (Asmwriter):

    pub fn write(mut self) -> Result<(), Box<dyn Error>> {
        for vmcmd in self.vmcmds {
            let mut asm_code = vmcmd.code()?;

            asm_code = if vmcmd.arg1.is_some_and(|segment| segment == Segment::Static) {
                asm_code.replace("FILE", self.filename)
            } else {
                asm_code
            };

            writeln!(self.file, "{}", asm_code)?;
        }
        Ok(())
    }
}

What I learnt about Rust programming

New things I learnt about Rust and implemented in Project 7 that I did not in the previous Project 6:

Followed the idiomatic way of defining a constructor for struct Parser and struct Asmwriter structs and, thus, keeping the main module simple.
Used the iterator methods peekable() and peek() to keep track of the current line in the Vec of lines constructed from BufReader. This helped the design conform to the suggested API of advance() in the parser.
Used lifetimes for the first time to yoke the lifetime of a raw string to the struct it was embedded in (struct Asmwriter) and to store the Peekable() line iterator in struct Parser.
Used the raw string literal r#"<string>"# syntax to define multiline strings for each of the phrases. This made the asm sequences easy to write and iterate on.
Added tests for the very first time: only for the parser, though – didn't have time for any other module.

Test script

I wrote a short shell script to ensure that the VM Translator passes all test cases.

$ ./projects/7/test.sh && echo "All test cases passed"
    Finished `release` profile [optimized] target(s) in 0.14s
     Running `target/release/vmt -f projects/7/MemoryAccess/BasicTest/BasicTest.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -f projects/7/MemoryAccess/PointerTest/PointerTest.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -f projects/7/MemoryAccess/StaticTest/StaticTest.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -f projects/7/StackArithmetic/SimpleAdd/SimpleAdd.vm`
End of script - Comparison ended successfully
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/vmt -f projects/7/StackArithmetic/StackTest/StackTest.vm`
End of script - Comparison ended successfully
All test cases passed

Postscript

I find that writing code is easy while blogging about it is hard and seemingly more time-consuming. Perfection is the enemy of progress, so I would much rather have an imperfect blogpost that gets published in a reasonable amount of time than a fully-fleshed one that takes forever to be published.

The next project, Project 8, extends the VM Translator to add program flow constructs such as IF, RETURN, FUNCTION, GOTO, et c. Looking forward to hacking on this in the new year!

No, not that virtual machine; rather, the Hack computer virtual machine conceptualized as a stack machine. ↩
The use of the Mutex may be unnecessary and even overkill as this program is only single-threaded. Using std::sync::atomic may be sufficient. ↩
Code excerpt showing labels in conditional phrases being made unique. ↩

TIL: The fzf fuzzy finder is amazing

2024-12-14T17:28:27-08:00

fzf is a general-purpose fuzzy text matching tool at its core that takes in a list of newline-separated text and allows the user to search for something within those lines and select one or more of the lines by typing just a few character sequences into a search field. It is highly customizable and the maintainer seems to have a good head on his shoulders. I recently started using it after a colleague introduced me to it and it has improved my developer workflow tremendously.

Here are some of the actions and workflows I use fzf for, multiple times on a daily basis.

Opening files in vim within current directory

From the shell, it is easy to search for a file within the current directory (recursively) and open it in vim. I have this Bash alias that does that for me with just one keystroke:

alias f='vim $(fzf)'

It is easy to do this from within an open vim session as well. For this, I use the fzf.vim plugin with these lines in my .vimrc:

nnoremap <C-f> :GFiles<CR>
let g:fzf_action = {
        \ 'ctrl-l': 'split',
        \ 'ctrl-h': 'vsplit' }

The above config allows me to use Ctrl + F from within vim to bring up a list of all the files known to Git in that directory (GFiles = Git Files). From the plugin's documentation, one can remap that key combination to just :Files as well for directories that are not git repositories. I can then search for the file I want and then open it in a horizontal or vertical split with Ctrl + L or Ctrl + H.

ag integration from within vim

I use ag (the "silver searcher") instead of grep, and the fzf.vim plugin supports this natively. I just had to make a couple of customizations for my convenience:

I renamed the invocation command from :Ag to :A, thereby saving one precious keystroke.
The vimscript helper function that fashions the invocation parameters to ag sets the search query to '^(?=.)' by default if a search term is not provided. The effect of this is that ag will print out every single line of every single file in the repo if :A is invoked as such without the search parameter, which makes no sense.
So instead, I changed the default search query to expand("<cword>") in case of an empty search query. The effect of this is that I can just place my cursor on a word and then say :A to search for that word, which is highly effective.

Bash alias to add filename in REPL prompt

Adding this line to ~/.bashrc adds the default fzf bindings for bash:

[ -f ~/.fzf.bash ] && source ~/.fzf.bash
# Set up fzf key bindings and fuzzy completion
eval "$(fzf --bash)"

Now, Ctrl + T brings the power of fzf to locating deeply-nested build artifact files and using them in a command you're building on the fly at the REPL shell prompt.

Bash alias for git log

The single most frequent use case is browsing git log output with fzf. This is how I've hooked things up - and a warning: it's not that pretty to look at, but gets the job done, and well.

export FZF_O_OPTS="--ansi --color=16 --no-mouse --multi --track --tac --disabled --no-print-query --bind j:down,k:up,q:abort"

function do_o() { git log --color=always --oneline --decorate ${2:-} ${1:-} | fzf $FZF_O_OPTS --preview "git show --color=always {1}"; }
function do_O()
{
    local num_commits="${2:-}"
    local path=${1:-}
    local commit=$(do_o "$path" "$num_commits" | awk "{print \$1}")
    if [[ -n $commit ]]; then
        git show --color=always -U10 --stat $commit
    fi
}

function o() { local path=${1:-}; do_O "$path" "-10"; }
function oo() { local path=${1:-}; do_O "$path" "-20"; }
function ooo() { local path=${1:-}; do_O "$path" "-30"; }
function O() { local path=${1:-}; do_O "$path"; }

Saying o at the terminal thus produces a fzf window containing the 10 most recent commits with the highlighted commit git-shown in the preview pane for easy review. One can easily cycle up and down the commits via the j and k keys, vim-style.

Since I do not care about searching for a specific commit in this mode, I pass --disabled to the fzf invocation options. Instead, I've chosen to make browsing through the list easier by binding the j and k keys vim-style, saving on having to hold Ctrl down otherwise.

If there is a specific commit I want to inspect further, I simply finalize the fzf selection via Enter, which then prints out the whole commit with git show -U10 --stat for full review.

Since I typically make lots of tiny commits as I develop a feature, I end up with dozens of atomic commits that I later would need to squash together or prune, which requires reviewing all of them in rapid succession. For this reason, I have the functions oo() and ooo() that show me the latest 20 and 30 commits, respectively, with the same behaviour as above. Finally, there's also O() (capital O) that just lists all the commits of the repo without any limits.

I came across the fzf maintainer's suggested plumbing for this same feature, but I have not tried it out personally as I am quite happy with my own solution.

Nand2Tetris - Project 6 (Assembler)

2024-10-13T19:32:02-07:00

I wrote a two-pass assembler for the Hack assembly language!

This is the sixth and final project of the first half (Hardware) of the Nand2Tetris course. In Project 6, the task is to write an assembler for the Hack assembly language. Unlike the previous projects which could only be written using the custom hardware description language (HDL) used by the Nand2Tetris project, Projects 6 to 12 allow the use of any programming language that is convenient. I chose to write the assembler in Rust because I wanted to continue building substantial things with it and learn more of its features in the process.

Follow my progress on Github here: guru-das-s/nand2tetris.

Table of Contents

Introducing: The has assembler
Code walkthrough
- Main data structures used
- Code flow
Does has output match the stock assembler's?
Post-completion thoughts

Introducing: The `has` assembler

My Hack assembler is named has - a simple portmanteau of the two words. It has the following calling card:

$ cargo run --release --bin has -- --help
    Finished `release` profile [optimized] target(s) in 0.12s
     Running `/home/gurus/projects/nand2tetris/target/release/has --help`
Hack assembler

Converts Hack assembly files to Hack binary

Usage: has [OPTIONS] --filename <FILENAME>

Options:
  -f, --filename <FILENAME>
          Path to Hack ASM file (e.g. Max.asm)

  -o, --output <OUTPUT>
          Output filename (e.g. Max.hack)

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

The only required argument to the assembler is the input filename. The default output filename is the input filename with the extension replaced by .hack unless overridden by the --output flag.

Defining a separate binary [[bin]] for the assembler in Cargo.toml seemed a natural choice to keep using this same repository for all future projects and easily build separate binaries for them:

[[bin]]
name = "has"
path = "projects/6/has/src/main.rs"

Code walkthrough

I followed the same sequence of developing the Hack assembler as suggested in the project docs - first, developing the ability to assemble symbol-less asm files and then adding a symbol table and enabling the handling of symbols too. Source code here.

There are four modules in the project in addition to main.rs:

$ tree projects/6/has/src/
projects/6/has/src/
├── main.rs
├── parser.rs
├── spec.rs
├── symboltable.rs
└── to_binary.rs

Main data structures used

The fact that there is a finite number of valid values for each of the fields in a C-instruction and a max value for the number contained in an A-instruction led me to choose enums as my fundamental data structure to base all the logic on.

The members of Rust enums have two forms:¹

Regular C-like enum members/variants that are just equated to constant values, and
Arbitrary constructions of structs/tuples/enums of any datatype!

This makes it really intuitive to represent the various forms that a valid line of Hack assembly code can take:

pub enum HackLine {
    Whitespace,
    Comment,
    Label {
        label: String,
    },
    A {
        value: u16,
    },
    Variable {
        name: String,
    },
    C {
        dest: Option<Destination>,
        comp: Option<Comp>,
        jump: Option<Jump>,
    },
}

I chose to represent a raw A-instruction (with a value spelt out explicitly) and a variable separately even though the latter is also technically a valid A-instruction. The members of HackLine::C are all Options in order to differentiate erroneous None values from valid <enum>::Null values.

The HackLine enum is pub because I'm using modules to encapsulate code cleanly as separate files and I need the enum to be visible to all modules in the project. This enum lives in the parser.rs module.

Destination and Jump are C-style enums. Excerpts edited for brevity:

pub enum Destination {
    Null = 0,
    M = 1,
    D = 2,
    ...
}
pub enum Jump {
    Null = 0,
    JGT = 1,
    JEQ = 2,
    JGE = 3,
    ...
}

Comp is also one, too, but with a custom impl that defines helper functions to map synonym enum variants such as Comp::A and Comp::M to the same hex values and figuring out the corresponding a-value for an enum variants.

pub enum Comp {
    Zero,
    One,
    MinusOne,
    D,
    A,
    M,
    NotD,
    NotA,
    NotM,
    MinusD,
    ...
}
impl Comp {
    pub fn to_u8(&self) -> u8 {
        match self {
            Comp::Zero => 0b101010,
            Comp::One => 0b111111,
            Comp::MinusOne => 0b111010,
            Comp::D => 0b001100,
            Comp::A | Comp::M => 0b110000,
            Comp::NotD => 0b001101,
            Comp::NotA | Comp::NotM => 0b110001,
            ...
        }
    }
}

This impl was required because Rust does not allow multiple enum variants to be equated to the same numeric value.

These enums all live in the spec.rs module - which, originally, was intended to capture all the specifications of the Hack asm language but now appears to contain only the specifications of the C-instruction, the A-instruction being relatively trivial to represent.

The symbol table is represented as follows, and it lives in its own symboltable.rs module, along with its attendant impl methods ².

pub struct SymbolTable {
    next_free_ram_address: u16,
    m: HashMap<String, u16>,
}

Code flow

Starting from the main.rs entry point:

After the clap crate does its thing and parses command-line arguments passed to the program, we read the contents of the input asm file into a Vec and assemble() it, passing to assemble() a mutable reference to a symbol_table duly initialized with known arch-specific symbols.

assemble() shows the two-pass architecture of the assembler. Why are two passes required? It is primarily to handle labels in the asm source file that are forward references, i.e. the first use of the label in an instruction occurs before its definition, thus making it impossible to know at that first-use location what code memory location to translate the label to.

So in the first pass, the input file is iterated through line-by-line and only HackLine::Labels are processed, adding labels to the list of known symbols in the symbol_table and storing their code memory locations in the symbol table's hashmap. In the second pass, each line is parsed, including the labels and variables and converted to its binary representation.

The parser.rs module parses each line and returns the HackLine enum variants it maps to, or an Err() if the line is not a valid Hack asm line.

The to_binary.rs module then takes in the parsed line, i.e. the HackLine enum type and converts it to its binary representation. Here, Rust's automatic destructuring of enum types allows for easily accessing the parsed data of the HackLine enum type.

pub fn binary_of(line: HackLine, symboltable: &mut SymbolTable) -> Option<String> {
    match line {
        HackLine::Whitespace | HackLine::Comment | HackLine::Label { .. } => None,
        HackLine::A { value } => Some(binary_of_a_type_instruction(value)),
        HackLine::C { dest, comp, jump } => Some(binary_of_c_type_instruction(
            dest.unwrap(),
            comp.unwrap(),
            jump.unwrap(),
        )),
        HackLine::Variable { name } => Some(binary_of_variable(name, symboltable)),
    }
}

The implementation of binary_of_variable() bears illustration here (the other two functions are relatively straightforward):

fn binary_of_variable(variable: String, symbol_table: &mut SymbolTable) -> String {
    if !symbol_table.is_known(&variable) {
        symbol_table.add_new_variable(&variable);
    }
    let value = symbol_table.get_variable_address(&variable);
    return binary_of_a_type_instruction(value);
}

Does `has` output match the stock assembler's?

With the assembler in place, I also wrote a shell script to verify that it matches the output of the stock assembler that is provided by the Nand2Tetris team in the repo. It first runs the stock assembler, tools/Assembler.sh, on all the .asm files in the Project 6 directory to generate the "ground truth" .hack files. The stock assembler converts a Hack asm file named, say, Test.asm to Test.hack.

Next, I run my has assembler on the same files, taking care to specify a different suffix to the output binary in order to distinguish it from the ground truth files. The has assembler converts a Hack asm file named, say, Test.asm to Test.has.hack.

These two files are diff-ed in the next step. Since the set -e command is set in the test script, any mismatches in the files will cause the diff command to error out and, consequently, the test script itself. Let's run the test script to check:

$ ./projects/6/test.sh && echo "No diffs!"
Assembling /home/gurus/projects/nand2tetris/projects/6/rect/Rect.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/rect/RectL.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/max/Max.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/max/MaxL.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/add/Add.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/pong/Pong.asm
Assembling /home/gurus/projects/nand2tetris/projects/6/pong/PongL.asm
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/rect/Rect.asm -o projects/6/rect/Rect.has.hack`
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/rect/RectL.asm -o projects/6/rect/RectL.has.hack`
    Finished `release` profile [optimized] target(s) in 0.03s
     Running `target/release/has -f projects/6/max/Max.asm -o projects/6/max/Max.has.hack`
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/max/MaxL.asm -o projects/6/max/MaxL.has.hack`
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/add/Add.asm -o projects/6/add/Add.has.hack`
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/pong/Pong.asm -o projects/6/pong/Pong.has.hack`
    Finished `release` profile [optimized] target(s) in 0.02s
     Running `target/release/has -f projects/6/pong/PongL.asm -o projects/6/pong/PongL.has.hack`
No diffs!

Yes - no diffs.

The script also cleans up after itself using the Bash trap <command> EXIT method. This causes <command> to run at the end of the Bash script's execution - either graceful or erroneous. In this case, I clean up all the generated .hack files in a cleanup() routine upon EXIT.

Post-completion thoughts

Writing this assembler was really rewarding not just in terms of learning how a basic two-pass assembler works but also because it afforded me the means and opportunity to learn more about Rust:

Rust enums and their automatic destructuring in matches
Rust associated methods vs plain functions in impls
Smartly using None to indicate an error condition that should not happen vs a valid "empty" value such as, say, Destination::Null
The compiler forcing me to learn about chaining .as_ref() to continue accessing a variable more than once
Working with modules and setting visibility to functions via pub
Adding #[cfg(debug_assertions)] to println!() macros to effectively make them debug prints
Using the clap crate for parsing command line arguments and doc comments
Appreciating how Rust forces you to contend with all possible values of an enum and also errors

The end of Project 6 marks the half-way point of the entire Nand2Tetris course. I'm half way done! This is so cool. With the so-called "Hardware" half of the course behind me, now I'm focussed on the next half, the "Software" half. Since the official website does not offer PDFs of the relevant chapters from the textbook for the remaining projects, I bought the textbook instead.

During the course of the next projects, I'll be building a virtual machine translator, a compiler for a Java-like high-level language, and an operating system for the Hack computer. I am really excited to read the textbook and learn about compilers and a "stack machine" in detail.

Finally, since I recently added tags to all my blog posts, it is now convenient to view all my Nand2Tetris-related posts by checking tag: nand2tetris.

Nand2Tetris - Projects 4 to 5

2024-08-28T00:03:11-07:00

Continuing from where I left off in my Nand2Tetris journey, I went ahead and completed Projects 4 (Machine Language) and Project 5 (Computer Architecture) and I now have "built" my very own Hack computer from scratch!

Follow my progress on Github here: guru-das-s/nand2tetris.

Project 4: Machine Language

This project introduces the Hack instruction set architecture (ISA) and assembly language corresponding to the Hack computer we're building, and tasks the student with writing two programs that provide a good understanding and feel for the platform.

The first program, Mult.asm is quite straightforward - to multiply two numbers.

The second program, Fill.asm has a particularly satisfying end result - to turn the screen in the CPU Emulator completely black or white based on keyboard input.

I haven't written a lot of assembly code, so it was very gratifying to reason about a problem in terms of assembly and write code while conforming to the constraints of the ISA.

Project 5: Computer Architecture

This is the first big project in the course according to me, and the most climactic so far, as we progressively build:

a Memory unit from the RAM chips built earlier,
a CPU from the ALU built earlier,

and finally wire them both together along with a ROM unit (separate instruction and data memories), thus creating a full-fledged Hack computer that can load and execute any arbitrary program!

The textbook chapter says it best:

The computer that will emerge from this project will be as simple as possible, but not simpler.

It's true - this computer does not have fancy stuff like pipelining, or branch predictors or caches of any kind. It also implements the Harvard architecture - separate instruction and data memories instead of the Von Neumann architecture model.

The Memory unit

The memory unit consists of a 16K RAM, a memory-mapped screen, and a memory-mapped keyboard all laid out adjacent to each other with no holes.

The CPU

This is the crux of the whole project - decoding an instruction from its bits and figuring out what action needs to take place for it. To that end, various bits and bitfields of the instruction encode specific pieces of information, like the type of ALU operation that needs to take place, where to store the results of the operation, and what kind of jump is requested.

The challenge in this part of the project is to route various bits and bitfields of the 16-bit instruction to the A, D and PC registers and implement the hardware-software contract.

There are a couple of gotchas that make things interesting when testing the code using the provided tests. I caught one edge case only when running the test for the next part of the project, i.e. building the Computer.

DMux8Way saved my ass in both the Memory and CPU parts of this project. It took me a while to recognize that the logic for powering the jump detection could easily make use of this building block - a side effect of not being a hardware guy, I suppose.

The Computer

After the rigours of the CPU design part of the project, this turned out to be relatively straightforward. Just three lines of code wiring the ROM, RAM and CPU together.

Looking ahead

The next project, Project 6, is to build an assembler that converts Hack assembly code into binary. I've not attempted something of this nature before, and am really excited about this!

TIL: Execute current line in vim

2024-07-29T22:52:04-07:00

I write a lot of notes in Markdown on a daily basis as a sort of personal wiki. Almost all of them contain code snippets - shell commands, mostly. Today I learnt that there is a much better way to directly execute a shell command listed on its own in a single line in vim.

Two ways, in fact.

The first

.w !bash

The fullstop character used here has special significance - it refers to the current line. From :help cmdline-ranges:

Some Ex commands accept a line range in front of them.  This is noted as
[range].  It consists of one or more line specifiers, separated with ',' or
';'.

...

Line numbers may be specified with:     :range {address}
    {number}        an absolute line number  E1247
    .               the current line                :.

The space between the w and the ! is important here. Without it, the current file would be saved with filename bash, which is not what we want to happen.

From :help !:

4.1 Filter commands                 *filter*

A filter is a program that accepts text at standard input, changes it in some
way, and sends it to standard output.  You can use the commands below to send
some text through a filter, so that it is replaced by the filter output.
Examples of filters are "sort", which sorts lines alphabetically, and
"indent", which formats C program files (you need a version of indent that
works like a filter; not all versions do).  The 'shell' option specifies the
shell Vim uses to execute the filter command (See also the 'shelltype'
option).  You can repeat filter commands with ".".  Vim does not recognize a
comment (starting with '"') after the `:!` command.

                            *!*
!{motion}{filter}   Filter {motion} text lines through the external
                    program {filter}.

The second

The second method is also documented a few lines from above:

                            *!!*
!!{filter}          Filter [count] lines through the external program
                    {filter}.

Just need to say !!bash (note: no preceding :).

But this will overwrite the contents of the current line with the output of the command in that line.

Addendum

This same trick can be used to insert a date in a Markdown file: !!date

Build your own shell - Codecrafters style

2024-07-28T23:26:59-07:00

I created my own shell in Rust using the Codecrafters Build your own shell challenge.

As part of my ongoing efforts to learn by doing, I came across the course on LinkedIn and resolved to try it out using Rust, another goal of mine. Here's my experience of completing the challenge, and also my thoughts on the Codecrafters platform as a whole.

Capabilities of the shell

But first, what can the shell do?

REPL: Read-Evaluate-Print-Loop structure
Handle the following builtin commands:
1. exit: exit shell with desired exit code
2. echo
3. type: identify command as a shell builtin, or print its absolute path
4. pwd
5. cd: both absolute and relative paths, and also ~
Run arbitrary programs present in $PATH and print their output.
Gracefully handle unidentified commands or programs not in $PATH.

The code is available on Github: https://github.com/guru-das-s/codecrafters-shell. Compiling and running it is quite straightforward, with cargo build and cargo run.

There are more "extensions" in the pipeline: piping, history, autocompletion of commands, et c. Really excited to do those when they're available!

The Codecrafters platform

I am thoroughly impressed by the Codecrafters platform.

Sign up was smooth, with Github login
They provide skeleton code to begin the challenge with
The shell is built up, stage by stage, with clear instructions for each stage
A convenient one-stop shell script to compile and run your code is provided too
The cli tool runs all the tests for a stage locally before push-ing
The tool also runs tests for all the previous stages to ensure nothing breaks
git push-ing to their server (origin already configured in the clone step) advances to the next stage

Working on the challenge

Making time to work on the challenge was challenging in itself with a full time job and other quotidian demands on my time. It was only during my vacation in India that I had the time and mindspace to finally take the plunge and start.

Visual Studio Code is a great coding tool for Rust, especially with the Rust extensions. Being able to see the return types of functions and of function parameters is highly useful, as are the real-time compilation errors and the compiler diagnostic messages. The vim mode emulation is adequate for my needs too.

I chose to go with the C-style mental model of having a list of function pointers for every shell builtin:

enum CmdHandler {
    Exit(fn(i32)),
    Echo(fn(&[&str])),
    Type(fn(&HashMap<String, CmdHandler>, &str)),
    Pwd(fn()),
    Cd(fn(&str)),
}

And a hashmap to map builtin commands to their handlers:

let mut builtins: HashMap<String, CmdHandler> = HashMap::new();

This allowed me to add new commands easily and keep things neat and organized.

It was great to discover the Rust equivalents of a few well-known C library functions:

stat() – fs::metadata()
exec() – std::process:Command::new()
getenv() – std::process:env::var()

A pleasant surprise was that I did not have to write any code to support relative filepaths for the cd builtin - the code for the absolute filepaths was sufficient. The difficulty level for that stage is marked as Hard but I literally had to write zero new code to pass that!

Closing thoughts

I feel quite proud of the fact that I got to build something tangible in Rust.

My earlier attempts at learning Rust were limited to solving Advent of Code 2023 problems. In my experience, it was quite thrilling to work on them in real time, i.e. in December when every day would bring forth a new problem, but once I "fell off the wagon" I found it difficult to keep my motivation levels up and feel excited about solving them. Being part of an active community seems like a great motivator for me.

Now that I'm done with the challenge, I've been checking out other solutions in Rust on Github to see what I can learn from them. I realize that my solution is very C-like (because I'm primarily a C programmer) and doesn't use many Rust-y features like attributes or traits. Learning a new programming language is like learning a new language in many ways - immersing oneself in well-written code accelerates and supplements self-learning.

I also appreciated the autogenerated email I received upon completion which had great suggestions on what to do next: retrying the challenge in a different language, sharing the experience (and code) with others, or trying a new challenge. I'm seriously considering taking up the final suggestion!

Nand2Tetris - Projects 1 to 3

2024-04-13T11:53:34-07:00

Nand2Tetris is a very interesting "learning by doing" free and open-source course that enables one to learn how a computer is designed from the ground up, and also how it is programmed from the very lowest of levels. It is a fantastic course offered in two parts (first building the hardware for the computer, and then the software) and is very rewarding to those who finish it. I started working on it this year after having discovered it via Hacker News and have completed Projects 1 to 3 of 12 projects in total. You may follow my progress on Github here: guru-das-s/nand2tetris.

The projects in this course start from designing various boolean gates using NAND gates as a building block and slowly build upon each other to create a full-fledged computer, complete with its own corresponding assembler, compiler and operating system, thus allowing for running rich and complex full-featured programs such as Tetris.

My motivation for starting and committing to this project is to fill in any gaps in my understanding of how computing systems work, from the application layer down to the individual components. Modern-day computer architectures such as x86 and ARM have many layers of optimization and complexity to achieve industrial levels of performance. This course pares away all inessential elements during the design process, retaining only clean and simple elements for illustrating the basic concepts. Most of all, I love the 100% hands-on approach because I learn better by doing.

Here are some thoughts and observations on each of the projects I've completed so far:

Project 1: Boolean logic

This project is about creating gates such as OR, NOT, and MUX along with their 16-bit versions using basic Boolean logic. The NAND gate is provided as a fundamental building block (no need to build it).
This introduces the Hardware Simulator Java program, the custom Hardware Description Language (HDL) and the whole write-test-debug iteration cycle.
Being an software guy myself, writing and thinking in HDL was a pleasant challenge. I didn't get to do much of this during my undergrad which was not all that electronics-focused, so it was quite enjoyable to do it now.
I noticed that a couple of test scripts and "ground-truth" comparison files provided in the project's source code were incorrect and had to fetch the right versions from poking around on Github. I also sent the Nand2Tetris admins an email about this; Prof. Shimon responded, acknowledging a recent issue with the software suite.
Overall, this project was very satisfying and straightforward.

Project 2: Boolean arithmetic and building an ALU

The concept of two's complement was explained clearly with motivation for its existence, and illustrated with examples of subtraction and addition with overflow.
The most thrilling part of this project was to build an Arithmetic and Logic Unit from scratch, and really understand how various operations could be built using just addition and subtraction.
The ALU is to be built in two stages. I thought the first stage was going to be difficult (getting all the operations right) and the second stage (detecting negative numbers and zero in the output), easy. It was the opposite - I struggled with the second stage mostly because I didn't know a quirk of the HDL's syntax. I had to read the HDL Guide and also consult the forum for guidance.

Project 3: Building RAM chips

This project introduces sequential logic and building a 16K Random Access Memory (RAM) chip using banks of registers, which are built from individual registers which, in turn, are built from Bits. The D Flip-flop is provided as a fundamental building block (no need to build it).
This was a very satisfying project to do because working with sequential logic was a nice change after two projects of combinatorial logic.
Once the basic RAM8 was complete (RAM chip containing just 8 registers), it was a breeze to complete the other larger RAM chips, mutatis mutandis.
Creating the Program Counter (PC) was also very interesting because of the clever ways in which internal pins need to be used.

Looking ahead

Things start to get even more interesting from Project 4 onward. It is about learning the assembly language of the particular computer that is being built and writing programs in it using the reference assembler provided. Project 5 is to create a full computer using the ALU and RAM that were just built earlier, and Project 6 is to write an assembler in any high-level language, to replace the provided assembler.

Really excited for the upcoming projects!

TIL: Easily cycle through all open tmux sessions

2024-03-30T08:32:00-07:00

My tmux workflow thus far has been strictly windows- and panes-based, meaning I usually launch only one tmux session and then create named windows and panes (horizontal and vertical splits) as necessary to deal with the set of projects for the day. While this works well, things get quickly out of hand when there is some firefighting to be done and context-switching between the projects is frequent, involving creating new windows to explore new ideas.

I draw the line at nine open windows because that's the maximum number of windows I can quickly jump to with one keystroke using the prefix + <number> keyboard shortcut. I have no interest in dealing with double-digit numbers of windows. But what if I have, say, three different projects to work on during the day, and I "run out" of windows? This week, I realized that there is a better way to effect a clean context switch and also organize my tmux windows better: by using the native sessions feature.

man tmux has this to say about sessions:

When tmux is started, it creates a new session with a single window and displays it on screen. A status line at the bottom of the screen shows information on the current session and is used to enter interactive commands.

A session is a single collection of pseudo terminals under the management of tmux. Each session has one or more windows linked to it. [...] and any number of windows may be present in the same session. Once all sessions are killed, tmux exits.

Command	Comment
`tmux`	Launch new unnamed `tmux` session
`<prefix> + $`	From within `tmux`, rename current session
`tmux new -s <session>`	Launch new session named `<session>`
`tmux ls`	List all open `tmux` sessions
`<prefix> + s`	List all open `tmux` sessions from within a session
`tmux attach -t <session>`	Attach to an open session named `<session>`
`<prefix> + d`	From within `tmux`, detach from current session

Adding these lines to my .tmux.conf solved my issue:

# Easily cycle between open tmux sessions
unbind (
bind-key 0 switch-client -p
unbind )
bind-key = switch-client -n

Turns out that <prefix> + ( or ) enables switching to the previous or next open session. I find this cumbersome because of the need to use the Shift key and so I remapped them to single keys. Now I can easily cycle through all my open sessions using <prefix> + 0 or <prefix> + =.

And, since I have this ability, I can happily create as many new sessions as I need - one per project - without having to fear using the Shift key to type in the paranthesis characters each time I want to switch to another session I created.

You may well wonder how I could access the 0-th pane if <prefix> + 0 has been re-bound as above. I don't, because I don't have a 0-th pane at all. I number my panes starting from 1 and not 0 via these lines:

# Start windows and panes at 1, not 0
set -g base-index 1
setw -g pane-base-index 1

And for good measure and clean pane number management:

# Automatically renumber all windows/panes when one is killed
# e.g. 1 2 [3] 4   .... 3 is killed, results in:
#      1 2 3       .... and not 1 2 4, which would call for a :movew -r
set -g renumber-windows on

And I chose = to switch to the next session because of its proximity to the Backspace key (easier to reach).

Bottom line: my workflow is now the following.

Open terminal, navigate to project directory, launch first tmux session
Create/kill as many windows and panes I need
If I need to switch to a new project, detach from current session
Navigate to new project directory, launch new session
Cycle through open sessions using <prefix> + 0 or <prefix> + =
- I mostly end up using only the latter because of its proximity to the Backspace key.

The explicit switching from one session to another not only separates one project's windows from another in tmux, making for clean organization, but also helps me context-switch in my mind more effectively.

TIL: Show current C function in vim status line sans plugins

2024-03-13T00:10:11-07:00

I primarily work on the Linux kernel (which is written almost entirely in C) and I frequently find myself needing to know which function I'm currently in while browsing the codebase. There is, of course, the dictum that functions should be as small as possible and not more than a screen's length long; this is mostly just a guiding principle.

There are plugins to do this sort of thing, but I do not want to become overly dependent on them. I am more partial towards things I can use that are just plain vanilla vimscript because I can just stick them in my .vimrc when setting up a new workstation or logging into a remote server somewhere.

The function

Here's the function in all its inscrutable beauty:

" Show function name on demand, mapped to key ','
fun! ShowFuncName()
    echohl ModeMsg
    echo getline(search("^[^ \t#/]\\{2}.*[^:]\s*$", 'bWn'))
    echohl None
endfun

nmap , :call ShowFuncName()<CR>

I couldn't locate the StackOverflow answer I had sourced this from (even though this post says TIL, I did not learn this literally today) so I used Gemini AI to break this down for me.

The breakdown

echohl ModeMsg: Sets the highlighting for the echo output to the mode message style. This ensures the function name visually stands out.

getline(search("^[^ \t#/]\\{2}.*[^:]\s*$", 'bWn')): This line is the core logic for finding the function name. Let's break down the search pattern:

^[^ \t#/] - Matches the beginning of the line (^) and excludes whitespace ( and \t), tabs (/), and comments (#).
\\{2} - Matches exactly two characters (the function name can have at least two characters).
.* - Matches any characters (captures the function name).
[^:] - Excludes the colon (:) which might follow the function name.
\s*$ - Matches whitespace at the end of the line.
bWn - Flags:
- b - Starts searching backward from the current line.
- W - Considers word boundaries.
- n - Doesn't include the matching pattern in the returned line.

This search essentially finds the line before the current line that starts with two non-whitespace, non-tab, non-comment characters, followed by any characters (the function name), excluding the colon and ending with whitespace.

echo: Prints the captured line containing the function name.

echohl None: Resets the highlighting back to the default.

The demo

If you were at this line in drivers/mfd/qcom-pm8008.c (which is a driver I authored) and pressed , to trigger the above command, you would see this in the vim statusline:

static int pm8008_probe(struct i2c_client *client)

which is, in my view, a very clean way of doing things.

TIL: How to unify all three of Ubuntu's clipboards

2024-03-02T18:24:00-08:00

Yes, you read that right - Ubuntu (which uses the X server) has a grand total of three clipboards: As the Inter-Client Communication Conventions Manual (ICCCM) explains:

Selection Atoms

There can be an arbitrary number of selections, each named by an atom. To conform with the inter-client conventions, however, clients need deal with only these three selections:

PRIMARY

SECONDARY

CLIPBOARD

Of these, only PRIMARY and CLIPBOARD are commonly used by applications. As freedesktop.org explains:

There are two historical interpretations of the ICCCM:

a) use PRIMARY for mouse selection, middle mouse button paste, and explicit cut/copy/paste menu items (Qt 2, GNU Emacs 20)

b) use CLIPBOARD for the Windows-style cut/copy/paste menu items; use PRIMARY for the currently-selected text, even if it isn't explicitly copied, and for middle-mouse-click (Netscape, Mozilla, XEmacs, some GTK+ apps)

No one ever does anything interesting with SECONDARY as far as I can tell.

What led me to this discovery?

There are two main ways to copy and paste stuff from one application to another on Ubuntu:

Select text using mouse, Ctrl-C, Ctrl-V
Select text using mouse, paste using middle click of mouse.

It was when I found myself naturally mixing up these two methods and expecting this unsupported hybrid scheme to work that I grew frustrated and decided to investigate why the scheme that occurred to me naturally did not, in fact, work. Allow me to explain using a couple of scenarios.

Terminal → Terminal

As a heavy terminal and tmux user, my workflow involves frequently copying stuff I use tmux, so I need to copy stuff from one tmux pane to another. To achieve this, I use the second method above.

This is intuitive to me because the two operations occur in the same application and what the mouse selects, it copies, too. The first method also works, albeit with a twist: need to use Ctrl-Shift-C and Ctrl-Shift-V. So this is technically a third way of copying and pasting on Ubuntu.

Non-terminal application → non-terminal application

Example: copying from one Google Chrome tab to another.

Here, the first method is intuitive to me because it is what has been ingrained in me from a young age thanks to Windows.

Non-terminal application → Terminal

Example: copying from Google Chrome to the terminal.

Here, too, the first method is intuitive to me, albeit with a twist. The twist is that you need to paste using Ctrl-Shift-V in the terminal. So this is technically a fourth way of copying and pasting on Ubuntu.

Terminal → non-terminal application

Example: copying from the terminal to Google Chrome.

Here is where my brain crosses out disruptively. What is intuitive to me is copying by selecting the text using the mouse, and pasting using Ctrl-V. This does not work. This is what is intuitive to me because I used PuTTy on Windows for a long time for work and this is how it works there.

Really, this was the main use case that frustrated me because I frequently have to look up a compilation error message, or copy some logs to a chat client. I would select to copy using the mouse, paste using Ctrl-V, fail, then middle-click. I did this so many times that I found it disruptive to my workflow and had to fix this.

The fix: unify both clipboards

Install autocutsel and add this to your .bashrc:

autocutsel -s CLIPBOARD &
autocutsel -s PRIMARY   &

Now, Ubuntu behaves as though there is only a single, common clipboard for all applications - terminal or otherwise. To add to the above list, you now can:

Select text using mouse, Ctrl-C, paste using middle click of mouse.
Select text using mouse, paste using Ctrl-V

You can also go one step further and unify the third clipboard too, but I have not tried this out personally.

autocutsel -s SECONDARY &

StackOverflow helped me a lot ¹ ² while researching this fix and learning more about the underlying architecture.

Fun with git rebase --interactive

2023-12-02T23:41:29-08:00

This is the second in a series of blog posts related to some powerful features of git I've used over the years. Previously, I wrote about git rebase --onto which you can read here.

git offers many powerful features, and one such feature I have used extensively in kernel development is git rebase --interactive. Here is what the man page has to say about it:

-i, --interactive
    Make a list of the commits which are about to be rebased. Let the user edit that
    list before rebasing. This mode can also be used to split commits (see SPLITTING
    COMMITS below).

That this feature exists is impressive on its own: in the list of commits that appears, one can reorder commits, delete them, rework their commit messages, combine ("squash", "fixup") them, and even merge them in recent versions of git. But where this really shines is how it allows, via an optional flag, for executing any arbitrary shell command for one or even all of the commits in the list. I'm talking about git rebase --interactive --exec.

Usecase: Pick a bunch of `linux-next` commits

Frequently we run into the situation where we need some patches that are accepted by maintainers into their own trees but are not merged by Linus into his main repository. This occurs when a developer posts a patch to the mailing lists ("pushes a patch upstream") that is an urgent fix of some kind, required in a product kernel ("downstream"). The downstream kernel, such as the one the Android Open Source Project uses (the Android Common Kernel), usually has strict requirements of the commit messages of patches it accepts for merging. One of these requirements is:

If the patch is not merged from an upstream branch, the subject must be tagged with the type of patch: UPSTREAM:, BACKPORT:, FROMGIT:, FROMLIST:, or ANDROID:

Some projects may even require specific git-trailers ¹ in the commit message, e.g.:

Git-commit: 26b4ca3c39012368ab22b06cd35dd6b77f0f3e00
Git-repo: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

This means every single patch being pushed to the downstream kernel project would need to be tagged this way. If we need to get, say, 10 or 20 patches merged from a maintainer's tree or linux-next, we need a scalable way of tagging all of them with one of the above subject prefixes and adding those git trailers. This is where git rebase --interactive --exec really shines.

From within the downstream kernel, add the linux-next remote:

git remote add linux-next https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git fetch linux-next

Gather list of commits required. These may be abbreviated SHAs too, for convenience.

base_sha=$(git rev-parse HEAD)
pick_commits=(<sha1> <sha2> ... <shaN>)

Cherry pick each commit and add the first trailer:

for commit in "${pick_commits[@]}"; do
    git cherry-pick $commit
    git commit --amend --no-edit --trailer "Git-commit: $(git rev-parse $commit)"
done

Now, add the subject prefix FROMGIT: to all the commits that were just cherry picked:

git rebase -i --exec 'git commit --amend -m "FROMGIT: $(git show -s --format=%B)"' $base_sha

Next, add the second trailer to all the commits that were cherry picked:

git rebase -i --exec 'git commit --amend --no-edit --trailer "Git-repo: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git"' $base_sha

Finally, add one's own Signed-off-by to all the cherry picked commits:

git rebase -i --exec 'git commit --amend --no-edit -s' $base_sha

git fixup and `git rebase --interactive --autosquash`

This blog post: Auto-squashing Git Commits goes into great detail on what this feature is, and how it may be made use of.

Git - git-interpret-trailers Documentation ↩

How I used git rebase --onto

2023-11-09T23:25:28-08:00

On one of my daily trawlings of Hacker News, I came across Julia Evans' blogpost on confusing git terminology that lists git rebase --onto as one such confusing command. I had never used this --onto flag before and her description didn't really help me visualize the problem or the solution.

I shrugged and let it go, not thinking much of it at the time.

I ended up using it recently, when I had to rebase a feature branch based off a couple of weeks old, stale, master branch onto the up-to-date tip of the same master branch.

Since I was using Gerrit, my feature branch had a topic:XYZ set. This XYZ topic series of mine had about ~20 commits, and the master branch had advanced quite a bit with almost ~15K commits ¹, some of which were also large merge commits.

My erstwhile strategy for cherrypicking the whole series onto the moving tip of the master branch was as follows:

git config pull.rebase true
git pull <server/project> <refs/changes/12345/4>

(where 12345 was the last change in the series)

This worked at the time I had published this strategy to our internal documentation at work so that other developers could make use of my XYZ series on top of the latest, up-to-date master branch. Over time, though, the series ran into a merge conflict once the master branch had advanced sufficiently enough. And that is when I was asked to rebase the series to fix this merge conflict and update the documentation.

When I tried the above instructions on top of the up-to-date master branch, it did not work because git tried to interactively rebase all the ~15K changes one by one and then somehow ran into merge conflicts on changes unrelated to mine. Resolving merge conflicts unrelated to my changes was a clear sign that I was doing something wrong — there had to be a better way.

After a bit of Googling, I found this answer ² on Stack Overflow that succintly expressed what I had to do:

git rebase --onto <final_base_commit> <initial_base_commit> <head>

This was interesting. The fact that I had come across this recently was fresh in my mind, and hence I quickly consulted the man page for git-rebase:

First let’s assume your topic is based on branch next. For example, a feature
developed in topic depends on some functionality which is found in next.

        o---o---o---o---o  master
             \
              o---o---o---o---o  next
                               \
                                o---o---o  topic

We want to make topic forked from branch master; for example, because the
functionality on which topic depends was merged into the more stable master branch.
We want our tree to look like this:

        o---o---o---o---o  master
            |            \
            |             o'--o'--o'  topic
             \
              o---o---o---o---o  next

We can get this using the following command:

    git rebase --onto master next topic

Aha! When read alongside the man page, the Stack Overflow answer's recommendation started to make sense.

final_base_commit would be my current HEAD, i.e. the current (up-to-date) tip of the master branch.

initial_base_commit would be the parent of the first patch in my series. In my case, this was <SHA>^ where <SHA> was that of the first patch - easily available by inspection on Gerrit. Let's say this was 43487e7b567cfb^.

head would be the FETCH_HEAD after running git fetch on the last change in my series, i.e.

git fetch <server/project> <refs/changes/12345/4>

Thus, the final command became:

git rebase --onto HEAD 43487e7b567cfb^ FETCH_HEAD

The merge conflict I had to fix was fairly trivial, and I was done in no time.

I was really impressed by the power of this feature as I was dreading having to manually cherry-pick all of the ~20 changes in my series.

This large number of changes is because the project was the Linux kernel actively being developed by a global team of developers. ↩
Git: Interactively rebase a range of commits - Stack Overflow ↩

Learning Go

2023-09-19T00:23:04-07:00

I recently had to learn Go. I was on a tight timeline of just about 4 or 5 days. I am familiar with learning Rust, and Rust by Example, so I found Go by Example. I found the format wanting for the absolute beginner level I was at, so I found the absolutely beautiful A Tour of Go instead. It has links to the official Go blog for some language features, plus a live playground, which is really great.

I learnt enough to pass a Hackerrank test, which is immensely gratifying :) It's such a lovely language - very non-threatening syntax and approachable. I hope to transition out of the embedded space and into pure application/web development one day, and I hope to use Go that day.

Guru Das Srinagesh

Nand2Tetris - Project 8 (VM Translator Part 2)

Preparatory refactoring

Task 1: Emit code for the control flow VM commands

The label and goto commands

The if-goto command

Task 2: Emit code for the function-related VM commands

Changes to phrases that occur within a function

The function command

The call command

The return command

Task 3: Emit bootstrapping code

Reflections

Test script

Postscript

TIL: Firefox supports back and forward touchpad gestures by default

Nand2Tetris - Project 7 (VM Translator Part 1)

Problem description

VM Translator – Code architecture

Development and testing workflow

Interesting problems

Labels must be unique for conditional operator phrases

To increment SP, or not to increment SP?

Implementing the pop operation

Implementing the static code segment

What I learnt about Rust programming

Test script

Postscript

TIL: The fzf fuzzy finder is amazing

Opening files in vim within current directory

ag integration from within vim

Bash alias to add filename in REPL prompt

Bash alias for git log

Nand2Tetris - Project 6 (Assembler)

Introducing: The has assembler

Code walkthrough

Main data structures used

Code flow

Does has output match the stock assembler's?

Post-completion thoughts

Nand2Tetris - Projects 4 to 5

Project 4: Machine Language

Project 5: Computer Architecture

Looking ahead

TIL: Execute current line in vim

The first

The second

Addendum

Build your own shell - Codecrafters style

Capabilities of the shell

The Codecrafters platform

Working on the challenge

Closing thoughts

Nand2Tetris - Projects 1 to 3

Project 1: Boolean logic

Project 2: Boolean arithmetic and building an ALU

Project 3: Building RAM chips

Looking ahead

TIL: Easily cycle through all open tmux sessions

TIL: Show current C function in vim status line sans plugins

The function

The breakdown

The demo

TIL: How to unify all three of Ubuntu's clipboards

What led me to this discovery?

Terminal → Terminal

Non-terminal application → non-terminal application

Non-terminal application → Terminal

Terminal → non-terminal application

The fix: unify both clipboards

Fun with git rebase --interactive

Usecase: Pick a bunch of linux-next commits

git fixup and git rebase --interactive --autosquash

How I used git rebase --onto

Learning Go

The `label` and `goto` commands

The `if-goto` command

The `function` command

The `call` command

The `return` command

Implementing the `pop` operation

Implementing the `static` code segment

Introducing: The `has` assembler

Does `has` output match the stock assembler's?

Usecase: Pick a bunch of `linux-next` commits

git fixup and `git rebase --interactive --autosquash`