When it comes to our dependencies, we simply tend to “trust” them — not real trust, of course, but some vague and fuzzy kind of trust, after all they are probably tested and code reviewed. And they’re open source, other people use them. We’re all professionals. Dependencies of dependencies? Potato potato!

You should probably trust your dependencies as much as you trust your own code. If your trust is high enough, this post may not be for you. If you care about doing the hard work, welcome aboard. Let’s get all our dependencies reviewed ... piece by piece.

If you’re working on any medium-sized project, it’s entirely likely that you can’t vet all of your dependencies on your own. Rust and Node.js are famous for transiently pulling in gazillions of them and it only takes one bad actor to cause a lot of trouble.

Enter Crev: a distributed code review system

Crev aims at helping with that issue by distributing the work and the reviews. The idea is that if everyone only vets a couple of their own dependencies we can build a network of trust (and distrust).

In short, when Crev scans your dependencies, it’ll inform you about the consensus of other reviewers for each dependency. You can then inspect and review individual dependencies yourself, and Crev will cryptographically sign your review and publish it on GitHub for others to consume.

You can then assign a level of trust to other people’s reviews, and through that network of trust get a higher review coverage — or, when you start from scratch, some review coverage to begin with.

At the time of writing this, crev provides cargo-crev, reference implementation for Rust, and initial implementations of npm-crev for Node, as well as pip-crev for Python.

Today, I’ll walk you through using cargo-crev to demonstrate the workflow. In it’s own words, it can (amongst others):

• warn you about untrustworthy crates and security vulnerabilities,
• allow you to review most suspicious dependencies and publish your findings, and
• increase trustworthiness of your own code.

You can also find a short version of the below information in the Getting Started Guide.

A worked example

The first step is to get it, of course: cargo install --locked cargo-crev. It’ll take a bit, after which it provides you with the cargo crev command.

Creating a proof repository

At the heart of it, Crev relies on proof repositories — Git repositories in which reviews are published. You can either fork the template on GitHub or create a repo from scratch; the canonical name for the repo is crev-proofs, and it must be publicly available over https, e.g.

https://github.com/your-username/crev-proofs

Here’s mine, if you want to see an example.

Setting up an identity

Each user of cargo-crev is identified by a unique identity, allowing the same repository to be used by multiple people at the same time. This might be interesting if you’re using Crev in a business or team context, but it has implications if you’re using cargo-crevon multiple machines (see below for more on that).

Setting up a new identity involves calling cargo crev new id and cargo crev publish, first creating the identity and then publishing it to the repository. You will be asked to provide a passphrase. Do so and keep it around safely.

At the end of it, you’ll be presented with a summary and asked to write it down. It’s a good idea to keep a backup, but if you don’t, you can always get it back. Apart from that, you won’t need this information anytime soon.

If needed, you can create multiple IDs on the same machine using the same process. Use cargo crev id current to see all available IDs.

Setting up an existing identity on the other hand does require the above output. Take your backup or run cargo crev id export on the existing machine/account to get it back, then run cargo crev id import and paste the entire output.

Trust a proof repository

On your new setup, start by trusting someone. The guidelines suggest to highly trust dpc/crev-proofs (a leap of faith in and of itself), but let’s do so by using cargo crev trust:

Later on you will want to extend your network of trust. In that case, repeat the cargo crev trust step with a proper trust level and a proof repository of your choice.

To give an example: If you wanted to trust my reviews (low trust, unless you know me), you’d run the following line.

Fetching current reviews for your repo

To fetch all current review repos, use cargo crev repo fetch all. Depending on when you run it, it’ll either say no updates or gives more information.

Verifying your dependencies

To verify your dependencies, switch into your Git repository directory and run cargo crev verify --show-all. It will fetch your dependencies and then list the review status; this is either none if no reviews exist, pass if there’s some level of trust, and flagged or dangerous for reported issues:

In the output above, the flagCB informs you about custom builds, where UM would inform you about an unmaintained dependency. The geiger count informs you about the number of lines of unsafe code in that dependency.

In a terminal, it would look roundabout like this:

Reviewing a dependency

The review process consists of two steps: Opening a dependency and actually reviewing it, then writing a review and publishing it.

To open the dependency, call cargo crev open $(name-of-the-crate); this will simply open a directory showing the crate contents.

To write a review, call cargo crev review $(name-of-the-crate). Keep your passphrase ready. Your editor of choice will open with something along these lines:

In reality, the file is much longer and thankfully self-describing, providing information about each field. The key aspects here however are the thoroughness of your review, your understanding of the dependency code and your overall rating of it, plus an additional comment.

Thoroughness encodes time and effort spent reviewing and comes in the following flavors

• high: This was a long, deep, focused review or even a formal security audit. Rule of thumb: “an hour or more per file”.
• medium: A standard and focused code review: “15 minutes per file”.
• low: A low intensity review: “2 minutes per file”.
• none: No actual review, just skimming: “seconds per file”. This is your fallback with common, established projects (think tokio) or when you just want to flag a dependency.

Understanding informs others about how well you actually understand the code:

• high: You completely understand the code.
• medium: You have a good understanding of the code.
• low: Some of the code parts are unclear to you.
• none: You lack understanding of the code. (We’ve all been there.)

Finally, rating informs about your verdict:

• strong: The code is secure and good in all aspects, for all applications.
• positive: The code is secure and okay to use, maybe some minor issues.
• neutral: The code is secure but with flaws.
• negative: The code has severe flaws and is not fit for production use.
• dangerous: The code is unsafe to use, has severe flaws or possibly malicious intent.

After checking the dependency’s source repository you can flag the dependency as either unmaintained or not (the default). If there are alternatives, you can add them in the alternatives section — remove the section if there are none.

After saving and closing the editor, your review is signed and committed to your (local) proof repository.

Re-reviewing a dependency after it changed

When the dependency code changed, you do not have to perform a full review again. Instead, you can use cargo crev crate diff and cargo crev review --diff to review only the parts changed since your last review.

And then you rinse and repeat

Now that you know

• that crev and cargo-crev exists,
• how to set up an identity and proof repository,
• how to trust other people’s review (to some degree),
• how to fetch reviews for your dependencies and
• how to publish your own reviews for others to use,

your work really has just begun.

While each review is only a little step in and of itself, each individual contribution does help the system grow. Tooling will get easier over time, and even if it never reaches general popularity — a rather unlikely thing, given the nature of the endeavor — it’s still a perfect chance to do some good. After all, you did just review one dependency more than you had before. Now on to the next one.

Thanks for making it here. Stay safe and healthy.

The following describes a basic setup for an STM32F3 Discovery board using the STM32F303VCT6 MCU, but should apply to all boards that can be programmed and debugged using OpenOCD. I have recently used it in a toy project here.

Preparing the Rust environment

This article focuses specifically on CLion, so I recommend following The Embedded Rust Book to get you started. There also is an older version of the book available specifically for the STM32F3 Discovery, which you can read here.

In particular you will need

• The proper Rust target installed (e.g. thumbv7em-none-eabi),
• OpenOCD installed,
• GDB Multi-Arch (or any GDB for your target platform) installed,
• udev rules set up, and
• User permissions to communicate with the UART device.

The book will guide you through all of this.

Set up the target toolchain

In CLion, go to Settings > Build, Execution, Deployment > Toolchains and create a new toolchain. In this window, select the GDB for the hardware, in this case arm-none-eabi-gdb. This step may not be strictly required as we will do a similar selection later on, but it keeps things nice and tidy.

Set up the build targets

Go to Settings > Build, Execution, Deployment > Custom Build Targets and create a new target for each binary and flavor to build. In the picture below, I have created Cargo Debug and Cargo Release targets and selected the toolchain created above. The Build and Clean steps will be created right after.

This configuration is stored in .idea/customTargets.xml.

Clicking the three-dot button next to Build and Clean will bring up the External Tools dialog.

Click the + button and create a single target Clean as well as multiple Build targets, one for each flavor or binary you want to build. This will bring up the Edit Tool dialog.

The Group name can be freely specified in the Edit Tool window, I selected Rust Build. You can choose a generic name or a specific name for each binary you build, e.g. to bundle debug and release builds per binary.

You can omit the --target ... argument if your project uses a custom .cargo/config.toml with a build.target configuration. My project uses:

[build]
target = "thumbv7em-none-eabihf" # Cortex-M4 and Cortex-M7 (no FPU)

If you do not provide a configuration like this, the Argument value must be something along the lines of:

build --target thumbv7em-none-eabihf

Cargo Workspaces

When setting up the project as a Cargo Workspace rather than a single binary, the variable expansions won’t be as useful. For example, $ContentRoot$ refers to the workspace directory, but so does $ProjectFileDir$. To build the right binary, you'll have to either specify default-members in the Cargo.toml or specifically provide the --bin your-project parameter to the build commands.

A complete value for Arguments would then look like:

build --target thumbv7em-none-eabihf --bin led-roulette

Run / Debug Configurations

Lastly we need to set up the run configurations. For this, navigate to Edit Configurations in your Debug/Run Configuration selection. This, unsurprisingly, brings up the Run/Debug Configurations dialog.

In here, create a configuration of type OpenOCD Download & Run.

• For Target select the build target created above. In the example below, I selected Cargo Debug.
• For Executable binary you select the build binary artifact from the target directory. In my case, the binary resided at target/thumbv7em-none-eavbihf subdirectory debug since the project builds for the thumbv7em-none-eavbihf target.
• Select the Debugger by picking the Toolchain you created above, or by pointing to the correct debugger on disk if you didn’t create a toolchain.
• For Board config file, click the Assist button. This will bring up the Select Board Config File dialog, allowing you to select the relevant configuration.
• Leave GDB port and Telnet port as they are unless they clash with other configuration.
• For Download select Always or If updated.
• For Reset select Halt. This provided the best debugging experience for me, anyway.
• For Before launch, keep Build.

Select the correct board configuration from the Select Board Config File dialog:

Flashing and running the application

When starting the selected Run / Debug Configuration, the Debug window should open. A tooltip will inform you about the firmware being uploaded through OpenOCD and you can now freely place breakpoints and step through them as usual.

At times, restarting the debugger results in compilation communications errors. Restarting the debugging session a second time resolves these.

System Viewer Description (SVD)

The Peripherals tab in the Debug view allows you to select the SVD file relevant to your board. I have selected mine from the github.com/stm32-rs/stm32-rs repo’s svd subdirectory (here).

Register View

Similarly, the Registers tab in the Debug view allows you to inspect the current register values.

That’s all folks

It’s not the best development experience in the world, but it’s a reasonable starting point. Have fun, take care.

I did so, only to find out that HSEStatus never switched to 0x01 because the RCC_CR_HSERDY flag was never asserted in the first place.

Obviously no one else in the whole wide web had trouble with this. Cold water? Let’s dive! Someone at the ST forums pointed me to the trick to output the RCC clock signal to the board’s PA8 pin, which I did like so:

Turned out … well, nothing. Flatline on that pin. So I took my multimeter and went upstream from the oscillator pins. Solder bridge SB12, of course bridged, working fine, SB17 open as requested, and then — silence on RB48. No beeps on my meter, no value, just plain high impedance.

To make a long story short: That 100 Ω resistor was borked, so I replaced it with some spare parts of an old scanner board I had floating around in the to-do stash. I’m not exactly known for massive soldering skills, but this video helped a lot here.

Final result:

Ugly but effective. Worked like a charm.

Of course, there is no such thing. When searching for hiking time rules, two candidates pop up regularly: Naismith’s rule (including Tranter’s corrections), as well as Tobler’s hiking function.

William W. Naismith’s rule — and I couldn’t find a single scientific source — is more a rule of thumb than it is exact. It states:

For every 5 kilometres, allow one hour. For every 600 metres of ascend, add another hour.

which reads as

where |w⃗ | is the walking speed, Δs the length on the horizontal plane (i.e. “forward”), Δa the ascend (i.e. the difference in height) and θ the slope.

That looks like

Interestingly, this implies that if you climb a 3 km mountain straight up, it will take you 5 hours.

By recognising that 5km/0.6km≈8.3≈8, the 8 to 1 rule can be employed, which allows the transformation of any (Naismith-ish) track to a flat track by calculating

So a track of 20km in length with 1km of ascend would make for km+8⋅1km=28km of total track length. Assuming an average walking speed of 5km/h, that route will take 28km/5km/h=5.6h, or 5 hours and 36 minutes. Although quite inaccurate, somebody found this rule to be accurate enough when comparing it against times of men running down hills in Norway. Don’t quote me on that.

Robert Aitken assumed that 5 km/h might be too much and settled for 4 km/h on all off-track surfaces. Unfortunately the Naismith rule still didn’t state anything about descent or slopes in general, so Eric Langmuir added some refinements:

When walking off-track, allow one hour for every 4 kilometres (instead of 5 km). When on a small decline of 5 to 12°, subtract 10 minutes per 300 metres (1000 feet). For any steeper decline (i.e. over 12°), add 10 minutes per 300 metres of descent.

Now that’s the stuff wonderfully non-differentiable functions are made of:

That is:

It should be clear that 12 km/h is an highly unlikely speed, even on roads.

So Waldo Tobler came along and developed his “hiking function”, an equation that assumes a top speed of 6 km/h with an interesting feature: It — though still indifferentiable — adapts gracefully to the slope of the ground. That function can be found in his 1993 report “Three presentations on geographical analysis and modeling: Non-isotropic geographic modeling speculations on the geometry of geography global spatial analysis” and looks like the following:

It boils down to the following equation of the walking speed |w⃗ | “on footpaths in hilly terrain” (with s=1) and “off-path travel” (with s=0.6):

where tan(θ) is the tangent of the slope (i.e. vertical distance over horizontal distance). By taking into account the exact slope of the terrain, this function is superior to Naismith’s rule and a much better alternative to the Langmuir bugfix, especially when used on GIS data.

It however lacks the one thing that makes the Naismith rule stand out: Tranter’s corrections for fatigue and fitness. (Yes, I know it gets weird.) Sadly these corrections seem to only exists in the form of a mystical table that looks, basically, like that:

where the minutes are a rather obscure measure of how fast somebody is able to hike up 300 metres over a distance of 800 metres (20°). With that table the rule is: If you get into nastier terrain, drop one fitness level. If you suck at walking, drop a fitness level. If you use a 20 kg backpack, drop one level. Sadly, there’s no equation to be found, so I had to make up one myself.

By looking at the table and the mesh plot it seems that each time axis for a given fitness is logarithmic.

I did a log-log plot and it turns out that the series not only appear to be logarithmic in time, but also in fitness. By deriving the (log-log-)linear regression for each series, the following equations can be found:

Visually, this renders as:

These early approximations appear to be quite good, as can be seen in the following linear plot. The last three lines t30 , t40 and t50 however begin to drift away. That’s expected for the last two ones due to the small number of samples, but the t30 line was irritating.

My first assumption was that the t40 and t50 lines simply are outliers and that the real coefficient for the time variable is the (outlier corrected) mean of 1.2215±0.11207. This would imply, that the intersect coefficient is the variable for fitness.

Unfortunately, this only seems to make things better in the log-log plot, but makes them a little bit worse in the linear world.

Equi-distant intersect coefficients also did not do the trick. Well, well. In the end, I decided to give the brute force method a chance and defined several fitting functions for the use with genetic algorithm and pattern search solvers, including exponential, third-order and sigmoidal forms. The best version I could come up with was

This function results in a least squared error of about 21.35 hours over all data points. The following shows the original surface from the table and the synthetic surface from the function.

A maximum deviation of about 1 hour can be seen clearly in the following error plot for the t30 line, which really seems to be an outlier.

For comparison, this is the synthetic correction table:

So now you know!

it is quite annoying to issue the push command twice — advanced git-fu to the rescue. Some dude over at Stack Overflow pointed out that Git supports the notion of a pushurl, being an endpoint for pushing to a given remote. The fun thing is that every remote may have multiple push URLs, which is exactly what I needed.

It needs to be said that despite the usage of the --add flag in the following snippet, a push URL always overwrites the default URL, so adding only one URL results in the original entry being overruled. So, for the situation given in the example above:

And that’s it. By pushing to origin Git instead pushes to both registered URLs.

Given a function f(x), find a quadratic interpolant q(x)=ax²+bx+c such that f(x) and q(x) share two points and have the same derivative on the first of them — i.e., the interpolant fulfills the conditions f(x0)=q(x0), f(x1)=q(x1) and f′(x0)=q′(x0). Basically this:

(The tangent looks a bit off, but you get the idea.)

So I took out my scribbling pad, wrote down some equations and then, after two pages of nonsense, decided it really wasn’t worth the hassle. It turns out that the simple system

for

has the solution

Instead of ruining your time on the paper, it can be obtained more easily in Matlab using

Obviously — given that this is a line search problem — the whole purpose of this operation is to find an approximation to the local minimiser of f′(x). This gives

We also would need to check the interpolant’s second derivative q′′(xmin) to ensure the approximated minimiser is indeed a minimum of q(x) by requiring q′′(xmin)>0, with the second derivative given as:

The premise of the line search in minimization problems usually is that the search direction is already a direction of descent. By having 0>f′(x0) and f′(x1)>0 (as would typically be the case when bracketing the local minimiser of f(x)), the interpolant should always be (strictly) convex. If these conditions do not hold, there might be no solution at all: one obviously won’t be able to find a quadratic interpolant given the initial conditions for a function that is linear to machine precision. In that case, watch out for divisions by zero.

Last but not least, if the objective is to minimize φ(α)=f(x⃗ k+αd⃗ k) using q(α) , where d⃗ k is the search direction and x⃗ k the current starting point, such that

then the above formulas simplify to

and, more importantly, the local (approximated) minimiser at αmin simplifies to

If q(α) is required to be strongly convex, then we’ll observe that

for an m>0, giving us that a must be greater than zero (or ϵ , for that matter), which is a trivial check. The following picture visualizes that this is indeed the case:

The article Why Machine Learning Doesn’t Work Well for Some Problems? (Shahab, 2017) describes the effect of Emergence as a barrier for predictive inference.

Emergence is the phenomenon of completely new behavior arising (emerging) from interactions of elementary entities, such as life emerging from biochemistry and collective intelligence emerging from social animals.

In general, effects of emergence cannot be inferred through a priori analysis of a system (or its elementary entities). While weak emergence can be understood still by observing or simulating the system, emergent qualities from strong emergence cannot be simulated with current systems.

Sheikh-Bahei suggests interpreting emergence — in a predictive context — as an additional dimension, called the E-Dimension, where moving across that dimension results in new qualities emerging. Crossing E-Dimensions during inference leads to reduced predictive power as emergent qualities cannot be necessarily described as a function of the observed features alone.

The more E-Dimensions are crossed during inference, the lower the prediction success will be — regardless of the amount of feature noise. Current-generation algorithms do not handle this kind of problem well and further research is required in this area.

Shahab, S.-B. (2017, July 6). The E-Dimension: Why Machine Learning Doesn’t Work Well for Some Problems? Retrieved March 4, 2018, from https://www.datasciencecentral.com/profiles/blogs/the-e-dimension-why-machine-learning-doesn-t-work-well-for-some

Conway’s Game of Life is a self-contained simulation game that evolves “living” cells over multiple generations according to four easy rules.

While the game is trivially implemented by sequentially evaluating the rules for every cell on the board, a rather beautiful highly parallelizable solution exists that uses 3×3 kernel convolutions and some clever observations. In the following, we will explore that algorithm and see a GPU-accelerated implementation of it in Rust using the ArrayFire library. At its core, it will look like this:

Recap: The rules to the game.

The game state is evolved over multiple generations. Each cell has two distinct states: Alive (1) or dead (0). A cell also has a 3×3 Moore neighborhood of eight neighbors from the top left (north-west) to the bottom right (south-east).

Starting from a randomly initialized board, a single evolution is performed by applying these steps:

1. Any living cell with fewer than two neighbors dies out due to underpopulation.
2. Any living cell with two or three living neighbors lives on to the next generation due to its stable environment.
3. Any living cell with more than three live neighbors dies due to overpopulation.
4. Any dead cell with exactly three living neighbors becomes alive due to reproduction.

In repeatedly applying these steps, the game may either converge to a steady state, cycle between two states or continue to change infinitely.

Simplifying the rules

After transferring the rules to pseudo-code, it looks somewhat like this:

From here, we can make some observations:

1. If a cell has less than two neighbors, the result will always be 0 regardless of the cell’s current value; this is due to the Under-population rule.
2. If the cell has exactly three neighbors, the result will always be 1 regardless of the cell’s current value; this is due to both the Stable Environment and Reproduction rules.
3. We do need to check for exactly two neighbors to ensure a cell keeps surviving (1) according to the Stable Environment rule: Note that with exactly two neighbors no new cell is born, so the outcome depends on the cell’s current state.
4. More than three neighbors always cause a cell to be 0 regardless of the cell’s current value; this is due to the Overpopulation rule.

This allows us to ignore a couple of tests for the cell’s previous value; in pseudo code:

When we ignore the cell’s current value for a moment, we can see the following pattern:

Upon closer inspection however, we see that both lines are simply the negation of each other:

As a consequence of that, only the Stable Environment and Overcrowding rules are required to determine the entire state progression. The missing piece being, of course, the current state of the cell:

• As we know from observation 2 and the Overpopulation rule, a cell will always exist if there are exactly three neighbors. We’ll call this the must_exist condition; it is additive (i.e. we “add life” to a cell).
• From observation 3 and the Stable Environment rule, a cell only continues to exist if it existed before. We’ll call this the can_exist condition; it is multiplicative (i.e. we “zero out” dead cells).

Mathematically, we can now express the resulting state as simply a multiplication and addition:

How do we get the number of neighbors of the cell? This is where a 2D convolution enters the stage: Convolving the board with the 8-neighborhood kernel shown below simply sums up all living cells (valued 1) around the center, which itself is ignored.

The result of that operation is therefore a direct measurement of the neighborhood size. All we need to do then is to compare that value against our thresholds of two and three as established above — and that’s it. If you want a nice refresher on how convolutions work particularly in image processing, head over to to the excellent post Image Kernels explained visually.

Putting it together in Rust and ArrayFire

As just explained, we want to determine the neighborhood size using a 2-dimensional convolution with the neighborhood kernel via convolve2. To specify our kernel in ArrayFire we use a 3×3×1×1 dimensionality, which reads approximately as height, width, number of channels and a “batch” dimension we can safely ignore here. When used against a multi-channel input such as an RGB image, this format will process each color channel independently of the others, as shown in the image in this post’s header.

Due to GPU requirements we cannot operate on a binary image directly and instead use a 32-bit floating point array. As a consequence, the addition used by our update will eventually send the cell values outside their allowed range. To mitigate that, we introduce the method clamp_range that limits the value range to the range 0..1 using ArrayFire’s clamp function:

Finally, we update the state using the method you already know by comparing counts exactly using ArrayFire’s eq function:

With that, Conway’s Game of Life runs on your GPU.

As usual, you can find the entire code on GitHub.

GitHub - sunsided/rust-arrayfire-experiments: Toying around with ArrayFire in Rust

Take care, stay safe and have fun!

To the majority of us, currencies and wallets make sense from our everyday experience in the physical realm. As such, I too had a basic idea of what cryptocurrencies are and how they work — buy them, send them, keep your wallet in a vault — but things get slightly weird once you look under the hood.

So let’s do that real quick.

Misconception #1: Those coins are in your wallet.

They’re not, and it’s easy to believe that this is how it works since that’s what we’ve been seeing all day in real life: You open your wallet and put a coin in. Interestingly, with crypto this is not at all how it works.

When you create a crypto wallet on a blockchain — let’s say Binance Smart Chain, a sibling of Ethereum — you will be handed a “wallet address” that looks somewhat like this:

0xC3DF3fe97a0d6054Da7f89262b19285a9eEf3C2A

Other blockchains use different formats, but the idea on all of them is the same: This address is the publicly visible part of your wallet, meant to be shared, while the other part (a secret key) is kept private to you, and only you.

However, cryptocurrency tokens are never actually sent to your wallet — instead it is your wallet address that is sent to the cryptocurrency. Those cryptocurrencies store your wallet, not the other way around.

Let me explain.

Misconception #2: Crypto contracts are complex.

Once you start learning about the potential and scope of modern blockchain technology, cryptocurrencies appear to be complicated beasts even to programmers, complex “smart contracts” running on hundreds of systems …doing finance, somehow.

What I didn’t expect is what actually happens behind the curtain. One recent example for this is a dear project in the making, FartCoin. A “token” — or cryptocurrency — is simply a small program running “on the blockchain”, and that program is also the piece that knows about your wallet address.

Like your wallet, the token itself has an address which in case of FartCoin looks like this:

0xdfaa1f2ba4550a3f099ca26ac73e4e4f27cf5ca3

This address in particular can be investigated on a website called BscScan, a Binance Smart Chain related explorer that allows to view smart contracts, their transactions and wallets on that blockchain. In some cases — as with FartCoin here — it also allows you to inspect the actual source code of the token’s smart contract.

It is this code that represents the entire cryptocurrency aspect of it. Here is the FartCoin code, in all its glory. If you don’t understand it code just yet, bear with me.

That’s all.

The first thing to take away from this is that the entire smart contract for a valid, working cryptocurrency is not more than 44 lines of code, whitespace included.

The key pieces here are in lines 4 and 24–26. Line 4 contains a table that stores the number of tokens held by every address, as a positive number. You can imagine it somewhat like this:

This table is modified by two standard cryptocurrency “functions” of the contract, transfer and transferFrom. Apart from serving two slightly different purposes, they essentially perform the same operation: Transferring “tokens” from one wallet address to another. They do that by changing numbers in the table:

Here’s how it works:

1. At first, the wallet address of the caller is tested for a high enough balance to perform the transfer in the first place.
2. Then, the balance of the target wallet is increased by the specified amount (value). This is simply an addition on the value in the table.
3. Next, the balance of the caller is decreased by the specified amount. This is a subtraction on the value in the table.
4. Lastly, a statement about this change is emitted to the blockchain for the world to see; this acts as a proof of the transaction.

And that’s really all there is to it; this is what makes you “have” a cryptocurrency “in” your wallet.

Interestingly, no actual token ever “left the contract” or “entered your wallet”. Your wallet balance simply is a number in a table held by that program, and all you do is prove to the program that this number belongs to you.

Misconception #3: Your wallet knows your balances.

It will eventually, but it has to cheat a bit.

You might have noticed that your wallet software like MetaMask, TrustWallet or Ledger Live is aware of some cryptocurrencies but not of others. Specifically very-much-altcoins usually require you to first make themselves known to your wallet by importing a “custom token”.

The key here is to realize that these tokens were never “in your wallet” in the first place. Instead, your wallet software is aware of some well-known cryptocurrency contracts and will actively query them for any balance assigned to your wallet address. All it does is showing these results to you.

When you import a custom token, you simply make another contract known to the wallet software that it will then look up for you every once in a while.

While this is slightly annoying, it is both necessary and safe: As you will find out below, not every cryptocurrency contract is benign. By not being able to see any arbitrary contract in the first place, your wallet protects you from scams by requiring you to actively make the decision to register a “nonstandard” contract.

That said, even if your wallet would not have that mechanism: There simply are too many cryptocurrencies out there and scanning all of them for your balance individually is a close to impossible task.

Misconception #4: Burning tokens destroys them.

Sooner or later you’ll someone speak of a “burning event”, where some amount of a cryptocurrency is “burned” to reduce the amount of available tokens in circulation. It evokes the idea of setting a pile of money on fire in order to remove it for good.

Let’s revisit the balances table example from above:

In here, you’ll see a special address called

0x000000000000000000000000000000000000dead

In Ethereum-like blockchains like Binance Smart Chain, this address is commonly known as the “Null Address” or “Burn Address” and like any other address it can be observed on your blockchain’s explorer. Take this example from FartCoin:

The trick here is that this address does not belong to any accessible wallet, so any balance sent there is lost for good. However, as the common theme of this story goes, the tokens still exist and therefore the total supply didn’t change; they’re just taken out of circulation as they can’t be transferred anywhere else anymore —that is, if the token’s contract is benign, of course.

Let’s talk about safety a bit.

If you lose your wallet, your tokens are gone.

That’s the short version.

What happens under the hood is that the thing you lose is the proof that an address belongs to you, and only by extension that some tokens in a contract’s “balances” table are yours. The “private key” of your wallet always generates the same “public address”, and as long as you have that private key, you can identify yourself to a contract — so keep it safe.

The “tokens”, in either case, will always remain in the contract, until the end of days. A soothing thought, in a way.

Airdrop scams: If it’s too good to be true, it’s probably not.

As a consequence of the balances table existing, cryptocurrency tokens might still “show up” in your wallet without requiring any interaction on your side. Is this what’s called an Airdrop; for it to happen, all it takes is for someone — like the owner — to instruct the contract that your address now has a nonzero balance, and now you “possess” it.

So far that’s nice, we all like free money. The dangerous part is that scammers do use this to their advantage by luring you into interacting with token contracts you don’t know.

The ways of scamming people are plenty, but here are some:

• You may think the value of an airdropped cryptocurrency is high, so you buy more. The contract might be forged however to only let the owner sell. As people buy the price goes up and the owner leaves with a fortune. Before you interact with an unknown token, look it up on a blockchain explorer. If you see a bunch of failed transactions like here, steer away.

• The token might have a “minting” function of any name that allows the owner to create new tokens at will. When enough people have bought the cryptocurrency, the owner can increase his own balance and cash it in, leaving everyone else with a worthless token and their money gone.
• The token name might indicate a website you will then want to visit. A Web3 website might have malicious code that interacts with your wallet software directly after asking you to connect your wallet to it. In this case, it is the website itself that will execute contracts on your behalf, like swapping your tokens for cash, then sending it to the scammer. The obvious solution: Never connect your wallet to websites you don’t know or trust, and never blindly sign or approve any transactions without double-or triple-checking their contents.

• Every operation you perform on the cryptocurrency contract yourself — like sending any amount of ETH or BNB to the contract, or executing any other function — can trigger arbitrary code you might not be aware of. This can lead to entirely unexpected results as the next point will highlight.
• A scammer might instruct you to send a zero-balance (i.e. 0 ETH, 0 BNB, …) payment along with some cryptic data. Since you do not appear to be sending any money here it appears safe, but this data might just be an encoded instruction. Approaches like this are likely to trick you into actually calling another function of the contract without your knowledge — for example, it could make you accept and perform a swap trade of an arbitrary token you possess such that the contract can drain your wallet on your behalf.

Opaque token contracts.

Note that a token contract isn’t always clearly visible as in the example I made above. Instead, it might show up as a bunch of hexadecimal values:

This isn’t a problem in itself, like DogeCoin’s contract shows. Trustworthy creators will however generally make it an effort to increase transparency by validating and showing the original source code, as is the case in the FartCoin example.

Be careful however that just because the code is listed still doesn’t mean the contract is trustworthy — well-crafted traps are hard to detect even in plain sight, even by professionals, and you might not want to pay for that lesson.

A prime example of a very well documented contract is that of Shiba Inu on the Ethereum blockchain. You will find that it is about 500 lines long, but most of that are comments on the code, explaining the implementation.

The silver lining. 🙂

A contract cannot steal any other crypto tokens from your wallet on its own, because it would have to have your private key to prove itself to the other contract as the rightful owner.

As a result, receiving a coin typically isn’t dangerous in itself. Different blockchains may have different rules however; Stellar, for example, requires you to actively establish a “trust line” before you can even receive a new kind of cryptocurrency because of that.

So — if you find a token in your wallet that you didn’t expect to be there, simply ignore it. Your wallet is safe.

TL;DR

A cryptocurrency, and in particular the abundance of very-much-altcoins in circulation these days, is often nothing more than a very small program shifting around a lot of value. To be on the safe side, when you hear “new cryptocurrency”, think of it as a computer program you just downloaded from the internet. You might not want to click that one right away.

That said, while a token’s contract might just be some fourty lines long, it may still change the world. Go FartCoin, you little rascal.

Take care, stay safe and have fun!

Doge, Shiba, FartCoin — What exactly IS a cryptocurrency? was originally published in CryptoStars on Medium, where people are continuing the conversation by highlighting and responding to this story.

public static SomeTypeDTO ToDTO(this SomeType data) { ... }

how can one write a generic method

public static TDTO ToDTO<TDTO, TData>(TData data) { ... }

that forwards to the right extension method(s). Some ten minutes into thinking about how to explain why it’s complicated and why it might not even be a good idea to begin with but how AutoMapper might solve the issue … the OP deleted the question.

But sometimes, the only reason to really do something is because you can, and so I set off to explore the solution space. In this post, I will go through implementing some approaches using

1. Simple Generics (TL;DR: won’t work)
2. Reflection with MethodInfo invocations,
3. Reflection with runtime compilation of Lambda expressions,
4. Compile-time Source Generation and
5. Using AutoMapper, for the sake of sanity.

You can find the source code for this blog post on GitHub:

GitHub - sunsided/medium-absurd-conversion: Code for the Converting between types in increasingly absurd ways medium post.

To have a starting point, these are the data classes and DTOs the author used:

namespace ExtensionMethods70642141;

public class Student
{
    public string Name { get; set; }
}

public class StudentDTO
{
    public string Name { get; set; }
}

public class Teacher
{
    public string Name { get; set; }
}

public class TeacherDTO
{
    public string Name { get; set; }
}

To convert, the author provided these extension methods:

namespace ExtensionMethods70642141;

public static class PeopleExtension
{
    public static StudentDTO ToDTO(this Student student) => new()
    {
        Name = student.Name
    };

    public static TeacherDTO ToDTO(this Teacher teacher) => new()
    {
        Name = teacher.Name
    };
}

And to recap, the question is:

namespace ExtensionMethods70642141;

public static class GenericPeopleConversion
{
    public static TDTO ToDTO<TDTO, TData>(TData data)
    {
        throw new System.NotImplementedException("how?");
    }
}

The code used for testing would look like this:

using Xunit;

namespace ExtensionMethods70642141;

public class SmokeTests
{
    [Fact]
    public void Smoke()
    {
        var student = new Student { Name = "Student Name" };
        var teacher = new Teacher { Name = "Teacher Name" };

        var studentDto = student.ToDTO();
        var teacherDto = teacher.ToDTO();

        Assert.Equal(student.Name, studentDto.Name);
        Assert.Equal(teacher.Name, teacherDto.Name);
    }
}

Approach #1: Generics won’t help us much

At a first glance, one might be tempted to convert the arguments to constrained generics first, such that

namespace ExtensionMethods70642141;

public static class PeopleExtension
{
    public static StudentDTO ToDTO<TData>(this TData student) 
        where TData : Student
        => new()
    {
        Name = student.Name
    };

    public static TeacherDTO ToDTO<TData>(this TData teacher) 
        where TData : Teacher
        => new()
    {
        Name = teacher.Name
    };
}

But the moment you do that, you’ll be greeted with compiler error CS0111 informing you that there already is a method with the same name and argument:

Nope.cs(12, 30): [CS0111] Type 'PeopleExtension' already defines a member called 'ToDTO' with the same parameter types

One way to resolve this issue is to distribute the extension methods across different classes, e.g.

public static class PeopleExtension1
{
    public static StudentDTO ToDTO<TData>(this TData student)
        where TData : Student
        => new()
        {
            Name = student.Name
        };
}

public static class PeopleExtension2
{
    public static TeacherDTO ToDTO<TData>(this TData teacher)
        where TData : Teacher
        => new()
        {
            Name = teacher.Name
        };
}

Older compilers would fail even with this, but at some point in the recent past (as of 2022), a tie breaker was implemented to assist. Now in order to get the TDTO output type, we have to introduce both a new() constraint, as well as one on the proper DTO type as we would be unable otherwise to create an instance, nor assign the property:

public static class PeopleExtension1
{
    public static TDTO ToDTO<TDTO, TData>(this TData student)
        where TData : Student
        where TDTO : StudentDTO, new()
        => new()
        {
            Name = student.Name
        };
}

public static class PeopleExtension2
{
    public static TDTO ToDTO<TDTO, TData>(this TData teacher)
        where TData : Teacher
        where TDTO : TeacherDTO, new()
        => new()
        {
            Name = teacher.Name
        };
}

Now calling the extension methods still works:

[Fact]
public void Smoke()
{
    var student = new Student { Name = "Student Name" };
    var teacher = new Teacher { Name = "Teacher Name" };

    var studentDto = student.ToDTO<StudentDTO, Student>();
    var teacherDto = teacher.ToDTO<TeacherDTO, Teacher>();

    Assert.Equal(student.Name, studentDto.Name);
    Assert.Equal(teacher.Name, teacherDto.Name);
}

No sadly, we didn’t really gain anything: We still have to know the exact type in order to pick the right extension method. The moment we erase the actual types …

public TDTO Convert<TDTO, TData>(TData data) =>
    data.ToDTO<TDTO, TData>();

… the compiler calls us a clown using error CS0314:

SmokeTests.cs(21, 14): [CS0314] The type 'TData' cannot be used as type parameter 'TData' in the generic type or method 'PeopleExtension1.ToDTO2<TDTO, TData>(TData)'. There is no boxing conversion or type parameter conversion from 'TData' to 'ExtensionMethods70642141.Student'.

And since we just split the extension methods into multiple classes, we also cannot call the method directly anymore.

Scratch that.

Approach #2: Reflection with MethodInfo invocations

The next best thing to typing out an if cascade of every combination of accepted input and output type is to look types up at runtime. Since reflection is costly, we can cache our lookup results in a static dictionary, and now the only question is whether we want to use look up each specific method pessimistically once it is required, or to optimistically look up all ToDTO methods once, and only once. Below I will go with the first approach as it is slightly less convoluted.

The main part here will be the introspection of the PeopleExtension class for all its public static methods, finding every candidate named ToDTO that has an output type matching TDTOand exactly one argument matching the TData type:

var methodInfo = typeof(PeopleExtension)
    .GetMethods(BindingFlags.Static | BindingFlags.Public)
    .Where(method => 
           method.Name.Equals(nameof(PeopleExtension.ToDTO)))
    .Where(method => outputType == method.ReturnType)
    .FirstOrDefault(method => 
         inputType == method.GetParameters()
                            .SingleOrDefault()?
                            .ParameterType);

This provides us with the MethodInfo of the relevant method, if one exists (methodInfo would be null otherwise).

Since we are only looking at one specific class (namely PeopleExtension), there can be at most one Y ToDTO(X data) method per type X as otherwise the methods would differ only in their return type, which is forbidden. We can therefore introduce a static cache dictionary keyed by the concrete type X that stores the relevant MethodInfo, allowing us to skip the reflection call next time around:

private static readonly ConcurrentDictionary<Type, MethodInfo> _cache = new();

If we were to look up all methods at this point, a simple Type key might still be sufficient. The worst case is a call with unrelated arguments, e.g. ToDTO<TeacherDTO, Student>() in which case we would happily match with the StudentDTO ToDTO(Student data) method, then fail trying to cast the return values. But that is the main issue with runtime lookups in the first place, so the only problematic thing here is that the exception would look weird to the caller — but one could still double-check the retrieved MethodInfo.

Now the only thing left to do is to call the actual method, and we can achieve this with this beauty:

(TDTO)methodInfo.Invoke(null, new object?[] { data })!;

The null cue here is to inform the Invoke method that we are operating on a static class, and the array we’re passing in contains the arguments.

To sum it up:

private static readonly ConcurrentDictionary<Type, MethodInfo> _cache = new();

private static TDTO ToDTO<TDTO, TData>(TData data)
{
    var inputType = typeof(TData);
    var outputType = typeof(TDTO);
    if (!_cache.TryGetValue(inputType, out var methodInfo))
    {
        methodInfo = GetMatchingMethodInfo(outputType, inputType);
        if (methodInfo is null)
        {
            throw new InvalidOperationException($"No conversion from {inputType} to {outputType} was registered");
        }

        _cache.TryAdd(inputType, methodInfo);
    }

    return (TDTO)methodInfo.Invoke(null, new object?[] { data })!;
}

private static MethodInfo? GetMatchingMethodInfo(
    Type outputType, Type inputType) =>
    typeof(PeopleExtension)
        .GetMethods(BindingFlags.Static | BindingFlags.Public)
        .Where(method =>
            method.Name.Equals(nameof(PeopleExtension.ToDTO)))
        .Where(method => outputType == method.ReturnType)
        .FirstOrDefault(method => 
            inputType == method.GetParameters()
                               .SingleOrDefault()?.ParameterType);

Approach 3: Reflection with runtime compilation of Lambda expressions

The caching already removed the majority of concerns with the reflection-based approach, but we are still left with a dynamic invocation from the MethodInfo. One way to improve on this is to somehow get a delegate (i.e. function pointer) to call the method directly, and we’ll be doing this by wiring up a method call expression, then compiling it — at runtime.

The spot where we put the scalpel is right between having obtained a MethodInfo and storing it in the dictionary. If we were to write a specialized TDTO ToDTO<TDTO, TData>(TData data) method ourselves, knowing precisely that TDTO and TData are always referring to the specific types StudentDTO and Student, we would probably be doing something like this:

private TDTO ToDTO<TDTO, TData>(TData data) =>
   (TDTO)ConvertWithTypesErased((object)student);

private object ConvertWithTypesErased(object data)
{
    var student = (Student)data!;             // unary conversion
    var dto = PeopleExtension.ToDTO(student); // method call
    return (object)dto;                       // unary conversion
}

First, we’d have to forget about the TData and StudentDTO type signatures by indirecting through an object cast. If we wouldn’t do it, CS0030 would be there to greet:

ReflectedWithDelegateTests.cs(71, 23): [CS0030] Cannot convert type 'TData' to 'ExtensionMethods70642141.Student'

We then call the actual method, and convert back to the generic TDTO type. The ConvertWithTypesErased method above follows a Func<object, object> signature, and this is exactly what we’ll be using for our cache:

private static readonly ConcurrentDictionary<Type, Func<object, object>> _cache = new();

In order to model the ConvertWithTypesErased method above using System.Linq.Expressions we will start by defining a function parameter of type object dubbed dataObj. We will use this twice, once for defining the method and once for calling it. We then convert from object to our known type and pass that into a method call expression.

var inputObject = Expression.Parameter(
    typeof(object), "dataObj");

var inputCastToProperType = Expression.Convert(
    inputObject, inputType);

var callExpr = Expression.Call(
    null, methodInfo, inputCastToProperType);

When done, we convert the result back to object and wrap the entire tree into a lambda expression (reusing the aforemention dataObj parameter):

var castResultExpr = Expression.Convert(callExpr, typeof(object));

var lambdaExpr = Expression.Lambda<Func<object, object>>(
    castResultExpr, inputObject);

The only thing left to do here is to call Compile on the result and store it in the cache:

Func<object, object> toDto = lambdaExpr.Compile();
_cache.TryAdd(inputType, toDto);

From that point on, all we do is pass in our TData data value (as it trivially casts to object) and map the result back to TDTO when we return. To wrap it up:

private static readonly ConcurrentDictionary<Type, Func<object, object>> _cache = new();

private TDTO ConvertToDTO<TDTO, TData>(TData data)
{
    var inputType = typeof(TData);
    var outputType = typeof(TDTO);
    if (_cache.TryGetValue(inputType, out var toDto))
    {
        return (TDTO)toDto(data!);
    }

    var methodInfo = GetMatchingMethodInfo(outputType, inputType);
    if (methodInfo is null)
    {
        throw new InvalidOperationException($"No conversion from {inputType} to {outputType} was registered");
    }

    toDto = CompileLambda<TDTO, TData>(inputType, methodInfo);
    _cache.TryAdd(inputType, toDto);

    return (TDTO)toDto(data!);
}

private static Func<object, object> CompileLambda(
    Type inputType, MethodInfo methodInfo)
{
    var inputObject = Expression.Parameter(
        typeof(object), "dataObj");
    var inputCastToProperType = Expression.Convert(
        inputObject, inputType);
    var callExpr = Expression.Call(
        null, methodInfo, inputCastToProperType);
    var castResultExpr = Expression.Convert(
        callExpr, typeof(object));
    var lambdaExpr = Expression.Lambda<Func<object, object>>(
        castResultExpr, inputObject);
    return lambdaExpr.Compile();
}

Still not good enough.

Approach 4: Compile-time Source Generation

The fundamental problem with the above approaches is that they all operate at runtime. There is nothing stopping us from typing up any wild combination of types only to figure out days later in production — or during testing, possibly— that this doesn’t actually work. There might be structural ways to resolve this (tricks like the one employed by the Visitor Pattern do away very nicely with their compile-time safe double indirection), but one thing is for sure: We would like to make sure right away that impossible combinations never compile.

Given the nature of the problem we will not be able to achieve this: The author originally asked for a method where TDTO and TData are strictly generic, and C# doesn’t always give us enough flexibility to work around that. There are still two things we can do here:

• Build an analyzer that sanity-checks all calls and emits a compiler error where needed. I won’t do this here.
• Build a source generator that moves the type lookup logic to compile time, rather than runtime.

Is the second approach slightly pointless and overkill? Yes. Let’s go!

We first create a new netstandard2.0 project hosting our source generator and reference the Microsoft.CodeAnalysis.CSharp and Microsoft.CodeAnalysis.Analyzers dependencies as private assets.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>netstandard2.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
        <LangVersion>10</LangVersion>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.0.1" PrivateAssets="all" />
        <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3" PrivateAssets="all" />
    </ItemGroup>

</Project>

We then reference this new project in our original one specifying both OutputItemType=”Analyzer” and ReferenceOutputAssembly=”false”:

<ItemGroup>
    <ProjectReference 
        Include="..\SourceGenerators\SourceGenerators.csproj" 
        ReferenceOutputAssembly="false"
        OutputItemType="Analyzer" />
</ItemGroup>

<PropertyGroup>
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
    <CompilerGeneratedFilesOutputPath>
        $(BaseIntermediateOutputPath)Generated
    </CompilerGeneratedFilesOutputPath>
</PropertyGroup>

If we were to skip ReferenceOutputAssembly, our source generator would end up as a regular runtime dependency, which is not what we want. It does, after all, simply generate source. The EmitCompilerGeneratedFiles bit is not required, but helps a lot when debugging the generated code. The generated source files will end up in the obj directory of the project.

If you’re new to source generators, there’s a lot to untangle and I will only go about things briefly. There is excellent material both on YouTube (e.g. here) and in blogs, so take a deep breath and have a dive.

As far as we are concerned, we’ll be using these concepts:

• ISourceGenerator and the [Generator] attribute; this is the bare minimum,
• partial classes and methods,
• ISyntaxReceiver to speed up compilation time,
• MethodDeclarationSyntax syntax nodes and IMethodSymbol from the semantic model to find the main extension methods, as well as
• InvocationExpressionSyntax syntax nodes to detect calls to the conversion method.

To begin with, we will create the following placeholder type:

namespace ExtensionMethods70642141;

public static partial class GenericPeopleConversion
{
    public static partial TDTO ToDTO<TDTO, TData>(TData data);
}

This method implements the code requested by the original StackOverflow author; it is the job of the source generator to actually provide the partial implementation of this method.

The boilerplate for our source generator will look like the following: We implement the ISourceGenerator interface and tag the class with the [Generator] attribute. In itself, this generator would be called for every piece of code syntax observed by the compiler. This might end up being prohibitively slow for large code bases (i.e., not ours), so we assist the compiler by taking note only of very specific pieces of code using an ISyntaxReceiver that we register in the Initialize method of the source generator. The main job of our syntax receiver will be to find possible ToDTO extension methods, but we will go a step further by detecting calls to our TDTO ToDTO<TDTO, TData>(TData data) method as well. This way, we can generate dispatching code specific to the types actually used in the code, and not for everything that might occur. Here’s how it looks so far:

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;

namespace SourceGenerators;

[Generator]
public sealed class ToDTOGenerator : ISourceGenerator
{
    public void Initialize(GeneratorInitializationContext context)
    {
        context.RegisterForSyntaxNotifications(() => 
            new SyntaxReceiver());
    }

    public void Execute(GeneratorExecutionContext context)
    {
        // ...
        context.AddSource("GenericPeopleConversion.Generated.cs", 
                          source: "/* TODO */");
    }

    private sealed class SyntaxReceiver : ISyntaxReceiver
    {
        public HashSet<MethodDeclarationSyntax> CandidateMethods { get; } = new();
        public HashSet<InvocationExpressionSyntax> CandidateInvocations { get; } = new();

        public void OnVisitSyntaxNode(SyntaxNode syntaxNode)
        {
            // TODO ...
        }
    }
}

The implementation of the syntax receiver benefits drastically from C# 8+’s pattern matching. TheOnVisitSyntaxNode method will be called by the compiler for each syntax element of the code, and we will be saving the interesting bits for later. As far as the ToDTO extension methods are concerned,

• the syntax node must resemble a method declaration (MethodDeclarationSyntax); moreover,
• the method declaration must have the identifier ToDTO,
• it must have exactly one argument (the data bit) and
• it must have at least one modifier (static)

When it comes to calls to our conversion method to be,

• the syntax node must resemble a method call (InvocationExpressionSyntax) with exactly one argument (the data bit), and
• it must resemble an access of the (static) class’ member ToDTO.

private sealed class SyntaxReceiver : ISyntaxReceiver
{
    public HashSet<MethodDeclarationSyntax> CandidateMethods { get; } = new();
    public HashSet<InvocationExpressionSyntax> CandidateInvocations { get; } = new();

    public void OnVisitSyntaxNode(SyntaxNode syntaxNode)
    {
        // Find candidates for "ToDTO" extension methods.
        // We expect exactly one input parameter as
        // well as a "static" modifier.
        if (syntaxNode is MethodDeclarationSyntax
            {
                Identifier.Text: "ToDTO",
                ParameterList.Parameters.Count: 1,
                Modifiers:
                {
                    Count: >= 1
                } modifiers
            } mds &&
            modifiers.Any(st => st.ValueText.Equals("static")))
        {
            CandidateMethods.Add(mds);
        }

        // Likewise, the method invocations must be to a
        // "ToDTO" method with exactly one argument.
        if (syntaxNode is InvocationExpressionSyntax
            {
                ArgumentList.Arguments.Count: 1,
                Expression: MemberAccessExpressionSyntax
                {
                    Name.Identifier.ValueText: "ToDTO",
                }
            } ie)
        {
            CandidateInvocations.Add(ie);
        }
    }
}

I’ll be upfront: There is currently no sane way to debug any of this and not everything makes immediate sense. The best way I found here is to repeatedly run dotnet clean and dotnet build and write fake code (wrapped in /* */comment blocks) to be inspected in the obj directories — hence the EmitCompilerGeneratedFiles property mentioned earlier on.

With the syntax receiver in place, we can flesh out the Execute method of the source generator. I’ll show the code first, the explanation follows immediately:

public void Execute(GeneratorExecutionContext context)
{
    var compilation = context.Compilation;
    var syntaxReceiver = (SyntaxReceiver)context.SyntaxReceiver!;

    // Fetch all ToDTO methods.
    var extensionMethods = syntaxReceiver.CandidateMethods
        .Select(methodDeclaration => compilation
            .GetSemanticModel(methodDeclaration.SyntaxTree)
            .GetDeclaredSymbol(methodDeclaration)!)
        .Where(declaredSymbol => declaredSymbol.IsExtensionMethod)
        .ToImmutableHashSet<IMethodSymbol>(SymbolEqualityComparer.Default);

    // Fetch type type arguments of all calls to the ToDTO methods.
    var usedTypeArguments = syntaxReceiver.CandidateInvocations
        .Select(methodDeclaration => compilation
           .GetSemanticModel(methodDeclaration.SyntaxTree)
           .GetSymbolInfo(methodDeclaration).Symbol as IMethodSymbol)
        .Where(symbol => symbol?.TypeParameters.Length == 2)
        .Select(symbol => 
            new InputOutputPair(
                symbol!.TypeArguments[0], symbol.TypeArguments[1]))
        .ToImmutableHashSet();

    var code = GenerateConversionMethodCode(extensionMethods, usedTypeArguments);
    context.AddSource("GenericPeopleConversion.Generated.cs", code);
}

As mentioned initially, the goal is of a source generator is to generate source code, which is exactly how the method ends. I will get into the GenerateConversionMethodCode method later on; at this point it’s only interesting to know that it returns the generated source code as a string.

As for the rest: Not everything makes sense on the syntax node level, so the first thing we do is obtaining semantic information from the syntax by getting the Compilation from the context and calling its GetSemanticModel method on every interesting node’s SyntaxTree.

For method declarations, we then grab the IMethodSymbol from said semantic model and check its IsExtensionMethod property; together with the syntax receiver, this ensures that we end up with exactly the methods we need. We pass both sets into the GenerateConversionMethodCode method, allowing it to see all possible conversion methods, as well as all their (actual) usages.

For the method invocations, things are a bit more complected, but follow the same approach: We get the semantic model, obtain the IMethodSymbol and then verify that the method we are calling has exactly two type parameters, namelyTDTO and TData. We stash away all type arguments, leaving us with a set of all used concreteTDTO-TData combinations. The InputOutputPair here is simply a readonly struct that uses SymbolEqualityComparer.Default internally:

private readonly struct InputOutputPair: IEquatable<InputOutputPair>
{
    public InputOutputPair(
        ITypeSymbol dtoType, ITypeSymbol dataType)
    {
        DtoType = dtoType;
        DataType = dataType;
    }

    public ITypeSymbol DtoType { get; }
    public ITypeSymbol DataType { get; }

    public bool Equals(InputOutputPair other) => 
        DtoType.Equals(other.DtoType,
                       SymbolEqualityComparer.Default) && 
        DataType.Equals(other.DataType,
                        SymbolEqualityComparer.Default);

    public override bool Equals(object? obj) => 
        obj is InputOutputPair other && Equals(other);

    public override int GetHashCode()
    {
        unchecked
        {
            return (SymbolEqualityComparer.Default
                        .GetHashCode(DtoType) * 397) ^ 
                    SymbolEqualityComparer.Default
                        .GetHashCode(DataType);
        }
    }
}

Note that HashCode.Combine isn’t available for netstandard2.0, so I’m going with Rider’s default implementation.

Now for the ugly bit: The GenerateConversionMethodCode method. In order to refer to the correct method and type names, we use a handful of helper methods that generate the fully qualified type and method names (Namespace.Type and Namespace.Type.Method) from the IMethodSymbol interfaces:

private static string GetMethodFullName(IMethodSymbol methodSymbol)
{
    var methodReceiverType = methodSymbol.ReceiverType!;
    return
        $"{methodReceiverType.ContainingNamespace.Name}.{methodReceiverType.Name}.{methodSymbol.Name}";
}

private static string GetReturnTypeFullName(IMethodSymbol methodSymbol)
{
    var returnTypeNamespace = methodSymbol.ReturnType
        .ContainingNamespace.Name;
    var returnTypeName = methodSymbol.ReturnType.Name;
    return $"{returnTypeNamespace}.{returnTypeName}";
}

private static string GetParameterTypeFullName(
    IMethodSymbol methodSymbol)
{
    var parameterTypeNamespace = methodSymbol
        .Parameters.Single()
        .Type.ContainingNamespace.Name;
    var parameterTypeName = methodSymbol
        .Parameters.Single()
        .Type.Name;
    return $"{parameterTypeNamespace}.{parameterTypeName}";
}

With that in place, our GenerateConversionMethodCode method simply iterates all entries in the extensionMethods set, tests whether it is actually referenced by the code, obtains the type names and builds up the partial method by adding more and more if statements checking the types, then dispatching to the appropriate ToDTO method.

private static string GenerateConversionMethodCode(
    ImmutableHashSet<IMethodSymbol> extensionMethods,
    ImmutableHashSet<InputOutputPair> usedTypeArguments)
{
    var sb = new StringBuilder();

    sb.Append(@"
using System;

namespace ExtensionMethods70642141;

public static partial class GenericPeopleConversion {
public static partial TDTO ToDTO<TDTO, TData>(TData data)
{");

    foreach (var methodSymbol in extensionMethods)
    {
        if (!usedTypeArguments.Contains(
                new InputOutputPair(
                    methodSymbol.ReturnType, 
                    methodSymbol.Parameters.Single().Type)))
        {
            continue;
        }

        var methodName = GetMethodFullName(methodSymbol);
        var parameterType = GetParameterTypeFullName(methodSymbol);
        var returnType = GetReturnTypeFullName(methodSymbol);

        sb.AppendLine($@"
    if (typeof(TData) == typeof({parameterType}) && typeof(TDTO) == typeof({returnType})) {{
        return (TDTO)(object){methodName}(({parameterType})(object)data);
    }}");
    }

    // TODO: Add an analyzer that prevents this from happening. :)
    sb.Append(
        @"        throw new InvalidOperationException(""No method found to convert from type {typeof(TData)} to {typeof{TDTO}}"");");
    sb.AppendLine(@"
}
}");

    return sb.ToString();
}

It also throws an exception for good measure (I did promise it’s not going to help much!) to ensure no invalid combination is called — and reminds you to write an analyzer.

The only thing left to do now is to actually call the method:

using System;
using Xunit;

namespace ExtensionMethods70642141;

public class SourceGeneratedTests
{
    [Fact]
    public void Works()
    {
        var student = new Student { Name = "Student Name" };
        var teacher = new Teacher { Name = "Teacher Name" };

        var studentDto = GenericPeopleConversion.ToDTO<StudentDTO, Student>(student);
        var teacherDto = GenericPeopleConversion.ToDTO<TeacherDTO, Teacher>(teacher);

        Assert.Equal(student.Name, studentDto.Name);
        Assert.Equal(teacher.Name, teacherDto.Name);
    }

    [Fact]
    public void InvalidConversionFails()
    {
        var student = new Student { Name = "Student Name" };

        var invalidCall = () => GenericPeopleConversion.ToDTO<TeacherDTO, Student>(student);

        Assert.Throws<InvalidOperationException>(invalidCall);
    }
}

and when used like this, the source generator creates an GenericPeopleConversion.Generated.cs of this content:

using System;

namespace ExtensionMethods70642141;

public static partial class GenericPeopleConversion 
{
    public static partial TDTO ToDTO<TDTO, TData>(TData data)
    {
        if (typeof(TData) == typeof(ExtensionMethods70642141.Teacher) && typeof(TDTO) == typeof(ExtensionMethods70642141.TeacherDTO)) {
            return (TDTO)(object)ExtensionMethods70642141.PeopleExtension.ToDTO((ExtensionMethods70642141.Teacher)(object)data);
        }

        if (typeof(TData) == typeof(ExtensionMethods70642141.Student) && typeof(TDTO) == typeof(ExtensionMethods70642141.StudentDTO)) {
            return (TDTO)(object)ExtensionMethods70642141.PeopleExtension.ToDTO((ExtensionMethods70642141.Student)(object)data);
        }

        throw new InvalidOperationException("No method found to convert from type {typeof(TData)} to {typeof{TDTO}}");
    }
}

Comment out both calls to GenericPeopleConversion.ToDTO and the source generator creates an empty implementation:

public static partial class GenericPeopleConversion {
    public static partial TDTO ToDTO<TDTO, TData>(TData data)
    {    
       throw new InvalidOperationException("No method found to convert from type {typeof(TData)} to {typeof{TDTO}}");
    }
}

And that’s how to convert between types in increasingly absurd ways. That said, writing an actual Analyzer isn’t going to be that much harder after what we just pulled off, so I’ll leave it as an exercise to the reader.

Approach #5: Using AutoMapper

Now arguably there are insights to be gained in the above experiments. That said, if we make our peace with runtime resolution, then AutoMapper is a superior solution to the problem; in the example below, automatic name-based mapping is applied, but that’s obviously configurable.

var configuration = new MapperConfiguration(cfg =>
{
    cfg.CreateMap<Student, StudentDTO>();
    cfg.CreateMap<Teacher, TeacherDTO>();
});

#if DEBUG
configuration.AssertConfigurationIsValid();
#endif

var mapper = configuration.CreateMapper();

var student = new Student { Name = "Student Name" };
var teacher = new Teacher { Name = "Teacher Name" };

var studentDto = _mapper.Map<StudentDTO>(student);
var teacherDto = _mapper.Map<TeacherDTO>(teacher);
var invalidCall = () => mapper.Map<TeacherDTO>(student);

Assert.Equal(student.Name, studentDto.Name);
Assert.Equal(teacher.Name, teacherDto.Name);
Assert.Throws<AutoMapperMappingException>(invalidCall);

As usual, use the right tool for the job.

Thanks for reading, stay safe, stay vaccinated and stay curious!