simst.im

Porting a Fortran EMP Simulation for Henry Newton

2026-08-16T00:00:00-05:00

Sometime around 2016, a guy named Henry Newton started showing up to our local meetup in New Orleans. He had a pretty interesting career and some cool stories. The reason he came out, I think other than just talking with some like minded people, was he wanted to revive an old project he worked on. The way he told it, he had some Fortran code from the 70s for simulating EMP attacks. He was interested in porting it to a modern language and studying how it could impact the electrical grid.

I found this kind of interesting, but it sounded like a big task at the time. And he said he needed help, but had a team working on it already. It felt like it was too much to help with and sounded like other people were helping anyway. So I forgot about it.

At some point he stopped coming around and we unfortunately learned in 2023 he died. I was thinking the other day about this as I was reading about the bun port. He had plenty of knowledge but just needed to find some willing Fortan, or at least Fortran-curious, developers (not easy to do). I feel if he was alive today he’d probably be excited to just drive a coding agent and build this out himself.

Agents are particularly good at porting code. So I decided why not give it a shot for him. Luckily it seems he posted the code online already.

The workflow behind the prompt I made was:

Get this code to compile and output reasonable data
Use this as an oracle to port the code to typescript
Create a simple webpage allowing me to manipulate the parameters and simulate an attack

I ran this and went to bed. Claude Code was able to get this running pretty easily. Here is the site it made and here is the code.

I honestly didn’t scrutinize this much or iterate on it. It seemed to follow the instructions correctly and the output seems reasonable. Just thought it was an interesting experiment, but don’t go making national security policy based on this.

Regrets

I realized actually, from looking at the Fortran code, it was surprisingly small and elegant and I probably could have figured this out back then on my own. Look at this subroutine for runge-kutta integration:

      SUBROUTINE RNGKUT (E1,E,R,H,SIGMA,COMPTJ)                         RNKT1010
C                                                                       RNKT1020
C         E(I+1) IS CALCULATED FROM E(I)                                RNKT1030
C         USING THE RUNGE-KUTTA METHOD                                  RNKT1040
C                                                                       RKKT1050
      DATA C/3.0E8/,RMUO/12.56637E-7/                                   RNKT1060
      EFUN(R,E)=-(1./R+C*RMUO*SIGMA/2.)*E-COMPTJ*C*RMUO/2.              RNKT1070
      RK1=H*EFUN(R,E)                                                   RNKT1080
      RK2=H*EFUN(R+H/2.,E+RK1/2.)                                       RNKT1090
      RK3=H*EFUN(R+H/2.,E+RK2/2.)                                       RNKT1100
      RK4=H*EFUN(R+H,E+RK3)                                             RNKT1110
      E1=E+(RK1+2.*(RK2+RK3)+RK4)/6.                                    RNKT1120
      RETURN                                                            RNKT1130
      END

I really regret not volunteering to at least look at his code, but this a lesson learned I suppose.

How to get recruited

2026-06-21T00:00:00-05:00

As both an IC and executive, I’ve been recruiting for software roles for about 10 years, off and on. I’m starting to recruit again for my new role, and I find myself giving the same advice over and over so I thought I’d write it down. This post is about the mechanics of recruiting. It’s about how to make it smooth and easy for the recruiter, how to give hiring signals, and how to avoid red flags. It’s not about how to cultivate an interesting career or choose a company to work at. Those are different topics. This is from my subjective standpoint so take time to consider who you’re talking to when applying it. This advice would work best when interacting with a hiring manager or IC (someone who actually works at the company or on the team), not necessarily an HR person or third party recruiter.

How to reach out

When recruiting, you’ll usually see someone post a link to a role around their networks. If you have an interest in the role or company, the best approach is to DM or email that person directly. Don’t start a public conversation. The goal here is to appear thoughtful (like you’ve actually thought about why you’d take this job over something else) and respectful of their time by giving them everything they need. Your initial message should be as short as possible, but should cover these things:

Who are you and how do I know you?

Ideally you can reference mutuals here for credentialing. If you do have mutuals, it helps the recruiter more easily vet you. If not, that’s okay, but check. Provide at least one link that gives a clear overview of who you are, like a website, portfolio, or resume.

Demonstrate knowledge and interest in the person or the company

Take some time to research both the recruiter and the company. Have something thoughtful to say or ask. It sounds trite but 90% of people who reach out to me want to apply without even knowing what the company does.

Provide a clear call to action

At the end of the message, arrange for some kind of next step and provide the organizing mechanism (like provide days you are available or a link to your schedule). Aim for a zoom call or in person chat. State you want to take this time to learn more about the company and role.

Don’t apply for the job unless asked

Don’t submit an application unless you’re asked to. You want to come across that you are considering this over other options, which ideally, you should be. Also, our primary goal here is to skip the front door which is just a black hole for resumes. If we follow the right steps then applying will become a formality.

Interview the recruiter

Before meeting up, take time to research what the company and team does. Have some theories about how you’d fit in. Prepare some questions that allow you fill in the gaps. Ask long term questions about company direction and growth. Employment can be a pretty serious, long term commitment. Lots of your financial and mental health can be tied into the company, you should be equally as (if not more) careful about joining a company than the other way around. Presenting yourself as someone who thinks that way, will come across as a good hiring signal. You should present as someone who takes time to think before making a commitment and that you have other options to consider.

At the end of this meeting, ask them for feedback on your resume and how you can better tailor it to the job. Also ask them if you can speak to someone else on the team, ideally the hiring manager. If it’s a small startup ask to speak with a founder or CEO. Repeat this as much as you want before actually applying. Utilize the feedback in each step to hone your message.

Ask for a referral

Most big tech companies offer referral bonuses to employees as an incentive to spend time recruiting. This is pretty easy money for an employee. I’ve gotten bonuses as high as $10k for recruiting single, senior engineer. If you get a referral as a candidate, you often automatically skip some of the filtering steps in the process and increase your likelihood of getting an interview. You also have an “inner champion” in the process. When the company is dragging their feet, you can back channel the referral to push them along. Oftentimes if the interview panel is divided, they will bring on the referrer to argue your case and push you over the edge.

All this to say, get a referral. Ask the recruiter what is needed for a referral and advice on how to get one. Usually someone has to have worked with you or can vet your work, but that’s not always a strict rule. There is almost always a way to get one.

Applying for the job

If you do all this, applying for the job will become a formality. Doing this has yielded a near 100% success rate of at least getting an interview for me. I’ve done interviews and even gotten offers this way without ever formally applying for the job. Of course your experience and skills will matter a lot to all this. But the goal here is to maximize your chances of getting an interview. The rest is on you.

Final Note: This feels icky

It feels kind of cringey writing about things you should do to “appear” to be a thoughtful person. It may feel cringey to read too. For many of us in tech, it also feels icky to perform this kind of social engineering. The way I feel about it is, if you’re reading this you’re already a thoughtful person and probably a good hire. And I just want to help you signal that and better sell yourself.

It’s getting harder and harder to break through the noise. If you’ve ever been on the receiving end of this hiring process, you’ll know that you can get thousands of submissions for a single role a week. Many of them real people using bots to apply to 100s of jobs, many of them just AI applying to scam you, and a small handful of thoughtful people who really want this specific job. But there isn’t enough human labor to find those people in the pile. If you blindly submit a resume it probably won’t get looked at. Or it will be looked at for just a brief moment by someone non-technical or a bot. With a lot of experience on the other end, I can say the good news is the tiniest little bit of effort actually makes you stand out.

Why I Joined Figma

2025-12-07T00:00:00-06:00

A few months ago I left my role as CTO of Dylibso. After lots of conversations with dozens of companies, I’ve decided to take a staff engineering role at Figma. I wanted to take a moment to write about why I chose Figma and what may come next.

First, I’ve admired Figma’s engineering culture for a long time. Figma was an early innovator in WebAssembly and extensibility. Both of these bets on Wasm make them spiritually aligned with my work on projects like Extism and Chicory and the beliefs behind them. As I started to interview people at the company, I came to see that it’s packed with Wasm knowledge and many kindred spirits.

I really respect their company culture as well. At its core, the company is building tools for people who breathe new products and experiences into existence. As someone who has always served developers, I’m highly motivated when I’m enabling people to create new things.

Second, I believe Figma is still in its early days as a business and platform. When I joined Datadog, I recognized that they were undervalued and had all the ingredients to win the observability space, and I see Figma as having the same ingredients for its market. I think over my time at Figma, it will continue to evolve as a platform and capture more of the lifecycle of creating digital experiences. And regardless of where we are on the AI S-curve, the value will continue to accrue up the stack to design.

What’s after Figma?

I’d like to be at Figma for as long as I can, but I do have bigger goals, and the decision to join Figma had to fit into that. Ultimately, I want to start another company. This could change as my life changes out from under me, but that’s my current aim. I do see Figma as a good springboard for this. The talent at the company, and exposure to new ideas, feel limitless.

How Extism Built SDKs for Dozens of Languages in Months, Not Years

2025-11-06T00:00:00-06:00

When we started the Extism ecosystem in 2022, the main challenge we had was the scale of SDK languages we needed to support. The goal for 1.0 was to run Wasm plug-ins from inside any application written in any language.

Since the core of Extism is a rust library, that means writing and maintaining a bindings SDK for every language out there. Without some kind of unified approach, each SDK requires a lot of overhead. It takes particular knowledge of writing an extension for the language, managing memory and ownership concerns, and maintaining it as the core library is evolving and churning in the early days. But, with our approach and a team of three, we were able to do this in a matter of months.

I’ve seen this type of deployment pattern come up a lot, with a single native core and a language ecosystem around it, so I thought I’d write a little about how we tackled the problem and how I might do it differently if we started over today.

Extism started with a C ABI

The core idea was, instead of writing custom extensions for each language, we distributed the library as a single, native shared object we called libextism. Each SDK is a light wrapper that loads this library via dlopen or equivalent and speaks to it via FFI.

Shipping the library with a C ABI interface is the lowest common denominator of all languages. Every language can call into C functions and most have very mature tools to manage this.

Step 1: Write the C ABI

If you’re writing C or C++, creating this shared object is pretty simple. If you’re using rust, you’ll need to create a module that exports all your c types and maps to and from rust using extern "C" For example:

// Types must have a c representation
#[repr(C)]
pub struct ExtismVal {
    t: ValType,
    v: ValUnion,
}

// Functions can be exported using `extern "C"`
#[no_mangle]
pub unsafe extern "C" fn extism_current_plugin_memory_length(
    plugin: *mut CurrentPlugin,
    n: ExtismMemoryHandle,
) -> Size {
  // ...
}

Step 2: Distribute

Set up a GitHub action to compile to all the targets you want to support and create release artifacts. You want to compile with the crate type cdylib which outputs a target specific shared object file.

Here’s an example release with all the architectures we support.

Step 3: Create the SDKs

Most languages have built-in FFI support. We ship libextism with a header file libextism.h which acts as the IDL to the ABI. Lean heavily toward using the standard bindgen for your language as they will automate the binding code for you. Bindings aren’t too hard to write, but you can get bogged down in details and mistakes regarding memory management and ownership. The code it generates is not always pleasant, but you can create an idiomatic native layer to decouple from generated code.

Note: We started off putting all the libraries in the monorepo for building and testing simplicity, but eventually moved the SDKs out to their own repos.

Step 4: Install Script

Extism ships with a CLI for a few purposes. One core features we added is pullling down the latest release for your target machine. and putting it in a place where the SDKs can easily find it.

Iterating and Scaling

This initial setup let us build SDK support very quickly and easily. It made maintenance easier as it allowed us ship updates to the core Extism runtime as we improved it without changing any of the SDKs. It also made it easy for the community to pick up new languages and experiment. They just had to write or generate FFI bindings to the shared object, which you can mostly do with limited low-level knowledge.

Eventually, you’ll find yourself wanting a more integrated experience. You can start streamlining languages as they pick up adoption. For example, some package managers might allow you to invoke or replace the need to use the Extism CLI to get the shared object. Or maybe you can bundle the shared object inside the package, a “fat” package. Or maybe you decide to upgrade and write some typesafe bindings in rust, and now you’re relying on the rust library directly and statically compiling it in.

This is mostly how Extism grew. As languages picked up steam we focused on improving the DX for each in iterative steps. But we always have the fallback shared object process to bootstrap new platforms.

How I’d do it today

The Extism philosophy of “running everywhere” is fairly extreme. In practice, especially if you are shipping some kind of end product, it makes sense to focus your efforts on a few languages up front. These days I would recommend having a good strategy for your native language (rust in our case), python, and javascript. That’s going to cover the majority of your early users. It’s also going to force you to think about supporting multiple concurrency paradigms up front.

But still, I think releasing a shared object allows for easy experimentation and low maintenance overhead and is worth doing up front, especially if you have a developer focused project. If I were to start over today, I’d consider a system to manage this for you like UniFFI.

Note: I can’t recall exactly who originated this architecture, but I can’t take credit. It was either Zach or Steve.

The AI Coding Rorschach Test

2025-11-01T00:00:00-05:00

“Today, more than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers. This helps our engineers do more and move faster.”

— Sundar Pichai, Forbes

If you work as a software engineer, you’ve probably seen many statements like this recently. They’re vague enough to be interpreted in a hundred different ways, and may even be saying nothing at all. But I’ve noticed that reactions to these statements tend to fall into two broad categories.

Two Perspectives

This diagram is an oversimplification, but it illustrates two ways to interpret this kind of statement.

Perspective 1 reflects a zero-sum mindset that views coding as a static block of “work” to be done. It imagines the job of software engineering as shoveling a pile of dirt. If AI is shoveling 25% of the dirt, then you now need only 75% of the engineering capacity.

This is an uninformed view, but it’s understandable to feel this way, especially if you’re inexperienced or struggling to find a job. The media amplifies this narrative because executives want to communicate it to investors. Massive layoffs can then be spun as “embracing AI” rather than as the result of overhiring during ZIRP or shrinking profit margins.

Perspective 2 is closer to how companies actually operate. It’s important to understand that code itself has no inherent value; it’s an enabling input. It’s a resource that produces value only once it’s compiled, deployed, and integrated into real systems. Economically, code has similar properties to other enabling inputs like energy.

First, it exhibits a property of non-satiation. Even at the company level, there is a nearly infinite demand for code. The amount of code a company writes is proportional to its capacity to produce code, not its demand to consume that code. We know this is true because we’ve been making coding more efficient for decades, yet demand continues to grow. This likely reflects a form of Jevons Paradox and will continue on into the future.

Second, the efficiency of code, kind of like energy, is highly elastic. A lightbulb at the turn of the century required far more watts to produce the same light output. Code is even more dynamic. As a thought experiment, imagine writing millions of lines of code that accomplish the same thing as a single line. It’s so easy to write wasteful code. The amount of code you have says nothing about the value it creates.

Conclusion

If you’re new to the industry or an outsider, you should resist all the market forces that want you to doom spiral into a zero sum mindset. We’ve seen this play out for decades: a world in which coding is easier and more abundant creates more value for the world; and being a software engineer allows you to capture that value. Even if your slice may be proportionally smaller, it will still grow as the pie becomes much bigger.

I Am Unemployed

2025-09-10T00:00:00-05:00

I left my role as CTO of Dylibso at the end of August. I’m not really doing anything with the exception of a few short term contracts for fun. I’m talking to lots of founders and engineering leaders about what’s next but I’m not in a hurry to take a job. I went to a meetup and someone asked me what I do and I said “I am unemployed”. I’ve never really done nothing before. I’ve been between jobs, but never without a role to attach myself to.

I’m fortunate to now be in a place where I don’t really need a full-time job. We had a daughter last August. I’ve been spending the time with her as she’s learning to walk and talk. It’s really a great time to not have a job.

One Year of Good Sleep

2024-02-03T00:00:00-06:00

About one year ago, I wrote about getting control of my sleep. Anecdotally, it feels like it’s working. I’ve been feeling much better, but there are a lot of confounding variables that may be effecting my feelings. I’m also not sure if my quality of sleep is actually improving or if I’m just feeling like it is. I decided to pull out pandas and look at some data from my fitbit.

Deep Sleep and REM Sleep

Looking at total minutes of deep and REM sleep: They seem to be going up.

These sleep stages are the most important for health. As I expected in my last post, they were dangerously low during the pandemic.

Percentages

I also normalized the data into percentages (relative to total minutes slept). Those are going up as well. The formula for this is: 100 * minutes-<stage>/ minutes-asleep:

This means I’m getting less light sleep and more restorative deep and REM sleep. My REM is approaching 25% which I believe is good. My deep percentage is still lower than it should be. I’m currently getting ~75 mins a night which is still less than recommended.

Total Time Asleep

My total time asleep hasn’t shifted much. I was cycling b/w 400 and 420 minutes regularly. I hit a low at the end of 2022 right before I wrote my last blog post.

I need to ask a sleep doctor, but I think this is still not enough. I know some people don’t need a full 8 hours but I’m not sure if I am one of them or not.

Goals for the year

My goals are to increase my sleep time per night as well as increase my deep sleep percentage. I think the two main levers for this goal will be going to bed earlier and getting more consistent exercise. I’ll check back in next year.

We Already Tried That

2024-01-02T00:00:00-06:00

Early in my career, I had a bad habit. When someone new to the team verbalized a recognizable idea, I’d gleefully say:

“yeah, we already tried that and failed”

Naturally, they’d trash the idea and ask what they should do instead. I justified it to myself that I was using my experience to guide the team and protect them from wasting time or publicly failing.

One day a new junior engineer decided not to ask, but to just try one of these things. He came to me and said:

“I thought it would be helpful if we had X so I started a PR, is this a good idea?”

I looked at the PR. He succeeded where we all failed, and did so in one sitting. The PR was small and simple. How did he do this? Was he a genius? Did we miss something obvious?

Problems Change Over Time

When I looked at the code, I realized the constraints that prevented us from succeeding were no longer there. Things changed in the year since we last tried. This made his change trivial. The only thing that changed was we were living in a different time.

From that point on, I’ve tried my best to never say those words again. First, I think protecting people from failure is a bad model and even failing to do something ambitious that is your idea has a lot of value. Second, more often than not, people surprise you. They either succeed or get way further than you think, and it feels amazing to see someone shatter your perceptions of them. Finally, you’re discouraging people from being vulnerable and sharing their ideas and they are less likely to share the next one.

Isn’t this just…

As I’ve gotten more involved in open-source communities, I’ve noticed a macro version of this. When you’re creating something new, experienced people will come out of the woodworks with comments like:

“Isn’t this just a modern version of X”

where X is some previous solution that rhymes with yours or prior art.

These kind of comments are profoundly lazy at best and malicious at worst. Looking at it objectively, the comment is practically tautological. You’re not introducing any new information, all you’re doing is pointing out the obvious: that all progress (societal, technological, business, etc) is a series of cycles and rhyming ideas and there are few, if any, truly original ideas. Just because something has been tried, doesn’t guarantee it will fail. It says almost nothing about how the idea interacts with the world today or the people implementing it.

Why do we do this?

I’m not confident I understand why we do this, but let me be an armchair psychologist for a moment. I suspect we do this because we are protecting our ego.

If I point out that your idea is unoriginal, I’ve demonstrated my knowledge and experience and that increases my reputation of being a smart person.

More importantly, I’ve successfully convinced myself that the logical thing for me to do next is comfortable, blissful, inaction. I can safely say to myself:

“I could easily do this myself, but I am a logical, rational thinker and have calculated that it would be a waste of time. Thus I will save my effort for a truly original idea!”

I don’t need to exert any effort or risk my reputation, and I still get some of the credit for “having” the idea in the first place.

Stop this for your sake and ours

Look, as someone who sometimes is this person: you don’t want be this person! From a civic perspective, you should recognize that ideas are fragile, and you should want to contribute to a world where we have more people trying more ambitious ideas that they believe in. From a selfish, personal perspective, learn that you will be a much happier person if you genuinely desire to see other people succeed. You will be even happier when you learn to believe in your own ideas and don’t treat failure as an existential threat to your identity.

Running Wasm in Elixir & Erlang

2023-09-09T00:00:00-05:00

I think someone is going to need to write a native Elixir or Erlang Wasm runtime. Let me explain why.

Background

One of the first projects I took on with Extism was to build an Elixir Host SDK. The architecture is fairly simple. I use Rustler to create a NIF in Rust. The NIF depends directly on the Extism runtime, which is itself written in Rust. So when you pull down and compile the Extism Elixir package, it compiles the Extism runtime and wraps it in a NIF which is further wrapped by some Elixir code.

Host Function Support

I built this before we had support for Host Functions in the Extism Runtime, but it still works just fine. For example, we built all of our GameBox demo application with just this functionality.

There were some things we had to work around without host functions, but we did it with some hacks. Now that we’re grocery shopping features for Extism 1.0, I went to see what it would cost to add host functions to the Elixir SDK. That’s where I ran into an unfortunate limitation.

Async Host Functions

Implementing host functions would require the NIF to invoke a function at the Erlang / Elixir level. This appears to be impossible. Instead folks recommend calling enif_send to a process with the callback. It seems this is effectively what wasmex does. They spin up a thread, send a message to the callback then, wait for the result.

I don’t mean to criticise this solution. I think it’s a fairly elegant solution within the confines of the NIF. And I’m sure it works for simple cases where you just want a Wasm module to trigger some action outside of it’s memory boundary. But I think not being able to do fast, efficient, and synchronous host functions will eventually rule out a class of modules.

Starting From Scratch

The only way I can see to solve this is to write a native Erlang or Elixir Wasm runtime. From a native runtime you’d be able to interop very deeply with Erlang and calling synchronous host functions wouldn’t be a problem. You may even be able to write some kind of Wasm to BEAM Bytecode JIT to make it relatively fast.

I don’t intend to pick up this project, but if you do, please reach out to me! Also, please reach out if you think I’m wrong or know of some other ways around this problem.

On Wasm Use Cases

2023-08-26T00:00:00-05:00

The goal of Extism was to show that Wasm could be the “Last Plug-In System We Need” as Steve likes to say. I think we have made a good case to developers in the know, but simultaneously, I have had difficulty explaining to developers what a plug-in system is and why you need one.

I think part of the problem is that we forgot how and why to build plug-in systems when we moved to the web. Another issue is that Wasm is more than just a plug-in system and it can be tough to express how and what the use cases are exactly.

Instead of enumerating all the possible things you could do with Wasm, I wanted to attempt to distill the use cases to their simplest form, like a pattern. This pattern doesn’t cover everything, but it focuses on what I have said is the most compelling and differentiating property of Wasm: its sandboxing capabilities.

Application = Computation & Data

An application is, in its simplest form, two things: compute and data. Modern applications are themselves composed of many other applications, all of which exist across the internet and belong to different organizations. Each of these applications also have their own code and their own data.

Moving Data to Compute

The primary way we “integrate” these applications is by moving data from one application which owns it to the computation that needs it.

Let’s say one of our applications we are integrating with is Stripe. We might make HTTP calls to their API and do something with, or store, the result as data. Or they might, unprompted, send us a data payload representing an event in their system (a webhook).

There are many downsides to doing things this way. A couple examples:

Data Governance
- Once the data leaves your system, you’ve lost control of it. And remember, these applications are spread out through space which includes multiple legal jurisdictions.
High Latency & Low Reliability
- If an event occurs and the compute needed to handle it belongs to a company across the world, how can you resolve the event in a timely and reliable manner?

Moving Compute to Data

The other option is to move the compute to the data. This isn’t a new idea and is used frequently in the “big data” world. One example that comes top of mind is Apache Hadoop.

But why is this so rare in the web-development world? The primary issue again comes down to isolation. All of these cloud applications are multi-tenant. Shipping code to Stripe for them to run on our behalf when a customer event happens opens them up many risks. If we ship them Ruby code to run in their Rails application, what is to stop us from extracting secrets or data from them or their customers? The current solutions for isolation, e.g. a container, are too heavyweight for most of these scenarios. Imagine if we must send over a whole OS and VM to Stripe so they can boot it up and decide what discounts to apply to an invoice. The result has been that the downsides of sharing data are less than the downsides of sharing compute.

Wasm Makes This Option Viable

Of course, I think Wasm has changed the balance here. We can easily accept Wasm modules from our partners because they are small, fast, isolated, language independent, and can be run in process. They present as a function to the host developer and are easy to invoke. Because you control the whole world as the host, you can also ensure some level of determinism and reliability as well.

What Does the Future Look Like?

So, should we move all our applications to this new model? Not necessarily. But what we need to do is take stock of the last 20 years of development and ask ourselves what is possible now that we have this option.

Consistent Sleep

2023-01-22T00:00:00-06:00

I never slept consistently well. My primary issue is not being able to fall asleep. I tried to get it under control in my 20s, but once I would get into a rhythm, it would be destroyed by a bad night.

The COVID pandemic gave me a newfound dedication to fix some of these problems that affect my health. Finally, now in my mid 30s, I’m consistently hitting my sleep goals. When I do have an off night, I naturally correct course.

This has significantly changed my life for the better. I was afraid to make some of these sacrifices before, but I see now that it makes you healthier and more productive in the long run.

I thought it would be helpful to share the most important changes I made. Of course this is extremely personal and subjective, but it could be helpful to you if you share my same weaknesses and lifestyle.

I’ll list them in order of importance. The first two are vitally important and must strictly be obeyed. The others are optional reinforcers.

Deep Thought Curfew

Do not let your mind wander at night. I often get a second wind before bedtime and I spend the night focusing deeply on something. I used to see this as a kind of super-power. It was a wave of inspiration I could ride that came long after everyone had left for the day. It turned out to be the cornerstone of my poor relationship with sleep. Once my brain latches onto something, it is very hard to wind it down. Even if I made it to bed at a reasonable time, I’d still be thinking about it.

I see my brain as a flywheel now. At the end of the day, about 2-3 hours before bed, I try to decelerate the flywheel and keep it at a low RPM. This is a very hard thing to do but can work with practice. I no longer meditate, but the technique is the same. Once your brain starts to attach to something problematic, you stop it.

These intrusive thoughts depend on what you personally fixate on. For me, it’s programming and complex, interpersonal communication. There are techniques you can use like write it down and tell yourself you will pick it up tomorrow. But I find just being diligent and aware helps.

Consistent Wake Time

Do not set a bedtime, set a waketime. Set an alarm for the same time every day. This is important because it reinforces your circadian rhythm. It also ensures consistency. You may have a bad night where you cannot fall asleep, but waking up early will create a bullwhip effect for your next night’s sleep and keep you on track.

Set this time to something earlier than it ever needs to be. This prevents anxiety when you are having trouble sleeping and worried about not getting enough sleep. If I’m having trouble I tell myself I have enough time to sleep in if I need to, but I never do. Feeling like I can though, relieves the pressure.

Eventually, you will start waking up before your alarm, but always keep it set.

Melatonin

Set a notification for your wind-down period. My Fitbit sends me a notification at 10PM. This lets me know to shut down any deep thoughts I am dealing with. I also use it as a reminder to take melatonin (5 or 10 mg dosage). This isn’t necessary, but I feel it helps reinforce my circadian rhythm by giving me a rush of sleep drive that might otherwise be dulled by things like coffee during the day.

No Caffeine After Noon

I try to have one coffee a day, and only early in the morning. My new sleeping habits have allowed me to be more liberal with this rule than previously thought. I can sometimes have two coffees and be perfectly fine, but I try to stick to one for health reasons and never after noon.

Why WebAssembly

2023-01-01T00:00:00-06:00

I’ve done many disparate things in the first decade of my career. For the next decade, I want to narrow my focus on one domain. I believe WebAssembly is this domain. I’m writing this post to explain my thoughts to those who’ve been asking me and for my future self.

There are two perspectives to this question of “Why Wasm?”. Why does the industry need it? And why is this problem well suited for me?

Why we need Wasm

There are a few things that make Wasm different than say, the Java bytecode, but I want to focus on what I think is the most compelling: Sandboxing. By default, Wasm programs have no capabilities unless explicitly granted to them by the host. The program’s memory access is contained to a contiguous block of linear memory.

It can give you a similar level of isolation as an operating system with the lightweight nature of a thread. This makes it ideal for running untrusted code in your own infrastructure. My feeling is that running untrusted code has always been a very hard and scary feature that has been left to the domain of browser and OS engineers. What suboptimal patterns and architectures have we built based out of fear that can now be relaxed? One of the big accomplishments of Wasm will be to enhance security of existing patterns; but the security is also an opportunity in that we can more safely do things we instinctively see as unsafe today.

Why I want to work on it

A few times in my life I’ve encountered an idea so simple but so powerful that it changed the way I think. I remember feeling this way the first time I learned to program. I learned how to do loops and print text to the screen. With that tiny bit of knowledge, I spent the entire night making little console games that animated text.

I felt similarly when I first hand built a TTL computer. I made a toolchain and emulator to go along with it. I believe doing this re-invigorated my passion for computers. These moments always come with a feeling of fun and an uncontrollable outpouring of ideas. Everytime I get this feeling I’ve been rewarded for following it.

I feel similar feelings about WebAssembly. I still believe it’s a 10 yr journey to mass adoption and there will be challenges and pivots along the way. I’m not really sure where I’m going to end up when I come out of this, but I know I’m going in the right direction.

Cracker Barrel Data Science

2020-05-02T00:00:00-05:00

As a kid growing up in South-West Louisiana, my family spent a few weeks every winter in West Texas camping and hunting deer. A regular occurrence of these trips was eating at Cracker Barrel. Having spent the whole morning sitting in the woods alone and missing my N64, I got obsessed with the “jump all but one” game present on every table.

The the game begins by placing a tee in every hole except one (of your choice). You then “jump” the tees (similar to checkers) until no jumps can be made. The goal is to leave just one tee. The more tees you leave on the board, the more “ignorant” you are (as the rules state). Given that we went there so often, I was able to work out a solution and memorized it. I liked challenging myself every following year to see if I could remember the solution (or resynthesize it).

I went back to Cracker Barrel recently for the first time in 15 years. The game was still there. I gave solving it a shot but left 2 tees on the board. While I was playing, a lot of questions were going through my head. Stuff like:

Out of all possible games, how are they distributed? Is leaving one tee really that rare? It almost seemed harder to leave more than 4.
Does it matter where you leave the first hole?
How hard is it for the last tee to land in the hole you started with? (something I could do as a kid)

I realized that there probably aren’t that many possible games and we should be able to brute force some of these answers by simulating the game and enumerating every possible move. Perhaps there are purely analytical answers to these questions, but I’m personally able to express this much more easily in code than in math.

Creating a notation

For some reason, I got the itch to start this while I was in a cabin in Mississippi without my computer. So the first thing I did was write down a notation and a set of legal moves on a bookmark and I snapped a pic of it with my phone.

I chose a notation similar to chess (rank and file). The common board orientation will be with one of the tips of the triangle pointing at you. So each row will be labeled going up (A, B, C, D, E) and each column (1,2,3,4,5).

Because we will be storing the positions in a 2D list, we need a way to store positions and convert them to and from their cartesian point system and their chess-like notation.

@dataclass
class Position:
    file: int
    rank: int

    @staticmethod
    def from_str(pos):
        f, r = list(pos)
        r = int(r) - 1
        f = ord(f) - 97
        return Position(f, r)

    def __repr__(self):
        return "%s%s" % (chr(self.file + 97), self.rank + 1)

Example use:

p = Position.from_str('b2')
print(p.file) #=> 1
print(p.rank) #=> 1
print(p)      #=> b2

The Game object will then be constructed with an open_position (the hole you are leaving open). There will be a _board variable which contains the 2D list of Holes which are either filled or not.

class Hole:
    def __init__(self, pos):
        self.position = pos
        self.filled = True

class Game:
    def __init__(self, open_position='a1'):
        self._board = []
        rank = 5
        file = 1
        for r in range(rank):
            self._board.append([Hole(Position(f, r)) for f in range(file)])
            file += 1
        self._hole_at(Position.from_str(open_position)).filled = False
        self.moves = []

    # Allows us to fetch a hole at a position
    def _hole_at(self, pos):
        if pos.rank >= 0 and pos.rank < len(self._board):
            rank = self._board[pos.rank]
            if pos.file >= 0 and pos.file < len(rank):
                return rank[pos.file]

    def __repr__(self):
        s = []
        for r in reversed(range(len(self._board))):
            s.append("".join(["x" if hole.filled else "o" for hole in self._board[r]]))
        return "\n".join(s)

Example use. The game pretty-prints an ASCII board:

print(Game())
#=>   xxxxx
#=>   xxxx
#=>   xxx
#=>   xx
#=>   o

We now need a way to encode the rules of the game, and a way to get from one game state to another. First let’s describe the directions which a piece may possibly move as an Enum:

class Direction(Enum):
    NW = 0
    NE = 1
    E  = 2
    SE = 3
    SW = 4
    W  = 5

We can encode a “move” in the game as an object with 2 positions:

@dataclass
class Move:
    start: Position
    end: Position

    def __repr__(self):
        return "%s %s" % (self.start, self.end)

Even though there are 6 directions a peg may move, certain holes limit which direction a peg can move. For instance, from the tip of the pyramid, only 2 directions are possible.

Instead of calculating this during the runtime of the game, I decided to encode all possible “jumps” into each hole.

class Hole:
    def __init__(self, pos):
        # ...
        self.jumps = {}

@dataclass
class Jump:
    via: Hole
    dest: Hole

    def is_valid(self):
        return self.via.filled and not self.dest.filled

class Game:
    def __init__(self, open_position='a1', rank=5):
        # ...
        self._build_jumps()

    # ...

    def _build_jump(self, direction, rank, file):
        via = None
        dst = None
        if direction == Direction.NW:
            via = self._hole_at(Position(file, rank+1))
            dst = self._hole_at(Position(file, rank+2))
        if direction == Direction.NE:
            via = self._hole_at(Position(file+1, rank+1))
            dst = self._hole_at(Position(file+2, rank+2))
        if direction == Direction.E:
            via = self._hole_at(Position(file+1, rank))
            dst = self._hole_at(Position(file+2, rank))
        if direction == Direction.SE:
            via = self._hole_at(Position(file, rank-1))
            dst = self._hole_at(Position(file, rank-2))
        if direction == Direction.SW:
            via = self._hole_at(Position(file-1, rank-1))
            dst = self._hole_at(Position(file-2, rank-2))   
        if direction == Direction.W:
            via = self._hole_at(Position(file-1, rank))
            dst = self._hole_at(Position(file-2, rank))        

        if via and dst:
            return Jump(via, dst)

    def _build_jumps(self):
        for rank in range(len(self._board)):
            for file in range(len(self._board[rank])):
                for direction in Direction:
                    jmp = self._build_jump(direction, rank, file)
                    if jmp:
                        self._board[rank][file].jumps[direction] = jmp

Now we need a way to calculate which moves are valid and the ability to make them:

class Game:

    #...

    def move(self, move):
        src_hole = self._hole_at(move.start)
        dst_hole = self._hole_at(move.end)
        jump = [jmp for jmp in src_hole.jumps.values() if jmp.dest == dst_hole][0]
        if jump.is_valid():
            src_hole.filled = False
            jump.via.filled = False
            dst_hole.filled = True
            self.moves.append(move)
        else:
            raise Exception('Invalid move')

    def valid_moves(self):
        moves = []
        for rank in self._board:
            for hole in rank:
                if hole.filled:
                    for jmp in hole.jumps.values():
                        if jmp.is_valid():
                            moves.append((hole, jmp))

This should be enough to play the game:

g = Game()
print(g)
#=>   xxxxx
#=>   xxxx
#=>   xxx
#=>   xx
#=>   o

p1 = Position.from_str('c3')
p2 = Position.from_str('a1')
g.move(Move(p1, p2))
print(g)
#=>   xxxxx
#=>   xxxx
#=>   xxo
#=>   xo
#=>   x

Now that we can play a game, we need some code to play every possible game. We’ll do this by doing a depth-first search through every move. At each “branchng” in this tree of moves, we’ll clone the game state and the moves we made so far so we can save all the games in memory.

I’ve decided to call this thing that plays all possible games a “Session” due to lack of creative energy:

class Game:
  # ...
  def clone(self):
      return copy.deepcopy(self)


class Session:
    def __init__(self):
        self.log = []

    def walk(self, game=Game()):
        mvs = game.valid_moves()
        if not mvs:
            if (len(self.log) % 10000 == 0):
                print("Completed Games: %d" % len(self.log))
            self.log.append(game.clone())
            return
        for move in mvs:
            start, jmp = move
            checkpoint = game.clone()
            # Make the move
            game.move(Move(start.position, jmp.dest.position))
            # Perform all other future moves in this branch
            self.walk(game)
            # restore the game from checkpoint and execute next move
            game = checkpoint

You run the session like so:

sess = Session()
sess.walk()

And now sess.log contains all possible games.

print(len(sess.log))
#=> 568630

So it appears there are 568630 possible games that start with this configuration.

What about the distribution of scores?

def remaining_tees(game):
    holes = [hole for rank in game._board for hole in rank]
    return len(list(filter(lambda h: h.filled, holes)))

tees = np.array(list(map(remaining_tees, sess.log)))
plt.hist(tees, bins=np.arange(tees.min(), tees.max()+1))

In number of tees remaining:

29760 (5.2% of games)
139614 (24.5% of games)
259578 (45.6% of games)
123664 (21.7% of games)
14844 (2.6% of games)
844 (0.1% of games)
326

What’s pretty interesting is that there are 2 games that end in 8:

games = list(filter(lambda g: remaining_tees(g) == 8, sess.log))
len(games)
#=> 2
games[0]
#=> xxxxx
#=> oooo
#=> xxx
#=> oo
#=> o
games[1]
#=> xxxxx
#=> oooo
#=> xxx
#=> oo
#=> o

Here are the moves to achieve those game states:

games[0].moves
#=> [a3 a1, c5 a3, d4 b4, a4 c4, c3 c5, a1 c3]
games[1].moves
#=> [c3 a1, c5 c3, a4 c4, d4 b4, a3 c5, a1 a3]

We now have a framework for analyzing the game. Below is the full code (it’s meant to be run in a Jupyter notebook):

%pylab inline
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import scipy
import numpy as np
from scipy.stats import beta
import pandas as pd

import math
from enum import Enum
from dataclasses import dataclass
import random
import copy

class Direction(Enum):
    NW = 0
    NE = 1
    E  = 2
    SE = 3
    SW = 4
    W  = 5
  
@dataclass
class Position:
    file: int
    rank: int

    @staticmethod
    def from_str(pos):
        f, r = list(pos)
        r = int(r) - 1
        f = ord(f) - 97
        return Position(f, r)
        
    def __repr__(self):
        return "%s%s" % (chr(self.file + 97), self.rank + 1)
    
@dataclass
class Move:
    start: Position
    end: Position
    
    def __repr__(self):
        return "%s %s" % (self.start, self.end)
    
class Hole:
    def __init__(self, pos):
        self.position = pos
        self.filled = True
        self.jumps = {}

@dataclass
class Jump:
    via: Hole
    dest: Hole
        
    def is_valid(self):
        return self.via.filled and not self.dest.filled

class Game:
    def __init__(self, open_position='a1', rank=5):
        self._board = []
        file = 1
        for r in range(rank):
            self._board.append([Hole(Position(f, r)) for f in range(file)])
            file += 1
        self._hole_at(Position.from_str(open_position)).filled = False
        self._build_jumps()
        self.moves = []
            

    def move(self, move):        
        src_hole = self._hole_at(move.start)
        dst_hole = self._hole_at(move.end)
        jump = [jmp for jmp in src_hole.jumps.values() if jmp.dest == dst_hole][0]
        if jump.is_valid():
            src_hole.filled = False
            jump.via.filled = False
            dst_hole.filled = True
            self.moves.append(move)
        else:
            raise Exception('Invalid move')
    
    def valid_moves(self):
        moves = []
        for rank in self._board:
            for hole in rank:
                if hole.filled:
                    for jmp in hole.jumps.values():
                        if jmp.is_valid():
                            moves.append((hole, jmp))
        return moves        
        
    def clone(self):
        return copy.deepcopy(self)
        
    def _hole_at(self, pos):            
        if pos.rank >= 0 and pos.rank < len(self._board):
            rank = self._board[pos.rank]
            if pos.file >= 0 and pos.file < len(rank):
                return rank[pos.file]
   
        
    def _build_jump(self, direction, rank, file):
        via = None
        dst = None
        if direction == Direction.NW:
            via = self._hole_at(Position(file, rank+1))
            dst = self._hole_at(Position(file, rank+2))
        if direction == Direction.NE:
            via = self._hole_at(Position(file+1, rank+1))
            dst = self._hole_at(Position(file+2, rank+2))
        if direction == Direction.E:
            via = self._hole_at(Position(file+1, rank))
            dst = self._hole_at(Position(file+2, rank))
        if direction == Direction.SE:
            via = self._hole_at(Position(file, rank-1))
            dst = self._hole_at(Position(file, rank-2))
        if direction == Direction.SW:
            via = self._hole_at(Position(file-1, rank-1))
            dst = self._hole_at(Position(file-2, rank-2))   
        if direction == Direction.W:
            via = self._hole_at(Position(file-1, rank))
            dst = self._hole_at(Position(file-2, rank))        
            
        if via and dst:
            return Jump(via, dst)
    
    def _build_jumps(self):
        for rank in range(len(self._board)):
            for file in range(len(self._board[rank])):
                for direction in Direction:
                    jmp = self._build_jump(direction, rank, file)
                    if jmp:
                        self._board[rank][file].jumps[direction] = jmp
    
    def _coords(self, pos):
        f, r = list(pos)
        r = int(r) - 1
        f = ord(f) - 97
        return f, r
    
    def __repr__(self):
        s = []
        for r in reversed(range(len(self._board))):
            s.append("".join(["x" if hole.filled else "o" for hole in self._board[r]]))
        return "\n".join(s)
    
class Session:
    def __init__(self):
        self.log = []

    def walk(self, game=Game()):
        mvs = game.valid_moves()
        if not mvs:
            if (len(self.log) % 10000 == 0):
                print("Completed Games: %d" % len(self.log))
            self.log.append(game.clone())
            return
        for move in mvs:
            start, jmp = move
            checkpoint = game.clone()
            # Make the move
            game.move(Move(start.position, jmp.dest.position))
            # Perform all other future moves in this branch
            self.walk(game)
            # restore the game from checkpoint and execute next move
            game = checkpoint

Avoiding race conditions in GenServer

2017-03-02T00:00:00-06:00

Suppose we are making a multi-player game. We are choosing to store the state of the game in a struct (Game). The game will start without any players and players will individually choose to join the game. We’ve determined that there must be a maximum number of players (@max_players) and the players will be represented by a list of strings (player nicknames).

defmodule Game do
  @max_players 4
  defstruct players: []

  def add_player(game, nick) do
    Map.update!(game, :players, &([nick | &1]))
  end

  def max_players?(%{players: ps}) when length(ps) < @max_players, do: false
  def max_players?(_game), do: true
end

Let’s try this out in iex.

iex(1)> game = %Game{}
%Game{players: []}
iex(2)> game = Game.add_player(game, "ben")
%Game{players: ["ben"]}
iex(3)> game = Game.add_player(game, "harry")
%Game{players: ["harry", "ben"]}
iex(4)> game = Game.add_player(game, "ralph")
%Game{players: ["ralph", "harry", "ben"]}
iex(5)> Game.max_players?(game)
false
iex(6)> game = Game.add_player(game, "tom")
%Game{players: ["tom", "ralph", "harry", "ben"]}
iex(7)> Game.max_players?(game)
true

Seems to be working! Now we need a way to wrap this in a process. OTP provides a number of ways to do this. In this case we are going to use a GenServer. Each game will get it’s own isolated process which can be identified by a PID. Let’s create our GenServer which we will call GameServer.

defmodule GameServer do
  use GenServer

  def start_link do
    GenServer.start_link(__MODULE__, %Game{})
  end

  def get_state(pid) do
    GenServer.call(pid, :get_state)
  end

  def add_player(pid, nick) do
    game = get_state(pid)
    if Game.max_players?(game) do
      {:error, :max_players}
    else
      GenServer.call(pid, {:add_player, nick})
    end
  end

  # Callbacks

  def handle_call(:get_state,  _from, game) do
    {:reply, game, game}
  end

  def handle_call({:add_player, nick}, _from, game) do
    {:reply, :ok, Game.add_player(game, nick)}
  end
end

We have two calls here

get_state returns the current game state.
add_player returns {:error, reason} if the player couldn’t be added and :ok otherwise.

Let’s try this out in iex now:

iex(1)> {:ok, pid} = GameServer.start_link
{:ok, #PID<0.91.0>}
iex(2)> GameServer.add_player(pid, "ben")
:ok
iex(3)> GameServer.add_player(pid, "harry")
:ok
iex(4)> GameServer.add_player(pid, "ralph")
:ok
iex(5)> GameServer.add_player(pid, "tom")
:ok
iex(6)> GameServer.add_player(pid, "richard")
{:error, :max_players}

A hidden race condition

This seems to be working in the repl, but if you deployed this into production you may find that every once in a while a 5th or even 6th player will sneak in. This code has a race condition! Let’s prove this by adding a temporary function to GameServer which adds n players simultaneously:

defmodule GameServer do
  # ...
  def add_players_simultaneously(pid, nicks) do
    Enum.each(nicks, fn nick ->
      Task.async(fn ->
        GameServer.add_player(pid, nick)
      end)
    end)
  end
  # ...
end

This function takes a list of nicks and calls add_player simultaneously from one process per nick. Let’s try it out with 5 players attempting to join.

iex(1)> {:ok, pid} = GameServer.start_link
{:ok, #PID<0.100.0>}
iex(2)> nicks =  ["ben", "harry", "ralph", "tom", "richard"]
["ben", "harry", "ralph", "tom", "richard"]
iex(3)> GameServer.add_players_simultaneously(pid, nicks)
:ok
iex(4)> GameServer.get_state(pid)
%Game{players: ["richard", "tom", "ralph", "harry", "ben"]}

Wait, how did richard get in there?

The erlang process model is a great way to handle concurrent operations on state and if done right it’s quite hard to mess up your state in this way. You just need to think a little up front when you are mutating state.

In order to demonstrate how this is happening, let’s add some puts into our GameServer.add_player function.

defmodule GameServer do
  # ...
  def add_player(pid, nick) do
    game = get_state(pid)
    IO.puts("Players in game: #{inspect game.players}")
    if Game.max_players?(game) do
      {:error, :max_players}
    else
      IO.puts("Add player #{nick}")
      GenServer.call(pid, {:add_player, nick})
    end
  end
  # ...
end

And when we run again.

iex(1)> {:ok, pid} = GameServer.start_link
{:ok, #PID<0.119.0>}
iex(2)> nicks =  ["ben", "harry", "ralph", "tom", "richard"]
["ben", "harry", "ralph", "tom", "richard"]
iex(3)> GameServer.add_players_simultaneously(pid, nicks)
Players in game: []
Players in game: []
Players in game: []
Players in game: []
Players in game: []
:ok
Add player ben
Add player harry
Add player ralph
Add player tom
Add player richard

This is a classic check-then-act race condition. It’s important to remember that the way GenServers work is they operate on each message in their mailbox one at a time and in the order they are received. The 5 get_state messages come in and the server processes those and returns the current state. Then every player sees a stale state and 5 new messages come in to add the players to the game state.

In order to fix this, we need to make add_player an atomic operation. GenServer makes this easy for us because the process model ensures that everything that happens in the callback is atomic! So all we need to do is move this logic into the add_player callback.

defmodule GameServer do
  # ...

  # Removed the max_players? check
  def add_player(pid, nick) do
    GenServer.call(pid, {:add_player, nick})
  end

  # ...

  # Checking inside the callback now
  def handle_call({:add_player, nick}, _from, game) do
    if Game.max_players?(game) do
      {:reply, {:error, :max_players}, game}
    else
      {:reply, :ok, Game.add_player(game, nick)}
    end
  end

  # ...
end

Trying again now.

iex(1)> {:ok, pid} = GameServer.start_link
{:ok, #PID<0.137.0>}
iex(2)> nicks =  ["ben", "harry", "ralph", "tom", "richard"]
["ben", "harry", "ralph", "tom", "richard"]
iex(3)> GameServer.add_players_simultaneously(pid, nicks)
:ok
iex(4)> GameServer.get_state(pid)
%Game{players: ["ralph", "tom", "harry", "ben"]}

Now our state is consistent like we’d expect! When richard tried to join he would see the max_players error. The nice thing about the semantics of GenServer is that you can think about your code in terms of clients and servers. All public API functions on the top of the module happen in the calling process or client side. All handle_* callbacks happen in the server process or server side. If you maintain this mental model it becomes obvious which side your code belongs in.

On bottlenecks

While you may be tempted to throw everything in the server side for safety, you need to consider the performance implications of doing so. When writing callbacks you should be asking yourself

What are the minimal operations I need to determine the next state?

It’s important to remember that callbacks can only happen one at a time so they can easily become a bottleneck. Be careful about performing long operations (particularly network operations) in the callbacks if you can get away with doing them somewhere else.

A holistic approach

Although this works as an example on GenServers and state, I’d suggest that this problem has a deeper, root cause. There is a design flaw with the Game datastructure. When designing a new datastructure, it is a good idea to guard it from ever getting into an invalid state. The logic of keeping the structure consistent belongs in the module itself. You should then encourage the programmers to only use that interface to manipulate the structure and return descriptive errors when something is wrong. In this case, I might change the interface of the Game module instead to something like this:

defmodule Game do
  @max_players 4
  defstruct players: []

  def add_player(game, nick) do
    if max_players?(game) do
      {:error, :max_players}
    else
      {:ok, Map.update!(game, :players, &([nick | &1]))}
    end
  end

  defp max_players?(%{players: ps}) when length(ps) < @max_players, do: false
  defp max_players?(_game), do: true
end

This will leave your GenServers pretty bare but I believe this is a good thing. You can consider something like exactor to make them even smaller.

How I have avoided "javascript fatigue"

2016-12-08T00:00:00-06:00

I keep hearing about “javascript fatigue” on Hacker News and amongst my friends and I wanted to write something about an exercise anyone can do to avoid it.

What is javascript fatigue?

The idea behind “javascript fatigue” is that all of the available tools and libraries have somehow made frontend web development a more difficult and frustrating thing to do than in the past. At first, I thought it was just a little inside joke or venting amongst experienced frontend developers, but now I’m starting to see all this once harmless venting has become toxic and harmful.

There are two casualties to this. The first is the open source maintainer who is burned out because it’s not a joke to them. In some cases it’s their life’s work. Read this article by James Kyle to see what I mean. We all know about our community’s problems with burnout, depression, and suicide. We don’t need to do that to people who dedicate so much time and energy into trying to help us.

The second is the new person who is coming into this world thinking this is the norm. They didn’t get the privilege to learn all this stuff when frontend was allegedly easier and are now told that their confusion is normal and will probably last forever. It’s soul crushing and driving people to quit. Learning this stuff should be fun and easy and it always has been.

The Exercise

This exercise is aimed at people who are just getting started as well as people who have been doing this for so many years that they forgot you can even still do things this way. Even if you know what is going to happen, you should physically do these steps anyway.

Start by opening a text editor. Don’t use your editor of choice if you have one. Use what comes with your computer. For example, use TextEdit on Mac, Notepad on Windows. Do shift+command+T in TextEdit to get into plain text mode
Save the file on your desktop as page.html. Make sure the extension is html.
Paste the code below into the file
Save the file again and double click the file on your desktop

<html>
  <body>
    <ul id="counts">
    </ul>
  </body>

  <script>
    var ul = document.getElementById("counts");
    var count = 0;

    setInterval(function() {
      // make a list item <li>123</li>
      var li = document.createElement("li");
      li.appendChild(document.createTextNode(count));

      // append to our list
      ul.appendChild(li);

      // increment counter
      count++;
    }, 1000); // wait one second
  </script>
</html>

View the results

Congratulations, you just created a javascript application with no tools! You can just use the things that come with virtually every computer. You even have a debugger.

It’s honestly still impressive to me how simple and accessible the web really is as a platform and when I see something like this run I still laugh a little bit. I’d also argue that this is a perfectly fine way to build things for most people. It works on all devices I’ve tried and I still use lots of useful applications built in this way.

Let’s keep going

You might be thinking that having a library like jQuery would make you more comfortable and the raw vanilla JS API is too much. Instead of setting up npm, why not just include it with another script tag. You can even use a fast and free CDN and choose between minified or not. You can choose and freeze the version just like npm would.

<html>
  <head>
    <!-- no need for minified, why complicate it? -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.js"></script>
  </head>
  <body>
    <ul id="counts">
    </ul>
  </body>

  <script>
    $(document).ready(function() {
      var $ul = $("#counts");
      var count = 0;

      setInterval(function() {
        $ul.append("<li>"+count+"</li>");
        count++;
      }, 1000);
    });
  </script>
</html>

You might be tempted now to reach for a cool new framework like React. Keep in mind it would be perfectly fine to continue on with just jQuery. In fact, if you are just getting started, I’d say learn jQuery and build a couple giant spaghetti applications before you bother looking at anything else. But if you are really in the mood to learn it, or maybe need to for a job, React can be a great library to learn and you still don’t need any tools!

<html>
  <head>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.1/react.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.1/react-dom.js"></script>
  </head>
  <body>
    <div id="count"></div>
  </body>

  <script>
    var Count = React.createClass({
      render: function() {
        var c = this.props.value;
        return React.createElement("li", null, c);
      }
    });

    var CountList = React.createClass({
      getInitialState: function() {
        return {counts: [0]};
      },
      increment: function() {
        var c = this.state.counts[this.state.counts.length - 1];
        this.state.counts.push(c + 1);
        this.setState({counts: this.state.counts});
      },
      componentDidMount: function() {
        setInterval(this.increment, 1000);
      },
      render: function() {
        var liElements = this.state.counts.map(function(c) {
          return React.createElement(Count, {value: c}, null);
        })
        return React.createElement("ul", null, liElements);
      }
    });

    $(document).ready(function() {
      ReactDOM.render(React.createElement(CountList), document.getElementById("count"));
    });

  </script>
</html>

So what about ES6? Maybe you think ES5 is too verbose. Should we start configuring a .babelrc now? No need, ES6 is actually a standard that your browser probably already supports. It’s still patchy but I’d be willing to bet that all the devices (and most browsers) you own already support it. Let’s even add in JSX for fun. It’s not supported anymore but we can include a JSX transformer from the CDN and it works fine in this case.

<html>
  <head>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.1/react.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/react/15.4.1/react-dom.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/react/0.13.3/JSXTransformer.js"></script>
  </head>
  <body>
  </body>

  <script type="text/jsx">
    class Count extends React.Component {
      render() {
        return <li>{this.props.value}</li>
      }
    }

    class CountList extends React.Component {
      constructor(props) {
        super(props)
        this.state = {counts: [0]}
      }
      increment() {
        let c = this.state.counts[this.state.counts.length - 1]
        this.state.counts.push(c + 1)
        this.setState({counts: this.state.counts})
      }
      componentDidMount() {
        setInterval(this.increment.bind(this), 1000)
      }
      render() {
        let els = this.state.counts.map(c => <Count value={c} />)
        return <ul>{els}</ul>
      }
    }

    $(document).ready(() => {
      ReactDOM.render(<CountList />, document.body)
    })
  </script>
</html>

Let’s take stock of what we have. We have a working React/JSX + ES6 application that works on most of our devices and browsers. Let’s point out what we didn’t need to accomplish it.

Nothing installed or configured on the system
No dependency management (npm)
No transpiler (Babel)
No minification
No crazy build system with separate “plugins” (Gulp/Grunt)
No node.js server that only serves up static content
No IDE
No testing frameworks
No javascript module tools (e.g. Webpack)

My thoughts on tools and libraries

Even though you absolutely can build great software without tooling, I’m not advocating this is the way everyone should be doing it (though if you are just starting out, I think you should be). What the exercise is meant to do is remind you (or maybe show you for the first time) that all of these tools don’t really help you build good, working frontend applications. They can be useful for deploying and maintaining applications, but none of them are required. If your application grows in users and programmers you will likely find you want at least some of these tools. We use some of these at work and couldn’t do without them. But add them one at a time and understand how they work and why they were made. Don’t think that you have to tame a Gruntfile just to try React for instance.

The beauty of doing it this way is it’s really easy to add these tools later. I’m not kidding. While most things are easier to learn from the top down, I think the unique thing about the web right now is it’s probably actually easier to learn from the bottom up. Everytime you just accept that you need one of these tools you are missing out on the fact that it doesn’t really do much for you and is probably just getting in your way at the moment.

Where to go from here

I don’t think that reading an html file straight from your desktop as we did in the exercise is a pleasant way to develop but I think there are some tools that come close in the freedom they give you. I like to start prototyping my idea in something like jsbin. It supports things like dependencies, versioning, and even JSX. You can immediately share the url with a friend for feedback or a device you want to test. You can literally copy and paste it out of there later. Don’t waste your time setting up a node server and some wacky Gulp plugins for something that no one but you may ever see. You’ll be so much happier, your system will be leaner, and you’ll understand every piece of it.

Visualizing Walkability in New Orleans

2016-08-16T00:00:00-05:00

I read this article on Geoff Boeing’s blog on using network analysis with OSM data to visualize the walkability of Berklee, CA. I thought it would be interesting to run it on data from New Orleans. Since it’s based on OSM data (which is probably incomplete here), it may not be the most complete picture.

Improving a Visualization of Housing Prices in New Orleans

2015-08-16T00:00:00-05:00

Last week I read an article in the New Orleans Advocate on the increase of the price of housing since Katrina. The article was fine, but they linked a visualization of the data that I really had trouble interpreting.

The idea is clear enough, the price per square foot values are displayed hovering over each respective zip code. The article was titled “Why New Orleans-area home prices are skyrocketing; check out where increases are biggest” so naturally I first attempted to do that. I looked at my old neighborhood (70130) and saw that it went from $173 to $253. Okay, so subtract and that’s an increase of $80. But what about the other ones? How would I see where the increases are biggest like the article suggests? Or what if I want to know the biggest decrease instead? I suppose I could subtract them all and compare them, but that would be a misleading comparison because I think what I really want to see is the percent changed, not just the difference. It’s like comparing apples to oranges otherwise.

This map looks very nice but I think it fails to show the trends. I decided that taking a shot at building my own interactive map would be a fun weekend project. It didn’t turn out the way I wanted, and I would have liked to spend more time making it look good, but it was still fun and I thought other people might enjoy it. The following article will explain the techniques and decisions involved in building it but if you find all that boring you can just view the map here.

Collecting And Processing the Data

As is typical in web news stories, the link to the source data appeared to be dead or incorrect. I realized all the trouble of finding the data or trying to parse it out of the pdf or whatever it is wouldn’t be worth it since hand transcribing into a CSV would only take about 30 minutes.

In the spirit of sharing here is the csv with the processed data.

After doing that I needed to process and explore the data so I can do some pre-computation as well as help make decisions about the visualization. When exploring data I always start by creating a notebook in ipython notebook (now jupyter).

First, some code to pull the CSV and calculate the percentage changes between the years using pandas. I only worked with the pre-katrina to 2015 range for this project because that is what I found interesting.

import pandas as pd

df = pd.read_csv('data/nola_house_data.csv', index_col=['zip'], na_values=['n/a'])

# dropping unless all 3 columns filled (incomplete data)
# this drops a couple zips (70112 for instance)
df = df.dropna(thresh=3)

def pct_change(c1, c2):
    return (df[c2] - df[c1]) * 100. / df[c1]

df['pct_change_2005_2015'] = pct_change('2005', '2015')

I was then able to sort on that column and I found that the biggest decrease was 70058 at -21.9% and the biggest increase was 70117 at 108.0%. I also wanted to know how the data was distributed over this range so I created a distplot

As you can see, most of the data is distributed around the -20% to 25% range. (The mean being 11.3% and the median being 3.9%) I also used this opportunity to apply a function to calculate the Freedman Diaconis size which was 10. Both of these details will be important for later on.

Preparing the Data for a map

The original visualization used color purely for classification of the year. The color choices were arbitrary and did not actually encode any information. What I wanted was a choropleth map. Each zip code will be colored to encode it’s percentage change in price. In order to do this we need to choose a color progression. When you have data that is meant to show some tendency away from a center, what you are looking for is a bi-polar or (diverging) color progression. That way, the center of the information spectrum (no change in price) and the side that the data sits on can easily be understood at a glance. There are ways to generate these but I found this neat tool to help with this process and I prefer to use their design advice. So I opened this up and choose diverging for the colors. Then i chose 10 for the number of classes (from our Freedman Diaconis rule). I also selected colorblind safe. Then I just copied all of them into a javascript object so I could experiment later.

If I was going to overlay and color the zips on a map, I’d need the data to describe the shapes and locations of the zip codes. I was able to get them in geojson format from this project.

wget https://github.com/jgoodall/us-maps/blob/master/geojson/zcta5.json

The only problem is this is all the zip codes in the US. Also, the properties have some junk in them. Since my map is going to consume geojson anyway, I just wanted to parse out the zips i care about and insert the data into the properties portion of the feature (while clearing out the other junk). I did this using our dataframe of zip data (df) from earlier.

import geojson

with open('data/zcta5.json', 'rb') as allzips:
    feature_collection = geojson.load(allzips)

def orleans_only(feat):
  return int(feat.properties['ZCTA5CE10']) in df.index

def transform(feat):
  """remove extra props and put on the props from the dataframe"""
  z = int(feat.properties['ZCTA5CE10'])
  data = df.ix[z]

  feat.properties = {
    'zip': z,
    'pct_change_2005_2015': data['pct_change_2005_2015'],
    '2005': data['2005'],
    '2015': data['2015']
  }

  return feat

orleans_features = filter(orleans_only, feature_collection.features)
orleans_features = map(transform, orleans_features)
orleans_collection = geojson.FeatureCollection(orleans_features)

f = open('data/orleans_zips.geojson', 'w')
f.write(geojson.dumps(orleans_collection))
f.close()

By the way, I posted this raw geojson data as a gist.

Creating the map

I had plans to finally learn d3 for this project but skipped that idea once again. In fact, the resulting map code is really quite horrible. Don’t look at it. I just wanted to be done with it in one night (kind of like a hackathon). It would be nice in the future to investigate using React with leaflet for use in a bigger project.

So what I did was basically just take some leaflet code I made for a previous experiment and just typed and refreshed my browser until it worked. I made some hacky little controls so I could experiment with colors and binning methods and I decided to just leave them in. I also wanted to have a hover window with the details but once I realized that would take more than 30 minutes I gave up so the details just sit in the top right.

The mapping code is the least interesting. The heart of it is a leaflet function that takes a geojson feature and applies a style:

function style(feature) {
  return {
    weight: 2,
    opacity: 1,
    color: 'white',
    dashArray: '3',
    fillOpacity: 0.65,
    fillColor: getColor(feature.properties.pct_change_2005_2015)
  };
}

The two most difficult things were deciding on the binning range and the base map. Unlike say, the shape of a US state, the shape of a ZIP code isn’t very recognizable at a glance so a base map was needed so you can orient yourself. All of the greyscale and black and white maps were too bright so I just chose a dark one. It looks kind of weird but it’s the best I could do.

Choosing the bin values was strange because of the skewed distribution of the data. I created two methods, zero and fit. Zero is as you would expect with a bi-polar progression where 0% is pure white and anything above or below it takes on the respective hue. Fit evenly fits the bins given the min and the max values but because it’s so skewed to the positive side it ends up a sea of blue. It does help you see the differences at the extreme level though. Perhaps a higher class number would have been appropriate? I didn’t take the time to find out. I’m sure talking to a GIS expert would yield a good way to deal with this problem but I will save that for another weekend.

Generating Bitcoin Keys From Scratch With Ruby

2015-03-15T00:00:00-05:00

I’ve recently been looking into some newer features in the bitcoin protocol and feeling a little overwhelmed. I’ve only ever worked with higher level abstractions of the protocol via libraries. In order to get a better understanding of the fundamentals I started reading as much as I could. I found a lot of articles but I felt like each one was missing pieces that helped me understand what was going on. I decided I would try to write a bitcoin transaction from scratch (no bitcoin related libraries) and document the process. I used ruby. I like ruby and I knew I wouldnt get much help from Google with it (which is good). My goal was create a bitcoin address, send some BTC to it, then spend that BTC by creating a new transaction and publishing to the network. So far I have only successfully generated keys so consider this part I.

Bitcoin primitives

We aren’t going to talk about mining and the bitcoin network. It’s not really important to understanding how to make a transaction. I expect you to understand bitcoin on a high level. These are the core primitives we are going to explore.

Private Key
Public Key
Address
Transaction (eventually)

Private Keys

The most important primitive is the private key. All other keys are derived from the private key. Here is a diagram explaining the process for creating keys:

The private key is, in essence, just a really big random number (256-bits) which can easily be generated offline with a sufficient source of randomness. Because of that, the public key and the address can also be generated offline. Of course, due to the laws of public key cryptography, it is impossible* to derive the private key from the public key or the address. Bitcoins are tied directly to the private key. The owner of the private key cotrols the bitcoins. Because of this, secrecy and security of the private key is of utmost importance. In general, the only thing that you should ever willingly expose is the Address. You don’t even want to reveal your public key until you have to.

Generating Private Keys

Like I said before, the private key is just a 256-bit (32 byte) random number. You can create a private key pretty much any way you can generate random numbers. If you have openssl installed on your machine, try running this command a few times:

$ openssl rand 32 -hex
f9a9c3825fe4f90385763d3a95b85a3c081410367bcb6c15bb550214bf1faf1a

These are valid private keys. You could even flip a coin 256 times, record each result as 0 or 1, then turn that bit array into a hex string that would look just like one from openssl. You can do pretty much anything you can think of as long as your source of randomness is cryptographically secure.

You may be wondering how you can do this offline since if someone were to somehow accidentally generate a private key that you have also generated, they could control the funds on that address. Believe it or not, every time you generate one of these numbers you can very safely bet your life this is the first time the world has seen that number because the keyspace is so massive even when considering the birthday paradox. For this reason, you can safely generate a key offline and be sure that it does not collide with someone else’s key.

Let’s generate a private key in ruby using securerandom. We are going to use another method later but I’ll explain when we get there. This will spit out 256-bit (32 byte) hex strings much like the openssl command above did.

require 'securerandom'

# Generate a 256-bit (32 bytes) secure random number
def generate_key
  SecureRandom.hex(32)
end

Private Key WIF

WIF, the wallet import format is a format for encoding a private key into ASCII space. It uses Satoshi’s own base58check encoding. You may wonder why Satoshi wouldn’t just use base64. Here is the explanation:

// Why base-58 instead of standard base-64 encoding?
// - Don't want 0OIl characters that look the same in some fonts and
//      could be used to create visually identical looking account numbers.
// - A string with non-alphanumeric characters is not as easily
//      accepted as an account number.
// - E-mail usually won't line-break if there's no punctuation to break at.
// - Doubleclicking selects the whole number as one word if it's all alphanumeric.

This encoding also includes a checksum using SHA-256 which can very quickly verify the integrity of an Address and potentially even allow you to correct it. This is a pretty responsible implementation considering one misplaced character could cost you all your bitcoins.

Let’s look at some ruby code to do base58. I took this from the utils in the bitcoin-ruby project:

def int_to_base58(int_val, leading_zero_bytes=0)
  alpha = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
  base58_val, base = '', alpha.size
  while int_val > 0
    int_val, remainder = int_val.divmod(base)
    base58_val = alpha[remainder] + base58_val
  end
  base58_val
end

def encode_base58(hex)
  leading_zero_bytes = (hex.match(/^([0]+)/) ? $1 : '').size / 2
  ("1"*leading_zero_bytes) + int_to_base58( hex.to_i(16) )
end

In order to do the checksum we will need a function capable of creating a SHA-256 hash from a hex string. Ruby’s digest module comes with this capability:

require 'digest'

# SHA-256 hash
def sha256(hex)
 Digest::SHA256.hexdigest([hex].pack("H*"))
end

Now the checksum is simply the first 4 bytes of the twice SHA’d hex string (The WIF format requires doing the double SHA).

# checksum is first 4 bytes of sha256-sha256 hexdigest.
def checksum(hex)
  sha256(sha256(hex))[0...8]
end

PRIV_KEY_VERSION = '80'
def wif(hex)
  data = PRIV_KEY_VERSION + hex
  encode_base58(data + checksum(data))
end

The WIF format also requires that the first byte be the WIF version. Here we have set it to 0x80. It would be 0xef if we were using testnet.

Now let’s make some WIFs!

# Remember #generate_key returns a new 256 bit
# private key as a hex string

puts wif(generate_key)
# => "5JNHWufka5pPaZ2SViwXEUARU67aqZrqJvB6bmj9Xd1rC91vo31"
puts wif(generate_key)
# => "5K5KeFgcbNzppWmz6zqVDe1NmWX4hLMdcDz6LtaoFCfLmB4dZwu"
puts wif(generate_key)
# => "5K14yHUGCQkTm9qRgVVgEpBXHdF1kcZefksprfTRJSd85NwdWiY"

According to the spec, a base58 encoded private key must start with either '5H', '5J', or '5K' and these all do. These are valid WIFs you could import into your wallet software now if you wanted! Please don’t do that though :)

Deriving Public Keys

The next step is deriving the public key. Bitcoin uses ECDSA (Elliptic Curve Digital Signature Algorithm) for it’s cryptographic signatures. I suggest you either know a little about or research the following topics before continuing:

Public Key Cryptography
Digital Signatures
Eliptic Curve Cryptography

Ruby doesn’t really make doing this part easy for us. But, I knew this would be the case coming in and decided that might make using it a good learning experience. There is a great, simple ecdsa library for python but not really one for ruby. I did find this library but the idea of someone implementing their own crypto, and in ruby of all languages, kind of freaked me out. To be fair, he does state in his readme that his library is for learning purposes. My code is also only for learning purposes so I am less concerned about security but I am instead concerned about an experimental library generating something in a weird way. The reason is that, since bitcoin is a cryptographic protocol, it’s very opaque and if even one bit is off it could result in errors with no explanation of what the problem is.

Because of these reasons, I chose to stick with openssl. It has a bad reputation right now (and an even worse API) but it at least has a lot of eyes on it and I can trust that it isn’t going to do anything too weird. It’s also default with ruby so there is no need to install any third party libraries.

We’re going to be using the EC class in the OpenSSL::PKey module. Because it generates it’s own private key we will no longer be using our generate_key method.

require 'openssl'

# Bitcoin uses the secp256k1 curve
curve = OpenSSL::PKey::EC.new('secp256k1')

# Now we generate the public and private key together
curve.generate_key

puts curve.private_key
# => #<OpenSSL::BN:0x007f3e2e63e520>

puts curve.public_key
# => #<OpenSSL::PKey::EC::Point:0x007f3e2e55fdc0

We now have our public and private key but they aren’t really in a good format for us to work with. We’d prefer hex strings.

The private_key is of the type OpenSSL::BN or (Big Number). It is what it sounds like, a big integer. Here is an example:

curve.private_key.to_s
=> "59349466814215569740836827258207701086591329864926689993285822531257454935241"

Ruby’s to_s takes a base. So to get the hex string we just have to ask for base 16 and we get back our 32 byte key in hex format:

curve.private_key.to_s(16)
=> "83369B99791B5F3E7008C897CFA3AE4899BD2E256E4EE0D3020501977F35BCC9"

The public_key is of the type OpenSSL::PKey::EC::Point because in elliptic curve crypto, the public key represents a point on the curve. The Point class has a method to_bn we can use to convert it to a Big Number. This will be a 65 byte number. The first byte will be 0x04. The next 32 will represent x, the next 32 will represent y. Fortunately this is the format Bitcoin wants it in so we just need to turn it into a hex string:

curve.public_key.to_bn.to_s(16)
# => "04FDA9AD13F927257D26365DF9ED5B8734B2468EFBB0DD29037AE18F08CF7442FD0139355A1EAD51932E0CBBC43F2D1DDDE2F287FD4A0E06F054E3360243320CA1"

Signing and verifying messages

Before we move on to generating the address let’s explore the properties of this curve object. As you should know, a digital signature allows us to sign and verify data. Let’s say we have some data in string format:

message = 'Hello World!'

I want to pass this message on to someone else but I want to provide proof that:

I am the actual author of the message (someone has not just forged my signature and pretended to be me)
The message has not been altered in transit

This is generally what would be called authentication in crypto and using a signature algorithm can help provide this proof. “Signing” our data is a function of our private key and the data. In this case the curve object has abstracted that away. Just know that it uses the private key under the hood:

signature = curve.dsa_sign_asn1(message)
# => "0F\x02!\x00\x8C\x18b\xF1\x96\xC7V\bARH\xA3\x05ES\v\xFF\x8F_D7\xE10\x16\xD0>UI5\f\xAA\xEE\x02!\x00\xB2h\xC2\b\xFE\x87uD\x98\xD7Z\x94YL\xF63\xC4\x06\x1C\xE1\x9E\x15\xAE\x05\xAF}5\xC9\x95\xEE\x9C\x97"

The result here is a signature in ASN1 format.

Now, given our public key, the message, and the signature, a reciever of the message can verify that the message was in fact signed by our private key and has not been tampered with. We call this “verification” of the message.

First suppose we export the big number of the public key.

pub_key_bn = curve.public_key.to_bn

Also suppose the receiver of the message already has this on his computer and we have verified it is my public key.

# Now let's pretend we just sent [message, signature] to
# the receiver of our message and this is now their computer

# The reciever creates their curve object using the same secp256k1 group
group = OpenSSL::PKey::EC::Group.new('secp256k1')
curve = OpenSSL::PKey::EC.new(group)

# They only have the public key (point) and set it
curve.public_key = OpenSSL::PKey::EC::Point.new(group, pub_key_bn)

# See, no private key
curve.private_key
# => nil

# Now they can use the public key to verify the message
curve.dsa_verify_asn1(message, signature)
# => true

In order to drive this home, let’s see what happens if we modify the message:

message << '!'
curve.dsa_verify_asn1(message, signature)
# => false

I leave it to the reader as an exercise to see that modifying the public key or the signature also results in a failure to verify the message.

Generating an Address

The public key is the address for all intents and purposes but for convenience and some security reasons we must first compute the Public Key Hash [a 160 bit hash of the public key]. Then we base58check encode that for integrity and readability. This base85 encoded result is the Address.

The Public Key Hash is just a SHA-256 hash of the public key which is then run through RIPEMD-160. We already have a sha256 function, so let’s make one for RIPEMD-160. Ruby provides this hashdigest in the same Digest module we used with SHA-256.

# RIPEMD-160 (160 bit) hash
def rmd160(hex)
  Digest::RMD160.hexdigest([hex].pack("H*"))
end

Now the public key hash:

# Turns public key into the 160 bit public key hash
def public_key_hash(hex)
  rmd160(sha256(hex))
end

Using our public key from earlier, the result should look something like this:

pub_key_hex = curve.public_key.to_bn.to_s(16)
# => "04FDA9AD13F927257D26365DF9ED5B8734B2468EFBB0DD29037AE18F08CF7442FD0139355A1EAD51932E0CBBC43F2D1DDDE2F287FD4A0E06F054E3360243320CA1"

public_key_hash(pub_key_hex)
# => "291b6b709bd2917afd72b490eda1cf3ffd272b53"

Now that we base58check encode the public key hash to get the address:

ADDRESS_VERSION = '00'
def generate_address(pub_key_hash)
  pk = ADDRESS_VERSION + pub_key_hash
  encode_base58(pk + checksum(pk))
end

generate_address(public_key_hash(pub_key_hex))
# => "14kMZ24Zm8ARfotFdyuJvKi1PpZ2bvfdjC"

And there is our bitcoin address. Our address version here is 0x00. For testnet it’s 0x6f. The spec says that version zero addresses should start with 1.

Visualizing Absenteeism in Congress

2014-08-24T00:00:00-05:00

Since I left New Orleans, I’ve continued to watch Cedric Richmond’s absenteeism climb. I started wondering what the data on attendance looked like in the House and the Senate.

Collecting Data

I let my computer run last night pulling down all the vote data I could capture from 1989 to present day. I loaded it up into a large pandas DataFrame this morning (about 2 Gigs) and started splitting some data. The first thing I thought about was breaking the data up into states. I decided I would first aggregate the total votes for each state. Then I would aggregate the total Not Present votes for each state. Then I would divide those two state-indexed vectors to get the percentage of missed votes per state. That code looks something like this:

total_votes = senate_votes.state.value_counts()
not_voting = senate_votes[senate_votes.result == 'Not Voting'].state.value_counts()
senate_missed_percentages = not_voting / total_votes

The same calculation was performed on House data. The result field may have a few different values depending on the vote and on whether it’s the Senate or the House. The 4 primary values for a typical vote are:

Present
Not Voting
Aye/Yea
Nay/No

A vote of Present means the legislator is present for the vote, but is choosing to abstain from making a choice. Not Voting means the legislator is not present at all. Aye and Nay are self-explanatory. Aye/Yea and Nay/No are different conventions used in different procedures but mean the same thing.

Visualizatons

I knew I wanted to make an interactive choropleth map to show the data. I had hoped on modifying Mike Bostock’s d3 example but I was not patient enough to learn d3 and couldn’t get it to work. Instead I just used Leaflet and removed the map tiling.

Unfortunately, I could not figure out how to imitate Bostock’s nice projection (with Hawaii and Alaska in the frame) so you will need to scroll or zoom a bit to see them. The maps support pinching and dragging. The darker the value here the more missed votes. Hover over each state to see the true percentage.

Senate

The highest 3 states for the Senate are:

MA (7.73%)
NC (6.76%)
AZ (6.26%)

The lowest 3 states are:

ME (0.36%)
WI (0.76%)
MI (0.91%)

House

The highest 3 states for the House are:

WY (12.31%)
AK (12.10%)
LA (6.74%)

The lowest 3 states are:

DE (1.39%)
ME (1.93%)
NE (2.01%)

Thoughts

While this was fun and I plan to keep exploring the data, the results don’t necessarily tell a story on their own. It’s clear that some states’ legislators are not showing up to vote more than others, but it’s not clear what they are doing instead or whether or not they have a choice (perhaps committee duties interfering with votes?). I have to also consider that while I see missing votes as an unambiguously bad thing, many people in this country see disengaging in national politics as a point of pride.

Precisely Delaying Jobs With Dead-Letter Exchanges in RabbitMQ

2014-07-19T00:00:00-05:00

Everyone who is familiar with RabbitMQ knows it’s a great tool to execute jobs that may happen out of band of some normal user process. Typically you want to execute these jobs right away, such as sending some kind of notification email, but there may be times in which you want the execution of these jobs to be delayed by specific times set by the client. RabbitMQ does not support this directly, but I encountered an interesting trick using Dead-Letter Exchanges to achieve this functionality.

How It Works

Instead of publishing directly to an exchange as we normally would, the producer creates a new, unique queue and publishes it’s message to it. In our code we are calling it send.later.<timestamp>.

When we create this queue, we set some special options:

{
    "x-dead-letter-exchange" : "immediate"
  , "x-message-ttl"          : 5000  // 5 seconds
  , "x-expires"              : 10000 // 10 seconds
  , "x-dead-letter-key"      : "right-now-queue"
}

The x-message-ttl is a time in milliseconds that you want to delay the job by. This can be unqiue per message. The x-expires is the time in milliseconds you wish for that temporary queue to exist. The x-dead-letter-exchange is the exchange you wish for the message to be published to after it expires. So, we are sortof doing a semantic hack here. Instead of actually expiring the message we are actually activating it on expiration. x-dead-letter-key is the routing key to be used when publishing into the immediate exchange.

We have previously created an exchange called immediate in which the consumers are listening to run jobs from immediately. When the message expires, it will be thrown into this exchange.

The Code

The code is written in node.js and uses node-amqp to speak to RabbitMQ. It is comprised of two parts (a producer and a consumer) which I have written into two separate programs.

The Producer (producer.js)

var amqp         = require('amqp')
  , conn         = amqp.createConnection()
  , PUBLISH_RATE = 1000 //every second
  , count        = 1
  ;

conn.on('ready', function() {
  console.log('ready');

  conn.exchange('immediate', options={
    durable: true,
    autoDelete: false
  }, function (exchange) {
    setInterval(function() {
      var key = "send.later." + (new Date().getTime()).toString();
      conn.queue(key, {
        arguments: {
            "x-dead-letter-exchange": "immediate"
          , "x-message-ttl": 5000  // 5 seconds
          , "x-expires": 10000     // 10 seconds
          , "x-dead-letter-key": "right-now-queue"
        }
      }, function() {
          console.log('publish');
          conn.publish(key, {
            v: count++
          }, {
            contentType: 'application/json'
          });
      });
    }, PUBLISH_RATE);
  });

});

The Consumer (consumer.js)

var amqp = require('amqp')
  , conn = amqp.createConnection()
  ;

conn.on('ready', function() {
  console.log('ready');
  conn.queue('right.now.queue', {
      autoDelete: false
    , durable: true
  }, function(q) {
    q.bind('immediate', '#');

    q.subscribe(function (msg, headers, deliveryInfo) {
      console.log(msg);
      console.log(headers);
    });
  });
});

Running

You’ll need node-amqp installed and you’ll need a rabbit server running on localhost with a default configuration.

In one terminall session run:

node consumer.js

This will start listening for messages ready to be run and print them out to the screen.

In another terminal run:

node producer.js

This will publish a message every 1 second, but they won’t come out of the consumer end until 5 seconds after they were published.

Altering This Code

You don’t have to do this exactly as I have. For instance, I am using a topic exchange but I think this should work with any kind of exchange. You also could share all the send later messages in a single queue that never expires. It really depends on your use case how you wish to configure it.

Adding Bluetooth Support to an FM Transmitter

2014-07-19T00:00:00-05:00

A few years ago I wanted to build an FM transmitter from a kit. I settled on the FM25B from Ramsey. It’s delivers really nice stereo quality for the price. I did a few experiments with it for a while and when I got tired of that I started using it as a way to broadcast music around my house. My main stereo system has a radio and I also have a shower radio. It’s really nice to play music or internet radio in the morning and be able to walk around or take a shower and continue to listen.

The way I had set it up before was that I just always had a hanging 1/8th cable coming off my bookshelf. This really felt like a hassle sometimes and it’s unsightly, so this weekend I decided to try to retrofit the transmitter with a bluetooth audio receiver.

Parts

I looked around on the internet and I found what I think would be a good device for this but I didn’t want to wait for shipping so instead I walked over to Best Buy and bought one of these. This thing has a battery (which we won’t need).

Since I wanted to power the device with the same power source as the radio, I knew i would need some kind of voltage regulator. I am assuming this thing runs off 3.3V because it is powered by USB so I stopped by RadioShack to look for a 3.3V regulator but could only find a 5V regulator. I decided to just try and make this work anyway.

Build

First step was to find some good points to leach off the radio. I looked around on the circuit and tested some spots until I found a place located after the filtering caps that was providing a clean power signal. I suspected that the 5V might be fine if the device had an internal regulator. I built this test rig and let it sit for a while to see if anything got hot.

Turns out it handled it just fine. Next thing I did was freeform solder the circuit then stick it into the transmitter. I ran the audio cable out the back and looped it back into the input. I can easily hardwire this later but for now it’s not noticeable.

Now when I am at home, I can just choose my transmitter in my bluetooth options and it will route all the audio to all the speakers in my house.

An Analysis of Crosswalk Lights in New Orleans

2014-05-11T00:00:00-05:00

I think one thing that noticeably separates other major cities from New Orleans are well thought-out and well maintained infrastructures for pedestrians. When you visit a city like San Francisco or Washington DC, there is hardly a crosswalk without a pedestrian light (even in residential zones). They usually do nice things like count downs and make clicking noises for the visually impaired.

I’ve been traveling back and forth lately between New Orleans and San Francisco and I’ve been doing a lot of walking in both cities. I have noticed not just infrastructure differences, but behavioral differences. In San Francisco I rarely see people jaywalking or vehicles failing to yield to pedestrians (your mileage may vary). Meanwhile, back in New Orleans, both of those things are so commonplace they are expected. I’ve even noticed my own behavior changes from city to city. I started to wonder, is my behavior being affected by the infrastructure?

I was sitting at the coffee shop yesterday thinking about this hypothesis. The first question that popped into my mind was “What is the current state of our pedestrian infrastructure?”. I only had anecdotal information from my 5 years of walking around the city so I tried googling around for some hard data. I could not find any, but I did find this report which I highly suggest you view now to familiarize yourself with the problem and what the city is planning to do about it.

The statement that stuck out to me was this: “About a dozen lights at pedestrian crosswalks are broken along Canal Street”.

That dozen number seemed significantly low. I did a quick estimate in my head, I figured there are roughly 100 lights on canal and about 3 out of 4 of the lights I’ve encountered in the city are broken. Well, now I knew I was going to have to go count them myself. Since I am in between jobs and homeless, I have some idle time on my hands. So I hacked together an app, picked up some road beers, and started zig-zagging the streets like a confused tourist to collect the data myself.

Collecting Data

I looked around for some mobile software that would help me collect the data but I couldn’t find anything that wasn’t cumbersome or too granular. Luckily I was able to design and hack together a database-backed mobile-web app in a few hours. It is ugly but gets the job done.

The way this works is pretty simple. I have this big blue marker that I use as my cursor. It starts off pinned to your physical location, but you can move it by pressing and dragging it. It’s so huge and ugly because you need to be able to see it under your thumb and I didn’t have time for a more elegant solution. You drag it to a location then hit the Add Light button. That presents you with 3 options Working, Broken, or Unusable. Once you choose one, it sends the information off to the server for storage then renders the marker on the map with the respective color (green, yellow, red).

Criteria

I tried to keep the criteria as objective as possible but I admit some of it had to be subjective. Here are my criteria for the 3 categories:

- Unusable

I would mark a light Unusable [red] when it was completely broken or I thought there was no way to figure out whether I could cross or not. For example, if the light was completely blank at any point of its cycle, or it has a single state that does not show, it’s Unusable. Also keep in mind that I am NOT counting lights that aren’t even there (there are plenty).

- Broken

I would mark a light Broken [yellow] when it was basically working but it was unclear whether I could cross or not. This would include situations where both states are lit at the same time. You could figure out what you were supposed to do but only by waiting long enough and using deductive logic. Broken also includes very low visibility (almost imperceptible) and situations where the light was working but turned away from view.

- Working

I would mark a light Working [green] when it appeared to be operating normally. I was very forgiving with this category. The minimum criteria was that I just needed to be able to figure out whether I was supposed to cross or not. Most of the lights that made it into this category still had some problems that should be addressed. Many of them had partially broken lights or their protective grate was shattered. I didn’t penalize them as long as I could figure it out.

I also decided that since I wouldn’t be able to walk the entire town, I would just walk the Central Business District. The way I saw it, the CBD is one of the highest foot-traffic areas in town. It has a high concentration of businesses and employees, tourists, and city officials moving about on foot.

Analysis

The end goal was to analyze the data to create a map and some statistics. First let’s look at the stats through some matplotlib plots.

Here is the breakdown per status of the 171 lights I recorded:

So I counted 90 (65 + 25) broken overall (59 on the parts of Canal I checked). And here is a pie chart of the percentages:

As you can see, over half of the lights are broken or unusable. Here is an interactive map of the raw data:

The blue polygon is the region that I walked looking for lights and the markers are colored according to the red-yellow-green color code established earlier.

Conclusions and More Questions

I can’t say whether or not this data actually helped support my original hypothesis. Unfortunately the questions I had in mind were governed by too many independent variables. Not that I thought it would, the data and the collection of the data were really just to elevate the discussion and illicit better questions, not provide answers.

At the end of the day though, I think I learned more while collecting the data than I did analyzing it. I spent about 3 minutes at each major intersection. While standing there I wasn’t just observing the lights, I was observing the people. While waiting for the lights to change, I would watch people’s reactions to broken or confusing lights. I am not exaggerating when I say that at a majority of the intersections I experienced at least one troubling situation.

At one intersection a young family misstepped into a street with a broken crosswalk light. The father pulled the little girl back from an oncoming car just in time. They had a nervous chuckle about it but the mother was mortified. At another intersection some bachelor party guys drunk from the casino got tired of waiting to see if the light would turn on or not (it wasn’t going to because it was completely shut off) and just bolted across the intersection only to have a taxi cab slam on its brakes and honk loudly. The taxi driver had that look on his face, “typical drunk bros ruining our city”.

At nearly every intersection I experienced tourists looking worriedly confused and muttering to themselves and each other “Should we just go?”. Nearby a local more familiar with the situation just ignored the rules altogether and casually ran across as soon as there was a gap in the cars.

The drivers don’t know how to handle it very well either. As many times as I have walked through it, I was surprised to notice that Poydras has no crosswalk lights. Instead they have these signs posted above the crosswalks urging drivers to yield to pedestrians. Unfortunately, the drivers rarely comply. I tested a few cars only to be honked at or aggressively hurried along. If drivers don’t follow this rule, what’s even the point of a crosswalk? Why don’t we all just cross the street where we wish and try our best to avoid cars? Is this why everyone behaves as if the rules don’t matter anyway?

And once again, I was kind of shocked at my own behavior too. When I come up to a crosswalk I just try to cross it if no cars are coming, even if the lights appear to be working (I just assume they are broken or it doesn’t matter). I’m still unsure why I have no problem doing that here and not in San Francisco though. Is it the laws? Is it just because everyone else is doing it? Is it just an Every-Other-Coast vs South-Coast attitude? Or is it because the infrastructure is hopeless and we have just given up on it? I am still not sure.

Resources

All the code and data for this experiment are up on a public github repo.

Proving the answer to the Monty Hall problem with a simulation

2014-03-26T00:00:00-05:00

The Monty Hall Problem is a well known statistical paradox. Even knowing the mathematics behind it, it can be difficult to accept that switching doors will change your odds from 33.3% to 66.6%. There are plenty of ways to prove it mathematically but why not see it in action? We could play out the game with 2 people, say n times. We could do one round where the player does not switch their choice, then we could do another round where they do switch. At the end we could tabulate the win percentage and if n was sufficiently large, the percentages should match up to theory. Or we could just write some python to play this game for us.

The Code

This code requires numpy so if you want to run it you will need numpy installed. First let’s import it into a namespace np:

import numpy as np

Now we need a function which can generate n prizedoors where n is the number of simulations we want to run. We are just going to use an integer in the set {0, 1, 2} to represent the index of the door. Let’s just generate a numpy array of evenly distributed, random choices:

def simulate_prizedoors(n):
    return np.random.random_integers(0, 2, size=n)

We can do the same thing to simulate the player’s guesses:

def simulate_guesses(n):
    return np.random.random_integers(0, 2, size=n)

Now we need a slightly more complicated function to generate the “goat door”. This is the door that the host chooses to reveal (which is always a goat). This function takes the prizedoors and the guesses as arguments, then returns the first door that isn’t the guess or the prizedoor.

def goatdoors(guesses, prizedoors):
    return [np.setdiff1d([0, 1, 2], [prizedoors[i], guesses[i]])[0] for i in range(len(prizedoors))]

We now need one more function which can simulate the player switching their guess. This does a very similar thing to goatdoors. It takes the guesses and the goatdoors and chooses the first door that is neither of those:

def switch_guess(guesses, goatdoors):
    return [np.setdiff1d([0, 1, 2], [guesses[i], goatdoors[i]])[0] for i in range(len(guesses))]

And one more function to take the results of the simulations and calculate the percentage of wins. If this function looks a bit naive it’s because my python is rusty:

def win_percentage(guesses, prizedoors):
    count = 0
    for i in range(len(guesses)):
        if guesses[i] == prizedoors[i]:
            count += 1
    return 100.0 * count / float(len(guesses))

Let’s run 10,000 simulations!

N = 10000

prizedoors = simulate_prizedoors(N)
guesses = simulate_guesses(N)

print "Wins without switching %f" % win_percentage(guesses, prizedoors)
#=> prints 33.3% +- 1.0%

prizedoors = simulate_prizedoors(N)
guesses = simulate_guesses(N)
goatdoors = goatdoors(guesses, prizedoors)

print "Wins with switching: %f" % win_percentage(switch_guess(guesses, goatdoors), prizedoors)
#=> prints 66.6% +- 1.0%

I also have a gist of the full code.