penelope.zone

A silly thing you can do with the Ruby parser

2019-12-22T00:00:00+00:00

Here’s a silly thing you can do with the Ruby parser:

begin
  raise "omg"
rescue =>
  (class Box
   class << self
     attr_accessor :contents
   end
end; Box).contents
end

puts Box.contents

# => "omg"

You… what?

So let’s break this down. In Ruby, the way one catches exceptions is with a begin...rescue...end expression. The rescue part of that looks like this: rescue <class_name> => <assignable_expression>. Usually you’d see something like:

begin
 ... whatever
rescue SomeClass => some_local_variable_name
  do_something_with_local_variable(some_local_variable_name)
end

However, it doesn’t have to be this way. In fact, the Ruby parser permits any valid assignment expression in Ruby! So we can for example set a key in a hash from a rescue:

h = {}
begin
  raise "omg"
rescue => h[:the_key]
end

puts h.inspect
# => {:the_key=>#<RuntimeError: omg>}

So, to come back to our first example:

begin
  raise "omg"
rescue =>
  (class Box
   class << self
     attr_accessor :contents
   end
end; Box).contents
end

puts Box.contents

# => "omg"

Here we, in the rescue clause, define a class called box, define a class level attribute called contents, and then set it. You shouldn’t ever do this in you production code, but it is possible.

Why is this even possible?

As far as I can tell, the reason why this works as it does has to do with the history of the implementation of Ruby. Before Ruby 1.9, Ruby used a “tree walking” interpreter. This means that Ruby code was executed from the syntax tree of the program, with no intermediate steps. Putting the rescued exception “somewhere”, conceptually, is the same thing as variable assignment, even though it doesn’t look like local_name =, so the code was shared between rescue and variable assignment.

Neat!

If you enjoyed this post, please consider sending me a follow on twitter: @penelope_zone.

What’s next for Rubyfmt?

2019-11-26T00:00:00+00:00

So today I was (happily) able to announce that the Rubyfmt unit suite is working in the Rust backend. This is a huge milestone for the library, but I also wanted to take a little time to talk about what’s next.

Implement a few remaining syntax nodes

The Rubyfmt unit suite is designed to be a comprehensive set of programs that show a minimal use of every possible node in Ruby’s grammar. I definitely missed a few while building out the first implementation, and this is evidenced quickly while running on larger programs. I’ll be working through filling those gaps as the next priority. I’m also intending to remove one very large program (rspec_core_notifications_actual copied directly from RSpec’s source code). I’ll be replacing that with a full execution over the RSpec source code

“fixup” renderer

Right now the Rubyfmt architecture is designed to walk a Ripper syntax tree and produce a queue of tokens. Those tokens have a “naieve” stringification. This naieve stringification prints very ugly Ruby programs. The pure Ruby architecture features a backend called “fixup” which introspects groups of those tokens to actually nice syntax. I need to implement that before anyone can get their hands on it.

Proper integration between the Ruby program and the Rust library

What I have right now is a total hack to just make something work. In order to be release worthy we need a story around integrating the Rubyfmt rust library and the Ruby shell.

CI / builds / QA

Right now the CI for rubyfmt is totally busted, and based on a beta of GitHub actions. I’ll be moving that to a more modern version of GitHub actions and doing things like automating rust compilation and clippy checks.

First things first

My first milestone will be the same as the pure Ruby version of Rubyfmt. Getting the RSpec Core test suite passing against formatted files. Once that’s done, we then move on to ensuring the formatted code looks good in most scenarios, and aim towards our next release.

You shouldn’t expect that any time right soon, as there’s plenty of work still to do. I appreciate, however, you following along for the journey. As always, day to day updates can be found on my twitter: @penelope_zone

Retiring from RSpec Core

2019-10-07T00:00:00+00:00

Hi Folks,

I’ll keep this one short and sweet. As of a little while ago, I messaged the other RSpec maintainers to let them know that I’d like to move on from the project, instead focusing on rubyfmt.

Many people don’t know this, but I essentially got started programming Ruby by working on RSpec. This has given me a very different perspective on how to work on Ruby than most folks I meet. I owe many parts of my career to the work that I did there and the help I got from the others getting started. Especially David, Andy, and Myron. It’s been a fun several years, but as of now, I’m no longer describing myself as “on the RSpec core team”.

Thanks :)

Minimum viable example of calling Rust from Ruby without a gem.

2019-07-13T16:39:13+00:00

For Rubyfmt I’m currently looking at rewriting significant sections in Rust, because there’s some performance issues. Broadly speaking, it turns out that cleaning up a very large Ruby parse tree (e.g. 4000 lines) can be slow because of the number of comparisons that have to be made. Rust can do this a lot faster.

See https://github.com/penelopezone/rubysoexperiment if you’d like to try it out yourself.

Starting with Rust

To start with, let’s imagine that I’ve got this rust function that I want to call from Ruby:

pub fn return_3() -> i32 {
  return 3;
}

First, we’ll need to prepare it to be used outside of Rust. By default the Rust Compiler will rename functions, and also make them incompatible with non-rust languages in order to be able to perform various kinds of optimisations. I makes a lot sense that the default is optimised for pure rust, but that’s no what we’re doing today.

So, I’ll rewrite my function like this:

#[no_mangle]
pub extern fn return_3() -> i32 {
    return 3;
}

#[no_mangle] means that the name is preserved for calling in c (or really any language that can load functions from a native library), extern means that it’ll be exported for use from whatever we compile.

Now we’ll need to modify our Cargo.toml to allow us to build a static library, that we can compile in to our Ruby extension eventually:

[package]
name = "rubysoexperiment"
version = "0.1.0"
authors = ["Sam Phippen <penelopezone@googlemail.com>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
[lib]
name = "foo"
crate-type = ["staticlib"]

Once we do cargo build, we’ll now see that target/debug contains a file called libfoo.a. .a means that the file is a collection of native functions that can be called by other languages, but must be compiled in with whatever program is going to call those functions (AKA a static library).

Making `return_3` available to Ruby

Ruby’s primitive types like Fixnum, String, etc aren’t C or Rust types by default, but rather are Ruby objects. The C type for these is called VALUE, which Ruby uses as a type under the hood to wrap every type of object that it can represent. So, we need a function that can call return_3 in a Ruby compatible way. We’ll do this in C so it can be more easily compiled in to a Ruby extension:

#include <ruby.h>
#include <stdint.h>

extern int32_t return_3();

VALUE foo_rb_return_3(VALUE klass) {
    int32_t three = return_3();
    return LONG2FIX(three);
}

This block of C includes ruby.h, a C header file that gives us various functions to work with the Ruby interpreter, and stdint.h, which declares fixed width integer types (in this case we need int32_t, because we declared our function to be an i32 in Rust). It then declares that return_3 is a function from outside of the C file (because it’s declared in Rust). Then it declares the function foo_rb_return_3, which calls return_3, and then wraps it in the LONG2FIX macro. This macro comes from ruby.h and converts a C int32 (really any numeric type) in to a Ruby Fixnum.

Making this file a Ruby extension.

Now that we’ve done that, we need to do a few more things to make this C file a valid Ruby extension. When Ruby loads native extensions it calls Init_<libraryname>, which for us is going to be Init_foo. So let’s define that function:

#include <ruby.h>
#include <stdint.h>

extern int32_t return_3();

VALUE foo_rb_module_foo = Qnil;

VALUE foo_rb_return_3(VALUE klass) {
    int32_t three = return_3();
    return LONG2FIX(three);
}

void Init_foo(void) {
    foo_rb_module_foo = rb_define_module("Foo");
    rb_define_module_function(foo_rb_module_foo, "return_3", foo_rb_return_3, 0);
}

We’ve made a few changes here. Firstly, we’ve defined a new variable: VALUE foo_rb_module_foo = Qnil. This is essentially declaring that we have a new Ruby object called foo_rb_module_foo. When Init_foo is called, we call foo_rb_module_foo = rb_define_module("Foo");. This will declare in Ruby a module called Foo that we can hang all of our C functions off.

Then, we put our Ruby-ified foo_rb_return_3 in our module with rb_define_module_function(foo_rb_module_foo, "return_3", foo_rb_return_3, 0). rb_define_module_function is the underlying C function that is equivalent to the module_function method in Ruby. The first argument is module we want to define that function on (in this case Foo, which is foo_rb_module_foo in C). Then we give it a C string, which is the name of the function in Ruby (return_3), then the C function to call (foo_rb_return_3) then the number of positional arguments the function takes (in this case 0).Foo, which isfoo_rb_module_fooin C). Then we give it a C string, which is the name of the function in Ruby (return_3), then the C function to call (foo_rb_return_3`) then the number of positional arguments the function takes (in this case 0).

Building everything

Now that we’ve got our Rust and Ruby modules set up, we need to build everything. Building the rust piece is easy, we can just run cargo build. That’ll create the libfoo.a we referenced earlier.

However, the Ruby-C-Rust bit is a little bit more involved. Linking C code against the Ruby interpreter isn’t as easy as just calling clang and expecting everything to work. But fortunately, Ruby provides a convenient tool for us, called “mkmf” (Makemakefile for long). To use it, we write a Ruby file called extconf.rb (it isn’t required to be called this, but this is the conventional name for this file).

#!/usr/bin/env ruby
# extconf.rb
require "mkmf"

$LDFLAGS << " -Ltarget/debug -lfoo "

# preparation for compilation goes here
create_makefile("foo")

The only line of this file that is doing anything particularly interesting is $LDFLAGS << " -Ltarget/debug -lfoo ". This line basically tells the Ruby extension compiler where to find the Rust extension we created earlier.

create_makefile("foo") creates a Makefile that’ll compile all the C files in the current directory, expecting to find an Init_foo function. Makefiles are somewhat complex scripts for compiling C programs that have a varied set of dependencies.

Once that’s done we can run make, and on my mac we’ll get an output file called foo.bundle.

Running it

So! Does it work? (Yes.)

Here’s how you use it:

$: << "."
require "foo.so"

p(Foo.return_3)

We can require foo.so and it’ll load the foo.bundle because Ruby knows how to translate between the various operating system dynamic library names.

So, there you have it, a very simple C extension, in Ruby, without rubygems.

Getting the most mileage out of Docker and Rails

2019-02-06T13:59:54+00:00

Hi Folks,

Recently I was working with a friend on their Rails app, which had been fully wrapped in a docker-compose setup. They were quite rightly asking me questions about the best way to do things like run the server, run tests, migrate the database and other common Rails tasks. While I’ve seen a number of different ways of Dockerizing Rails in development, my preferred way is not to Dockerize my Rails app itself. In development I prefer to use Docker to manage any complex dependencies my application has, leaving my app running bare on my machine. When it comes to production time, that’s the ideal place to Dockerize your application. Let’s look at why!

It starts with the “Rails Way”

The “Rails Way” is all the stuff that Rails is designed to make really easy, for example:

Using generators to quickly build out standard pieces of your app
The Rails console to quickly modify development data
Code reloading
Being able to get in to your database if you really need to

In order to do get all the benefits of Rails’s finely honed development tools you typically need a postgres (or mysql) database, and a redis instance (because you’re using sidekiq right ;)). Those dependencies are all pretty easy to get going on a Mac or Linux machine these days. So: if I’m building a Rails app that has an eventual Docker target in production, but only those simple dependencies, it is my preference to not use Docker in development at all. This way, I get the experience of developing Rails as it is intended, and all the benefits that come with it being local to my machine.

I have my editor set up so that I can drop in to a Rails console or execute the tests with a quick flick of my hand. Those shortcuts work because the commands execute on the local machine. I’ve never found a way where I can achieve the same level of seamlessness using any combination of Docker or docker-compose.

But what about if my Rails app needs a constellation of microservices to be realistic

This is becoming a more common case. Let’s say that you’re a company that grew up on a Rails monolith. You’ve extracted functionality to a microservice that means you can’t realistically develop your Rails app without the extracted microservice up and running. Also, your microservices are a huge pain to coordinate without Docker and docker-compose. You also have this case if you have a more complex dependency like Elasticsearch, or Neo4j, or some other component to your application that isn’t trivial to brew install and have work.

So your Rails app is dependent on something that you can’t for whatever reason run cleanly bare on your machine. What I do in these cases is build a docker-compose.yml file which knows how to spin up all my app’s dependencies, and then use .env to point my development server running on my bare machine at those services. This way, I get all the benefits of local Rails development, and all the benefits of Docker for the things that are hard to run on my machine.

I love using Docker for Rails in production

For clarity, I do think you should still Dockerize your Rails app if you have a good production container runtime story. Containers provide you with the ability to deploy a single static artifact of your application. This artifact ensures that every server is running the exact same code, has the exact same assets, and if configured properly, is talking to the exact same dependencies. There’s a mountain of benefit there. In production, I don’t need many of those “Rails Way” things, a console, a reloader, running the tests are all not useful in that environment. It’s quick and easy to get started with Docker for Rails. It’s a very appropriate way to get your app to production.

Conclusion

Working with Rails for the last 10 years of my life has been an absolute joy. I want to keep using the amazing tooling that I’m familiar with. I want to get as much of the benefit as I can from Docker, and all the tooling that comes with it. In my opinion, though, Docker isn’t well suited for Rails development because the local development experience has been so thoroughly optimized. Instead: using Docker for what it’s good at which is isolating dependencies makes a tonne of sense in a traditional Rails development context. Docker is well suited to deploying Rails apps to production, however, and you should by no means take this as a dislike for the technology. Docker is great, and using it in production has genuinely made my life significantly easier. Your optimised development environment is Rails, with docker supporting, and not the other way around.

If you liked this post, please consider following me on twitter

Questions for a prospective employer about on call (2/3)

2019-01-14T13:10:42+00:00

Hi Folks,

This is part two in my series about talking to prospective employers about on call. In the first part we covered a bunch of general questions. In this part, we’ll dig down in to specific questions about incidents and how they function. An incident is something that happens that causes you to get paged when you’re on call, and so asking questions around this is pretty important.

How many people are on the smallest team that is on call, how frequently do they paged for out of hours incidents? Are you doing anything to help alleviate their pain?

By definition the smallest on-call team has their engineers rotate most frequently. If you get an answer which is a really small number (say 2 people in an engineering org of 50) that’s a big problem. Unless the platform that you’re working on is ridiculously stable, you’re going to have a bad time. Really this question is aimed to help you get an understanding of how the engineering leadership in the business thinks about the health of their incident management program. Are they working to remove single points of failure? Are they working to support that small team by sharing on-call with another team? Are they rotating engineers or cross training people back in? These are all good questions to ask to get a feeling for how much pain the worst off engineers in the company are. More importantly if you were to end up in the same situation, how likely it is that you’d have a leader you could talk to to make sure something is done about it.

Does your company have distributed employees that can take on call for times that make sense for them?

I was once lucky enough to work on an on-call team that split between East Coast America and European time zones. The five/six hour time shift meant that we as a group could take some of each other’s on-call burden. We’d specifically level it in such a way that the most obnoxious times for people to receive a page were covered by a more reasonable time in the other time zone. This gets easier as you have wider distribution.

Many companies do not have the luxury of doing this, because everyone works in a single time zone in a single office. If you do have a distributed employee base taking advantage of that in order to alleviate some of the pages at the worst hours can be really great. Talk to individual engineers and individual engineering managers about how they deal with remotes and timezones.

Do teams routinely create and put in to place newly automated alerts?

This is more common as a practice than it was, say, five years ago. Automated alerts are most important around new services or new features that are going into production. It sort of maps back into the severity scale discussion. Are you defining your severity scale by something automated, or relying on some kind of human at your back?

One of the things this question is designed to hint at though is which tooling a company is using to get these alerts in place. Do they have something like Datadog integrated all the way to their Pagerduty? Are they using the CNCF Prometheus/Grafana/AlertManager stack? How easy is it for an engineer to synthesize a hypothesis about what a good alert is, test it, and then put it into production if the alert is meaningful? Organizations that don’t have a good strategy around automated alerting are generally resistant to adding newly automated alerts, and this is what we’re trying to bare out here. How much work is it for you as an engineer to add a new alert in production?

Do you conduct any kind of review on your automated alerts to ensure they are not becoming noisy?

Businesses grow over time, operational characteristics of our systems change, third-party providers change their service levels. The metrics against which a given alert fires are going to drift over time. An alert that was very indicative of a problem six months ago might suddenly become noisy because of this drift. An alert which used to be sensitive and accurate may fail to fire when data starts to change.

Conducting some kind of alert review on a regular cadence is a great way to prevent this. Validate that the alert thresholds are still tight, that we’re not too close to suddenly firing and waking somebody up at 4am when we didn’t need to. If your teams have a reasonable workload this entire exercise can consume less than an hour a month, and that’s totally worth it.

This is one of those questions where if you get a “no” answer, it might not be a problem. The business might not be changing fast enough to require this kind of review. There may be mitigating circumstances. Here you’re looking, again, for thoughtful leadership responses and or good mitigations in place, not just a flat out “this is not a problem for us” with no reasoning.

Do you have a support/operations/customer success team that can call a page in lieu of an automated alert?

If something’s going horribly wrong, you want to be notified. Sometimes you’ve failed to cover your service in enough observability, something gets missed, and you’re down and not automatically woken up. Hopefully, you’ve got a support team who is awake to notice. Hopefully, they’ve got a path to escalate to you.

This can be a double-edged sword. A human making the decision to file a JIRA versus waking you up in the middle of the night can be a fraught thing. They don’t want to be a jerk for waking you up, and you don’t want to be woken up. Similarly, they’re incentivized to keep the customers happy, and so filing a ticket and waiting until morning might be a really shitty thing for them to have to deal with on their end. This is where having a severity scale can help, but when it comes down to human determination, there’s always going to be room for some error. The important thing for you as the engineer to know is that if they do wake you up for something that isn’t as big of a deal as it seems at first, that’s a systemic failure, it’s not the fault of that person. You need to collaborate with them to clarify on either the severity scale, how to triage incidents, or how the system works.

You’re going to get a lot of varied answers to this question because every organization puts these teams together differently, and gives those teams different objectives. This is a good opportunity for you to dig into the structure of the company that you’re joining such that it exists beyond the engineering organization. Engineering doesn’t exist in a vacuum, and so when you’re thinking about this, understand how communication from the company’s customers makes its way to engineers. When something is broken for a customer, what exactly happens before an engineer gets involved, and how frequently do humans make a call to bring an engineer in out of hours?

Do you have any kind of 24 by 7 team (e.g. a NOC, SOC, or “cloud operations” team) that does first level triage on behalf of engineers?

A yes answer to this question will make your life significantly easier. To take a quick diversion for a second: at DigitalOcean there is a 24x7 operations team called CloudOps (short for cloud operations) that works in three shifts. They’re technical folks who mix engineering, sysadmin, and SRE skills. They exist as a point to which support teams can escalate issues, make severity determinations if engineers aren’t already on top of a problem. They coordinate the start of an incident to get everyone needed to resolve it. This team is great (I love you if you’re reading this CloudOps) and it was one of the best tools in my operational arsenal. They would frequently prevent something from getting through to my team by executing a playbook against an incident for us, and for that, I’m eternally grateful.

Many engineering organizations are too small to have a team like this. If you’re big and you don’t have a team like this it might be worth considering what the impact of a 24 hour operations team would be. In my experience having a series of operators and communicators who are continually fresh can significantly reduce the time to resolution on any incident.

Getting a “no” answer to a question like this is pretty standard, and it’s not really a red flag. In addition to the usual 24x7 support and success operation that many organizations have, having a 24x7 technical team as well can be unnecessary overkill. If you don’t have a team like this, though, it might be worth considering you could form a rotation of this form. If your business has a significant operational burden (not everyone is a cloud hosting provider, or of similar class), it really can be a lifesaver.

Have you ever had an incident where you felt like a severity decision was made for an incident that was too high, causing people to get paged when they shouldn’t have? How did you remediate that?

This is one that I’d aim squarely at the engineering leadership of a company. Keeping incidents appropriately sev’d is going to show a strong difference between a healthy and unhealthy engineering organisation. When everything’s a SEV-0 biggest most important fire in the universe nothing is. It’s a natural response though: any little defect in the product is probably a big deal to somebody. Most likely your customer-facing teams are going to take some flak for even the smallest of bugs and it’s natural to want to fix them as quickly as possible.

Conversely: engineers getting paged too frequently causes them to burn out and leave, so striking a balance is important. When a call for severity is made it is of course open to inspection and review. If any person genuinely feels like an incident was over or under sev’d, they should absolutely speak up about it. It’s then up to the leadership of the company, or someone who’s responsible for running the incident management program to make a decision as to how that call ultimately sits.

Usually, when a “bad” call of this form is made, it’s due to a lack of communication. What that means is that probably some documentation needs updating, how a product works needs to be clarified, or the engineering team needs to improve their automated alerting or metrics gathering such that it’s more obvious what’s going wrong when something’s going wrong.

You may find that this hasn’t happened, because it’s not humans that are dealing with alerts, but in general, making sure that someone’s got an eye on this is a good idea.

Conclusion

Incidents are painful for everyone involved. When they trigger, who triggers them, and what happens when they trigger are really important for you to understand. Healthy engineering organizations have a strategy around alerts that goes beyond “customers start complaining and then we respond”. Digging in to the alerting strategy, alerting review, and process for support escalations will help you get a better handle on whether or not a company is careful and considered about its on call.

Questions for a prospective employer about on call (1/3)

2019-01-04T14:45:23+00:00

Hi Folks,

This is the first in a series of three blog posts. These contain questions that I asked prospective employers when I was interviewing recently and why I asked them. On-call is one of the biggest determiners in whether or not a job is particularly painful. It also seems to me to be becoming a more “normalized” practice. These questions are designed to give you a feel for how seriously engineers, and, importantly, engineering leadership take their on-call responsibilities.

The reason you want to determine how engineering leaders think about on-call is that they’re ultimately responsible for how well or badly on-call will go. Things like how much engineering pain they’re aware of, how they’re helping drowning teams, and what they personally do to ensure there is mature operational practice have an outsized impact on how on-call goes.

In this first part, I’ll address questions that have to do with what I’m calling “general maturity”. This is a quick health check on where an organization is at on the design of their on-call practice. Do they have good procedures, are they thinking about who’s on-call and when, do they have an understanding of the uptime requirements of their product?

In the next post, I’ll talk about the specific nitty-gritty of being on-call. In the final one, what the entire incident management program of a company looks like, not just the on-call part.

These questions are formed from serving close to three years as an on-call engineer at organizations of various sizes. Also serving as an engineering manager for the last year of a team of 3, all of whom were on-call. I’m by no means trying to declare this is the absolute best framework to assess a company by, but it will let you start discussions and have a lot of insight. I really do hope you enjoy it.

General maturity questions

Does $product have to be online 24/7? Do you expect engineers to be on-call out of hours

This may seem like a “dumb” question, but I’ve been surprised by companies that have answered no to this question. There are obvious places where the answer to this question is no: retail POS, many types of financial services businesses, etc. If you get a “no” answer to this question, and it doesn’t sit right with you: probe. You might find that this isn’t well considered and that the software, in fact, needs to be online 24 hours. If that’s the case that’s a big red flag.

You’ll often hear a “yes” answer as a simple default. Things like clouds, software that people might need to query information out of at any time of day, communications software, etc, it’s probably a well-founded reason. If you get a “yes” answer and you think that the product doesn’t need to be online 24/7, again, probe. Making sure that you get to a good reason for the availability expectations that the leadership of a company is placing on you is important. A 24/7 bound is basically the starting point for all of the questions in the rest of this post. Pages during “normal business hours” or slightly outside (say 7 am to 7 pm) are significantly less painful to deal with than those in the middle of the night (put your hands up 4 am ops club). If you’re entering into an organisation that expects its engineers to be on-call out hours, you want to know that.

You want to inquire here for uncertainty or bad reasons for uptime requirements. If you’re expected to do on-call for something that demonstrably has almost no users in the wee hours, that’s a problem. Waking up at 4am sucks, and you want to make sure it’s for good reasons, so this is really the start of the conversation, and your chance to probe both individual contributor engineers and engineering leadership (I’ve asked this question all the way up to CTO level) about how they think about on-call.

Is anyone at this company performance managed on the reliability of the software that is being produced?

No is the most common answer to this question. Most companies are small and require little process. As such, frequently, shipping features is performance managed and reliability often isn’t. If you’re joining a company that’s lucky enough to have a “real” SRE (Site Reliability Engineering) function (not a team of SREs that sit somewhere, but SREs embedded within product teams) the answer might be someone in SRE leadership. Sometimes a VP of engineering or Director will take on this role, and try to manage reliability holistically. That sometimes works, and sometimes doesn’t.

You should ask this question more as a determiner of attitude than a binary yes/no question on which to eliminate a company. If engineering managers and engineering leadership want to improve in this area, that’s a great signal. If it’s something to which they’re not putting a large degree of concern, that can be a red flag.

If you hear words or explanations to the effect of “we trust individual engineers to manage the reliability of the software that they write”, that’s a problem. No matter how well-intentioned an engineer is if the structure of their organization is focused on shipping software that engineer will eventually tune out their focus on reliability. Non-functional requirements, things like metrics, timeouts, retries, etc will go to the wind as you naturally incentivize those engineers to head down the “easy” path and write code that “just” works, apart from the myriad failure conditions they didn’t account for.

Is there any kind of cross organisational service level management that all teams aim for (e.g. support engineering on successful ticket submissions per hour, checkouts team on checkouts per hour etc). If so, when teams miss their prescribed service level objectives, what do you do?

This question is a good follow up to the previous one. Generally speaking, if nobody is being performance managed on service level, then service level won’t be being measured either. If service level is being managed, this is a great place to probe into what that practice looks like. Are they measuring business metrics (tickets submitted, emails sent, etc), or non-functional metrics (throughput, latency, error rate)? If so, does the organisation have a preference on service level of one kind of metric over the other? Neither is necessarily better but in my experience organizations that really care about their users tend to focus on the functional metrics.

Asking what organizations do if teams miss their prescribed service level is designed to probe at how the company holistically treats the health of their systems. Some companies will respond to failing service levels by making teams do reliability work until they are back within that service level. Some will take a look at any open postmortem action items and push those to the to top of their priority list. They’ll do this to improve their service level rather than continuing to ship features. Some will do nothing at all but insist that they really do care about reliability.

Use this as a good place to determine if reliability is being measured in any kind of consistent way throughout the organisation, or that it’s up to individual teams and engineers that care. When you’ve got a consistent organisational focus on reliability, that’s a sign that on-call is less likely to be hell.

How do teams at your company manage their on-call rotation?

The most likely answer that you will get to this question is that “we leave it to individual teams to manage their on-call rotations”. That’s a good answer but has a couple of pitfalls. If there’s no holistic review or individual teams within the organisation don’t have strong managers or a strong retrospective process, it’s easy for those teams to get stuck with a static on-call rotation. Being stuck like this makes people less adaptable to changes in circumstance, and can be a red flag.

By far and away the most common rotation I’ve seen is that teams do “on-call primary for 24 hours a day, for a week, and then rotate”, sometimes referred to as a 24 by 7. There’s usually some kind of secondary rotation which is a person who is the backup if the first person fails to respond. Sometimes that person is the lead of the team. As a final step: some teams opt to page everyone on the team if the first page is not responded to. This kind of rotation won’t work for everyone, and it’s important to make sure that a rotation of that form isn’t a blindly followed default. Look to see if the on-call rotation has actually been well considered when interviewing for a team. This is a great question to put to an individual engineering manager. As you go higher into engineering leadership, specific rotation details tend to melt away.

Another line of questioning here is how a team deals with swaps. Emergencies, flights, conferences, and vacations all happen. Determining if the team’s culture is healthy enough to support these kinds of things without scorekeeping can give you a really good lens into how healthy that team is.

A spin on this question for engineering leadership is to see what kind of health checks they’re conducting on their team’s rotations. Do they get any kind of reporting as to who’s getting paged the most frequently in their organisation? Do they insist that managers rotate an individual when that individual gets paged too many times in a week? What kind of high-level on-call rotation reviews are being conducted?

I’ll also note here that some people absolutely cannot deal with an out of hours call. This is for a variety of reasons ranging from being a single parent to mental health issues, and so on.

Sure. My problem is when that’s assumed to be enough flexibility for everyone.

I’m a single parent. My kids school still starts at 8 regardless of whether I was up the night before. There’s nobody else to take him. Getting up after midnight just is not an option.

— Sarah Mei (@sarahmei) December 28, 2018

These people also might be fantastic engineers, and excellent on-call responders, who don’t mind the interruptive work during their day to day. If you can determine that they’ve got some kind of process for dealing with this, or do 12 hours on-call slots where some people volunteer to take overnights on behalf of their team, that can also work. The point is, you should probe to determine if there’s any kind of flexibility there. Even if you don’t need it now, you may in the future, and it’s good to know it’s available.

Every answer that you hear to this question will be different. Red flags will include that someone in leadership can’t tell you holistically how different teams manage different on-calls. Also look to see if on-call is dictated across the entire organisation with little to no team specific flexibility. If a mid-level engineering manager has their on-call rotation all laid out but doesn’t have a specific reason as to why the on-call is like that, it might be worth probing to see if they’re open to changing it based on discussions and retros with the team.

Do you have a “Severity Scale”? If so, could you walk me through what the ratings are, at what severity people can be called out of hours, who makes severity determinations, etc?

This is a meaty question, and will likely consume the majority of the conversation with someone who’s in engineering leadership. There’s a lot of data that you can take away from this one. Firstly, you should aim to get the description of what the severity levels are. For example, you might have:

SEV-0: A critical business-ending event, multiple products are unavailable, all customers cannot log in, registrations are failing, the whole site is unavailable, etc. Engineers and engineering leadership paged immediately to coordinate response
SEV-1: A serious impact to a single product area, business function, etc. Entire site responding but outside of usual performance SLO, Product is regionally unavailable in a single country/area of the world. Engineers paged, potentially an engineering leader paged to coordinate incident response if needed
SEV-2: A serious workflow breaking bug in a single product that does not have a workaround, single view or part of site outside of performance SLO. Engineers paged immediately
SEV-3: A workflow breaking bug exists but support has been able to determine a workaround, JIRA ticket filed and engineers expected to fix when they come in the next day
SEV-4: minor bug or piece of functionality not working, either not serious enough to need a workaround or simple workaround exists, JIRA filed, engineers expected to fix as part of usual bug fixing/sprint work

You’re looking for something that is precise. Ideally you will have specific definitions based on the product and business units within the company that you’re interviewing for. The reason that you want something like this is that it clearly distinguishes when you can be interrupted out of hours to fix problems. Having shared definitions like this enables you, your engineering leadership, and your support and operations folks to agree on how to respond to any given problem that they’re seeing.

As to who makes severity decisions: you’ll want the majority of incidents to be triggered automatically. It’s worth acknowledging, however, that we can’t cover all possible things that would go wrong in automated alerts. As such we usually concede that as a last resort a human is going to make a call to bring in engineers to try and fix the problem. If that’s the case, you’ll want to follow up by probing into who can make severity decisions, and out of hours page decisions. When a decision to page someone is made who makes that decision: a support individual contributor? Support manager? Is there an operations team that can triage for you? An on-call incident manager who is an engineer from any team? Those things will determine how frequently you get a bad page and have to down-sev or redirect an incident.

It’s worth noting here that not having a sev scale isn’t necessarily a bad thing. If you’re joining a tiny company (n < 15 engineers on staff) you might not have a wide enough range of things that can go wrong as to need this much process and documentation. It’s worth considering whether the answers that you hear map to the amount of process you’d expect the company at which you’re interviewing to have.

A big red flag here is that if this document exists, but is stale, isn’t collaborated on by all stakeholders, etc. It should be a living document that accounts for new products, launches, teams, etc. Not a static document that was dictated by some engineering leader one time and then never changed again.

Is there anything similar to a holistic incident management program at this company? Is someone in charge of that?

This question goes right along with the severity scale question. Answers that you’re looking for here include maintaining the severity scale, postmortem process (if they have one!), pre- and post-incident management, etc. Not everyone has this! Larger companies that care about reliability tend to hire either a technical program manager into this position or make it the responsibility of one of the more senior engineers on staff.

Hearing a “no” here, again, isn’t a hard elimination. Not every company needs something like this, not every company has a service level requirement on their software that requires this role. If the program does exist, digging into why it was set up, and digging into why the person who is running it is running it can be a really good indicator of a healthy and mature attitude towards on-call.

Generally speaking, is the number of incidents that engineers are dealing with out of hours going up or down?

This is really the start of a discussion. Based on the answer that you hear you’ll want to ask what specific steps are being taken to improve the situation, what new practises the organisation is using, etc. Are your engineering leaders concerned with the state of this at all? Do they know? What are they doing to personally improve the lives of the engineers on the ground? Are those engineering leaders feeling any of the pain? Use this one as a point to establish whether or not you’re likely to encounter increasing or decreasing pain. I’ve not got a whole lot more to say here, because repeatedly asking “why?” at this stage is basically the best way to go.

Conclusion

This first round of questions will give you some insight in to the maturity of a specific organization’s on-call practise. If you hear strong consistent answers that have been clearly thought about through all these questions, that’s a really good sign. If you come away from the discussion worried, that’s something to follow up in with further conversations. I’ve found that when people are on top of on-call, you can bare it out really quickly. I do hope this is useful for you and helps you start to build a framework about what a good on call practice might look like for you.

How I organise my VIM hotkeys

2018-12-27T10:03:35+00:00

Intermediate to advanced VIM users love to extract common workflows to hotkeys. You might want to optimise workflows like running tests, adding files to git/doing a commit, or reformatting text. The thing is, as you add more and more workflows, organising these hotkeys can become a nightmare. For the past several years, I’ve had a philosophy to organising mine that has really helped me.

The leader key

For me it all starts with the leader key, \ by default, but some people use ,. In a VIM setup that isn’t at all customised, no hotkeys follow the leader key, and this makes it an excellent place to hang functionality from. In my .vimrc there’s a whole section filled with lines that look like this:

map <leader>jna :!nosetests <CR>

This says “when I press \jna run the shell command nosetests”.

The first protip I have for you is fairly simple: hanging everything off the leader key is a great place to start your hotkeying.

Prefix trees

Consider a world in which I had my leaders set up like this:

map <leader>j :!bundle exec rspec<CR>
map <leader>jf :!bundle exec rspec %<CR>

This creates an ambiguity, if you press <leader>j, VIM can’t know whether or not you want to execute the first or second command: until you press the f key, or wait a for some period of time. This creates an unnatural pause if you do want to run the j leader, which breaks flow. Additionally, if you press <leader>j and then pause for a second because you are interrupted for any reason, you’re going to run bundle exec rspec executing your entire test suite, instead of executing a command over a single file (the % character passes the current filename to the command, which RSpec will then execute as a single file test, rather than all of them).

This problem is because one leader j is a valid prefix of jf. The way to fix this is to use a “prefix tree”, where there are no valid commands which overlap in their pefixing. Rather every command is valid and distinct, causing you to type an entire sequence before anything will execute. In my dotfiles, I have all of my leader sequences mapped such that they form a prefix tree, have a look:

map <leader>jca :!bundle exec cucumber --no-color<CR>
map <leader>jra :call RunAllSpecs()<CR>
map <leader>jrf :call RunCurrentSpecFile()<CR>
map <leader>jrn :call RunNearestSpec()<CR>
map <leader>jrl :call RunLastSpec()<CR>
map <leader>jr! :call RunLastFailure()<CR>
map <leader>jna :!nosetests <CR>
map <leader>jnf :!nosetests %<CR>
map <leader>jma :!ruby test/test_helper.rb <CR>
map <leader>jmf :!bundle exec ruby % <CR>
map <leader>jta :!./script/test<CR>
map <leader>jmf :!ruby %<CR>
map <leader>rpf :!python3 % <CR>
map <leader>rrf :!bundle exec ruby %<CR>
map <leader>rb :!bash <CR>
map <leader>rv :so ~/.vimrc <CR>
map <leader>rm :!touch % && make <CR>
map <leader>rc :!ctags -R<CR>
map <leader>cr :!cargo run<CR>
map <leader>cb :!bash -c 'cargo build 2>&1 \| head -n 10'<CR>
map <leader>ct :!cargo test<CR>
map <leader>eir o<ESC>ogem "rspec-core", :github => "rspec/rspec-core"<CR>gem "rspec-expectations", :github => "rspec/rspec-expectations"<CR>gem "rspec-mocks", :github => "rspec/rspec-mocks"<CR>gem "rspec-support", :github => "rspec/rspec-support"<CR><ESC>
map <leader>eipr o<ESC>ogem "rspec-core", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-core"<CR>gem "rspec-expectations", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-expectations"<CR>gem "rspec-mocks", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-mocks"<CR>gem "rspec-support", :path => "/Users/sam/dev/rspec/rspec-dev/repos/rspec-support"<CR><ESC>

map <leader>eid orequire 'pry'; binding.pry<ESC>
map <leader>li gg/def.*init<CR>
map <leader>lcd gg/class.*<CR>
map <leader>lrp /^ *p <CR>
map <leader>arp :Ack "^ *p " <CR>
map <leader>apr :Ack "binding.pry " <CR>
map <leader>eal :Align & <CR>
map <leader>eap :Align => <CR>
map <leader>edc mzO<ESC>j:s/,/,\r/g<CR>(%i<CR><ESC>%a<CR><ESC>(%=%:%s/ *$//g<CR>:noh<CR>'zkdd
map <leader>ecl :silent! %s/\[x\]/\[\]/g<CR>gg
map <leader>eae :Align =<CR>
map <leader>ea{ :Align {<CR>
map <leader>erf i.fetch\x1Blr{r(f]r)
map <leader>erh xea:\x1Bllxxx
map <leader>epl :PromoteToLet<cr>
map <leader>etw mp:%s/ *$//g<CR>:noh<CR>'p
map <leader>ert :%s/\t/ /g<CR>
map <leader>orf :call OpenSpec()<CR>
map <leader>orv :call VsplitSpec()<CR>
map <leader>onrv :call VsplitSpec()<CR>irequire "spec_helper"<CR><ESC>
map <leader>ors :tabe .rspec<CR>
map <leader>ogf :tabe Gemfile<CR>
map <leader>ovr :tabe ~/.vimrc<CR>/map.*leader<CR>:noh<CR>
map <leader>bi :!bundle install<CR>
map <leader>ft zfa
map <leader>uft zR
map <leader>gt :!go test -tags=integration -ldflags -s ./...<CR>
map <leader>gb :GoBuild<CR>
map <leader>snp :set nopaste<CR>
map <leader>sp :set paste<CR>
map <leader><space> :noh<CR>

A quick scan of this will show you that there are no issues with command prefixing, where if I type part of a command, another command will execute.

We can visualise some of this as follows this as follows

What this shows is that there is no sequence of keys that can be pressed that is ambiguous to VIM as to whether or not it should run a command. In addition, not all commands sit at the same level. For example \snp and \sp are of different lengths, but both are terminal and unambiguous.

Conclusion

To quickly conclude: VIM leader commands are a useful tool for vimming faster. You can extract them by finding common sequences you type frequently, and putting them in your .vimrc. I’ve found this style of organisation, where each letter you type while building a command is a prefix that cannot ambiguously resolve to another command. Thanks for reading!

Rack middlewares that have saved me literally hours of my life

2018-12-25T15:49:33+00:00

In Ruby, Rack is our webserver baseline. It is an incredibly simple interface. A rack app is any object which has a public #call method that takes a single argument, typically called env, which represents the environment of a HTTP request (params, headers, etc) and returns a three item array containing:

An integer containing a response code
The response headers
An object responding to #each that emits strings to build the response body

which represents the response to the request.

The cool thing about this interface, is that it’s very easy to insert an intermediate object between the request and the app generating the response, “wrapping” it, to modify the behaviour of request processing. These intermediate objects are called Rack middlewares. They have the exact same interface as a rack app, but with one more refinement: their constructor takes the app that they are wrapping.

This contract between Rack and your code is designed to be easy to understand. What this means is that once you’ve got an idea for a rack middleware you can just type it out. You can quickly write them in config.ru as a Ruby class, and use them by invoking invoke use with the classname of the middleware you want. Once you’ve done those two things you’re good to go. This simplicity makes them very attractive for development and even production debugging.

What follows is the full source code of a couple of middleware that I’ve created and some explanation of how they’ve made my life much easier.

The `potato` middleware

I don’t know why I called it this. When I was working with one of my colleagues, they were like “WTF is potato”. Still, here we are. This is a debugging tool of last resort. Something that you break out when you’re beginning to lose faith that the Ruby programming language functions on a fundamental level.

class Potato
  def initialize(app)
    @app = app
  end

  def call(*args, &blk)
    @app.call(*args, &blk)
  rescue BasicObject => e
    require 'pry'; binding.pry
  end
end

use Potato
run Rails.application

Sometimes you can’t work out why an exception is blowing up your app. Sometimes the rails debugger doesn’t kick in. Sometimes you’re tearing your hair out in frustration and you need some help.

This middleware drops a pry at the very outer most layer of your app. I’m a big fan of the pry debugger, as it lets you deeply investigate the state of a bunch of things in your application. Putting it here at the very outermost level of your application lets you be certain that you’re bypassing anything Rails has introduced, or anything else that might be in the way making it harder for you to debug your application. This middleware has enabled me to better understand what’s going on in my application a number of times.

Stackprof middleware

class ProfilerMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    if env["QUERY_STRING"].to_s.include?("stackprof")
      StackProf.start(mode: :wall, interval: Integer(ENV.fetch("STACK_PROF_INTERVAL"], "250"), raw: true)
    end

    result = @app.call(env)

    if env["QUERY_STRING"].to_s.include?("stackprof")
      StackProf.stop
      file_name = "/tmp/foo.txt"
      StackProf.results(file_name)
      result = [200, { "Content-Type" => "text/plain" }, [File.read(file_name)]]
    end

    resul
  end
end

This one’s a little more complicated. Stackprof is one of my all time favourite profiling tools in Ruby. At DigitalOcean, we run a lot of our apps in docker containers on kubernetes. Which means that we can’t just SSH in and introspect running processes or the file system. I needed a way to be able to profile the execution of a request, in production, and get the results back. This is what I came up with.

When this middleware is installed if you include stackprof anywhere in the URL query string, the response body gets replaced with the result of a profiling run. You can then use this to generate flamegraphs or other useful profiling information. We don’t directly put this in to customer facing applications in production, but it’s been very useful for performance improvements in internal applications.

Unicorn worker killer

Unicorn worker killer can be configured to kill individual unicorn workers when they’ve either served a certain number of requests, or a certain amount of memory. I didn’t write this one, but it’s been useful to me to improve the stability of my production applications. Frequently, you’ll observe that a unicorn stops responding to requests to some reason. Generally speaking, I’ve found that rebooting processes in production frequently is a way to make them more stable, and so this is just a convenience wrapper for that. If you’re not using Unicorn, you probably don’t need this.

So there you have it, a few rack middlewares, two that you can write by hand, one that you can install from a gem, that are very useful. Thanks for reading!

Tensorflow journal, part 1

2018-12-25T15:49:14+00:00

Hi Folks. This is a little post where I’m leaving a bunch of single line notes of things I’ve found useful in tensorflow, for image based tasks. I mostly have no idea what I’m doing, but I’ve found some things while working on it that work well for me. Maybe you’ll find this useful!

The RMSPropOptimizer seems to be very good for a wide range of tasks. 0.001 seems to be a good default learning rate
Deeper networks need more training rounds to converge. Anecdotally, doubling the number of training rounds per layer (1000, 2000, 4000, etc) seems to be good
The inception module is a very good tool for image discrimination tasks (e.g. recognition, the discriminator in a GAN, the encoder layer in a VAE). Most of the implementations on the internet are complicated. here’s mine.
Directly connecting dense layers to undownsampled convolutional networks will cause a computational complexity/memory explosion. As a rule of thumb, a dense layer should have no more than 2048 inputs.
If you’re building a GAN or VAE, DCGAN has a very good decoder layer. here’s my implementation
When you’re building a GAN or VAE, all the output looking exactly the same for the first few hundred generations is pretty normal, don’t worry, give it some time.
Visualise everything
tf.nn.sparse_softmax_cross_entropy_with_logits is the right loss metric for when your classification problem has exclusive classes (e.g. MNIST)

please hit me up on twitter @penelopezone if any of this strikes you as wrong!

penelope.zone

A silly thing you can do with the Ruby parser

You… what?

Why is this even possible?

What’s next for Rubyfmt?

Implement a few remaining syntax nodes

“fixup” renderer

Proper integration between the Ruby program and the Rust library

CI / builds / QA

First things first

Retiring from RSpec Core

Minimum viable example of calling Rust from Ruby without a gem.

Starting with Rust

Making return_3 available to Ruby

Making this file a Ruby extension.

Building everything

Running it

Getting the most mileage out of Docker and Rails

It starts with the “Rails Way”

But what about if my Rails app needs a constellation of microservices to be realistic

I love using Docker for Rails in production

Conclusion

Questions for a prospective employer about on call (2/3)

Specific questions related to incidents

How many people are on the smallest team that is on call, how frequently do they paged for out of hours incidents? Are you doing anything to help alleviate their pain?

Does your company have distributed employees that can take on call for times that make sense for them?

Do teams routinely create and put in to place newly automated alerts?

Do you conduct any kind of review on your automated alerts to ensure they are not becoming noisy?

Do you have a support/operations/customer success team that can call a page in lieu of an automated alert?

Do you have any kind of 24 by 7 team (e.g. a NOC, SOC, or “cloud operations” team) that does first level triage on behalf of engineers?

Have you ever had an incident where you felt like a severity decision was made for an incident that was too high, causing people to get paged when they shouldn’t have? How did you remediate that?

Conclusion

Questions for a prospective employer about on call (1/3)

General maturity questions

Does $product have to be online 24/7? Do you expect engineers to be on-call out of hours

Is anyone at this company performance managed on the reliability of the software that is being produced?

Is there any kind of cross organisational service level management that all teams aim for (e.g. support engineering on successful ticket submissions per hour, checkouts team on checkouts per hour etc). If so, when teams miss their prescribed service level objectives, what do you do?

How do teams at your company manage their on-call rotation?

Do you have a “Severity Scale”? If so, could you walk me through what the ratings are, at what severity people can be called out of hours, who makes severity determinations, etc?

Is there anything similar to a holistic incident management program at this company? Is someone in charge of that?

Generally speaking, is the number of incidents that engineers are dealing with out of hours going up or down?

Conclusion

How I organise my VIM hotkeys

The leader key

Prefix trees

Conclusion

Rack middlewares that have saved me literally hours of my life

The potato middleware

Stackprof middleware

Unicorn worker killer

Tensorflow journal, part 1

Making `return_3` available to Ruby

The `potato` middleware