npf.io

Optimizing for Reviewers: The Three Step AI Dev Loop

Sun, 19 Apr 2026 12:01:33 -0400

With AI writing a lot more of our code, we need to optimize for the human code reviewer.

The key to this is good test coverage and minimizing diffs in changes that humans need to verify.

We should strive to separate refactors that don’t change logic from changes that intentionally change logic (bug fixes, feature additions, etc). Separating the two makes it significantly easier to understand if a change is correct. This was always best practice, but it was often burdensome because of the additional human effort required. Now with AI to do the boring parts, it’s more practical to put into practice.

With broad enough test coverage, you can refactor the code all you want and still ensure it behaves the same. Then the code reviewer only needs to review the general shape of the new code to know if it’ll be fit for upcoming features.

When adding new features, minimize the diff in production code as much as possible, even if the code is suboptimal. This makes it easier for reviewers to understand if the change is correct or not. You can always have the AI refactor it later.

The best way to optimize for reviews is to break changes into three steps, each in separate PRs / commits that can be reviewed and verified independently.

The Three Step AI Dev Loop

1. Ensure tests validate current behavior, add tests if needed.

Change the production code as little as possible here.
Write your tests to be as broad as possible, to make it easier to refactor later.

2. Refactor the code so that it’s easier to add the feature you want.

Refactor as much as you can without changing tests at all, or with minimal, easy to verify changes.
If you need to change the tests significantly to support a refactor, go back to #1.

3. Write the feature/bugfix.

Have the AI write the failing test for the new code first.
Have the AI then implement the feature/bugfix in a way that minimizes the diff from the old production code.

For #1, You want as close to zero production code changes as possible so that you know you’re just validating what already exists. You want tests that are likely to survive a refactor, or that will only need to change in minimal, easily verifiable ways.

For #2 we refactor to make the code cleaner, more maintainable, more performant, and/or easier for the AI and humans to understand. It’s important not to change the tests so that humans know the behavior of the production code hasn’t changed. If small changes are needed and can be done in a way that is easy for AI and humans to verify, that’s fine (replacing a concrete type with an interface, for example).

For #3, this is really just TDD, but the point is to minimize the diff, even to the point of suboptimal code with respect to performance, maintainability, etc. Since we’re changing behavior, we have to change both the tests and the production code at the same time, and this is where bugs can slip in. So we optimize for changes that are most obviously correct beyond all other metrics. You want simple, straight-forward changes that are easy for a human to verify.

After #3, you begin the loop again, and you can use #1 and #2 to refine the code from #3 to be actually-good code. Since the human knows the simple change in #3 was correct, and the tests fully constrain behavior, we can then refactor the obviously-correct code into more performant, more maintainable, more resilient code without worrying if the actual behavior is correct.

Conclusion

The key is,

When you change prod, don’t change the tests.
When you change the tests, don’t change prod.
Any time you change both, minimize the diff and refactor later.

Why Infer Types?

Thu, 26 May 2022 09:03:00 -0400

Someone at work asked the following question:

Why write code like this:

foo := getFoo()
bar, err := getBar()

instead of this:

var foo Type = getFoo()
var err error
var bar Type
bar, err = getBar(foo)

Isn’t the latter more explicit? Isn’t explicit better? It’ll be easier to review because you’ll be able to see all the types.

Well, yes and no.

For one thing, between the name of the function you’re calling and the name of the variable you’re assigning to, the type is obvious most of the time, at least at the high level.

userID, err := store.FindUser(username)

Maybe you don’t know if userID is a UUID or some custom ID type, or even just a numeric id… but you know what it represents, and the compiler will ensure you don’t send it to some function that doesn’t take that type or call a method on it that doesn’t exist.

In an editor, you’ll be able to hover over it to see the type. In a code review, you may be able to as well, and even if not, you can rely on the compiler to make sure it’s not being used inappropriately.

One big reason to prefer the inferred type version is that it makes refactoring a lot easier.

If you write this code everywhere:

var foo FooType = getFoo()
doSomethingWithFoo(foo)

Now if you want to change getFoo to return a Foo2, and doSomethingWithFoo to take a Foo2, you have to go change every place where these two functions are called and update the explicitly declared variable type.

But if you used inference:

foo := getFoo()
doSomethingWithFoo(foo)

Now when you change both functions to use Foo2, no other code has to change. And because it’s statically typed, we can know this is safe, because the compiler will make sure we can’t use Foo2 inappropriately.

Does this code really care what type getFoo returns, or what type doSomethingWithFoo takes? No, it just wants to pipe the output of one into the other. If this shouldn’t work, the type system will stop it.

So, yes, please use the short variable declaration form. Heck, if you look at it sideways, it even looks kinda like a gopher :=

Safer Enums

Fri, 13 May 2022 09:03:00 -0400

How to “do” enums is a common problem in Go, given that it doesn’t have “real” enums like other languages. There’s basically two common ways to do it, the first is just typed strings:

type FlagID string
const (
FooBar FlagID = “FooBar”
FizzBuzz FlagID = “FizzBuzz”
)
func IsEnabled(id FlagID) bool {

The problem with this is that string literals (really, string constants) in Go will get converted to the correct type, so you’d still be able to call IsEnabled(“foo-bar”) without the compiler complaining.

A common replacement is to use numeric constants:

type FlagID int
const (
FooBar FlagID = iota
FizzBuzz
)
func IsEnabled(id FlagID) bool {

This is nice, because it would be pretty odd to see code like IsEnabled(4). But the problem then becomes that you can’t easily print out the name of the enum in logs or errors.

To fix this, someone (Rob Pike?) wrote stringer, which generates code to print out the name of the flags… but then you have to remember to run stringer, and it’s a bunch of (really) ugly code.

The solution to this was something I first heard suggested by Dave Cheney (because of course it was), and is so simple and effective that I can’t believe I had never thought of it before. Make FlagName into a very simple struct:

type FlagID struct {
name string
}
func (f FlagID) String() { return f.name }
var (
FooBar = FlagID{ “FooBar” }
FizzBuzz = FlagID{ “FizzBuzz” }
)
func IsEnabled(id FlagID) bool {

Now, you can’t call IsEnabled(“nope”), because the constant string can’t be converted into a struct, so the compiler would complain.

There’s no size difference between a string and a struct{ string } and it’s just as easy to read as a straight string. Because of the String() method, you can pass these values to %s etc in format strings and they’ll print out the name with no extra code or work.

The one tiny drawback is that the globals have to be variables instead of constants, but that’s one of those problems that really only exists in the theoretical realm. I’ve never seen a bug from someone overwriting a global variable like this, that is intended to be immutable.

I’ll definitely be using this pattern in my projects going forward. I hope this helps some folks who are looking to avoid typos and accidental bugs from stringly typed code in Go.

Error Flags

Wed, 07 Apr 2021 23:03:00 -0400

Error wrapping in go 1.13 solved a major problem gophers have struggled with since v1: how to add context to errors without obscuring the original error, so that code above could programmatically inspect the original error. However, this did not – by itself – solve the other common problems with errors: implementation leakage and (more generally) error handling.

Fragile Error Handling

In 2016, Dave Cheney wrote a blog post that includes a section titled “Assert errors for behaviour, not type”. The gist of the section is that you don’t want code to depend on implementation-specific error types that are returned from a package’s API, because then, if the implementation ever changes, the error handling code will break. Even four and a half years later, and with 1.13’s new wrapping, this can still happen very easily.

For example, say you’re in an HTTP handler, far down the stack in your data layer. You’re trying to open a file and you get an os.ErrNotExist from os.Open. As of 1.13, you can add more context to that error without obscuring the fact that it’s an os.ErrNotExist. Cool, now the consumers of that code get a nicer error message, and if they want, they can check os.IsNotExist(err) and maybe return a 404 to the caller.

Right there, your web handler is now tied to the implementation details of how your backend, maybe 4 levels deep in the stack, stores data. If you decide to change your backend to store data in S3, and it starts returning s3.ObjectNotFound errors, your web handler won’t recognize that error, and won’t know to return 404. This is barely better than matching on the error string.

Dave’s Solution - Interfaces

Dave proposes creating errors that fulfill interfaces the code can check for, like this:

type notFound interface {
NotFound() bool
}
// IsNotFound returns true if err indicates the resource doesn’t exist.
func IsNotFound(err error) bool {
m, ok := err.(notFound)
return ok && m.NotFound()
}

Cool, so now you can ensure a consistent API without relying on the implementation-specific type of the error. Callers just need to check for IsNotFound, which could be fulfilled by any type. The problem is, it’s missing a piece. How do you take that os.NotExistErr and give it a IsNotFound() method? Well, it’s not super hard, but kind of annoying. You need to write this code:

// IsNotFound returns true if err indicates the resource doesn’t exist.
func IsNotFound(err error) bool {
n, ok := err.(notFound)
return ok && n.NotFound()
}
// MakeNotFound wraps err in an error that reports true from IsNotFound.
func MakeNotFound(err error) error {
if err == nil {
return nil
}
return notFoundErr{error: err}
}
type notFound interface {
NotFound() bool
}
type notFoundErr struct {
error
}
func (notFoundErr) NotFound() bool {
return true
}
func (n notFoundErr) Unwrap() error {
return n.error
}

So now we’re at 28 lines of code and two exported functions. Now what if you want the same for NotAuthorized or ? 28 more lines and two more exported functions. Each just to add one boolean of information onto an error. And that’s the thing… this is purely used for flow control - all it needs to be is booleans.

A Better Way - Flags

At Mattel, we had been following Dave’s method for quite some time, and our errors.go file was growing large and unwieldy. I wanted to make a generic version that didn’t require so much boilerplate, but was still strongly typed, to avoid typos and differences of convention.

After thinking it over for a while, I realized it only took a slight modification of the above code to allow for the functions to take the flag they were looking for, instead of baking it into the name of the function and method. It’s of similar size and complexity to IsNotFound above, and can support expansion of the flags to check, with almost no additional work. Here’s the code:

// ErrorFlag defines a list of flags you can set on errors.
type ErrorFlag int
const (
NotFound = iota + 1
NotAuthorized
// etc
)
// Flag wraps err with an error that will return true from HasFlag(err, flag).
func Flag(err error, flag ErrorFlag) error {
if err == nil {
return nil
}
return flagged{error: err, flag: flag}
}
// HasFlag reports if err has been flagged with the given flag.
func HasFlag(err error, flag ErrorFlag) bool {
for {
if f, ok := err.(flagged); ok && f.flag == flag {
return true
}
if err = errors.Unwrap(err); err == nil {
return false
}
}
}
type flagged struct {
error
flag ErrorFlag
}
func (f flagged) Unwrap() error {
return f.error
}

To add a new flag, you add a single line to the list of ErrorFlags and you move on. There’s only two exported functions, so the API surface is super easy to understand. It plays well with go 1.13 error wrapping, so you can still get at the underlying error if you really need to (but you probably won’t and shouldn’t!).

Back to our example: the storage code can now keep its implementation private and flag errors from the backend with return errors.Flag(err, errors.NotFound). Calling code can check for that with this:

if errors.HasFlag(err, errors.NotFound) {
// handle not found
}

If the storage code changes what it’s doing and returns a different underlying error, it can still flag it with that with the NotFound flag, and the consuming code can go on its way without knowing or caring about the difference.

Supporting Errors.Is and Errors.As

This is an update in 2022, and I realized that there’s an easier way to do this that properly supports errors.Is and errors.As. In go 1.20, there will be an errors.Join method that can let you combine two errors into one where either one will be found by errors.Is and errors.As. Until then, you can use github.com/natefinch/wrap. Then you can just define the flags as straight errors.

package flags
var (
NotFound = errors.New("not found")
AlreadyExists = errors.New("already exists")
// etc
)

Then, as long as the package wraps its errors with those flags (using a package like Wrap or the upcoming errors.Join), you can check for the flag with the normal functions:

 user, err := store.FindUser(id)
if errors.Is(err, flags.NotFound) {
// return 404
}

The nice thing is that you can still get the behavior of the original error because this is non-destructive wrapping. So if you need some low level detail of the underlying error, you can get it.

Indirect Coupling

Isn’t this just sentinel errors again? Well, yes, but that’s ok. In 2016, we didn’t have error wrapping, so anyone who wanted to add info to the error would obscure the original error, and then your check for err == os.ErrNotExist would fail. I believe that was the major impetus for Dave’s post. Error wrapping in Go 1.13 fixes that problem. The main problem left is tying error checks to a specific implementation, which this solves.

This solution does require both the producer and the consumer of the error to import the error flags package and use these same flags, however in most projects this is probably more of a benefit than a problem. The edges of the application code can easily check for low level errors and flag them appropriately, and then the rest of the stack can just check for flags. Mattel does this when returning errors from calling the database, for example. Keeping the flags in one spot ensures the producers and consumers agree on what flag names exist.

In theory, Dave’s proposal doesn’t require this coordination of importing the same package. However, in practice, you’d want to agree on the definition of IsNotFound, and the only way to do that with compile-time safety is to define it in a common package. This way you know no one’s going to go off and make their own IsMissing() interface that gets overlooked by your check for IsNotFound().

Choosing Flags

In my experience, there are a limited number of possible bits of data your code could care about coming back about an error. Remember, flags are only useful if you want to change the application’s behavior when you detect them. In practice, it’s not a big deal to just make a list of a handful of flags you need, and add more if you find something is missing. Chances are, you’ll think of more flags than you actually end up using in real code.

Conclusion

This solution has worked wonders for us, and really cleaned up our code of messy, leaky error handling code. Now our code that calls the database can parse those inscrutable postgres error codes right next to where they’re generated, flag the returned errors, and the http handlers way up the stack can happily just check for the NotFound flag, and return a 404 appropriately, without having to know anything about the database.

Do you do something similar? Do you have a totally different solution? I’d love to hear about it in the comments.

How to Pay Remote Workers

Fri, 26 Jul 2019 12:47:11 -0400

Pay remote workers the same as you’d pay local workers. Or vice versa if your local workers are cheap.

That’s it. That’s the blog post.

It’s 2019, folks. Average home internet speeds are more than enough for video conferencing and every single laptop has a built-in video camera. Conference room video hardware has come way down in price and gone way up in quality. Everyone collaborates via Slack and email and Jira and wikis and shared documents in the cloud anyway. Our code is hosted in the cloud, ci/cd in the cloud, deployed to the cloud. Why on earth would it matter where your desk is?

The truth is, it doesn’t matter. With extremely low effort, any company can hire remote folks and have them be productive, collaborative members of a team. I should know, I’ve done it for the last 8 years.

One thing that always comes up with remote employees is “how much should I pay them?” I’m not exactly sure why this is even a question…. actually, yes, I am sure. Because companies are cheap and want to pay employees as little as possible. They are, after all, a business. So I guess the question is more accurately asked “How can I justify paying my employees less while still getting great talent?”

The answer is always the same - cost of living adjustments. The theory is that you pay everyone equitably, so they all sustain the same standard of living. i.e. you pay the person in San Francisco enough for rent and food and spending money for a new XBox every month. You do the same for the person in rural Ohio - rent, food, Xbox every month.

Just like a meritocracy, on its face, this sounds perfectly fair. But peek under the surface, and it’s easily dismissed as false equivalence. Why is a SF apartment four times the cost of the same apartment in rural Ohio? Because of supply and demand. Because people believe the apartment in SF is worth more, so they’re willing to pay more. Why do they believe that? Because the apartment in SF is near awesome restaurants, easy public transportation, lots of great similar-minded folks, etc. etc.

These are attributes of the apartment that don’t fit on a spreadsheet of square footage, number of bedrooms, and lot size… but they have a huge effect on the price of the home. Clearly, that is what you’re paying for when you buy a $500k studio in SF.

So, if the house in SF is clearly more valuable than an equivalent-sized one in rural Ohio… why should the company subsidize paying for those invisible benefits that come with a house in SF? Would you pay someone more who lived in a bigger house in the same city? Why not? Why is it ok for companies to subsidize the location-based value of a home, but not the value derived from square-footage or lot size?

To put a finer point on it… would you pay someone less who lives on the wrong side of the tracks in the same city? That’s still location-based, isn’t it?

The thing is, the value of money isn’t actually different in SF and rural Ohio. Buying an XBox from Amazon costs the same in both places. $3000 a month in rent for a studio or $3000 a month in mortgage for a 4 bedroom house…. still costs you $3000. If you live in SF, you’re saying that studio’s location is worth $3000 a month to you. If you live in rural Ohio, you’re saying the extra bedrooms and big backyard are worth $3000 a month to you.

…so why would you pay the person in Ohio less?

Someone on Twitter mentioned they understood paying people more who live in high cost of living areas, but thought it would be weird to pay people less who live in low cost of living areas…. but it’s really the exact same thing. You pay the person in SF more, and you’re just paying everyone else less. You can’t have it one way and not the other.

Does this mean you have to compete with Google’s salaries if your company is in rural Ohio? Yes and no. It’s true that Google and the other big-five tech companies pay people a lot more. But that’s true even in Silicon Valley. I’ve interviewed at lots of SF companies that weren’t able to compete with those kind of salaries either, but they still get to hire a lot of great talent. The big five may have a lot of devs, but they can’t hire all the devs. And since hiring is really hard, they don’t even get all the best devs. The big five mostly pay a lot of money to keep the other four from poaching… i.e. they’re really only competing with each other.

So, you might not have to compete with the Googles of the world, but you probably do have to compete with the Salesforces, Stripes, and (previous to acquisition) Githubs. While those companies generally pay more than some random tech company, it’s not double or triple. It’s like 30% more. And honestly, developers are worth that much. Basically every company in existence needs developers, or needs to pay a service vendor for specialized software.

Hiring managers - the onus is on you to stop this predatory and unfair hiring practice. Don’t accept it as “just the way things are”. Speak up against it. Fight to get your remote developers the same salary and benefits your on-site folks get. Their work is just as valuable to the company as the local folks, paying them less is unfair, insulting, and wrong.

Do or Do Not

Fri, 07 Jun 2019 23:40:50 -0400

The proposal

There’s a new Go proposal in town - try(). The gist is that it adds a builtin function try() that can wrap a function that returns (a, b, c, …, error), and if the error is non-nil, it will return from the enclosing function, and if the error is nil, it’ll return the rest of the return values.

This is how it looks in code:

func doIt() (string, int, error){
return "Daisy", 45, io.EOF
}
func tryIt() error {
name, age := try(doIt())
// use name, age
return nil
}

In the above, if doIt returns a non-nil error, tryIt will exit at the point where try is called, and will return that error.

Complications

So here’s my problem with this… it complicates the code. It adds points where your code can exit from inside the right hand side of a statement somewhere. It can make it very easy to miss the fact that there’s an early exit statement in the code.

The above is simplistic, it could instead look like this:

func tryIt() error {
fmt.Printf("Hi %s, happy %vth birthday!\n", try(doIt())
// do other stuff
return nil
}

At first blush, it would be very easy to read that code and think this function always returns nil, and that would be wrong and it could be catastrophically wrong.

The Old Way

In my opinion, the old way (below) of the original code is a lot more readable. The exit point is clearly called out by the return keyword as well as the indent. The intermediate variables make the print statement a lot more clear.

func tryIt() error {
name, age, err := doIt()
if err != nil {
return err
}
fmt.Printf("Hi %s, happy %vth birthday!\n", name, age)
return nil
}

Oh, and did you catch the mismatched parens on the Printf statement in the try() version of tryIt() above? Me neither the first time.

Early Returns

Writing Go code involves a LOT of returning early, more than any other popular language except maybe C or Rust. That’s the real meat of all those if err != nil statements… it’s not the if, it’s the return.

The reason early returns are so good is that once you pass that return block, you can ignore that case forever. The case where the file doesn’t exist? Past the line of os.Open’s error return, you can ignore it. It no longer exists as something you have to keep in your head.

However, with try, you now have to worry about both cases in the same line and keep that in your head. Order of operations can come into play, how much work are you actually doing before this try may kick you out of the function?

One idea per line

One of the things I have learned as a go programmer is to eschew line density. I don’t want a whole ton of logic in one line of code. That makes it harder to understand and harder to debug. This is why I don’t care about missing ternary operator or map and filter generics. All those do is let you jam more logic into a single line, and I don’t want that. That makes code hard to understand, and easier to misunderstand.

Try does exactly that, though. It encourages you to put a call getting data into a function that then uses that data. For simple cases, this is really nice, like field assignment:

p := Person{
Name: try(getUserName()),
Age: try(getUserAge()),
}

But note how even here, we’re trying to split up the code into multiple lines, one assignment per line.

Would you ever write this code this way?

p := Person{Name: try(getUserName()), Age: try(getUserAge())}

You certainly can, but holy crap, that’s a dense line, and it takes me an order of magnitude longer to understand that line than it does the 4 lines above, even though they’re just differently formatted version of the exact same code. But this is exactly what will be written if try becomes part of the language. Maybe not struct initialization, but what about struct initialization functions?

p := NewPerson(try(getUserName()), try(getUserAge()))

Nearly the same code. Still hard to read.

Nesting Functions

Nesting functions is bad for readability. I very rarely nest functions in my go code, and looking at other people’s go code, most other people also avoid it. Not only does try() force you to nest functions as its basic use case, but it then encourages you to use that nested function nested in some other function. So we’re going from NewPerson(name, age) to NewPerson(try(getUserName()), try(getUserAge())). And that’s a real tragedy of readability.

Hiring Remote

Tue, 14 May 2019 23:05:55 -0400

I have been working remotely for about 8 years now. I’ve worked at companies that did it poorly, and companies that did it well. Let me define remote for a minute. I mean fully remote. Like, I can count on one hand the number of times per year I see my coworkers in person and have fingers left over.

I was the first remote employee in my division at Mattel, and I helped guide the culture toward supporting remote employees. Mattel, for its part, has been very supportive, and honestly did many things right without even really thinking about them as supporting remote employees.

I have a lot of thoughts about remote work, and I’ll probably turn this into a series of posts on the subject. For now, I’m going to start at the beginning - hiring.

I’ve interviewed at over a dozen companies that support remote employees. How the interviewing process goes tells me a lot about whether or not a company really supports remote employees.

When I interviewed at Canonical, the whole interview was remote. I never saw anyone in person until after I got my offer, and then it was just a run to the nearest office to sign paperwork. I literally never met any of my coworkers in person until our first offsite about three months in. And that’s totally ok.

When I interviewed at Mattel it was much the same, except I was brought on as a contractor first, which allowed me to prove myself, and then they were happy to just continue letting me do my thing as a full time employee 3000 miles away from the rest of the team. I pushed my boss to hire more remote devs, and we now have a team that is almost fully remote.

Many places I’ve interviewed want you to come onsite after some number of interviews to “meet the team”. While this is ok, it tends to make me think those places aren’t as fully bought into remote culture. There was no office to go into at Canonical. Meeting the team was getting on a google hangout (and that’s fine).

If you buy into remote culture, meeting someone in a video chat should be good enough. After all, that’s how you’re going to interact with them 99% of the time. Just as the whiteboard is a relic of another time, I believe the onsite interview is a relic if the job is remote. (If the job is not remote, then I think it’s pretty important to get a handle on how the person interacts in person with other people… but that’s not what we’re talking about.)

I don’t code on a whiteboard at work, and I don’t code in a meeting room with another developer at work either. And honestly, they almost never ask me to code in that meeting room. It’s all talk and drawing architecture on a whiteboard. Which, like, seriously, save yourself the plane ticket and hotel charge and just let me do that over hangouts.

The problem with having me come on site is that it’s a 2 day thing. I have to leave work early to catch a plane the night before, spend the night in a hotel, get up mildly jetlagged, interview all day, then take a redeye home unless I want to spend a second night in the hotel and get home at like 3 in the afternoon the next day. If I did that for every job I interviewed for last time I was looking, I would have had to take a full month off… it’s just not scalable…. and it’s rough on my family.

Speaking of family, let’s talk about onboarding. Some companies will onboard you remotely. This is great. Paperwork can be tricky, but it’s doable with a notary public (that’s what I did for Mattel). Otherwise, going onsite for onboarding is fine… you sign paperwork, get a company laptop, some swag, etc. All that could be mailed out, but I get that paperwork can be tricky remote.

But that’s like… maybe 3 days if everything goes really slowly. Many places want a week onsite for onboarding. Buh…. to do what? If there are significant things I can only do while onsite, we’re probably going to have problems when I go back to work at my house for months at a time. Also, aren’t many of my coworkers remote, so won’t most of them not even be there? One place even mentioned onboarding was two weeks onsite.

Two weeks is an eternity. I have young kids, and there is zero chance I’m going anywhere onsite for two weeks. I work remote so I can be with my kids. I’m sure a lot of more senior devs out there are in the same position. Making your onboarding process long makes you much less desirable to anyone with a family. Don’t draw it out any longer than necessary. As a mediocre white man who is used to getting his way, I might feel comfortable asking for a reduced onsite, but I bet many other developers who are not so privileged might not.

So, to sum up - if you really want to show you support remote developers, instead of just saying you do, start with the interview. Make as much of your interview process remote as possible, and then make your onboarding as painless as possible. It’ll save you time, it’ll save your candidates time, it’ll save the company money, and it’ll make everyone happier.

Retooling Retool

Sat, 11 May 2019 21:42:34 -0400

I was so happy when I discovered retool. It’s a go tool that builds and caches go binaries into a local directory so that your dev tools stay in sync across your team. It fixes all those problems where slight difference in binary versions produce different output and cause code churn. We use it at Mattel for our projects, because we tend to have a large number of external tools that we use for managing code generation, database migrations, release management, etc.

However, retool doesn’t work very well with modules, and trying to run it with modules turned off sometimes misbehaves, and some tools just fail to compile that way.

So what to do? Well, it turns out that in the module world, retool can be replaced by a very small mage script:

func Tools() error {
update, err := envBool("UPDATE")
if err != nil {
return err
}
if err := os.MkdirAll("_tools", 0700); err != nil {
return err
}
wd, err := os.Getwd()
if err != nil {
return err
}
env := map[string]string{"GOBIN": filepath.Join(wd, "_tools")}
args := []string{"get"}
if update {
args = []string{"get", "-u"}
}
for _, t := range tools {
err := sh.RunWith(env, "go", append(args, t)...)
if err != nil {
return err
}
}
return nil
}

This code is pretty simple — it ensures the _tools directory exists (which is where retool puts its binaries as well, so I just reused that spot since our .gitignore already ignored it). Then it sets GOBIN to the _tools directory, so binaries built by the go tool will go there, and runs go get importpath@<tag|hash>. That’s it. The first time, it’ll take a while to download all the libraries it needs to build the binaries into the modules cache, but after that it’ll figure out it doesn’t need to do anything pretty quick.

Now just use the tool helper function below in your magefile to run the right versions of the binaries (and/or add _tools to your PATH if you use something like direnv).

// tool runs a command using a cached binary.
func tool(cmd string, args ...string) error {
return sh.Run(filepath.Join("_tools", cmd), args...)
}

Now all the devs on your team will be using the same versions of their (go) dev tools, and you don’t even need a fancy third party tool to do it (aside from mage). The list of tools then is just a simple slice of strings, thusly:

var tools = []string{
"github.com/jteeuwen/go-bindata/go-bindata@6025e8de665b31fa74ab1a66f2cddd8c0abf887e",
"github.com/golang/protobuf/protoc-gen-go@v1.3.1",
"gnorm.org/gnorm@v1.0.0",
"github.com/goreleaser/goreleaser@v0.106.0",
}

For most maintained libraries, you’ll get a nice semver release number in there, so it’s perfectly clear what you’re running (but for anything without tags, you can use a commit hash).

I’m really happy that this was as straightforward as I was hoping it would be, and it seems just as usable as retool for my use case.

Init Is Bad and You Should Feel Bad

Fri, 12 Apr 2019 11:04:21 -0400

func init() in Go is a weird beast. It’s the only function you can have multiples of in the same package (yup, that’s right… give it a try). It gets run when the package is imported. And you should never use it.

Why not? Well, there’s a few reasons. The main one is that init is only useful for setting global state. I think it’s pretty well accepted that global state is bad (because it’s hard to test and it makes concurrency dangerous). So, by association init is bad, because that’s all it can do.

But wait, there’s more that makes it even worse. Init is run when a package is imported, but when does a package get imported? If a imports b and b imports c and b and c both have init functions, which one runs first? What if c has two init functions in different files? You can find out, but it’s non-obvious and it can change if you import code differently. Not knowing the order in which code executes is bad. Normal go code executes top to bottom in a very clear and obvious order. There’s good reason for that.

How do you test init functions? Trick question, you can’t. It’s not possible to test the state of a package before init and then make sure the state after init is correct. As soon as your test code runs, it imports the package and runs init right away. Ok, maybe that’s not 100% true, you can probably do some hackery in init to check if you’re running under go test and then not run the init logic… but then your package isn’t set up the way it expects, and you’d have to write a test specifically named to run first, to test init… and that’s just horrible (and nobody does that, so it’s basically always untested code).

Ok, so there’s the reasons not to use it… now what do you do instead? If you want state, use a struct. Instead of global variables on the package, use fields on a struct. The package-level functions become methods, and the init function becomes a constructor.

This fixes all the aforementioned problems. You get rid of global variables, so if you have two different parts of your code using the same package, they don’t stomp on each other’s settings etc. You can run tests without worrying that a previous test modifies global state for a later test. It’s clear and obvious how to test before and after a constructor gets called. And finally, there’s a clear and normal order to the initialization of things. You don’t have to wonder what gets called when, because it’s just normal go functions.

As a corollary… this means you shouldn’t use underscore imports either (since they’re generally only useful for triggering init functions). These imports (import _ "github.com/foo/db") are used for their side effects, like registering sql/db drivers. The problem is that these are, by definition, setting global variables, and those are bad, as we’ve said. So don’t use those either.

Once you start writing code with structs instead of globals and init, you’ll find your code is much easier to test, easier to use concurrently, and more portable between applications. So, don’t use init.

…Axel Wagner mentioned on Twitter that this looked too dogmatic, and he’s right. This is programming, there are infinite possible programs, and thus there will always be exceptions to every rule. I think it’s really rare that init is the right choice, and you should only come to that decision after trying other options and ensuring you take into consideration things like startup order, concurrent access, and testing.

Starlight

Fri, 07 Dec 2018 09:59:12 -0500

I’d like to announce starlight - https://github.com/starlight-go/starlight.

Starlight wraps google’s Go implementation of the starlark python dialect (most notably found in the Bazel build tool). Starlight makes it super easy for users to extend your application by writing simple python-like scripts that interact seamlessly with your current Go code… with no boilerplate on your part.

What is Starlark?

Starlark is a subset of python that removes some of the more advanced features, but keeps the easy to read-and-write feel. For the purposes of this article, to avoid confusion between starlight (my package) and starlark (the language), I’ll be referring to the code as python (since starlark code is a subset of python code), but there are some small differences (described in the previous link).

Parser by google

The parser and runner are maintained by google’s bazel team, which write starlark-go. Starlight is a wrapper on top of that, which makes it so much easier to use starlark-go. The problem with the starlark-go API is that it is more built to be a used as configuration, so it assumes you want to get information out of starlark and into Go. It’s actually pretty difficult to get Go information into a starlark script…. unless you use starlight.

Easy two-way interaction

Starlight has adapters that use reflection to automatically make any Go value usable in a starlark script. Passing an *http.Request into a starlark script? Sure, you can do name = r.URL.Query()["name"][0] in the python without any work on your part.

Starlight is built to just work the way you hope it’ll work. You can access any Go methods or fields, basic types get converted back and forth seamlessly… and even though it uses reflection, it’s not as slow as you’d think. A basic benchmark wrapping a couple values and running a starlark script to work with them runs in a tiny fraction of a millisecond.

The great thing is that the changes made by the python code are reflected in your go objects, just as if it had been written in Go. So, set a field on a pointer to a struct? Your go code will see the change, no additional work needed.

100% Safe

The great thing about starlark and starlight is that the scripts are 100% safe to run. By default they have no access to other parts of your project or system - they can’t write to disk or connect to the internet. The only access they have to the outside is what you give them. Because of this, it’s safe to run untrusted scripts (as long as you’re not giving them dangerous functions to run, like os.RemoveAll). But at the same time, if you’re only running trusted scripts, you can give them whatever you want (http.Get? Sure, why not?)

Example

Below is an example of a webserver that changes its output depending on the python script it runs. This is the full code, it’s not truncated for readability… this is all it takes.

First the go web server code. Super standard stuff, except a few lines to run starlight…

package main
import (
"fmt"
"log"
"net/http"
"github.com/starlight-go/starlight"
)
func main() {
http.HandleFunc("/", handle)
port := ":8080"
fmt.Printf("running web server on http://localhost%v?name=starlight&repeat=3\n", port)
if err := http.ListenAndServe(port, nil); err != nil {
log.Fatal(err)
}
}
func handle(w http.ResponseWriter, r *http.Request) {
fmt.Println("handling request", r.URL)
// here we define the global variables and functions we're making available
// to the script. These will define how the script can interact with our Go
// code and the outside world.
globals := map[string]interface{}{
"r": r,
"w": w,
"Fprintf": fmt.Fprintf,
}
_, err := starlight.Eval("handle.star", globals, nil)
if err != nil {
fmt.Println(err)
}
}

And the python handle.star:

# Globals are:
# w: the http.ResponseWriter for the request
# r: the *http.Request
# Fprintf: fmt.Fprintf
# for loops and if statements need to be in functions in starlark
def main():
# Query returns a map[string][]string
# this gets a value from a map, with a default if it doesn't exist
# and then takes the first value in the list.
repeat = r.URL.Query().get("repeat", ["1"])[0]
name = r.URL.Query().get("name", ["starlight"])[0]
for x in range(int(repeat)):
Fprintf(w, "hello %s\n", name)
# we can use pythonic truthy statements on the slices returned from the map to
# check if they're empty.
if not r.URL.Query().get("repeat") and not r.URL.Query().get("repeat"):
w.Write("\nadd ?repeat=<int>&name=<string> to the URL to customize this output\n")
w.Write("\ntry modifying the contents of output.star and see what happens.\n")
main()

You can run this example by running go get github.com/starlight-go/starlight and using go run main.go in the example folder. You can then update the python and watch the changes the next time you hit the server. This just uses starlight.Eval, which rereads and reparses the script every time.

Caching

In a production environment, you probably want to only read a script once and parse it once. You can do that with starlight’s Cache. This cache takes a list of directories to look in for scripts, which it will read and parse on-demand, and then store the parsed object in memory for later use. It also uses a cache for any load() calls the scripts use to load scripts they depend on.

Work Ongoing

Starlight is still a work in progress, so don’t expect the API to be perfectly stable quite yet. But it’s getting pretty close, and there shouldn’t be any earth shattering changes, but definitely pin your imports. Right now it’s more about finding corner cases where the starlight wrappers don’t work quite like you’d expect, and supporting the last few things that aren’t implemented yet (like channels).

Mage - make/rake for Go

Wed, 19 Sep 2018 22:50:51 -0400

A Brief History

A question came up at the Framingham Go meetup a while back about why something like Gradle hasn’t taken hold in the Go community. I can’t say that I know for sure what the answer is - I don’t speak for the community - but, I have some guesses. I think part of it is that many projects don’t need a full-fledged build tool - for your typical Go networked server or CLI tool, a single binary built with go build is probably fine.

For more complex builds, which may require more steps than just compile and link, like for bundling static assets in a web server or generating code from protobufs, for example, many people in the Go community reach for Make. Personally, I find that unfortunate. Makefiles are clearly pretty cool for a number of reasons (built-in CLI, dependencies, file targets). However, Make is not Windows friendly, and it has its own language and conventions that you need to learn on top of the oddity that is Bash scripting. Finally, it doesn’t let you leverage the Go community’s two greatest resources - go programmers and go code.

Maybe `go run`? Maybe not

The above is the start of a blog post I’ve had half written for two years. I started to go on to recommend using go run make.go with a go file that does the build for you. But in practice, this is problematic. If you want your script to be useful for doing more than one thing, you need to implement a CLI and subcommands. This ends up being a significant amount of work that then obscures what the actual code is doing… and no one wants to maintain yet another CLI just for development tasks. In addition, there’s a lot of chaff you have to handle, like printing out errors, setting up logging etc.

The Last Straw

Last summer there were a couple questions on r/golang about best practices for using Makefiles with Go… and I finally decided I’d had enough.

I looked around at what existed for alternatives - rake was the obvious pattern to follow, being very popular in the Ruby community. pyinvoke was the closest equivalent I saw in python. Was there something similar in Go? Well, sort of, but not exactly. go-task is written in Go, but tasks are actually defined in YAML. Not my cup of tea. Mark Bates wrote grift which has tasks written in Go, but I didn’t really like the ergonomics… I wanted just a little more magic.

I decided that I could write a tool that behaved pretty similarly to Make, but allowed you to write Go instead of Bash, and didn’t need any special syntax if I did a little code parsing and generation on the fly. Thus, Mage was born.

What is Mage?

Docs: magefile.org
Github: github.com/magefile/mage

Mage is conceptually just like Make, except you write Go instead of Bash. Of course, there’s a little more to it than that. In Mage, like in Make, you write targets that can be accessed via a simple CLI. In Mage, exported functions become targets. Any of these exported functions are then runnable by running mage <func_name> in the directory where the magefile lives, just like you’d run make <target_name> for a make target.

What is a Magefile?

A magefile is simply a .go file with the mage build tag in it. All you need for a magefile is this:

//+build mage

package main

Mage looks for all go files in the current directory with the mage build tag, and compiles them all together with a generated CLI.

There are a few nice properties that result from using a build tag to mark magefiles - one is that you can use as many files as you like named whatever you like. Just like in normal go code, the files all work together to create a package.

Another really nice feature is that your magefiles can live side by side with your regular go code. Mage only builds the files with the mage tag, and your normal go build only builds the files without the mage tag.

Targets

A function in a magefile is a target if it is exported and has a signature of func(), func()error, func(context.Context), or func(context.Context)error. If the target has an error return and you return an error, Mage will automatically print out the error to its own stderr, and exit with a non-zero error code.

Doc comments on each target become CLI docs for the magefile, doc comments on the package become top-level help docs.

//+build mage

// Mostly this is used for building the website and some dev tasks.
package main

// Builds the website. If needed, it will compact the js as well.
func Build() error {
 // do your stuff here
 return nil
}

Running mage with no arguments (or mage -l if you have a default target declared) will print out help text for the magefiles in the current directory.

$ mage
Mostly this is used for building the website and some dev tasks.

Targets:
 build Builds the website.

The first sentence is used as short help text, the rest is available via mage -h <target>

$ mage -h build
mage build:

Builds the website. If needed, it will compact the js as well.

This makes it very easy to add a new target to your magefile with proper documentation so others know what it’s supposed to do.

You can declare a default target to run when you run mage without a target very easily:

var Default = Build

And just like Make, you can run multiple targets from a single command… mage build deploy clean will do the right thing.

Dependencies

One of the great things about Make is that it lets you set up a tree of dependencies/prerequisites that must execute and succeed before the current target runs. This is easily done in Mage as well. The github.com/magefile/mage/mg library has a Deps function that takes a list of dependencies, and runs them in parallel (and any dependencies they have), and ensures that each dependency is run exactly once and succeeds before continuing.

In practice, it looks like this:

func Build() error {
 mg.Deps(Generate, Protos)
 // do build stuff
}

func Generate() error {
 mg.Deps(Protos)
 // generate stuff
}

func Protos() error {
 // build protos
}

In this example, build depends on generate and protos, and generate depends on protos as well. Running build will ensure that protos runs exactly once, before generate, and generate will run before build continues. The functions sent to Deps don’t have to be exported targets, but do have to match the same signature as targets have (i.e. optional context arg, and optional error return).

Shell Helpers

Running commands via os/exec.Command is cumbersome if you want to capture outputs and return nice errors. github.com/magefile/mage/sh has helper methods that do all that for you. Instead of errors you get from exec.Command (e.g. “command exited with code 1”), sh uses the stderr from the command as the error text.

Combine this with the automatic error reporting of targets, and you easily get helpful error messages from your CLI with minimal work:

func Build() error {
 return sh.Run("go", "build", "-o", "foo.out")
}

Verbose Mode

Another nice thing about the sh package is that if you run mage with -v to turn on verbose mode, the sh package will print out the args of what commands it runs. In addition, mage sets up the stdlib log package to default to discard log messages, but if you run mage with -v, the default logger will output to stderr. This makes it trivial to turn on and off verbose logging in your magefiles.

How it Works

Mage parses your magefiles, generates a main function in a new file (which contains code for a generated CLI), and then shoves a compiled binary off in a corner of your hard drive. The first time it does this for a set of magefiles, it takes about 600ms. Using the go tool’s ability to check if a binary needs to be rebuilt or not, further runs of the magefile avoid the compilation overhead and only take about 300ms to execute. Any changes to the magefiles or their dependencies cause the cached binary to be rebuilt automatically, so you’re always running the newest correct code.

Mage is built 100% with the standard library, so you don’t need to install a package manager or anything other than go to build it (and there are binary releases if you just want to curl it into CI).

Conclusion

I’ve been using Mage for all my personal projects for almost a year and for several projects at Mattel for 6 months, and I’ve been extremely happy with it. It’s easy to understand, the code is plain old Go code, and it has just enough helpers for the kinds of things I generally need to get done, taking all the peripheral annoyances out of my way and letting me focus on the logic that needs to be right.

Give it a try, file some issues if you run into anything. Pull requests more than welcome.

Handle and Check - Let's Not

Thu, 06 Sep 2018 13:49:33 -0400

There’s a new error handling design proposed here. It’s…. not great.

Handle is a new keyword that basically defines a translation that can be applied to errors returned from the current function:

func printSum(a, b string) error {
handle err { return fmt.Errorf("error summing %v and %v: %v", a, b, err ) }
x := check strconv.Atoi(a)
y := check strconv.Atoi(b)
fmt.Println("result:", x + y)
return nil
}

Check applies the handler and returns if the error passed into it is not nil, otherwise it returns the non-error value.

Handle, in my opinion is kind of useless. We can already do this today with functions thusly:

func printSum(a, b string) (err error) {
check := func(err error) error {
return fmt.Errorf("error summing %v and %v: %v", a, b, err )
}
x, err := strconv.Atoi(a)
if err != nil {
return check(err)
}
y, err := strconv.Atoi(b)
if err != nil {
return check(err)
}
fmt.Println("result:", x + y)
return nil
}

That does literally the same thing as check and handle above.

The stated reason for adding check and handle is that too many people just write “return err” and don’t customize the error at all, which means somewhere at the top of your program, you get this inscrutable error from deep in the bowels of your code, and you have no idea what it actually means.

It’s trivial to write code that does most of what check and handle do… and no one’s doing it today (or at least, not often). So why add this complexity?

Check and handle actually make error handling worse. With the check and handle code, there’s no required “error handling scope” after the calls to add context to the error, log it, clean up, etc. With the current code, I always have an if statement that I can easily slot more lines into, in order to make the error more useful and do other things on the error path. With check, that space in the code doesn’t exist. There’s a barrier to making that code handle errors better - now you have to remove check and swap in an if statement. Yes, you can add a new handle section, but that applies globally to any further errors returns in the function, not just for this one specific error. Most of the time I want to add information about one specific error case.

So, for example, in the code above, I would want a different error message for A failing Atoi vs. B failing Atoi…. because in real code, which one is the problem may not be obvious if the error message just says “either A or B is a problem”.

Yes, if err != nil { constitutes a lot of Go code. That’s ok. That’s actually good. Error handling is extremely important. Check and handle don’t make error handling better. I suspect they’ll actually make it worse.

A refrain I often state about changes requested for Go is that most of them just involve avoiding an if statement or a loop. This is one of them. That’s not a good enough reason to change the language, in my opinion.

Go2 Contracts Go Too Far

Wed, 05 Sep 2018 14:00:18 -0400

So, I don’t really like the contracts defined here. They seem complicated to understand, and duplicate a lot of what interfaces already do, but in a much clunkier fashion.

I think we can do 90% of what the design given can do, with 20% of the added complexity.

Most of my objection comes from two things:

First the syntax, which adds “type parameters” as yet another overloaded meaning for stuff in parentheses (we already have: argument lists, return values, function calls, type conversion, type assertion, and grouping for order of operations).

Second, the implicit nature of how contracts are defined by a random block of code that is sorta like go code, but not actually go code.

Syntax

This is a generic function as declared in the contracts code:

func Print(type T)(s []T) {
for _, v := range s {
fmt.Println(v)
}
}

The (type T) here defines a type parameter. In this case it doesn’t tell us anything about the type, so it’s effectively like interface{}, except that it magically works with slices the way we all thought interfaces should work with slices back in the day – i.e. you can pass any slice into this, not just []interface{}.

Are we now going to have func(type T)(input T)(output T){}? That’s crazy.

Also, I don’t like that the type parameters precede the arguments… isn’t the whole reason that we have Go’s unusual ordering that we acknowledge that the name is more important than the type?

Here’s my fix… since contracts are basically like interfaces, let’s actually use interfaces. And let’s make the contracty part last, since it’s least important:

func Print(s []interface{}:T) {
for _, v := range s {
fmt.Println(v)
}
}

So here’s the change in a nutshell. You use a real interface to define the type of the argument. In this case it’s interface{}. This cuts out the need to define a contract separately when we already have a way of defining an abstract type with capabilities. The : tells the compiler that this is a parameterized type, and T is the name given that type (though it’s not used anywhere).

More Complex Types

added this section to help remove some confusion people had with the proposal

More complex functions with multiple contract types are just as easily done:

func Map(vals []interface{}:X, f func(x X) interface{}:Y) []Y {
ret := make([]Y, len(vals))
for i := range vals {
ret[i] = f(vals[i])
}
return ret
}

:X defines a type in this scope which is constrained by the interface that precedes it (in this case, there’s no constraint). Y defines a separate type… then inside the scope you can reference those types.

Contract Definitions as Code Are Hard

Specifying contracts via example code is going to age about as well as specifying time formats via example output. -me on Twitter

The next example in the design is

contract stringer(x T) {
var s string = x.String()
}
func Stringify(type T stringer)(s []T) (ret []string) {
for _, v := range s {
ret = append(ret, v.String())
}
return ret
}

Wait, so we have to redefine the Stringer interface? Why? WHy not just use a Stringer interface? Also, what happens if I screw up the code, like this?

contract stringer(x T) {
s := x.String()
}

You think the error message from that is going to be good? I don’t.

Also, this allows an arbitrarily large amount of code in contract definitions. Much of this code could easily imply restrictions that you don’t intend, or be more general than you expect.

contract slicer(x T) {
s := x[0]
}

Is that a map of int to something? Or is it a slice? Is that just invalid? What would the error message say, if so? Would it change if I put a 1 in the index? Or -1? Or “1”?

Notably… a lot of really smart gophers who have been programming in Go for years have difficulty defining contracts that are conceptually simple, because there is so much implied functionality in even simple types.

Take a contract that says you can accept a string or a []byte… what do you think it would look like?

If you guessed this with even your second or third try…

contract stringOrBytes(s S) {
string(s)
s[0]
s[:]
S([]byte{})
}

…then I applaud you for being better at Go than I am. And there’s still questions about whether or not this would fail for len(s) == 0 (answer: it won’t, because it’s just type checked, not actually run… but, see what I mean about implications?) Also, I’m not even 100% sure this is sufficient to define everything you need. It doesn’t seem to say that you can range over the type. It doesn’t say that indexing the value will produce a single byte.

Lack of Names and Documentations

The biggest problem with contracts defined as random blocks of code is their lack of documentation. As above, what exactly a bit of code means in a contract is actually quite hard to distill when you’re talking about generic types. And then how do you talk about it? If you have your function that takes your locally defined stringOrByte, and someone else has theirs defined as robytes, but the contents are the same (but maybe in a different order with different type names)… how can you figure out if they’re compatible?

Is this the same contract as above?

contract robytes(t T) {
T([]byte{})
t[5:10]
string(t)
t[100]
}

Yes, but it’s non-trivial to see that it is (and if it wasn’t, you’d probably have to rely on the compiler to tell you).

Imagine for a moment if there were no io.Reader or io.Writer interfaces. How would you talk about functions that write to a slice of bytes? Would we all write exactly the same interface? Probably not. Look at the lack of a Logging interface, and how that affected logging across the ecosystem. io.Reader and io.Writer make writing and reading streams of bytes so nice because they’re standardized, because they are discoverable. The standardization means that everyone who writes streams of bytes uses the exact same signature, so we can compose readers and writers trivially, and discover new ways to compose them just by looking for the terms io.Reader and io.Writer.

Just Use Interfaces, and Make Some New Built-in Ones

My solution is to mainly just use interfaces and tag them with :T to denote they’re a parameterized type. For contracts that don’t distill to “has a method”, make built-in contract/interfaces that can be well-documented and well-known. Most of the examples I’ve seen of “But how would you do X?” boil down to “You can’t, and moreover, you probably shouldn’t”.

A lot of this boils down to “I trust the stdlib authors to define a good set of contracts and I don’t want every random coder to throw a bunch of code in a contract block and expect me to be able to understand it”.

I think most of the useful contracts can be defined in a small finite list that can live in a new stdlib package, maybe called ct to keep it brief. ct.Comparable could mean x == x. ct.Stringish could mean “string or []byte or a named version of either”… etc.

Most of the things that fall outside of this are things that I don’t think you should be doing. Like, “How do you make a function that can compare two different types with ==?” Uh… don’t, that’s a bad idea.

One of the uses in the contract design is a way to say that you can convert one thing to another. This can be useful for generic functions on strings vs []byte or int vs int64. This could be yet another specialized interface:

package ct
// Convertible defines a type that can be converted into T.
type Convertible:T contract
// elsewhere
func ParseUint64(v ct.Convertible:uint64) {
i, err := strconv.ParseUint(uint64(v))
}

Conclusion

The contracts design, as written, IMO, will make the language significantly worse. Wrapping my head around what a random contract actually means for my code is just too hard if we’re using example code as the means of definition. Sure, it’s a clever way to ensure that only types that can be used in that way are viable… but clever isn’t good.

One of my favorite posts about Go is Rob Napier’s Go is a Shop-Built Jig. In it, he argues that there are many ineleagant parts to the Go language, but that they exist to make the whole work better for actual users. This is stuff like the built-in functions append and copy, the fact that slices and maps are generic, but nothing else is. Little pieces are filed off here, stapled on there, because making usage easy matters more than looking slick.

This design of contracts as written does not feel like a shop-built jig. It feels like a combination all-in-one machine that can do anything but is so complicated that you don’t even know how to even approach it or when you should use it vs the other tools in your shop.

I think we can make a smaller, more incremental addition to the language that will fix a lot of the problems that many people have with Go - lack of reusable container types, copy and paste for simple map and filter functions, etc. This will only add a small amount of complexity to the language, while solving real problems that people experience.

Notably, I think a lot of the problems generics solve are actually quite minor in the scheme of major projects. Yes, I have to rewrite a filter function for every type. But that’s a function I could have written in college and I usually only need one or two per 20,000 lines of code (and then almost always just strings).

So… I really don’t want to add a bunch of complexity to solve these problems. Let’s take the most straightforward fix we can get, with the least impact on the language. Go has been an amazing success in the last decade. Let’s move slowly so we don’t screw that up in the next decade.

Comment Your Code

Fri, 17 Nov 2017 14:48:09 -0500

There’s a disturbing thread that pops up every once in a while where People On The Internet say that comments are bad and the only reason you need them is because you and/or your code aren’t good enough. I’m here to say that’s bullshit.

Code Sucks

They’re not entirely wrong… your code isn’t good enough. Neither is mine or anyone else’s. Code sucks. You know when it sucks the most? When you haven’t touched it in 6 months. And you look back at the code and wonder “what in the hell was the author thinking?” (and then you git blame and it’s you… because it’s always you).

The premise of the anti-commenters is that the only reason you need comments is because your code isn’t “clean” enough. If it were refactored better, named better, written better, it wouldn’t need that comment.

But of course, what is clean and obvious and well-written to you, today, while the entire project and problem space are fully loaded in your brain… might not be obvious to you, six months from now, or to the poor schmuck that has to debug your code with their manager breathing down their neck because the CTO just ran into a critical bug in prod.

Learning to look at a piece of code that you understand, and trying to figure out how someone else might fail to understand it is a difficult skill to master. But it is incredibly valuable… one that is nearly as important as the ability to write good code in the first place. In industry, almost no one codes alone. And even if you do code alone, you’re gonna forget why you wrote some of your code, or what exactly this gnarly piece of late night “engineering” is doing. And someday you’re going to leave, and the person they hire to replace you is going to have to figure out every little quirk that was in your head at the time.

So, throwing in comments that may seem overly obvious in the moment is not a bad thing. Sometimes it can be a huge help.

Avoiding Comments Often Makes Your Code Worse

Some people claim that if you remove comments, it makes your code better, because you have to make your code clearer to compensate. I call BS on this as well, because I don’t think anyone is realistically writing sub-par code and then excusing it by slapping a comment on it (aside from // TODO: this is a temporary hack, I'll fix it later). We all write the best code we know how, given the various external constraints (usually time).

The problem with refactoring your code to avoid needing comments is that it often leads to worse code, not better. The canonical example is factoring out a complicated line of code into a function with a descriptive name. Which sounds good, except now you’ve introduced a context switch for the person reading the code.. instead of the actual line of code, they have a function call… they have to scroll to where the function call is, remember and map the arguments from the call site to the function declaration, and then map the return value back to the call site’s return.

In addition, the clarity of a function’s name is only applicable to very trivial comments. Any comment that is more than a couple words cannot (or should not) be made into a function name. Thus, you end up with… a function with a comment above it.

Indeed, even the existence of a very short function may cause confusion and more complicated code. If I see such a function, I may search to see where else that function is used. If it’s only used in one place, I then have to wonder if this is actually a general piece of code that represents global logic… (e.g. NameToUserID) or if this function is bespoke code that relies heavily on the specific state and implementation of its call site and may well not do the right thing elsewhere. By breaking it out into a function, you’re in essence exposing this implementation detail to the rest of the codebase, and this is not a decision that should be taken lightly. Even if you know that this is not actually a function anyone else should call, someone else will call it at some point, even where not appropriate.

The problems with small functions are better detailed in Cindy Sridharan’s medium post.

We could dive into long variable names vs. short, but I’ll stop and just say that you can’t save yourself by making variable names longer. Unless your variable name is the entire comment that you’re avoiding writing, then you’re still losing information that could have been added to the comment. And I think we can all agee that usernameStrippedOfSpacesWithDotCSVExtension is a terrible variable name.

I’m not trying to say that you shouldn’t strive to make your code clear and obvious. You definitely should. It’s the hallmark of a good developer. But code clarity is orthogonal to the existence of comments. And good comments are also the hallmark of a good developer.

There are no bad comments

The examples of bad comments often given in these discussions are trivially bad, and almost never encountered in code written outside of a programming 101 class.

// instantiate an error
var err error

Yes, clearly, this is not a useful comment. But at the same time, it’s not really harmful. It’s some noise that is easily ignored when browsing the code. I would rather see a hundred of the above comments if it means the dev leaves in one useful comment that saves me hours of head banging on keyboard.

I’m pretty sure I’ve never read any code and said “man, this code would be so much easier to understand if it weren’t for all these comments.” It’s nearly 100% the opposite.

In fact, I’ll even call out some code that I think is egregious in its lack of comments - the Go standard library. While the code may be very correct and well structured.. in many cases, if you don’t have a deep understanding of what the code is doing before you look at the it, it can be a challenge to understand why it’s doing what it’s doing. A sprinkling of comments about what the logic is doing and why would make a lot of the go standard library a lot easier to read. In this I am specifically talking about comments inside the implementation, not doc comments on exported functions in general (those are generally pretty good).

Any comment is better than no comment

Another chestnut the anti-commenters like to bring out is the wisdom can be illustrated with a pithy image:

Ah, hilarious, someone updated the contents and didn’t update the comment.

But, that was a problem 20 years ago, when code reviews were not (generally) a thing. But they are a thing now. And if checking that comments match the implementation isn’t part of your code review process, then you should probably review your code review process.

Which is not to say that mistakes can’t be made… in fact I filed a “comment doesn’t match implementation” bug just yesterday. The saying goes something like “no comment is better than an incorrect comment” which sounds obviously true, except when you realize that if there is no comment, then devs will just guess what the code does, and probably be wrong more often than a comment would be wrong.

Even if this does happen, and the code has changed, you still have valuable information about what the code used to do. Chances are, the code still does basically the same thing, just slightly differently. In this world of versioning and backwards compatbility, how often does the same function get drastically changed in functionality while maintaining the same name and signature? Probably not often.

Take the bug I filed yesterday… the place where we were using the function was calling client.SetKeepAlive(60). The comment on SetKeepAlive was “SetKeepAlive will set the amount of time (in seconds) that the client should wait before sending a PING request”. Cool, right? Except I noticed that SetKeepAlive takes a time.Duration. Without any other units specified for the value of 60, Go’s duration type defaults to…. nanoseconds. Oops. Someone had updated the function to take a Duration rather than an Int. Interestingly, it did still round the duration down to the nearest second, so the comment was not incorrect per se, it was just misleading.

Why?

The most important comments are the why comments. Why is the code doing what it’s doing? Why must the ID be less than 24 characters? Why are we hiding this option on Linux? etc. The reason these are important is that you can’t figure out the why by looking at the code. They document lessons learned by the devs, outside constraints imposed by the business, other systems, etc. These comments are invaluable, and almost impossible to capture in other ways (e.g. function names should document what the function does, not why).

Comments that document what the code is doing are less useful, because you can generally always figure out what the code is doing, given enough time and effort. The code tells you what it is doing, by definition. Which is not to say that you should never write what comments. Definitely strive to write the clearest code you can, but comments are free, so if you think someone might misunderstand some code or otherwise have difficulty knowing what’s going on, throw in a comment. At least, it may save them a half hour of puzzling through your code, at best it may save them from changing it or using it in incorrect ways that cause bugs.

Tests

Some people think that tests serve as documentation for functions. And, in a way, this is true. But they’re generally very low on my list of effective documentation. Why? Well, because they have to be incredibly precise, and thus they are verbose, and cover a narrow strip of functionality. Every test tests exactly one specific input and one specific output. For anything other than the most simple function, you probably need a bunch of code to set up the inputs and construct the outputs.

For much of programming, it’s easier to describe briefly what a function does than to write code to test what it does. Often times my tests will be multiple times as many lines of code as the function itself… whereas the doc comment on it may only be a few sentences.

In addition, tests only explain the what of a function. What is it supposed to do? They don’t explain why, and why is often more important, as stated above.

You should definitely test your code, and tests can be useful in figuring out the expected behavior of code in some edge cases… but if I have to read tests to understand your code in general, then that’s red flag that you really need to write more/better comments.

Conclusion

I feel like the line between what’s a useful comment and what’s not is difficult to find (outside of trivial examples), so I’d rather people err on the side of writing too many comments. You never know who may be reading your code next, so do them the favor you wish was done for you… write a bunch of comments. Keep writing comments until it feels like too many, then write a few more. That’s probably about the right amount.

Code Must Never Lie

Tue, 29 Aug 2017 12:40:19 -0400

If you tell the truth, you don’t have to remember anything.

—Mark Twain

In a code review recently, I asked the author to change some of their asserts to requires. Functions in testify’s assert package allow the test to continue, whereas those in the require package end the test immediately. Thus, you use require to avoid trying to continue running a test when we know it’ll be in a bad state. (side note: don’t use an assert package, but that’s another post) Since testify’s assert and require packages have the same interface, the author’s solution was to simply change the import thusly:

import (
assert "github.com/stretchr/testify/require"
)

Bam, now all the assert.Foo calls would stop the test immediately, and we didn’t need a big changelist changing every use of assert to require. All good, right?

No.

Hell No.

Why? Because it makes the code lie. Anyone familiar with the testify package understands the difference between assert and require. But we’ve now made code that looks like an assert, but is actually a require. People who are 200 lines down in a test file may well not realize that those asserts are actually requires. They’ll assume the test function will continue processing after an assert fails. They’ll be wrong, and they could accidentally write incorrect tests because of it - tests that fail with confusing error messages.

This is true in general - code must never lie. This is a cardinal sin amongst programmers. This is an extension of the mantra that code should be written to be read. If code looks like it’s doing one thing when it’s actually doing something else, someone down the road will read that code and misunderstand it, and use it or alter it in a way that causes bugs. If they’re lucky, the bugs will be immediate and obvious. If they’re unlucky, they’ll be subtle and only be figured out after a long debugging session and much head banging on keyboard. That someone might be you, even if it was your code in the first place.

If, for some reason, you have to make code that lies (to fulfill an interface or some such), document the hell out of it. Giant yelling comments that can’t be missed during a 2am debugging session. Because chances are, that’s when you’re going to look at this code next, and you might forget that saveToMemory() function actually saves to a database in AWS’s Antarctica region.

So, don’t lie. Furthermore, try not to even mislead. Humans make assumptions all the time, it’s built into how we perceive the world. As a coder, it’s your job to anticipate what assumptions a reader may have, and ensure that they are not incorrect, or if they are, do your best to disabuse them of their incorrect assumptions.

If possible, don’t resort to comments to inform the reader, but instead, structure the code itself in such a way as to indicate it’s not going to behave the way one might expect. For example, if your type has a Write(b []byte) (int, error) method that is not compatible with io.Writer, consider calling it something other than Write… because everyone seeing foo.Write is going to assume that function will work like an io.Write. Instead maybe call it WriteOut or PrintOut or anything but Write.

Misleading code can be even more subtle than this. In a recent code review, the author wrapped a single DB update in a transaction. This set off alarm bells for me as a reviewer. As a reader, I assumed that the code must be saving related data in multiple tables, and that’s why a transaction was needed. Turned out, the code didn’t actually need the transaction, it was just written that way to be consistent with some other code we had. Unfortunately, in this case, being consistent was actually confusing… because it caused the reader to make assumptions that were ultimately incorrect.

Do the poor sap that has to maintain your code 6 months or two years down the road a favor - don’t lie. Try not to mislead. Because even if that poor sap isn’t you, they still don’t deserve the 2am headache you’ll likely be inflicting.

npf.io

Optimizing for Reviewers: The Three Step AI Dev Loop

The Three Step AI Dev Loop

Conclusion

Why Infer Types?

Safer Enums

Error Flags

Fragile Error Handling

Dave’s Solution - Interfaces

A Better Way - Flags

Supporting Errors.Is and Errors.As

Indirect Coupling

Choosing Flags

Conclusion

How to Pay Remote Workers

Do or Do Not

The proposal

Complications

The Old Way

Early Returns

One idea per line

Nesting Functions

Hiring Remote

Retooling Retool

Init Is Bad and You Should Feel Bad

Starlight

What is Starlark?

Parser by google

Easy two-way interaction

100% Safe

Example

Caching

Work Ongoing

Mage - make/rake for Go

A Brief History

Maybe go run? Maybe not

The Last Straw

What is Mage?

What is a Magefile?

Targets

Dependencies

Shell Helpers

Verbose Mode

How it Works

Conclusion

Handle and Check - Let's Not

Go2 Contracts Go Too Far

Syntax

More Complex Types

Contract Definitions as Code Are Hard

Lack of Names and Documentations

Just Use Interfaces, and Make Some New Built-in Ones

Conclusion

Comment Your Code

Code Sucks

Avoiding Comments Often Makes Your Code Worse

There are no bad comments

Any comment is better than no comment

Why?

Tests

Conclusion

Code Must Never Lie

Maybe `go run`? Maybe not