I became a software engineer because writing code is fun. Thinking through hard problems, designing elegant solutions, seeing the things you’ve built working for the first time… these moments are all deeply satisfying, so why in the world would I ever surrender them to AI?

I know the arguments for using AI to write code; I hear them constantly from all levels of the tech industry. I’m told that it’ll make me more productive. I could use Copilot to write boilerplate or unit tests for me so I can focus on more creative work. I could use Claude Code as a sounding board or planning tool for complex features, refactors, or other projects. I could run and coordinate multiple concurrent agents and have them write and maintain an entire application for me from scratch. I could do all of these things, but I won’t. Writing code is just too much fun!

When people use generative AI to produce images, videos, music, etc., they like to style themselves as “AI artists”. But they’re not artists. Even if you decide to be generous and call the final piece “art”, they’re commissioning that art, no matter how detailed or iterative they are with their prompts. They are, at best, art directors.

Likewise, the more you rely on AI to produce code for you, the less of a software engineer you become. Instead of spending your time solving problems and writing code, you spend your time reviewing AI-generated code. Maybe you’re relying on AI for code review, too, in which case you’re mostly just writing requirements, determining how best to format them into a prompt, and maybe managing AI agents. When someone’s primary job is to figure out and write requirements or manage the entities who are actually producing the code, we don’t usually call that person a software engineer. We call them a product or project manager. This isn’t to say that it’s bad to be a product or project manager! We sometimes need product and project managers just like the art world sometimes needs people to commission and direct art.

I’ve reluctantly tried various AI coding tools over the last few years and, while they have become more impressive, they remain deeply unfun. Writing code is, of course, only one duty of being a software engineer, but where code is concerned, there are two primary responsibilities: writing it and reviewing it. For people who enjoy being a software engineer, I think it’s safe to say that most of them would agree that writing the code is more fun than reviewing it, so I ask again: why cede the fun parts of your job to AI? Just to chase the increasingly demanding productivity requirements of the ruling class? Just to produce a bit more in a shorter amount of time?

The thing is, in all my attempts to use AI coding tools, they’ve never actually enabled me to move faster. They initially felt faster because the tangible output came more quickly but, in almost every case, I realized that I spent at least the same amount of time that I would have spent if I had just written the code myself (even just for boilerplate). AI produced the code more quickly than I would have, but between writing prompts, reviewing/scrutinizing code, and tweaking follow-up prompts to fix issues, I saved no time at all. I traded happiness for the illusion of speed (and more problems). I could surely spend a lot of time learning to use these AI tools more effectively, but why do something that’s worse and less enjoyable? I’d rather spend that time learning other things.

Writing code is fun and helps me improve my skills. Solving problems keeps my mind sharp and helps me learn even more. Using AI doesn’t help me do any of this; it only sucks the joy out of my work.

Ages ago, when I was still a student, I taught myself Ruby on Rails for my senior thesis and fell in love. Fifteen years later, and I’ve used Rails at every job I’ve ever held in the tech industry. Fifteen years, and I still love Rails! But there’s something rotten at its core, and we share a name.

Back in 2014, Tom Stuart delivered a pithy yet salient lightning talk at the Scottish Ruby Conference titled The DHH Problem, in which he succinctly describes the character of David Heinemeier Hansson, the creator of Rails. I recommend watching it because, in just three and a half minutes, he effectively provides context on who DHH was, who he still is, and why his future shift to the right would make so much sense and be so unsurprising. But, if you’d rather not, I’ll include a transcript below:

DHH is an intelligent and successful person. DHH invented Rails, and now I can get paid for writing Ruby, even though I enjoy it. So: thanks, DHH (sincerely). But, there’s a problem. DHH is “Ruby Famous”, which means that DHH is extremely visible (some might say disproportionately visible) in the Ruby community. Why is that a problem? Because DHH is the Fox News of Ruby.

He’s noisy, he’s reactionary, he’s anti-intellectual, he’s very sure that he is right, and he enjoys being rude. [Tom cuts to a photo of DHH delivering a conference talk in 2006, standing in front of a slide that just says “Fuck You”] That was eight years ago, but things haven’t changed much. Just like Fox News, DHH appeals to “common sense” and makes a show of being “fair and balanced” but, in reality, his arguments use aggressive rhetoric and rely on a fixed viewpoint.

To paint a topical example, is TDD hard in Rails apps because TDD is dead, or because Rails makes TDD hard? Is TDD not worth the effort because TDD is dead, or because the complexity of software can be managed more effectively if you only work on one product for which you control the requirements? If we only listen to DHH, then we’ll never know, because DHH is just one person and he only has DHH’s experiences.

All I’m saying is: the Ruby community is large and diverse and thoughtful, and that is why I love it. Please listen to DHH: his experiences are valuable. But DHH does not speak for me, and he probably doesn’t speak for you. My personal preference is for a bit less of this [Tom cuts back to the photo of DHH’s “Fuck You” slide] and a bit more of this [he then cuts to a different man delivering a conference talk, standing in front of a slide that says “What others do may be the stimulus of our feelings, but never the cause.”]

Please give DHH’s opinions the weight they deserve: they are what one man thinks. And if you disagree with him, please speak up—write a blog post, give a talk, create a web framework—so that we can all learn what the world looks like to you.

That was eleven years ago, but things haven’t changed much. DHH is still noisy, still reactionary, still anti-intellectual, still very sure that he is right, and still enjoys being rude. You need only look at his personal blog to witness this yourself, but doing so isn’t for the faint of heart; this iteration of DHH’s blog on Hey World started a mere four years ago but, in that time, he’s written 476 posts on a variety of topics. That’s too many posts for a deep dive, but the broad trends show a reactionary who has moved, at times steadily and at times quickly, to the far right.

Early on, DHH comes off as a man who held a variety of political beliefs. He would write about the dire state of our climate and then, a couple weeks later, drop hints of an increasingly present narrative around cancel culture. Early posts on this blog tended to follow a theme of his “natural political stance” ostensibly being left-leaning, while being open to right-wing views as a way to “find common ground”. I’ll be honest and say that I’m not sure I believe that he was ever truly left-leaning. My own observations show someone who is fairly centrist or even slightly right of center, which tracks with what standard Liberalism has become in America.

Over the next few years, we’d see posts from DHH covering a range of far-right talking points: anti-wokeness, anti-DEI, free speech absolutism, support of figures who range from right-leaning (like Joe Rogan) to far-right (like Andrew Tate or Jordan Peterson), to anti-trans fearmongering, all the way to outright nationalism (this post may seem innocuous on its face, but hints at his inability to recognize hate speech as violence and includes the common dog whistle of countries having a “strong national identity”). He rails against finding beauty in diversity rather than what’s “normal” or in people of all sizes, then cheers when companies cave to the conservative backlash against DEI and stop celebrating that beauty. But what I find most indicative of DHH’s shift (whether towards the right, or towards honesty that he’s always been on the right) comes from a quote in one of his earliest posts on this blog, Legacy without nostalgia:

I’m just not a nostalgic person. In fact, I’m deeply skeptical of nostalgia. It too often feels like a trap to romanticize the past at the expense of the future. If you fall in love with who you once were, it’s too easy to forget to keep going. I want to keep going.

This is deeply ironic given just how much DHH thirsts for nostalgia in his later posts. He frequently yearns for the world of the past, especially the 80s (but not the 90s, which he hates), before all of this pesky talk about DEI, before cancel culture, before “pessimism”. But no post encompasses this hypocritical desire for nostalgia more than his essay from a few days ago, As I remember London, a post which is quite simply fascism on full display.

The fascism of DHH

Some people really don’t like it when you throw the word “fascism” around or call people a fascist, and there are many cases where its usage is undue. This is not one of those cases. There is a long history of attempts to coalesce definitions of fascism (many of which delve into extreme nationalism), but I like Roger Griffin’s definition, which he argues can be condensed into one sentence:

Fascism is a political ideology whose mythic core in its various permutations is a palingenetic form of populist ultra-nationalism.

That’s admittedly a dense sentence, so why don’t we unpack it a little bit, starting with Palingenesis being the concept of a rebirth or recreation. Fascism has an obsession with a cult of national purity and the idea of a mythic past to which we must return through some form of national rebirth (palingenetic ultranationalism). Fascists will attempt to paint their nation as humiliated and threatened on the global stage and that it must be reborn through a populist, revolutionary, nationalist movement. This is a style of politics, not a form of government; a process, not an end result (even if the end results tend to be similar); and “As I remember London” fits this bill.

This post echoes that myth of a golden past, of London in the late 90s, and very quickly claims that London has strayed from it because “it’s no longer full of native Brits” in his view, citing statistical changes in London’s ethnic groups. This is an obvious and purposeful feint that equates “native Brits” with “white Brits” and falls into the fascist rhetoric of their national purity suffering as a result. DHH’s entire post is a subtextual frame for the hope that Britain can experience a national rebirth and return to a time of white purity.

After citing those statistics, DHH lionizes Tommy Robinson, calling his recent march “heartwarming”. Robinson is one of Britain’s most prominent far-right activists (with a string of multiple arrests and convictions) and has been involved in several hate-based organizations like the English Defence League, Pegida UK, and the British Nationalist Party. He cites misleading reports of “Pakistani Rape Gangs” despite later research making it clear that the majority of group-based child sex offenders are white. DHH’s quick reinforcement of his support for Graham Linehan (the notably anti-trans Irish comedian) is disgusting icing on an already disgusting cake.

DHH takes this growing multiculturalism in Britain and uses it to stoke fear of the same thing happening in Denmark, and Steve Klabnik was quick to notice the ridiculousness of saying that there’s “absolutely nothing racist or xenophobic in saying that Denmark is primarily a country for the Danes, Britain primarily a united kingdom for the Brits, and Japan primarily a set of islands for the Japanese” and, worse, how that verbiage disturbingly echoes the sentiments of the Ku Klux Kreed (America for Americans)

Something needs to change

This post was my breaking point. This post is why I’m here right now, writing something much longer than the social media quips I’ve written over the years in response to his ongoing descent into far-right white nationalism and fascist rhetoric (again, nationalism is the bedrock of fascism). Every time DHH spews a new piece of hatred, he alienates people that slowly decide to leave the Rails or greater Ruby communities. But I’ve noticed another common response over the years—often from multiple people—every time there’s a new controversy: “Wow, I had no idea DHH went fully mask off.” There’s a clear lack of awareness. Or, worse: people simply forget.

“As I remember London” should be a wake-up call for everyone in the Ruby and Rails communities. This is not a man who wants to keep going. This is a man who romanticizes a past that was predominantly good for white men. This is a man who has spent years railing against diversity, equity and inclusion and who spreads anti-trans rhetoric. This is a man who is deeply afraid of immigration changing countries’ cultures and “national identities”, despite this kind of change being a constant for the whole of human history. This is a man who is a white nationalist. And he is in sole control of Rails, this framework we all love.

There’s a growing discontent that I see on social media over leadership in Rails, most of which comes down to DHH’s continued involvement, waning trust in the organizations who continue to platform him (like Ruby Central), and the problematic reality of how much Ruby and Rails is beholden to Shopify money. But what’s the answer? What can we reasonably change? I don’t know! It doesn’t help that, as I was nearing the end of this post, yet another Ruby Drama unfolded live on social media. This latest drama (in a community that has a long history of drama the likes of which I’ve not seen in any other programming community) is that, over the course of the last week and a half, Ruby Central forcibly took control of the rubygems.org codebase on GitHub and revoked access from all of its maintainers in a massive shift of governance that came without advanced warning or communication of any kind. They responded several hours later with some very hand-wavey corpospeak describing their motive of “a fiduciary duty to safeguard the supply chain and protect the long-term stability of the ecosystem,” but Joel Drapper’s summary makes it clear that this came down to Shopify taking advantage of funding being pulled from Ruby Central due to their continued relationship with DHH. The whole situation has reeked of incompetence and I’ve lost any trust I had left in the organization, which had admittedly become very little. I’m tired of a lack of transparency from our community’s unelected leaders and board members. I’m tired of the vagueposting that makes it impossible to decipher what events like these are truly about. I’m just so tired in general, and now I’m thinking not only of Rails’ governance, but governance in the greater Ruby community as a whole.

I appreciate a lot of the work Ruby Central has done over the years in stewarding important projects and infrastructure that are central to Ruby and Rails, but we need something that is beholden to the community rather than a board of appointed officials who can become beholden to corporate sponsors. What I’d prefer to see is something more akin to a co-operative. What if we had a public organization with open membership (whether via dues or some other mechanism) and democratically elected leadership? I would feel much more comfortable with that kind of organization maintaining and stewarding important projects like RubyGems, Rails, Rack et al. and a democratic process would abate a lot of my negative feelings around our current state of corporate sponsorship in Rails.

At the same time, I recognize that this kind of change would be incredibly difficult. I don’t think there’s any chance of removing DHH from Rails, and I can’t see him stepping down of his own accord. He owns the copyrights to Rails, he’s the founder and chair of Rails’ current governing body, the Rails Foundation, and—despite his open politics—he continues to be platformed. He’s invited to speak on podcasts. He’s invited to speak or keynote at conferences. Even Ruby Central invited him back for the last RailsConf earlier this year after he threw a hissy fit over being asked to cede the keynote stage in 2022. Rails core members and organizations like Ruby Central continue to work with and collaborate with him closely and, though many have walked away in recent years, I wish more would follow. I get that doing so is an incredibly difficult decision to make, but people have had years to distance themselves from this man. But they don’t, so Ruby and Rails continue to be directed by far-right sympathizers like DHH and Tobi Lütke (and their money). And all of this is allowed because Ruby and Rails have a code of conduct that purposefully falls prey to the paradox of tolerance.

But is difficulty really an excuse not to try? Maybe, just maybe, we can do something about this. As I mentioned before, many prominent maintainers of projects like RubyGems and Rails have already stepped away due to ideological issues with DHH and RubyCentral. What if they were be open to returning with new governance? Of course, the unfortunate nature of copyright and current ownership means we would likely be facing a number of community forks (at the very least for Rails, and quite possibly for RubyGems as well), but community forks need not be doomed to fail! Starting the right conversations with the right people early on can do wonders for momentum. So, why collectively settle for this? I have to believe that we can do better than the leadership we currently have. We can do better than the Fox News of Ruby.

At xoxo.zone, we chose to block federation with Threads as soon as they’d publicized the threads.net domain they intended to use. There were a few questions but, thankfully, that decision wasn’t particularly controversial in our community. Outside of our community, though, I’ve seen a lot of skepticism about preemptively blocking Threads.

One of the first and most widely-shared takes I saw was from John Gruber of Daring Fireball, “Not That Kind of ‘Open’”, in which he linked to the Anti-Meta Fedi Pact and wrote…

The whole point of ActivityPub as an open protocol is to turn Twitter/Instagram-like social networking into something more akin to email: truly open. If Facebook were on the cusp of launching a Gmail-like email service, would you preemptively declare that your email server would block them? To me that’s what this “Anti-Meta Fedi Pact” is arguing for.

Maybe I’m wrong! I certainly don’t think the “let’s pledge to block Facebook before their Fediverse thing even starts” people are nuts. But to me this feels like convicting Facebook of a pre-crime. Is the goal of the Fediverse to be anti-corporate/anti-commercial, or to be pro-openness? I think openness is the answer. Others clearly disagree.

Gruber misses the point entirely. ActivityPub isn’t some critical communication protocol akin to email and, while openness is one goal of ActivityPub, the freedom of server admins to choose how they federate is another. More importantly, blocking Meta isn’t about being anti-corporate/anti-commercial (though I’m sure that’s partially at play for some), and the idea that it’s akin to “convicting Facebook of a pre-crime” is a naïve conclusion to draw. Our problem with Meta isn’t that we’re worried that they’ll be bad for the fediverse; our problem is that Meta is already bad for the entire internet and society at large.

Meta’s existing atrocities form a long list, but Erin Kissane provides a thoughtful and detailed exploration of Threads and federation. She approaches the argument from all angles while doing her best to check her own biases and, if you don’t understand why federating with Threads is a bad idea, you should stop here, read her article, and then come back. If you have even more time, you should read her entire Meta in Myanmar series. Either way:

Threads isn’t yet running ads-qua-ads, but it launched with a preloaded fleet of “brands” and the promise of being a nice, un-heated space for conversation—which to say, an explicitly brand-friendly environment. (So far, this has meant no to butts and no to searching for long covid info and yes to accounts devoted to stochastic anti-LGBT terrorism for profit, so perhaps that’s a useful measure of what brands consider safe and neutral.) Perhaps there’s a world in which Threads doesn’t accept ads, but I have difficulty seeing it.

That “preloaded fleet of brands” made Threads feel particularly soulless when I first joined, and a lot of the content I get served on Threads still falls into the category of “brands and blue checks post inane, thought-leadering engagement bait.” Most of it is just tedious to read and boring to interact with, but the worst kind of content on Threads is actually sinister and (surprise!) Meta does practically nothing in the way of content moderation to prevent it. This brings me to a particularly sobering excerpt from Kissane’s article:

I would note that most mainstream fedi servers maintain policies that at least claim to ban (open) harassment or hateful content based on gender, gender identity, race or ethnicity, or sexual orientation. On this count, I’d argue that Threads already fails the first principle of the Mastodon Server Covenant:

Active moderation against racism, sexism, homophobia and transphobia Users must have the confidence that they are joining a safe space, free from white supremacy, anti-semitism and transphobia of other platforms.

Don’t take my word for this failure. Twenty-four civil rights, digital justice and pro-democracy organizations delivered an open letter last summer on Threads’ immediate content moderation…challenges:

…we are observing neo-Nazi rhetoric, election lies, COVID and climate change denialism, and more toxicity. They posted bigoted slurs, election denial, COVID-19 conspiracies, targeted harassment of and denial of trans individuals’ existence, misogyny, and more. Much of the content remains on Threads indicating both gaps in Meta’s Terms of Service and in its enforcement, unsurprising given your long history of inadequate rules and inconsistent enforcement across other Meta properties.

Rather than strengthen your policies, Threads has taken actions doing the opposite, by purposefully not extending Instagram’s fact-checking program to the platform and capitulating to bad actors, and by removing a policy to warn users when they are attempting to follow a serial misinformer. Without clear guardrails against future incitement of violence, it is unclear if Meta is prepared to protect users from high-profile purveyors of election disinformation who violate the platform’s written policies.

That open letter was back in July, well before the ongoing humanitarian crisis and genocide in Gaza. Every time I’ve decided to check in on Threads recently, I’ve seen (in addition to everything observed above) a virulent mix of antisemitism, Islamophobia, and propaganda, all courtesy of Threads’ algorithm.

Like Kissane, I try to be pragmatic. I understand why people would choose to federate with Threads. I spent a lot of my life on Twitter and I still struggle with the fact that my community has scattered to the wind, with only a small percentage finding their footholds elsewhere. Some people think that Threads is best positioned to be the next large, text-based social network now that so many have fled Twitter. Personally, I don’t believe there will be another huge, centralized social network like Twitter (at least not any time soon).

For me (and, thankfully, the community I help moderate), federating with Threads isn’t an option. Meta’s long track record should make it painfully obvious that they care more about their business metrics than the community they profit from. They continue to avoid content moderation as much as possible and have proven time and time again that not only will they allow vile and hateful content in their networks; they’ll visibly proliferate it. Meta is not a good steward of their communities, and we’re under no obligation to allow them unfettered access to any of ours.

Steve’s post is cleverly titled to play on the fact that it was published on the tenth anniversary of his foray into Rust, but he mainly talks about how he rewrote his website to learn a new framework (Astro) and to use a language that he doesn’t frequently employ (TypeScript). Me, on the other hand? I’m mostly staying comfortable. You see, fifteen years ago, I was introduced to Ruby and Rails, and I’ve been working with them ever since. Fifteen years ago is also when I registered my first domain name (davidcelis dot com) so I could start hosting a personal site. All that to say: I’ve been rewriting this dang site for over fifteen years.

My first website was a WordPress blog hosted on NearlyFreeSpeech.NET so I could share some very amateur and very infrequent photography. During the next few years, as I got more involved in programming and was nearing my college graduation, I started to care more about my internet presence and having a more professional website I could share on my résumé. I played around with a second wordpress site centered more on myself than my photography, then a CNAME to my tumblr (easier to maintain), then a Jekyll blog powered by Octopress and, eventually, a simple, vanilla Jekyll blog.

Since then, I’ve rewritten and redesigned my Jekyll website every few years, and even switched from davidcelis dot com to my current Icelandic domain “hack”. My website has never blown me away (I’m not a designer or even a very good frontend developer), but each time did get a little better. Today marks a brand new era for my website. After fifteen years, I’m right back where I started in my programming journey: Ruby on Rails.

Why Rails? Why not keep it a static site? Or why not, like Steve, take this as an opportunity to stretch my legs, leave my comfort zone, and learn something new? These are reasonable questions. Static sites perform better and are generally easier to maintain and update. You don’t need to worry about a server or PaaS for running a Ruby process indefinitely. You can just build a static site, push it to GitHub, and add a CNAME. And this is exactly what I had already been doing for years. So why change this and replace it with a Rails app? Steve rewrote his website to learn something new and, seemingly in the process, to reinvigorate his desire to write. I want to reinvigorate my desire to write too! But the biggest reason I’m rewriting my website is that Twitter is dying.

I know Twitter has completely dominated news cycles and online discussions for months now, so I won’t spend a ton of time on that. But Twitter has been an incredibly important part of my life for a long time. I signed up in early 2007 and I’ve used it ever since. Sure, I’ve used it to shitpost and spew random thoughts into a void for meaningless internet points, but I’ve also used it to make friends, build communities, and even find multiple jobs that strengthened my career. I’ve maintained and rewritten and redesigned this website a lot, but Twitter has always been my home on the internet. I loved Twitter, but I don’t love it anymore, and most of my friends are leaving or have already left.

At this point, I’ve made Mastodon the place where I post what I would’ve posted on Twitter, but it still doesn’t scratch the same itch. Federation is messy, and Mastodon lacks a lot of the exploration features that Twitter had. Then again, with how addictive social media can be, maybe it’s a good thing that Mastodon can’t scratch the same itch Twitter did. Maybe that itch was bad for me. I say I loved Twitter but, like a lot of people, I also really hated Twitter. It was a terrible place with terrible people and it made me care too much about things like likes and retweets.

Twitter’s collapse has made me realize that what I really want is to reclaim some control. Right now, Mastodon seems to be the place where most of my network is migrating. Maybe that will stay true for a while, and maybe it will change. Maybe some other centralized platform will grab wide-spread attention. Maybe people, including my own disparate groups of online friends, will fragment and coalesce across multiple networks. The future is uncertain, but I’m trying to carve out a space for myself that I can rely on, and where friends can find me regardless of where they themselves end up: my own website. And, after all this time, this is what brings me back to Rails.

You see, while it can be really easy to maintain and deploy a static site, it’s not that easy to publish new content to it. I was never a frequent blogger, but when I had some motivation for longform writing, I used to write at least a couple times a year. That turned into once every couple years. Then, after my last post in 2017, I lost steam entirely. Meanwhile, on Twitter, I’d post almost every single day, often multiple times a day. With Twitter dying, and with an uncertain future for that kind of social media, I wanted to refocus on my own space and refocus on ways to reclaim my own content. But I had a few important critera:

1. It needs to be easy for me to post. With Twitter, it was so easy to have a thought, pull out my phone, instantly share it with my friends and followers, and feel like I was connecting with people. That’s a lot harder to do with a static site. I didn’t want to be fiddling around with complicated ways to publish short notes to a GitHub repository from my phone. Sharing photos would be even worse.
2. I want to be able share all sorts of things: text, yes, but also photos, videos, links, and maybe even things like location-based check-ins.
3. I want to reclaim control of my content while still being able to reach my friends. This means continuing to share my content on platforms like Mastodon (and, for now, Twitter) but treating my own website as the final source of truth, basically joining the IndieWeb.

For a bit, I tried out micro.blog, which gives you a static website but with a Twitter-like posting interface that lets you publish new posts and photos really quickly. It’s admittedly a great service that tackles those three requirements, but I’m really picky, and there were enough little ways in which it didn’t work in the way I wanted it to work, so I gave up on it pretty quickly. I also thought about keeping my Jekyll site, but it’s too hard to update a Jekyll site from a mobile device. I really didn’t want to have to do anything like fiddle with iOS shortcuts or complicated workflows that take text or photos from my phone and somehow put them in my website’s git repository. I just want to be able to pull out my phone, write something in Markdown, maybe add a photo or two (or four), hit “Send”, and have it available to read on my website and a few other websites I’m using. And, of course, I want all of it to work in exactly the way that I want it to work.

For all those reasons, I ended up doing something I hadn’t done since running through my very first Rails tutorial fifteen years ago. I opened up my terminal, typed in rails new blog, and hit enter. And for the first time in recent memory, I feel excited about my website again. I feel excited about writing again.

Here’s to another fifteen years.

What is REST?

In REST APIs, the server defines a specific set of resources that a client can request, and these resources are defined by unique URLs. For example, in the API for a generic microblogging platform, the URL /users/1 may denote the first user in the system, /users/1/posts could return a collection of all posts that user has written, and /users/1/posts/327 could return a single post. REST has many nuances and a well-documented specification for behavior, but URL-based resources cover the basic idea. What is ultimately important is that the server defines the structure of the data that the client can request.

What’s wrong with REST?

Imagine you work for the aforementioned Generic Microblogginator™ company as a mobile app developer. You’re given the task of writing the mobile view for a user’s profile, which needs to show information about the user and list their posts. This isn’t too difficult; just hit the /users/{id} endpoint to get the former, and /users/{id}/posts to get the latter.

You ship the mobile view and wait to be ✨ dazzled ✨ by all of the customer feedback and app reviews. Next week, once all of the reviews have poured in, you get a new requirement. That Other Microblogger™ shows a couple of comments on each post in their profile view. Why don’t we do that, too? Luckily, your API already has an endpoint to get a blog post’s comments: /users/{id}/posts/{id}/comments. You change the view to hit that endpoint for each post you show on a user’s profile page, and you’re done.

But now your app is slow, and this leads us to one of the major problems with REST APIs:

Too many HTTP requests

Let’s face it: client applications rarely stay simple. More often than not, each client has a fairly specific set of requirements that reflect what data they need from your system. If you provide only one absolute way to request data, you’ll get clients trying to ram a rhomboid peg into a diamond-shaped hole.

In our previous example, our mobile app will become slower and slower with each post a user writes. If a user has twenty posts listed on their profile, we’re issuing 22 API requests. One for information on the user, one for their list of posts, and then twenty requests to get each post’s comments.

As you add more components to your mobile app’s interface, this problem will get worse. With each new UI component comes a new API call or a new customization to existing API endpoints. You can nest objects within each other to avoid extra API calls, but as your view becomes more complex, you’ll inevitably start nesting irrelevant data. You’ll end up with endpoints that don’t describe a single resource but, instead, a view of multiple resources. Now your API doesn’t seem so RESTful anymore.

Even worse, you’ll need to support any old endpoints as long as there are old versions of clients in the wild lest you risk breaking those clients. This leads to another major problem with REST:

“Versioning” REST APIs is a pain

The structure of responses from REST APIs is important. Clients build themselves around the knowledge that each resource has a specific structure. When Generic Microblogginator™ first released their API, this is what the response for getting a single post looked like:

{
  "author_id": 1,
  "title": "Give it a REST: Use GraphQL!",
  "body": "In the world of API architecture, REST has been the reigning ruler for a decade or more.",
  "published_at": "Thu Jan 05 2017 14:45:10 GMT-0800 (PST)"
}

After some time has passed, you decide there are a couple things you want to improve about a post’s structure in the API. Posts are about to get categories, so you’ll need to add those as a new field. You’ve also received feedback that the format for published_at isn’t very friendly. JavaScript clients can parse it okay, but you’d rather any tool be able to parse your timestamps easily, so you decide to change it to an ISO-8601 format. When all is said and done, you want the new structure to look like this:

{
  "author_id": 1,
  "title": "Give it a REST: Use GraphQL!",
  "categories": ["tech", "programming", "graphql", "rest", "api"],
  "body": "In the world of API architecture, REST has been the reigning ruler for a decade or more.",
  "published_at": "2017-01-05T14:45:10-08:00"
}

Looking good! Unfortunately, one of your changes will break all of your existing clients. Every client expects published_at to be the less-friendly format, so that’s how they’ll try to parse it. If you want to update a field or remove a field, you have to version your API (whether it’s via the URL or an HTTP header) and try to get clients to upgrade. It’s unlikely you’d get every client to upgrade, so you have two choices:

1. Be okay with breaking old versions of clients (including your own app)
2. Support old versions of your API until the day your company decides to announce a new chapter in their incredible journey.

The easiest thing to do is simply leave your old code alone, which means piling more and more versions of your API versions on top of the old ones.

A challenger approaches

Enter GraphQL, a technology written by Facebook. Facebook was facing major problems with the data pipeline for their mobile applications. Their mobile apps used to be wrappers around web views and, as the mobile apps increased in complexity, they began to suffer performance problems and frequent crashes. Facebook turned to writing native applications and found themselves needing a new API to retrieve data for their native views. They evaluated REST and other options but, given problems like those described above, ultimately took the opportunity to produce something truly new.

What is GraphQL?

GraphQL is, as the name might suggest, a query language. It’s also perfect for APIs. It allows you to define your data using a fully-fledged type system, forming a schema that is self-documenting. It also gives clients full control over the data they request.

Too many HTTP requests? How about one HTTP request?

With GraphQL, clients can get all of the data they need to render a view using only one request. With our previous profile page example, a client would need to issue one request to get a user’s information, one request to get that user’s posts, and then another request for each post to get a few comments. With GraphQL, that client could get all of the above data with one request:

query {
  user(id: 1) {
    username
    fullName
    avatarUrl

    posts(first: 25) {
      title
      body
      publishedAt
      categories

      comments(last: 2) {
        author {
          username
        }
        body
      }
    }
  }
}

Boom! 💥 There are other benefits to this aside from the fact that we went from 22 HTTP requests to one. For instance, your User may have other information attached to it. Maybe you expose the timestamp of when a user signed up. Maybe another client doesn’t care about a post’s categories. If a client doesn’t need to query for a piece of data, neither does your server. So when a client saves, you can save too by simplifying your own database queries.

Versioning? Just deprecate!

As with (most) REST APIs, you can add fields to GraphQL types without fear. To remove functionality, GraphQL includes deprecation as a feature. Instead of fully removing a field and breaking clients, you can declare a field as deprecated and hide it from tools as it ages.

Documentation: you’ll barely need to worry about it

Let me be real for a second here: I can count the number of times I’ve used a well-documented API on one hand. Many times, APIs remain undocumented or poorly documented. With GraphQL, your schema is practically self-documenting. All you have to do is give your types and fields descriptions when necessary, and this happens in the code itself. Clients can issue special GraphQL queries to introspect on your application’s schema and know, in one query, all of the data they can request, what it’s called, and what it describes. Developers can also use tools that are built on this introspection like GraphiQL, which allows clients to test their queries with live syntax highlighting and error detection.

Get started with GraphQL

Are you sold enough to try out GraphQL? There are plenty of resources to help you get started on your journey:

• Check out GraphQL’s official website for documentation and examples
• Play around with a working example, like the Nook Stop API, an Animal Crossing: New Horizons themed “GiraffeQL” API that I built for instructive purposes (you can also check out the code that powers it on GitHub)
• If you’re into the nitty gritty, you can read the GraphQL Specification itself.

Until recently, my site was hosted on GitHub Pages. It’s a static site built using Jekyll so, because Pages is free, supports custom domains, and automatically builds Jekyll sites for you, it made sense. Unfortunately, ease often comes with caveats. In the case of Pages, there were three main caveats for me:

No SSL on custom domains

In the age of HTTPS Everywhere, it’s a major drawback not to be able to support SSL on your website. Even Google has said that HTTPS sites will get preferential placement in search results. I know that my site is just a static application with no forms or POSTs, but even having an SSL certificate that shows this domain is really owned by me is nice for verification purposes.

No custom caching headers

Now, to be fair, GitHub does use Fastly as their CDN. However, the speed benefits are apparently somewhat lost if you use a naked subdomain instead of something like “www”. Additionally, they set their max-age value for assets to only 600 seconds. The font files on my website will never change, and the CSS (and even existing pages) very rarely change. I wanted to be able to tell web browsers to fetch these files from the browser cache for up to one year and prevent retrieving them over the network.

Jekyll is forced into safe mode

GitHub runs Jekyll in safe mode by default, for obvious reasons. This means that you are unable to use Jekyll Plugins on GitHub Pages. This wasn’t actually a huge deal for me, as I was able to get by somewhat easily without introducing custom plugins. But there’s some nice stuff out there in the Jekll ecosystem.

Onward to AWS!

With these issues in mind, I decided to try moving my site to AWS. I knew that this would involve hosting my built site in an Amazon S3 bucket and, in order to have effective caching and serve my site over a CDN, I’d need to put a CloudFront distribution in front of it. I didn’t think this process would be as tricky to get right, but it ended up being a bit arduous. However, by the end, I was left with mostly what I wanted:

• My site now runs on SSL at https://davidcel.is/
• If a user visits my site on HTTP, they are redirected to HTTPS.
• If a user visits my site with a www subdomain, they are redirected to a naked subdomain.
• CSS files are versioned with a query parameter (I wanted fingerprints but couldn’t get existing third party plugins to work for my purposes).
• After the first visit, browser caching has most pages typically loading and fully rendered in less than 100ms (blog posts with embedded media are still a bit slower, but what are you gonna do).

Want the same sort of setup for your own site? Here’s how I did it:

s3_website

s3_website is a Ruby gem that facilitates the publishing of a static site to S3 and CloudFront. It supports Jekyll out of the box and automatically detects a site built and placed in the _site directory of a project. It also has a fairly sane default configuration and has decent usage documentation. I thought that my search was over when I found this project, but I hit a few snags and had to do too much reading in CloudFront’s documentation to let anybody else do the same. Anyway, start by installing s3_website and generating a configuration in your project:

gem install s3_website
s3_website cfg create

This will give you a YAML configuration file in the root of your project. Read through it to see various configuration options you might want to set, but I’ll go ahead and give you my final configuration in a bit. In the meantime, head over to the AWS Console. You’ll need to generate an access key and secret if you haven’t already. Under the “Security & Identity” group of apps, enter the “Identity & Access Management” (IAM) app. Click “Users” and create one. Then, click through to your new user and attach a policy. The one you’ll want is called “AdministratorAccess”. Or you can just give it access to S3 and CloudFront. Finally, visit your new user’s Security Credentials tab and create an access key. This is the only time that the access key’s secret will be shown, so make sure to grab them both before you leave.

Alright. Once you’ve gotten your credentials and read through the default configuration, you can take a look at what I’ve got:

s3_id: <%= ENV['DAVIDCELIS_S3_ID'] %>
s3_secret: <%= ENV['DAVIDCELIS_S3_SECRET'] %>
s3_bucket: <%= ENV['DAVIDCELIS_S3_BUCKET'] %> # Your bucket's name (e.g: example.com)
s3_endpoint: <%= ENV['DAVIDCELIS_S3_ENDPOINT'] %> # The S3 endpoint (e.g: us-east-1)

cloudfront_distribution_id: <%= ENV['DAVIDCELIS_CLOUDFRONT_DISTRIBUTION_ID'] %>
cloudfront_invalidate_root: true
cloudfront_distribution_config:
  default_cache_behavior:
    min_TTL: <%= 60 * 60 * 24 %>
    viewer_protocol_policy: redirect-to-https
  aliases:
    quantity: 1
    items:
      CNAME: davidcel.is

index_document: index.html
error_document: 404.html

max_age:
  "css/*": <%= 60 * 60 * 24 * 365 %>
  "fonts/*": <%= 60 * 60 * 24 * 365 %>
  "images/*": <%= 60 * 60 * 24 * 365 %>
  "*": <%= 60 * 60 * 24 %>

gzip:
  - .html
  - .css

I placed any sensitive configuration options in environment variables so that anybody viewing my site’s repository on GitHub could see what values I have set. s3_id and s3_secret are the Access Key/Secret that you created for your new AWS user. s3_bucket and s3_endpoint configure your bucket’s name and which of Amazon’s data centers it will be in (there’s a list of these endpoints available in Amazon’s documentation). The bucket doesn’t have to exist yet; s3_website will create it for you! In fact, you can leave cloudfront_distribution_id out too as s3_website will also create your CloudFront distribution for you. In the meantime, let’s walk through the rest of these values to get a sense of what’ll be going on.

This cloudfront_invalidate_root option is necessary if you’re using pretty URLs that don’t involve an index.html file (e.g. https://example.com/about/ instead of https://example.com/about/index.html). This makes sure that when the, say, /about/index.html file changes, CloudFront invalidates the cache for the resource located at /about/. Handy!

The cloudfront_distribution_config section contains settings for the CloudFront distribution and its behaviors. I’ve set the minimum caching TTL to one day, and stated that HTTP requests should be redirected to HTTPS. Then, I’ve provided one CNAME alias: my domain, davidcel.is.

Then I tell S3 that my site’s root document (located at https://davidcel.is/) is the top-level index.html file, and that any errors experienced should render the top-level 404.html file.

Finally, I’ve set some additional caching rules. CSS, fonts, and images will all get a max-age of a whole whopping year, so browsers are instructed to retrieve them from their local cache until that age has expired. This is great for font files and images which should never change. CSS changes do occur, so I end up versioning those files using query parameters (more on that later). Last but not least, I tell CloudFront to deliver my HTML and CSS files gzipped. Normally you’d include .js in that list, but the only JavaScript I use is for analytics.

Once you’ve got the configuration you want, you can run s3_website cfg apply. This command will use your AWS credentials on Amazon’s API to create your S3 bucket and CloudFront distribution with the configuration we just talked about.

Finally, you can build and publish your site:

jekyll build
s3_website push

Any time you update your website and re-build it, just use s3_website push to update it in CloudFront. The command will calculate the diff of your new site with the old site and update only the files that need to be invalidated. If you want to invalidate everything, you can add the --force option.

HTTPS

In order to serve HTTPS from your own domain name, you’ll need to get an SSL certificate. I won’t detail the process of obtaining one, but I will recommend two sources:

1. StartSSL offers free SSL certificates that last for a year. Sign-up is a bit time consuming and involves creating and saving an SSL certificate just to identify you and authenticate on their website, but I’d rather go through their process than pay exorbitantly for a certificate. Their certificates require a subdomain, but will also work on a naked domain. I’d suggest getting a certificate with the “www” subdomain which would, for example, work for both “www.example.com” and “example.com”.
2. Let’s Encrypt has been making the rounds lately, promising free, automated SSL certificates. They’ll even be issuing wildcard certificates! The caveat here is that it’s still a beta product, and certificates currently only last for 90 days.

Once you have your SSL certificate, you have to upload it to Amazon. I spent ages trying to figure out how to do this in their web console, but it’s much easier to do it from the command line. You’ll need a few files, the names of which I’ll assume:

• Your newly generated SSL certificate (ssl.crt)
• The SSL certificate’s private key (ssl.key)
• Your provider’s Certificate Authority (CA) bundle (ca-bundle.pem)

When you’ve got these files ready, install the AWS Command Line Interface and go to town (replacing example.com with your domain):

brew install awscli
export AWS_ACCESS_KEY_ID= # same value as s3_id from earlier
export AWS_ACCESS_KEY_SECRET= # same value as s3_secret from earlier
aws iam upload-server-certificate \
  --server-certificate-name example.com \
  --certificate-body file:///full/path/to/ssl.crt \
  --private-key file:///full/path/to/ssl.key \
  --certificate-chain file:///full/path/to/ca-bundle.pem \
  --path /cloudfront/example.com/ # /cloudfront/ is necessary for detection

Once the certificate is uploaded, head back to the AWS Console and navigate to the CloudFront app. Click into the distribution that was created for your site, click “Edit”, and select “Custom SSL Certificate (stored in AWS IAM)”. Make sure the certificate in the dropdown is selected, and save.

Hook up your DNS

The last step is pointing your domain’s DNS at your newly created CloudFront distribution. In the console, grab your distribution’s “Domain Name” value. It should look something like “a2ebi9s43z9v9o.cloudfront.net”. You’ll want the following two records, replacing example.com with your own domain and a2ebi9s43z9v9o.cloudfront.net with your actual CloudFront distribution’s domain name:

• An ALIAS record for example.com pointing to a2ebi9s43z9v9o.cloudfront.net. If your DNS provider doesn’t support ALIAS records, your only choice is a CNAME. Unfortunately, CNAMEs can’t be used to point naked domains at another domain, so you’ll need something like www.example.com or blog.example.com.
• A URL record for www.example.com pointing to example.com. This is only if you want to redirect a www subdomain to a naked domain. If your DNS provider doesn’t have this record type, the only solution I can think of is routing www.example.com to a lightweight VPS running nginx that just redirects to example.com.

There’s also a big caveat with that URL record. These records can’t redirect over SSL, so if somebody were to visit https://www.example.com, it’ll hang in the web browser. I personally want the redirection and, because my site has never been on SSL before, I doubt there are any links out there in the wild to the https://www version of my website. I’m willing to risk it, but you might not be. I also tried to set up a second origin in CloudFront for the www subdomain that would read out of a second S3 bucket named www.davidcel.is (merely a mirror for my original bucket) but I couldn’t get it to quite work right. If anybody else has, please email me with steps so I can put them here! Otherwise, I’d recommend the VPS solution if you don’t mind paying or if you already have a box. That Nginx configuration could be as simple as this:

server {
  listen 80;
  server_name www.example.com;
  return 301 https://example.com$request_uri;
}

server {
  listen 443;
  server_name www.example.com;
  return 301 https://example.com$request_uri;

  ssl on;
  ssl_certificate /path/to/ssl.crt;
  ssl_certificate_key /path/to/ssl.key;
}

Wait a bit…

Once CloudFront finishes deploying your cached distribution and DNS kicks in, you should have a nice, fast site running low-cost on AWS. Rejoice!

One of the most popular solutions for spatial databases is PostGIS. PostGIS is a PostgreSQL extension that adds support for spatial data types and queries. This should suit our needs perfectly! Let’s start by opening up your command line and installing both PostgreSQL and PostGIS, and then initializing a new database! If you’re on macOS, you can install both using Homebrew; if you’re on Ubuntu, you can use apt-get.

# Mac OS X
brew install postgresql postgis
createdb nutsy

# Ubuntu
sudo apt-get install postgresql-9.4-postgis-2.1
createdb nutsy

Alternatively, if you use Docker, there’s a community-maintained PostGIS Docker Image!

Anyway, now we’ve got a PostgreSQL database named nutsy. Normally you’d want to use some sort of migration utility to help you manage your database schema, but let’s just jump into our PostgreSQL console to play around. Just type psql -d nutsy in your command line. You should see something like:

psql (14.6)
Type "help" for help.

nutsy=#

What we want to end up with is a table we can use to keep track of your past foraging spots. PostGIS gives us a few special data types to store things like points, lines, or polygons on a coordinate. The main two data types are geometries and geographies. Geometries are used to represent objects in a Euclidean coordinate system. Geographies, on the other hand, are used to represent objects in the round-earth coordinate system. While either geometries or geographies could make sense for the task at hand, our specific calculations will be made easier if we use geographies.

CREATE EXTENSION "postgis";

CREATE TABLE foraging_spots (
  id          SERIAL                 PRIMARY KEY,
  coordinates GEOGRAPHY(POINT, 4326) NOT NULL,
  nuts        INT
);

In the first line of this file, we’re enabling usage of the PostGIS extension in our database, allowing us to use its functions and types. Then we create a table named foraging_spots with an auto-incrementing ID, coordinates, and an optional number of nuts found in that spot. The coordinates takes advantage of the GEOGRAPHY type given to us by PostGIS. That type takes two parameters:

1. A type modifier. Valid choices here are POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, and MULTIPOLYGON. We’re representing points on a coordinate system, so we passed in POINT.
2. A unique Spatial Reference Identifier (SRID). 4326 is the SRID for the geographic latitude/longitude coordinate system of the Earth and is a default for many PostGIS types, so we’ll keep that as our SRID as well.

Now that we have our geography column, we can start adding some locations that we’ve foraged in. I see you live in New York City’s Central Park. That wouldn’t be my first choice, but… Whatever makes you happy! Maybe you get a lot of free food from the tourists. Anyway, let’s add a few spots in the park that you’ve searched. Five should be enough:

--- A little island in Turtle Pond
INSERT INTO foraging_spots (nuts, coordinates) VALUES (4, ST_GeographyFromText('POINT(-73.968504 40.779741)'));

--- Around the Obelisk
INSERT INTO foraging_spots (nuts, coordinates) VALUES (9, ST_GeographyFromText('POINT(-73.965393 40.779640)'));

--- One of the softball fields in the Great Lawn
INSERT INTO foraging_spots (nuts, coordinates) VALUES (17, ST_GeographyFromText('POINT(-73.966256 40.780602)'));

--- In the Shakespeare Garden
INSERT INTO foraging_spots (nuts, coordinates) VALUES (12, ST_GeographyFromText('POINT(-73.969861 40.779859)'));

--- In "The Ramble", whatever that is!
INSERT INTO foraging_spots (nuts, coordinates) VALUES (7, ST_GeographyFromText('POINT(-73.969046 40.776146)'));

Okay! Now we have five foraging spots in our new table. This uses the ST_GeographyFromText function given to us by PostGIS. This takes a string representation of a POINT and uses it to construct a native Geography that can then be stored in our table. So this ends up taking the form of ST_GeographyFromText('SRID=4326;POINT(longitude latitude)')). In our examples, we left out the SRID declaration because an SRID of 4326 is assumed; we don’t have to specify it.

Now, let’s say you go out on another run. You remember that you found a good number of nuts out in the Great Lawn, so you venture there again. It’s full of softball fields! You remember that you searched near one of them, but you don’t remember which. You want your database to be able to tell you if you start searching near the same field as last time. Your first thought is probably to put a uniqueness constraint¹ on your coordinates field:

CREATE UNIQUE INDEX coordinates_gix ON foraging_spots (coordinates);

Now, let’s see what happens when we try to create a point with the same coordinates as last time:

--- That same softball field in the Great Lawn
INSERT INTO foraging_spots (coordinates) VALUES (ST_GeographyFromText('POINT(-73.966256 40.780602)'));
--- ERROR:  duplicate key value violates unique constraint "coordinates_gix"
--- DETAIL:  Key (coordinates)=(0101000020E61000009A982EC4EA63444015E46723D77D52C0) already exists.

Our index works! … If you input the exact same coordinates. However, you are quite thorough in your searching. When you’ve added a point to your database, you know that you conducted a search that extends in a 50 meter radius around that point. Because geographic coordinates are so precise, it’s unlikely that you would try to search in the exact same spot. But if you try to start a new search even a single meter away from the old one…

--- That same softball field in the Great Lawn, but shifted eeever so slightly...
INSERT INTO foraging_spots (coordinates) VALUES (ST_GeographyFromText('POINT(-73.966246 40.780612)'));
--- INSERT 0 1

Uh oh. That point is very close to our original point in the Great Lawn… Definitely within 50 meters of it. So how do we prevent ourselves from this sort of overlap? Unfortunately, PostGIS doesn’t seem to give us any sort of index or constraint that lets us do this easily… However, PostGIS does give us a nice function we can try to utilize: ST_DWithin().

ST_DWithin() takes two geographies and, in the case of our SRID, a number of meters. It will return true if the two geographies are within that number of meters of each other, and false if not. This isn’t usable from an index or uniqueness constraint, but we can use it in a different way. At a high level, we’re going to create a sort of check that raises an exception if a foraging spot is within 50 meters of an existing one. Then, we’re going to make PostgreSQL call that function every time a foraging spot has been added or modified.

CREATE FUNCTION check_distance() RETURNS trigger AS $check_distance$
  BEGIN
    IF (SELECT 1 FROM foraging_spots WHERE ST_DWithin(NEW.coordinates, foraging_spots.coordinates, 50)) THEN
      RAISE EXCEPTION 'You already search this spot! Go somewhere else!';
    END IF;

    RETURN NEW;
  END;
$check_distance$ LANGUAGE plpgsql;

CREATE TRIGGER check_distance
  BEFORE INSERT OR UPDATE ON foraging_spots
  FOR EACH ROW
  EXECUTE PROCEDURE check_distance();

Okay, there’s a lot going on here! First, we’ve just created a function named check_distance() that will compare a new foraging spot with all existing foraging spots. The new foraging spot is helpfully assigned to NEW because we’ll be using the function as a trigger (more on that in a bit). We execute a query as a conditional check:

SELECT 1 FROM foraging_spots WHERE ST_DWithin(NEW.coordinates, foraging_spots.coordinates, 50). If the new foraging spot is found to be within 50 meters of any existing foraging spot, that statement returns 1, passes the condition, and will raise an exception with a helpful message. If the two spots are sufficiently far apart, the condition fails and our function simply returns the new foraging spot.

Finally, we use the CREATE TRIGGER statement to tell PostgreSQL that we want this function called before any INSERT or UPDATE statement is called on the foraging_spots table. We want it to be called FOR EACH ROW as opposed to FOR EACH STATEMENT, since a single statement could insert multiple rows. For each row added in an INSERT statement or modified in an UPDATE statement, check_distance() will be called with NEW assigned with the new or updated values.

We should be good to go with this new distance check. Let’s remove that last foraging spot that we added in error, and try adding it again:

--- Delete our bad foraging spot (it should just be the last one)
DELETE FROM foraging_spots WHERE id IN (SELECT id FROM foraging_spots ORDER BY id DESC LIMIT 1)
--- DELETE 1

--- Now, try to add it again
INSERT INTO foraging_spots (coordinates) VALUES (ST_GeographyFromText('POINT(-73.966246 40.780612)'));
--- ERROR:  You already search this spot! Go somewhere else!

Great! Now when you add a location that you’re going to start foraging in, your database will tell you if it’s within 50 meters of a spot you already searched. If it is, you should probably go somewhere else! Happy hunting!

With the recent release of Discourse, an excellent piece of forum software by Jeff Atwood, Robin Ward and Sam Saffron, I was eager to get an installation up and running myself. With a simple layout it seemed like just what I needed as support and discussion forums for a side project. Discourse is still considered to be squarely in the beta phase, so I had a hard time finding any guides or tutorials that fit my needs for deploying Discourse to a VPS. I did find a nice tutorial by Christopher Baus, but I wanted to get a setup using Capistrano rather than init.d because that feels much more Railsy to me. Read on for how I got a Digital Ocean droplet running Discourse on thin with zero-downtime deployments.

Set up your VPS

If you’re a Chef wiz and already have some recipes to provision servers for running Rails apps, you can skip ahead to the section on setting up Discourse. Chef is still on my learning TODO list, so I provisioned a server from scratch.

I started with a Digital Ocean droplet running the latest release of Ubuntu 13.04, but presumably any VPS running a recent release (12 04 or 12.10) of Ubuntu should do. Make sure your server has at least 1GB of RAM; you’ll need it to compile all of Discourse’s assets. I chose the 1GB / 1CPU plan, and it’s served me well thus far.

Secure your server

Linode already has an excellent guide on hiding your server from prying eyes, so I suggest following it to the T. I don’t want to reinvent the wheel here, so this is what I followed exactly. I set my username to goodbrews for deploying anything related to that site.

Change your server’s hostname

Depending on what characters you included in your droplet’s name, DigitalOcean may not set the hostname correctly. To change my server’s hostname, I did the following:

echo forums.goodbre.ws | sudo tee /etc/hostname

Because I deployed my installation to forums.goodbre.ws, I changed the first line of my /etc/hosts/ file to:

127.0.0.1 localhost forums.goodbre.ws

Create a swapfile

If you’re running on a VPS service that automatically provisions a swap (such as Linode), you can skip this. On Digital Ocean, however, you’ll need to do this yourself. 1GB of RAM was enough for my first deployment but, when I tried to run a second, I wasn’t able to allocate enough memory to compile assets. Creating a swap, however, has fixed this issue for me. Let’s create a 512MB swapfile:

sudo dd if=/dev/zero of=/swapfile bs=1024 count=512k
sudo mkswap /swapfile
sudo swapon /swapfile

Then, you’ll need to edit /etc/fstab and paste in the following line:

/swapfile       none    swap    sw      0       0

Finally, prevent your swapfile from being readable:

sudo chown root:root /swapfile
sudo chmod 0600 /swapfile

Install Ruby 2.0.0-p195 and bundler

SSH into your VPS as your deployment user and install the necessary dependencies to build Ruby on Ubuntu:

sudo apt-get install build-essential openssl libreadline6 libreadline6-dev \
  curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev \
  sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev libgdbm-dev \
  ncurses-dev automake libtool bison subversion pkg-config libffi-dev

Now, I personally enjoy using rbenv and ruby-build, so this tutorial will follow installation prodecures around those tools. Feel free to use RVM, chruby, or some other Ruby version manager if you wish.

# Install rbenv
git clone git://github.com/sstephenson/rbenv.git ~/.rbenv

# Setup rbenv initialization
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile

# Restart your shell to use rbenv
exec $SHELL -l

# Install ruby-build
git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build

# Install Ruby 2.0.0-p195 and set it as your user's default
rbenv install 2.0.0-p195; rbenv global 2.0.0-p195

NOTE: the installation guide for rbenv mentions using ~/.profile instead of ~/.bash_profile, but I found the latter to work for me. You can try either if you have issues.

Finally, you’ll need bundler.

gem install bundler --no-rdoc --no-ri

Install nginx

I prefer to run my Rails apps under nginx. Plus, Discourse comes with a sample configuration file that requires little tweaking. No brainer!

sudo apt-get install nginx

Install PostgreSQL, Redis, and other required libraries

Discourse requires PostgreSQL and Redis to run, so let’s install those as well:

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 redis-server \
  libxml2-dev libxslt-dev libpq-dev make g++

Note that postgresql-contrib-9.1 will install the hstore extension for PostgreSQL, which Discourse requires. You’ll need to create a postgres role and Discourse’s production database:

# Create your postgres role (I named mine goodbrews, go figure)
sudo -u postgres createuser goodbrews -s -P

# Create discourse's production database (I named mine discourse_production)
createdb -U goodbrews discourse_production

And now your VPS should be set up and ready to run Discourse! That means it’s time to…

Set up Discourse

Because you’ll need to configure it yourself, Discourse is a prime candidate for forking. If you don’t have Ruby installed on your own computer, yet, I recommend installing rbenv locally. The same steps above will work (or follow rbenv’s installation guide), so make sure you have Ruby 2.0.0-p195 installed.

Then, clone your fork locally (on your own computer).

git clone git@github.com:$GITHUB_USERNAME/discourse.git
cd discourse
gem install bundler
bundle install

Edit the Discourse configuration files

Copy the sample configuration files…

cp config/database.yml.sample config/database.yml
cp config/nginx.sample.conf config/nginx.conf
cp config/redis.yml.sample config/redis.yml
cp config/environments/production.sample.rb config/environments/production.rb

… and edit them with your own information. database.yml should, of course, be edited to use the postgres role that you configured earlier (I also changed the name of the production database to discourse_production as I mentioned earlier).

redis.yml is unlikely to change if you’ve followed this guide and are running it on the same server.

production.rb is where you’ll want to configure how your Discourse installation’s email gets sent out. Personally, I use Mandrill and configured my app to hit their API.

nginx.conf is trickier depending on your needs, however. For instance, I set my discourse installation up to use HTTPS and provisioned SSL certificates from StartSSL. For the purposes of this guide, we’ll stick with basic HTTP. I prefer deploying my applications into the home directory of my deployment user to ensure proper ownership, so my nginx.conf file ended up looking like:

upstream discourse {
  server unix:///home/goodbrews/discourse/shared/sockets/thin.0.sock max_fails=1 fail_timeout=15s;
  server unix:///home/goodbrews/discourse/shared/sockets/thin.1.sock max_fails=1 fail_timeout=15s;
  server unix:///home/goodbrews/discourse/shared/sockets/thin.2.sock max_fails=1 fail_timeout=15s;
  server unix:///home/goodbrews/discourse/shared/sockets/thin.3.sock max_fails=1 fail_timeout=15s;
}

server {
  listen 80;
  gzip on;
  gzip_min_length 1000;
  gzip_types application/json text/css application/x-javascript;

  server_name forums.goodbre.ws;

  sendfile on;

  keepalive_timeout 65;

  location / {
    root /home/goodbrews/discourse/current/public;

    location ~ ^/t\/[0-9]+\/[0-9]+\/avatar {
      expires 1d;
      add_header Cache-Control public;
      add_header ETag "";
    }

    location ~ ^/assets/ {
      expires 1y;
      add_header Cache-Control public;
      add_header ETag "";
      break;
    }

    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  Host $http_host;

    # If the file exists as a static file serve it directly without
    # running all the other rewite tests on it
    if (-f $request_filename) {
      break;
    }

    if (!-f $request_filename) {
      proxy_pass http://discourse;
      break;
    }
  }
}

The max_fails=1 fail_timeout=15s statements on the end of the upstream thin servers is what will help us achieve zero-downtime deployments. As we’re phasing out old thin processes with new ones, any failures on an old thin process will cause nginx to cease handing it requests. It will, instead, hand the request to an old thin process that’s either still running or one of the new thin processes that has already started.

Configure thin

You’ll also want a thin configuration file to make your life easier:

thin config -C config/thin.yml --servers 4 -e production

This will generate a file, config/thin.yml, which you should then go edit. Make sure to set the chdir, log, pid, and socket keys to the correct paths. My thin configuration looks like this:

---
chdir: /home/goodbrews/discourse/current
environment: production
address: 0.0.0.0
port: 3000
timeout: 30
log: /home/goodbrews/discourse/shared/log/thin.log
pid: /home/goodbrews/discourse/shared/pids/thin.pid
socket: /home/goodbrews/discourse/shared/sockets/thin.sock
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
servers: 4
daemonize: true
onebyone: true

Make sure you add the onebyone: true key. This is the secret sauce that will make your Thin servers restart… You guessed it… One by one. Zero downtime!

Create a secret token

Discourse doesn’t ship with a secret token set, since it’s not recommended to be version controlled. Therefore, you’ll need to generate one and set it yourself:

bundle exec rake secret

Open up config/initializers/secret_token.rb and paste the generated secret somewhere (probably line 10). You can delete the rest of the file.

Set up Capistrano

Add the following to Discourse’s Gemfile:

gem "capistrano", require: nil
gem "capistrano-rbenv", require: nil

Create a Capfile in Discourse’s root directory:

load "deploy" if respond_to?(:namespace)
load "deploy/assets"
Dir["vendor/plugins/*/recipes/*.rb"].each { |plugin| load(plugin) }
load "config/deploy"

Create your deployment recipe file at config/deploy.rb. Here’s what mine looks like:

# Require the necessary Capistrano recipes
require "capistrano-rbenv"
require "bundler/capistrano"
require "sidekiq/capistrano"

# Repository settings, forked to an outside copy
set :repository, "git@github.com:goodbrews/forums.git"
set :deploy_via, :remote_cache
set :branch, fetch(:branch, "master")
set :scm, :git
ssh_options[:forward_agent] = true

# General Settings
set :deploy_type, :deploy
default_run_options[:pty] = true

# Server Settings
set :user, "goodbrews"
set :use_sudo, false
set :rails_env, :production
set :rbenv_ruby_version, "2.0.0-p195"

role :app, "forums.goodbre.ws", primary: true
role :db,  "forums.goodbre.ws", primary: true
role :web, "forums.goodbre.ws", primary: true

# Application Settings
set :application, "discourse"
set :deploy_to, "/home/#{user}/#{application}"

namespace :deploy do
  # Tasks to start, stop and restart thin. This takes Discourse's
  # recommendation of changing the RUBY_GC_MALLOC_LIMIT.
  desc "Start thin servers"
  task :start, :roles => :app, :except => { :no_release => true } do
    run "cd #{current_path} && RUBY_GC_MALLOC_LIMIT=90000000 bundle exec thin -C config/thin.yml start", :pty => false
  end

  desc "Stop thin servers"
  task :stop, :roles => :app, :except => { :no_release => true } do
    run "cd #{current_path} && bundle exec thin -C config/thin.yml stop"
  end

  desc "Restart thin servers"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "cd #{current_path} && RUBY_GC_MALLOC_LIMIT=90000000 bundle exec thin -C config/thin.yml restart"
  end

  # Sets up several shared directories for configuration and thin's sockets,
  # as well as uploading your sensitive configuration files to the serer.
  # The uploaded files are ones I've removed from version control since my
  # project is public. This task also symlinks the nginx configuration so, if
  # you change that, re-run this task.
  task :setup_config, roles: :app do
    run  "mkdir -p #{shared_path}/config/initializers"
    run  "mkdir -p #{shared_path}/config/environments"
    run  "mkdir -p #{shared_path}/sockets"
    put  File.read("config/database.yml"), "#{shared_path}/config/database.yml"
    put  File.read("config/redis.yml"), "#{shared_path}/config/redis.yml"
    put  File.read("config/environments/production.rb"), "#{shared_path}/config/environments/production.rb"
    put  File.read("config/initializers/secret_token.rb"), "#{shared_path}/config/initializers/secret_token.rb"
    sudo "ln -nfs #{release_path}/config/nginx.conf /etc/nginx/sites-enabled/#{application}"
    puts "Now edit the config files in #{shared_path}."
  end

  # Symlinks all of your uploaded configuration files to where they should be.
  task :symlink_config, roles: :app do
    run  "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
    run  "ln -nfs #{shared_path}/config/newrelic.yml #{release_path}/config/newrelic.yml"
    run  "ln -nfs #{shared_path}/config/redis.yml #{release_path}/config/redis.yml"
    run  "ln -nfs #{shared_path}/config/environments/production.rb #{release_path}/config/environments/production.rb"
    run  "ln -nfs #{shared_path}/config/initializers/secret_token.rb #{release_path}/config/initializers/secret_token.rb"
  end
end

after "deploy:setup", "deploy:setup_config"
after "deploy:finalize_update", "deploy:symlink_config"

namespace :db do
  desc "Seed your database for the first time"
  task :seed do
    run "cd #{current_path} && psql -d discourse_production < pg_dumps/production-image.sql"
  end
end

after  "deploy:update_code", "deploy:migrate"

These should be all of the necessary Capistrano recipes you need to run your first deployment. Let’s do that now!

Deploy Discourse

Okay, you’ve got your server set up and Discourse configured… Let’s do this!

# Set up the server's directory structure for Capistrano
cap deploy:setup

# Seed the database you created earlier
cap db:seed

# Do the first deploy! No server is running yet, so do a cold deployment
cap deploy:cold

Configure Discourse

Congratulations! You should now have Discourse running on your VPS. Now that your forums are up and running, you’ll want to follow the Quick Start Guide that the Discourse team has written up.

Keeping Discourse up-to-date

You’ll, of course, want to keep your copy of Discourse up-to-date with the main repository. To do so, cd into your local copy and set up an upstream remote:

git remote add upstream git@github.com:discourse/discourse.git
git fetch upstream
git merge upstream/master
git push origin master
cap deploy

And there you go. Whenever you want to update, just fetch upstream, merge it into master, push, and redeploy.

Anything here wrong?

Please let me know if the above steps didn’t work for you. It’s possible that I accidentally left a step out of my process or that the process has changed and I don’t know about it. Reply to my thread on meta.discourse.org where I’ve posted this guide, or shoot me an email using the link at the top.

In 2012, goodbre.ws was featured in The Huffington Post and Lifehacker; with those features came a small horde of new users, and I quickly found myself with 7000 new accounts. This was quite a change from humble beginnings with only a couple hundred friends, classmates and colleagues. Unfortunately, with all of these new people came a few problems. First, my background jobs to refresh recommendations slowed waaay down. I eventually discovered an I/O bottleneck in the background worker that was hitting both PostgreSQL and Redis more than it reasonably should have been. However, as more and more people were getting their recommendations, I saw my server’s RAM usage get worse and worse. It wasn’t long before the amount of RAM that Redis was trying to use had exceeded the amount of RAM on my server (1 GB). I couldn’t reasonably afford larger servers, especially at this rate of growth, and I was forced to take goobre.ws down.

I started doing a lot of thinking about my Redis usage and what could possibly be causing it to use so much memory. The first thing I considered was the length of my keys. Typical redis keys in my instance looked something like recommendable:users:1234:liked_beers. Okay. Multiply that by five for each user (for dislikes, bookmarks, hidden beers, etc.) and there’s a lot of repetition in the key names. They’re also quite long. Maybe Redis was eating memory by storing tens of thousands of really long key names in RAM? I decided to try shortening them to a more compact format: u:1234:lb for example.

With lots of hope, I renamed my keys and restarted Redis. Hopes dashed: that reduced memory usage by a meager 0.01 GB. That’s 10 MB which, for RAM, may be worth exploring again in the future. However, it obviously wasn’t my main problem.

Being a fairly junior engineer at the time, optimization wasn’t a rabbit hole I’d had to go down many times. I was hardly an expert, and I let my own self-consiousness and self-doubt get in the way of doing real testing. I immediately jumped to conclusions that maybe Redis wasn’t the tool I should be using. Maybe I should revert to storing ratings in PostgreSQL and accept what would certainly be a large performance hit during recommendation generation (Redis was perfect for this in my case because I was using set math in a binary rating system).

I toyed with the idea of finding some other data store. At the time, I couldn’t find a comparable key-value store that had the features I needed from Redis, namely both sets and sorted sets with the various operations I relied on for matching user similarities. The SET and ZSET data structures were just far too perfect for my usage. But what could I do? Redis obviously was becoming too expensive for me. I would have to find something else.

I thought about moving my ratings into a Neo4j graph database. It could make for an interesting way of generating recommendations, like a simple graph traversal out from a user to connected (similar) users to find beers that those users like frequently. That might even be faster, but I worried that the recommendations themselves wouldn’t be as good.

I also thought about moving the ratings back into PostgreSQL and initializing some sort of Ruby Set mapping when the Rails app booted up, but that would probably take just as much memory if not more. I’d just be moving RAM usage from Redis into Ruby.

Finally, the day before originally writing this post, I did what I should have done in the first place: I downloaded a memory profiling tool built for Redis that would give me key-by-key memory usage stats. What I discovered was surprising, only because it outlined a problem I remember thinking about so long ago that I thought I had already addressed it.

My issue was how much data I was retaining in the sorted sets (ZSETs) I was creating. Each user got two ZSETs. One was used to store user similarities, pairing other users’ IDs with a calculated similarity value as the rank. The other ZSET stored recommendations, pairing beer IDs with the probability of the user liking that beer. In each ZSET, I was keeping those values for every other user and for every other beer. Multiply that by what became a database of 7000 users and 60000 beers and, well, you can guess what happened. Let’s just say that a lot of these sets were over 1 MB each.

I thought I was already truncating the ZSETs filled with similarity values by using a k-Nearest-Neighbor setting that I had introduced to Recommendable. That setting uses some specified number of similar users when generating recommendations as opposed to every user. Enabling that setting reduced the size of each similarity set from around 7000 values to 200 (100 similar users and 100 dissimilar users).

Additionally, I implemented a setting to specify how many recommendations should be kept at any one time for each user. I only ever show 10 recommendations, so maintaining those probabilities for every single beer was ridiculous. I reduced that to 100 as well so people can immediately get more recommendations if they rate their current ones. After truncating all of the sets to their specified lengths, I watched in awe as the memory Redis had been consuming dropped from 1.5 GB to 50 MB.

If you yourself are a Recommendable user, definitely make use of the nearest_neighbors, furthest_neighbors, and recommendations_to_store settings!

Some people, when confronted with a problem, think, “I know, I’ll use regular expressions.” Now they have two problems.

— Jamie Zawinski

Here’s a fairly common code sample from Rails Applications with some sort of authentication system:

class User < ActiveRecord::Base
  # This regex is from https://github.com/plataformatec/devise, the most
  # popular Rails authentication library
  validates :email, format: {with: /\A[^@]+@([^@\.]+\.)+[^@\.]+\z/}
end

If you’re experienced at Regex, this seems simple. If (like me when I first saw this) you AREN’T experienced at Regex, it takes a while to parse. But believe me, it can get way worse…

class User < ActiveRecord::Base
  validates :email, format: {with: /^(|(([A-Za-z0-9]+_+)|([A-Za-z0-9]+\-+)|([A-Za-z0-9]+\.+)|([A-Za-z0-9]+\++))*[A-Za-z0-9]+@((\w+\-+)|(\w+\.))*\w{1,63}\.[a-zA-Z]{2,6})$/i}
end

Or even worse still…

class User < ActiveRecord::Base
  validates :email, presence: true, email: true
end

class EmailValidator < ActiveModel::EachValidator
  EMAIL_ADDRESS_QTEXT           = Regexp.new "[^\\x0d\\x22\\x5c\\x80-\\xff]", nil, "n"
  EMAIL_ADDRESS_DTEXT           = Regexp.new "[^\\x0d\\x5b-\\x5d\\x80-\\xff]", nil, "n"
  EMAIL_ADDRESS_ATOM            = Regexp.new "[^\\x00-\\x20\\x22\\x28\\x29\\x2c\\x2e\\x3a-\\x3c\\x3e\\x40\\x5b-\\x5d\\x7f-\\xff]+", nil, "n"
  EMAIL_ADDRESS_QUOTED_PAIR     = Regexp.new "\\x5c[\\x00-\\x7f]", nil, "n"
  EMAIL_ADDRESS_DOMAIN_LITERAL  = Regexp.new "\\x5b(?:#{EMAIL_ADDRESS_DTEXT}|#{EMAIL_ADDRESS_QUOTED_PAIR})*\\x5d", nil, "n"
  EMAIL_ADDRESS_QUOTED_STRING   = Regexp.new "\\x22(?:#{EMAIL_ADDRESS_QTEXT}|#{EMAIL_ADDRESS_QUOTED_PAIR})*\\x22", nil, "n"
  EMAIL_ADDRESS_DOMAIN_REF      = EMAIL_ADDRESS_ATOM
  EMAIL_ADDRESS_SUB_DOMAIN      = "(?:#{EMAIL_ADDRESS_DOMAIN_REF}|#{EMAIL_ADDRESS_DOMAIN_LITERAL})"
  EMAIL_ADDRESS_WORD            = "(?:#{EMAIL_ADDRESS_ATOM}|#{EMAIL_ADDRESS_QUOTED_STRING})"
  EMAIL_ADDRESS_DOMAIN          = "#{EMAIL_ADDRESS_SUB_DOMAIN}(?:\\x2e#{EMAIL_ADDRESS_SUB_DOMAIN})*"
  EMAIL_ADDRESS_LOCAL_PART      = "#{EMAIL_ADDRESS_WORD}(?:\\x2e#{EMAIL_ADDRESS_WORD})*"
  EMAIL_ADDRESS_SPEC            = "#{EMAIL_ADDRESS_LOCAL_PART}\\x40#{EMAIL_ADDRESS_DOMAIN}"
  EMAIL_ADDRESS_PATTERN         = Regexp.new "#{EMAIL_ADDRESS_SPEC}", nil, "n"
  EMAIL_ADDRESS_EXACT_PATTERN   = Regexp.new "\\A#{EMAIL_ADDRESS_SPEC}\\z", nil, "n"

  def validate_each(record, attribute, value)
    unless value =~ EMAIL_ADDRESS_EXACT_PATTERN
      record.errors.add(attribute, options.fetch(:message, "is invalid"))
    end
  end
end

Yeesh. Is something that complex really necessary? If you actually check the Google query I linked above, people have been writing (or trying to write) RFC-compliant regular expressions to parse email addresses for years. They can get ridiculously convoluted as in the case above and, according to the specification, are often too strict anyway.

Sections 3.2.4 and 3.4.1 of the RFC go into the requirements on how an email address needs to be formatted and, well, there’s not much you can’t do in your email address when quotes or backslashes are involved. The local string (the part of the email address that comes before the @) can contain any of these characters: ! $ & * - = ^ ` | ~ # % ' + / ? _ { }

But guess what? You can use pretty much any character you want if you escape it by surrounding it in quotes. For example, "Look at all these spaces!"@example.com is a valid email address. Nice. Technically, even the @ and domain aren’t strictly required. If omitted, the email address is assumed to be local and mail could still be deliverable to the same machine using that address.

For those reasons, I just run any email address against a much more lenient regular expression:

class User < ActiveRecord::Base
  validates :email, format: {with: /@/}
end

Simple, right? While email addresses don’t technically have to have an @ symbol, almost every application will require it; you’ll be sending email remotely, after all. This is often the most I do and, when paired with a confirmation field for the email address on your registration form, can alleviate most problems with user error. But what if I told you there were a way to determine whether or not an email is valid without resorting to regular expressions at all? It’s surprisingly easy, and you’re probably already doing it anyway.

Just send them an email already

No, I’m not joking. Just send your users an email. The activation email is a practice that’s been in use for years, but it’s often paired with complex validations that the email is formatted correctly. If you’re going to send an activation email to users, why bother using a gigantic regular expression?

Think about it this way: I register for your website under the email address qwiufaisjdbvaadsjghb@gmail.com. C’mon. That’s probably going to bounce, courtesy of the preeminent mail daemon, but the formatting is fine; it’s a valid email address. To fix this problem, you can implement an activation system where, after registering, I am sent an email with a link I must click. This is to verify that I actually own that email address before my account is activated. At this point, why keep parsing email addresses for their format? The result of sending an email to a badly formatted email address would be the same: it’ll bounce. If your user enters a bad email address, they won’t get the activation email and they’ll try to register again if they really care about using your site. Of course, having too many of your emails bounce can lead to your domain being marked as spam, so this approach should be combined with rate limiting to avoid being blacklisted.

So eschew your fancy regular expressions already. If you really want to check email addresses right on the signup page, you can include a confirmation field so they have to type it twice. Better yet, use a library like mailcheck to perform a bit of client-side validation that catches some potential common errors. It comes down to is this: if your user enters a bad email address, you shouldn’t make it more of a problem for yourself than you have to. A complex regex validation on the email address doesn’t introduce an additional solution, it introduces an additional problem. If you really, really want to make sure people are typing in an email address, just use the /@/ regular expression and call it done. If that makes you nervous, then check for the dot too: /.+@.+\..+/i. But then send an email! Anything more is overkill.

Previously, the Inflector could handle only one set of rules at a time. Rails provides a lengthy list of singularization and pluralization rules for English, but what if somebody wants to specify how certain words in a different language should be pluralized? In Rails 3, they had two options: they could define the words as irregularities in config/initializers/inflections.rb or add them one by one into locale files. The former is bad because it involves mixing two languages into one set of rules, and the latter can lead to large, cluttered locale files when internationalizing a website.

As of Rails 4, however, I’m happy to offer a better solution for Rails developers in the process of internationalization: a multilingual Inflector. It can manage a complete set of inflection rules for each locale! Rails will still only provide a list of inflections in English, but it’s now much easier to specify your own sets of rules for additional locales, or even to have them provided to you via a gem. You can, as previously, specify these rules in config/initializers/inflections.rb:

# You can pass any locale code as a parameter. This defaults to `:en`,
# but here we’re adding rules for Spanish (`:es`)!
ActiveSupport::Inflector.inflections(:es) do |inflect|
  inflect.plural(/$/, 's')
  inflect.plural(/([^aeéiou])$/i, '\1es')
  inflect.plural(/([aeiou]s)$/i, '\1')
  inflect.plural(/z$/i, 'ces')
  inflect.plural(/á([sn])$/i, 'a\1es')
  inflect.plural(/é([sn])$/i, 'e\1es')
  inflect.plural(/í([sn])$/i, 'i\1es')
  inflect.plural(/ó([sn])$/i, 'o\1es')
  inflect.plural(/ú([sn])$/i, 'u\1es')

  inflect.singular(/s$/, '')
  inflect.singular(/es$/, '')

  inflect.irregular('el', 'los')
end

After specifying our ruleset for Spanish, we can see it in action by passing the same locale code to methods defined by the inflector:

'avión'.pluralize      # => 'avións'
'avión'.pluralize(:es) # => 'aviones'
'luz'.pluralize        # => 'luzs'
'luz'.pluralize(:es)   # => 'luces'

Note that the Inflector will still default to English and use :en as the locale unless specified, despite the existence of the I18n.default_locale configuration. This is to avoid breaking applications that have been wired internally to use English pluralization rules for mapping between Model, Controller, and table names.

Where can I get rules for other locales?

I’m happy to offer a solution for this too! I’ve created a gem called Inflections that I’m hoping can serve as a central repository for inflection rules of all locales. Unfortunately, I’m not a master of linguistics. Aside from English, I am only comfortable with Spanish, and so that’s the only additional set of inflection rules I myself have provided (aside from a shorter list of English inflections that I believe to be stripped to the essentials). If you or anybody you know are a Rails developer and fluent in another language, please consider forking Inflections, making a list of inflections (lib/inflections/<locale>.rb) with tests (test/<locale>_test.rb) and opening a pull request.

There are a lot of grammatical rules to consider when converting between the singular or plural form of various words, so what does the Inflector consider? There must be some magic involved, right? According to the documentation, Rails defines these inflections directly in ActiveSupport… lib/active_support/inflections.rb to be exact. Let’s take a looksy, shall we?

#--
# Defines the standard inflection rules. These are the starting point for
# new projects and are not considered complete. The current set of inflection
# rules is frozen. This means, we do not change them to become more complete.
# This is a safety measure to keep existing applications from breaking.
#++
#--
# Defines the standard inflection rules. These are the starting point for
# new projects and are not considered complete. The current set of inflection
# rules is frozen. This means, we do not change them to become more complete.
# This is a safety measure to keep existing applications from breaking.
#++
module ActiveSupport
  Inflector.inflections(:en) do |inflect|
    inflect.plural(/$/, "s")
    inflect.plural(/s$/i, "s")
    inflect.plural(/^(ax|test)is$/i, '\1es')
    inflect.plural(/(octop|vir)us$/i, '\1i')
    inflect.plural(/(octop|vir)i$/i, '\1i')
    inflect.plural(/(alias|status)$/i, '\1es')
    inflect.plural(/(bu)s$/i, '\1ses')
    inflect.plural(/(buffal|tomat)o$/i, '\1oes')
    inflect.plural(/([ti])um$/i, '\1a')
    inflect.plural(/([ti])a$/i, '\1a')
    inflect.plural(/sis$/i, "ses")
    inflect.plural(/(?:([^f])fe|([lr])f)$/i, '\1\2ves')
    inflect.plural(/(hive)$/i, '\1s')
    inflect.plural(/([^aeiouy]|qu)y$/i, '\1ies')
    inflect.plural(/(x|ch|ss|sh)$/i, '\1es')
    inflect.plural(/(matr|vert|ind)(?:ix|ex)$/i, '\1ices')
    inflect.plural(/^(m|l)ouse$/i, '\1ice')
    inflect.plural(/^(m|l)ice$/i, '\1ice')
    inflect.plural(/^(ox)$/i, '\1en')
    inflect.plural(/^(oxen)$/i, '\1')
    inflect.plural(/(quiz)$/i, '\1zes')

    inflect.singular(/s$/i, "")
    inflect.singular(/(ss)$/i, '\1')
    inflect.singular(/(n)ews$/i, '\1ews')
    inflect.singular(/([ti])a$/i, '\1um')
    inflect.singular(/((a)naly|(b)a|(d)iagno|(p)arenthe|(p)rogno|(s)ynop|(t)he)(sis|ses)$/i, '\1sis')
    inflect.singular(/(^analy)(sis|ses)$/i, '\1sis')
    inflect.singular(/([^f])ves$/i, '\1fe')
    inflect.singular(/(hive)s$/i, '\1')
    inflect.singular(/(tive)s$/i, '\1')
    inflect.singular(/([lr])ves$/i, '\1f')
    inflect.singular(/([^aeiouy]|qu)ies$/i, '\1y')
    inflect.singular(/(s)eries$/i, '\1eries')
    inflect.singular(/(m)ovies$/i, '\1ovie')
    inflect.singular(/(x|ch|ss|sh)es$/i, '\1')
    inflect.singular(/^(m|l)ice$/i, '\1ouse')
    inflect.singular(/(bus)(es)?$/i, '\1')
    inflect.singular(/(o)es$/i, '\1')
    inflect.singular(/(shoe)s$/i, '\1')
    inflect.singular(/(cris|test)(is|es)$/i, '\1is')
    inflect.singular(/^(a)x[ie]s$/i, '\1xis')
    inflect.singular(/(octop|vir)(us|i)$/i, '\1us')
    inflect.singular(/(alias|status)(es)?$/i, '\1')
    inflect.singular(/^(ox)en/i, '\1')
    inflect.singular(/(vert|ind)ices$/i, '\1ex')
    inflect.singular(/(matr)ices$/i, '\1ix')
    inflect.singular(/(quiz)zes$/i, '\1')
    inflect.singular(/(database)s$/i, '\1')

    inflect.irregular("person", "people")
    inflect.irregular("man", "men")
    inflect.irregular("child", "children")
    inflect.irregular("sex", "sexes")
    inflect.irregular("move", "moves")
    inflect.irregular("zombie", "zombies")

    inflect.uncountable(%w(equipment information rice money species series fish sheep jeans police))
  end
end

As you can see from the comment, this file is mostly no longer touched; the last commit that did touch this file was in 2017, but Rails hasn’t accepted changes to the rules since possibly Rails 2 or even Rails 1. So, although this is a snapshot of Rails’ inflections in the most recent release (7.0.4), it’s unlikely to ever change. And, well… Yikes. This brings me to what I really want to discuss: the state of Rails’ Singularization and Pluralization rules. I think it’s a mess.

Pluralization in English is not regular

There are only a few basic rules in English for pluralization. Because we’re speaking in terms of text, I’ll try to keep these rules based on characters rather than sounds. However, one important rule does depend on “sibilant” sounds, which are defined as a sound made by directing air through the sharp edge of your teeth and your tongue (i.e. “sh”, “ss”, “dge”, etc.). While prevalent, this can be difficult to detect in text and there are definitely edge cases.

The rules

• If the word ends with a “sibilant” sound, the plural form ends with “es” (dish → dishes or kiss → kisses) or “s” if the word already ends with an “e” (such as fridge → fridges or judge → judges)
• Most words that end with an “o” preceded by a consonant pluralize as “oes” (potato → potatoes, avocado → avocadoes).
• Most words that end with a “y” preceded by a consonant pluralize as “ies” (lady → ladies, berry → berries)

Aside from these rules, however, all other regular plurals are achieved by adding an “s”.

Some exceptions

• Words of foreign origin are exempt from the “oes” rule (piano → pianos, zero → zeros, kimono → kimonos).
• Proper nouns that end with a y are exempt from the “ies” rule (Germany → Germanys, Cody → Codys).

These are just two sets of exceptions, however, and these are moreso rules that are exceptions to other rules. English pluralization is riddled with other exceptions that are inconsistent:

• Some words that end in an “f” have that “f” mutated to a “v” during pluralization (calf → calves, shelf → shelves, leaf → leaves) due in part to the evolution of old/middle English to standard English.
• Some words with double “o”s replace those “o”s with “e”s (goose → geese, foot → feet).
• Many words are both singular and plural (buffalo, money, sheep, series, fish, coffee) and are therefore uncountable.
• Some words can even be pluralized multiple ways depending on context (indices/indexes, staffs/staves)! That’s a case that the Rails Inflector can never hope to get right.
• And, of course, some words are just plain irregular (child → children, man → men, mouse → mice, datum → data, etc.).

How can Rails hope to consider all of these exceptions when English is such an irregular and fluid language? How should Rails handle the edge cases and irregularities? The answer is simple: it shouldn’t.

The inflector should be based on rules, not exceptions

The current inflections that Rails defines are riddled with both rules and exceptions. The file has become such a mess, and so many people were submitting pull requests (#7086 #345 #3930 #3910 #6820 #2457 and the list goes on and on and on…) to either fix inflections or add new ones, that inflections in Rails are now frozen. From the documentation for ActiveSupport::Inflector:

The Rails core team has stated patches for the inflections library will not be accepted in order to avoid breaking legacy applications which may be relying on errant inflections. If you discover an incorrect inflection and require it for your application, you’ll need to correct it yourself.

This makes sense, but I feel that this situation is unfortunate, and it seem like the Rails core team agrees. Don’t get me wrong, many of these pull requests should be closed. A common response to these patches is “Rails cannot possibly include all inflections by default,” and I completely agree. Rails has already found itself in a situation where it has defined way too many inflections that are exceptions or irregularities, such as ox → oxen, crisis → crises, and the aforementioned case of index → indices (even though this pluralization is purely contextual). Many of the “rules” defined in Rails’ inflections are really exceptions. Some of these exceptions are narrow and affect only one or two words. Some of the exceptions admittedly make sense, but should instead be defined as irregularities rather than singular/plural inflections. I’ll gloss over why some of the current inflections don’t make sense:

• axis → axes, testis → testes: these are special rules that should be defined as irregularities.
• octopus → octopi, virus → viri: these special rules are actually disputed, as octopuses and viruses are more used and accepted.
• octopi → octopi, viri → viri, oxen → oxen: these words are not singular, so pluralization should not even be attempted.
• buffalo → buffaloes, tomato → tomatoes, hive → hives, alias → aliases, status → statuses: these all follow regular pluralization rules and shouldn’t have needed to be defined as special cases.
• matrix → matrices, vertex → vertices, index → indices: indices and indexes are both accepted depending on the context, and it’s likely that neither matrices nor vertices are used enough in Rails applications to warrant a special rule.
• quiz → quizzes: similar to the above; this is an irregularity.
• mouse → mice, louse → lice: exceptional words that are unlikely see the light of day in most Rails applications.
• news ⇄ news: this could have just been defined as an uncountable rather than a pluralization rule.
• ox → oxen: A special rule that should be an irregularity, but is also unlikely to be used in most Rails applications

As you can see, we have numerous inflections defined as special cases even though they follow a regular pluralization rule. Many shouldn’t be defined in the first place, even as irregularities instead of singularization/pluralization rules. Many involve words that most Rails applications will never need to use. The “Zombie” rule, for example, was only added because a website devoted to Rails tutorials, Rails for Zombies, noticed that generators were singularizing “zombies” as “zomby” due to “zombies” being irregular. Perhaps a better idea would have been to take that as an opportunity to provide a quick lesson on inflections to new Rails users. Oh, and don’t even get me started on the ridiculously added, archaic plural form of “cow”: inflect.irregular('cow', 'kine')

Of course, there are some exceptions and irregularities that make sense to define. To name a few, “child” is a frequently-used term in programming and computer science, “person” is a somewhat frequently-used model name in Rails applications, and “half” or “life” are also used frequently in programming depending on the area. I won’t argue that they should be defined within the framework as irregularities.

Why not .unfreeze, or at least fix, the default inflections?

When I originally wrote this post, Rails 4.0 was on its way, which I thought would be a great opportunity to clean up the default inflections. Isn’t a major release the perfect time to eschew the worry of breaking existing applications for the betterment of the framework? Rails is a huge piece of software; any upgrades should be done with caution. In fact, most major (or even minor) version bumps of Rails now involve detailed instructions and tools to help users upgrade their applications. Do inflections really need to be backwards compatible as long as the CHANGELOG, documentation, and upgrade guides/tools make these changes clear? Upgrading Rails versions has required extreme caution for many years now.

Until then, a better set of defaults:

Until Rails core decides its time to clean up inflections, I’ve provided a more sane set of singularization and pluralization rules in the form of a gem:

davidcelis/inflections

Here’s the difference:

• 4 pluralization rules (down from 21)
• 5 singularization rules (down from 27)
• 3 irregularities (down from 7)
• 1 uncountable (down from 10)

Ahhh… Much better.

Collaborative filtering

If you aren’t familiar with collaborative filtering, its a technique used in recommendation engines to predict (filter) the interest of a user by collecting data about the interests of many users (collaborating). There are many different types of collaborative filtering, but for our system of likes and dislikes, we’ll be focusing on memory-based collaborative filtering, which uses ratings submitted by users to calculate the similarity between those users.

Even within just the scope of memory-based collaborative filtering, there are a number of algorithms or techniques to calculate similarity. A few of the more widely used algorithms or formulae include Euclidean Distance, Pearson’s Correlation, Cosine-based vector similarity, and the k-Nearest Neighbor algorithm. These are all well documented and multiple example implementations are available should you wish to know more. They’re all great for the heavily-used five-star system and, while they’d work fine for likes and dislikes, we have an interesting alternative that I feel is better suited! So let’s talk about an algorithm I don’t see used often but which works great for our binary system. Let’s talk about the Jaccard similarity coefficient!

Jean-Luc Jaccard?

No, no, no; we’re talking about Paul Jaccard, a botanist that performed research near the turn of the 20th century. Jaccard’s research led him to develop the coefficient de communauté, or what is known in English as the Jaccard index (or Jaccard similarity coefficient).¹ The Jaccard index is a simple calculation of similarity between sample sets. Where the aforementioned collaborative filtering algorithms can quickly become mathematically complex, the Jaccard index is rather simple! It can be described as the size of the intersection between two binary sample sets divided by the size of the union between the same sample sets. Whew! That description might be a little difficult to follow, so here’s how to represent it in math:

$$ J(u_1,u_2)=\frac{\left |u_1 \bigcap u_2\right |}{\left |u_1\bigcup u_2 \right |} $$

This formula can rather intuitively be used with likes and dislikes! Let’s say we’re comparing two users: u₁ and u₂. How does one intersect two users? How does one union them? Well, we don’t want to intersect or union the people themselves; this isn’t Mary Shelly’s Frankenstein! If we’re using the Jaccard index for collaborative filtering, we want both of these operations to deal with the users’ ratings. Let’s say that the intersection is the set of items which both users have rated. The union would then be the combined set of items that either u₁ or u₂ has rated. But how does this work with the actual ratings? Let’s modify the formula a bit to deal with the likes and dislikes themselves:

$$ J(u_1,u_2)=\frac{\left |L_{u1} \bigcap L_{u2}\right |+\left |D_{u1} \bigcap D_{u2}\right |}{\left |u_1\bigcup u_2 \right |} $$

Now we’re getting somewhere! What we’ve got now is looking more collaborative and filtery for sure. We find the number of items that both u₁ and u₂ like, add it to the number of items that both u₁ and u₂ dislike, and then divide that by the total number of different items that u₁ and u₂ have rated. This is a great start, but we can go even further to match users up.

Birds of a feather flock together, and opposites repel

We may be defining our similarity as our agreed upon interests and disinterests, but what about our disgreements? If our shared likes and dislikes are important factors in calculating our similarity, we can use discrepancies in our ratings to incorporate disimilarity into our calculations. To show you what I mean, let’s tweak the formula a bit more, shall we?

$$ J(u_1,u_2)=\frac{\left |L_{u1} \bigcap L_{u2}\right |+\left |D_{u1} \bigcap D_{u2}\right |-\left |L_{u1} \bigcap D_{u2}\right |-\left |D_{u1} \bigcap L_{u2}\right |}{\left |u_1\bigcup u_2 \right |} $$

Whew! This looks a lot more complex than the original formula, but we can walk through it together. Now, in addition to finding the agreements between u₁ and u₂, we’re finding their disagreements! The agreements between u₁ and u₂ are the same as before. Their disagreements are conversely defined as the number of items that u₁ likes but u₂ dislikes and vice versa. All we do is subtract the number of disagreements from the number of agreements, and divide by the total number of items liked or disliked across the two users.

Previously, when we were calculating similarity only based on agreements, our coefficient would have been bounded between 0 and 1. However, now that we’re accounting for disagreements, our bounds have expanded to being between -1 and 1. You would have a -1.0 similarity value with your polar opposite (e.g. your evil twin that has rated the same items as you, but each one differently) and a 1.0 similarity value with a very fresh clone of yourself (you have both rated the same items in the same ways).

Okay, read my mind!

Now that we can reduce the tender, loving relationship between two people to a cold, indifferent number, let’s use that number to predict whether you’ll like or dislike something. Neat! Let’s say we want to predict how you’ll feel about thing. We get every user in our system that has rated thing and start calculating a kind of hive-mind sum. Don’t be afraid, though; this isn’t really a hive mind or intelligent AI! Anyway, if a user liked thing, we add your similarity value with them to the sum. If they disliked it, we subtract instead! The idea behind this is that if someone with tastes similar to yours likes thing, you’ll probably like it too. If they dislike thing, you’re less likely to enjoy it. Likewise, if a user who has a low or negative similarity coefficient with you has rated thing, you’re likely to rate it in the opposite way. Finallym we take this sum and divide it by the total number of people that have rated thing. Done! Like before, let’s let the math speak too:

$$ P(you, thing)=\frac{\sum_{i=1}^{n_L} J(you, u_i) - \sum_{i=1}^{n_D}J(you, u_i)}{n_L + n_D} $$

In this equation: thing is the thing we want to know if you will like, n_L is the number of users that have liked thing, and n_D is the number of users that have disliked thing.

Math is cool but how about some code?

That’s fair. You’ve been very patient and I appreciate you reading all of that! Heck, even if you skipped all the way here, you’re here nonetheless. So here’s a little pseudo-implementation of Jaccardian collaborative filtering (in Ruby, of course)!

require "set"

class User
  # The collections of objects this user likes and dislikes. These are both
  # best represented using a Set.
  attr_reader :likes, :dislikes

  def similarity_with(user)
    # Set#& is the set intersection operator.
    agreements = (self.likes & user.likes).size
    agreements += (self.dislikes & user.dislikes).size

    disagreements = (self.likes & user.dislikes).size
    disagreements += (self.dislikes & user.likes).size

    # Set#| is the set union operator
    all_items = (self.likes + self.dislikes) | (user.likes + user.dislikes)

    return (agreements - disagreements) / all_items.size.to_f
  end

  def prediction_for(item)
    sum = 0.0
    item.liked_by.each { |user| sum += self.similarity_with(user) }
    item.disliked_by.each { |user| sum -= self.similarity_with(user) }

    rated_by = (item.liked_by + item.disliked_by).size

    return sum / rated_by
  end
end

This is more or less the way I do things in recommendable and, while it was still online, goodbre.ws. I did, however, tweak the algorithm in one major way. For example, in that last stage of calculating the similarity values, I actually divide by self.likes.size + self.dislikes.size. With this change, the similarity value becomes dependent on the number of items that self has rated, but not the number of items that user has rated. As such, this makes their similarity values not be reflective:

self.similarity_with(user) == user.similarity_with(self)
# => false unless self.ratings.size == user.ratings.size

My reasoning behind this is that newer users who have not had a chance to submit likes and dislikes for many objects shouldn’t be punished for simply being new; recommendations for new users can be pretty bad! Say I’ve submitted ratings for five items, you’ve submitted ratings for fifty, and four of these items are the same. If we share the same ratings for three of those items, I want our similarity coefficient to be high. I’m new here, and it’ll potentially help me get better recommendations faster. On the other hand, your fifty ratings means you’ve seen things. You don’t really need the same jump start that I do, so your similarity value with me can stand to be lower.

The Conclusioning

The Jaccard index can be a very intuitive way to compare people when your rating system is binary. The other algorithms I mentioned are pretty cool too, but likes, dislikes and set math were just made for each other. They’re like peanut butter and jelly. Bananas and Nutella™. Bored people and reality television. It’s a beautiful partnership that I hope can last forever.

The ★★★★★ scale

At its core, any star rating scale is just a numeric rating scale with some number of options. The five-star scale is arguably the most classic of its kind (psychologically, five options “feels” like a nice, round number), so it’s not surprising that a lot of websites use it. Most big e-commerce sites like Amazon, eBay, or stores powered by Shopify allow you to rate products using five stars. IMDB uses a ten-star scale, which may as well be a 5-star scale that allows half stars. Untappd uses a five-star scale but with the granularity of quarter stars, which gives you 20 options. There are a lot of numeric systems, but they all have the same, core issues.

Ambiguity and uncertainty of the scale

One of my big gripes about numeric scales like the five-star scale is the ambiguity behind the ratings that you are allowed to give. What exactly distinguishes between three stars and four stars? What’s enough to push your rating up to that next star? What’s enough to pull it down? Because of a lack of clarity, star ratings can end up being highly subjective. It’s easy to end up with two people who give an item the same three-star rating when they actually feel differently about that item. Some websites attempt to handle this reasonably; back when Netflix still used a five-star scale, they presented some explanatory text for each star when hovering over while rating a movie:

1. ★☆☆☆☆ (Hated it)
2. ★★☆☆☆ (Didn’t like it)
3. ★★★☆☆ (Liked it)
4. ★★★★☆ (Really liked it)
5. ★★★★★ (Loved it)

Eventually, Netflix stopped displaying this hover text, instead letting ambiguity creep back in. That being said, even the explanatory text itself can come off as subjective. What does it mean to “really” like a movie? Why are the intervals between the options unequal, with there being no “Really disliked it” option and with the typically neutral three-star text being very much not neutral? Explanatory text can help if done correctly, but it can also add to the subjectivity of submitted ratings.

Unreliability of ratings

Because a star rating scale iteslf is so ambiguous and uncertain, the ratings end up reflecting that ambiguity and uncertainty. Many users will not use this scale as intended even with intent given in the form of explanatory text. Some users will use the scale as intended, but that usage is always based on their subjective ability to understand the way the scale should be used.

Despite this, recommendation systems will accept these ratings as statistically accurate communications. Websites with huge samples of users and ratings are less likely to be negatively affected by the unreliable nature of these ratings; as sample sets grow, that unreliability can become normalized. Smaller websites and recommendation systems experiencing the cold start, however, will suffer due to the subjective nature of their small rating samples.

Binary voting is already happening

Despite being given a scale with five possible ratings, most people tend to vote in a binary fashion anyway. Back in 2009, YouTube published some interesting data concerning the ratings that videos had been receiving. As it turns out, a huge majority of videos would receive mostly five-star ratings. I think that YouTube’s takeaway from this data was spot on:

Seems like when it comes to ratings it’s pretty much all or nothing. Great videos prompt action; anything less prompts indifference.

The second highest rating was, of course, one star; this is a great example of binary voting in the works. A lot of people give mostly five-star ratings for things they like. If they don’t like a thing, they either give it one star or just bounce without rating the thing at all. I’ve also spoken to numerous friends and acquaintances who admit to giving almost exclusively four-star ratings to things they like, and three-star ratings to things that are “just ok”. In fact, this is a wide-spread phenomenon on Tabelog, a popular website used in Japan to rate restaurants. So many reviews on Tabelog stick to the middle of the road that, on average, most of its ratings are distributed between 3.1 and 3.5 stars.

YouTube toyed with the idea of switching their rating system to a “favorites” system to “declare your love for a video”, but ultimately settled on likes and dislikes.

The binary scale (and why it’s better)

Binary rating scales have gained a lot of popularity. As mentioned earlier, YouTube has now operated on a thumbs up or down rating scale for a long time. Netflix also eventually dropped their star-rating system in favor of likes and dislikes. Reddit and other social networking sites like it use upvotes and downvotes. But what makes a binary system better than a five-star system?

Less ambiguity

The binary rating scale removes a large amount of ambiguity present in the star rating systems. Five or more subjective rating options are aggregated down into two options based on words that are easily understandable by native speakers of the language. It is much easier for a person to declare, “Hey, I like this thing” than it is to determine, “Well, I like this thing… But do I ‘three stars’ like it, or do I ‘four stars’ like it?”

Less subjectivity

A large amount of subjectivity is also removed. Ratings given that are based directly on feelings are much more likely to match than ratings given based on numbers. This can simplify a lot of situations in which two people may have similar feelings about something but rated it different:

• Me: “I liked this thing and rated it four stars.”
  Friend: “I liked this thing and rated it five stars.”
• Me: “I liked this thing and rated it three stars.”
  Friend: “I didn’t like this thing, so I only gave it three stars.”

Our feelings about something are clearly not conveyed well by more granular ratings, and they also don’t match. As I posited earlier, this will normalize as data sets grow, but this does not change that we have no way of knowing whether or not the underlying ratings are truly indicative of agreement. Given a binary scale, however, agreement is much more clear: “We both liked this thing” or “we both disliked this thing.”

People are already doing this!

Remember, even within a five-star system or other numeric systems, people are pretty much already rating in this way. Why fight it?

No middle ground

Of course, the likes and dislikes are not without their own flaws. Most notably, unless its explicitly added, there isn’t an obvious neutral ground in a binary rating system outside of abstenance. It’s often an all-or-nothing situation in which you either like something or you don’t. This may or may not be an issue for you as the implementor. Personally, when I’m ready to rate an item, I can almost always manage to categorize it into a like or dislike even if its very close. However, if I were to truly feel 100% neutral about something, I would likely ignore that thing and move on rather than rate it. If I have no feelings either way, why would I want it affecting my recommendations?

tl;dr

Embrace the binary rating system. It’s much less ambiguous and subjective than its stellar cousin, and it’s much easier for the user to deal with in general. Feelings themselves are more easily comparable than numbers indirectly based on feelings and can lead to more accurate recommendations.