Trying out the new "Least Outstanding Requests" algorithm on AWS ALBs on Intercom's busiest endpoint
Brian Scanlan
5,327 posts
Brian Scanlan
@brian_scanlan
I work at Fin.
- I really like my team's use of variable names in this monitoring configuration - the consequences are very clear.
- We’ve been retiring cloud dev environments in favour of laptop based development at Intercom over the last few months. Modern Macs are serious machines, and the ergonomics are hugely better.
- At @IntercomEng we just went three weeks without any out of hours pages. Plenty of stuff alarmed when we were in the office, but it's still very satisfying that our product and infrastructure doesn't rely on constantly interrupting engineers' lives to keep it working well.
- 52,937 RPS on Intercom's Rails monolith at 14:01 UTC today. We're coming for you GitHub! 😀Rails scaaaaales
- No big deal - just my life partner getting awarded a Fulbright scholarship for research into diversity in technology. She'll be working in @UCBerkeley this time next year!
- Replying to @zackkanterI'm not a Musk fanboi, but there's a vast distance between the two. From what I've read, Theranos was built on very little real science. Tesla and SpaceX were ambitious for sure, but were built on top of incremental progress based largely in reality.
- Last week, Intercom shipped code to production 1073 times.
- Last week Intercom took the week off as a company holiday. I helped cover oncall and incident command, and we had to deal with the grand total of zero pages and incidents.
- Doing work in an actual office for the first time in 873,264 days.
- Replying to @brian_scanlan and @IntercomEngTo get to this state we: - build on reliable AWS services and re-use patterns and infrastructure where possible - Alarm on customer facing metrics not symptoms - Practice continuous improvement, reviewing every alarm that pages somebody and take action to ensure their value
- The data is in! We're now spending $18k a year on public IPv4 addresses due to the new AWS charge.
