lynn.zone (blog)

You Probably Don't Need a VPN

Sun, 13 Nov 2022 00:00:00 GMT

Given the nature of this blog, and where I usually share my content, chances are most of the people reading this probably already know everything I’m about to say here. That being said, I really wanted to write about this because I think it’s a big problem, and I think those of us who see what’s going on have a duty to call it out. If this post ends up being helpful to even one person, I’ll consider the effort well worth it.

Also, a quick clarification: This is mainly going to be about consumer VPN products, not business VPNs, which have very different use cases and aren’t really affected by issues like predatory marketing.

The problem

I’ll get right to it — something that’s become a huge issue in recent years is predatory VPN marketing, largely aimed at non-technical audiences, and misleading to the point where I’m pretty comfortable with just calling it a scam.

You’ve almost certainly seen examples of what I’m talking about here — ads for VPN products, especially in the form of stuff like YouTube video sponsorships, trying to convince you that your online privacy and security is in grave danger if you’re not using their product. Oh no!

Before we continue, here’s a really quick explainer of what a VPN does: It lets you tunnel your connection through one of their servers (usually, you can pick where this server is), with the tunnel between you and their server being encrypted. That’s it.

Now, keeping that in mind, let’s take a quick look at the claims some of the biggest VPN vendors make on their websites:

NordVPN:

A good VPN service provides you a secure, encrypted tunnel for online traffic to flow. Nobody can see through the tunnel, get their hands on your online data, or find your real IP address and location. NordVPN is the best VPN if you’re looking for peace of mind when on public Wi-Fi. Securely access personal information or work files, encrypt your internet connection, and keep your browsing history and online identity private.

ExpressVPN:

Going online doesn’t have to mean being exposed. Whether you’re shopping from your desk or just connecting at a cafe, keep your personal information more private and secure.

Private Internet Access:

Keep your data encrypted for extra security. Don’t let anyone get their hands on unencrypted emails, pictures, banking details, or anything else best kept private.

Proton VPN:

Our secure VPN sends your internet traffic through an encrypted VPN tunnel, so your passwords and confidential data stay safe, even over public or untrusted Internet connections.

Okay, so… none of these are technically saying anything that’s not true. However, if we read between the lines here even a tiny bit, there’s several wild claims being made:

Your online activity isn’t usually encrypted.
People can “get their hands on your online data” if you’re not using a VPN.
People can “get their hands on [your] unencrypted emails, pictures, [and] banking details” if you’re not using a VPN.
Using public Wi-Fi without a VPN is unsafe.
Shopping online without a VPN is unsafe.

All of these are patently, demonstrably untrue, and they know that, which is why they’re being careful not to explicitly make any of those claims.

Why are these claims untrue? The answer is pretty simple — in 2022, basically everything you do online is already encrypted, thanks to the near-ubiquitous presence of HTTPS.

Anytime you visit a website where the little padlock shows up in your browser, and the URL begins with https:// (which, again, is basically every website — certainly every website you’re going to be doing anything remotely sensitive on), or use an app that does the same thing under the hood, your connection to that site is already encrypted with, to borrow a phrase, “military-grade encryption”. It’s effectively impossible for anyone to get past this, at least not without compromising the device you’re using in some way.

Now, HTTPS doesn’t quite encrypt all of your traffic in the way a VPN does. Even if you’re using HTTPS, someone with unrestricted access to your traffic can usually still use it to see a few things, mainly:

The IP addresses you’re connecting to.
The names of the websites you’re connecting to* (not full URLs).
Any other unencrypted data you’re sending over the network, even though this should be quite rare in 2022, and largely limited to stuff like games that don’t bother to encrypt their network packets.

It’s legitimately the case that the extra layer of encryption you get with a VPN makes it so that nobody between you and the VPN server can see these anymore. That being said, it’s still clear that the reality is much less scary than the wild claims VPN vendors make about what can happen if you’re not using their product, which is inconvenient for them, so they just lie instead.

* There are ongoing efforts to close the remaining gaps in encryption on the web that make this possible, such as DNS over HTTPS, DNS over TLS, and TLS Encrypted Client Hello, but industry-wide changes like this take years to happen, so as of 2022, these haven’t been widely deployed yet.

What can we do here?

Honestly, I’m not sure what we can really do about this. The companies that do this are quite careful to not make these claims in a way that’d make it trivial to bag them for false advertising, and that’s without even getting into the fact that many of these companies are based in jurisdictions like the British Virgin Islands (ExpressVPN) and Panama (NordVPN), where I think it’s reasonable to assume consumer protections are a little weaker than in places like Europe or North America.

Perhaps the most effective thing we can do here is hold accountable the army of YouTubers and other content creators who take sponsorships from these companies and make these claims on their behalf, often even going above and beyond the companies’ own marketing, as well as blurring the distinction between what they’re being paid to say and what they genuinely believe.

To be clear, I don’t think there’s anything inherently wrong with content creators taking sponsorships, but they have a duty to vet these sponsorships to make sure they’re not selling snake oil to their audience, and that’s clearly not happening here. I really hope we’re slowly approaching a point where people wake up to how awful a lot of these YouTube sponsorships are (not even just the ones we’re talking about here), because it sure doesn’t feel like the situation can get much worse on that front.

So, should you use a VPN?

So, should you use a VPN? Maybe! Even knowing that they don’t do nearly as much as they claim to, there are still totally legitimate reasons you might want to use one:

You live in an authoritarian state where being found to have engaged in “undesirable activity” comes with potentially grave consequences, and/or want to get around the sweeping internet censorship inevitably done by such states.
You don’t live in an authoritarian state, but still have reason to be concerned about the possibility of a middleman (your ISP, employer, school, parents, etc.) having access to even very limited data on what you’re doing online, or you want to get around filters that rely on that data.
You want to be able to hide your real IP, or make it look like you’re in a different location. This is quite useful for gaining access to region-locked content on e.g. streaming services, and probably the most common real use case for a VPN.

That being said, if you’re just worried about your passwords getting stolen because you logged into your bank account while on Starbucks Wi-Fi… don’t waste your money. You do not need a VPN to enjoy a basic level of online security.

"Opting Out" of Tracing Headers on Google Cloud Platform

Tue, 09 Aug 2022 00:00:00 GMT

If you’re using Google Cloud Run (and, from what I understand, this also applies to Google Cloud Functions and Google App Engine standard), Google will automatically create a trace in Cloud Trace for each incoming request, and inject the trace context into the request via the standard traceparent HTTP header.

This is a cool feature… if you want to use Cloud Trace. If you want to send telemetry to a different vendor, though, like Honeycomb, this is going to be a problem, because if your app is set up to pick up the trace context from the incoming request’s headers, your telemetry will contain references to “missing” root spans that only exist in Cloud Trace.

In Honeycomb, that’ll look something like this, and will prevent a lot of queries from working properly:

In my Node.js services, instrumented using OpenTelemetry and its included Node.js auto-instrumentations, I “fixed” this issue by using the hooks provided by the HTTP instrumentation to discard any headers Cloud Tracing might be setting before the incoming request gets processed into a span. You can do this by replacing your call to getNodeAutoInstrumentations() with this code:

getNodeAutoInstrumentations({
  '@opentelemetry/instrumentation-http': {
    startIncomingSpanHook: (req) => {
      delete req.headers.traceparent;
      delete req.headers[`x-cloud-trace-context`];
      delete req.headers[`grpc-trace-bin`];

      return {};
    },
  },
}),

(If you’re setting up @opentelemetry/instrumentation-http separately, you can pass the same config object to the HttpInstrumentation constructor.)

This is, unfortunately, clearly a hack, but also the least painful way I could find of getting around this issue. Other potential solutions I considered were:

Disable Cloud Trace — You can’t outright disable Cloud Trace for a project without also disabling a bunch of other services that depend on it, and I’m not sure if doing that would even actually prevent this behavior.
Use a reverse proxy to strip the headers — On Cloud Run, you could theoretically solve this by adding some kind of reverse proxy to your container, and configuring it to strip these headers before they make it to your app. This would (in my opinion, anyway) be a massively overengineered solution, though, and also negatively impact your app’s startup time, which is especially important in a serverless context.

Please let me know if you’ve also run into this and figured out a better way of getting around it!

Update: Thanks to Andreas for the suggestion of using B3 Propagation as another potential solution!

B3 Propagation is a natively-supported method for trace context propagation by OpenTelemetry, but not by GCP, so configuring OpenTelemetry to use B3 Propagation rather than the W3C Trace Context spec is a convenient, robust and non-hacky way to make sure GCP won’t mess with your telemetry.

The main drawback to this method is that you may not be able to do this if you, for example, need to interface with other systems that expect W3C Trace Context headers rather than B3 headers. It’s also possible (but, in fairness, probably not super likely) that GCP will introduce support for B3 Propagation in the future, which would potentially break this solution.

I also personally have a bit of philosophical issue with perpetuating the status quo of there being multiple competing standards for doing the same thing, but realistically, even if you’re like me and agree that there should really be one standard for this, there isn’t really much harm being done by using a feature that OpenTelemetry has already committed to supporting for the foreseeable future.

On a final note, if B3 Propagation doesn’t work for you for whatever reason, everything I’ve said here theoretically holds true for any propagation standard natively supported by OpenTelemetry that isn’t W3C Trace Context, which (at time of writing), in addition to B3 Propagation, also includes W3C Baggage and the Jaeger propagation format.

Service-to-Service Communication on Google Cloud Run with tRPC

Thu, 04 Aug 2022 00:00:00 GMT

tRPC is a TypeScript library that’s usually used to enable boilerplate-free and typesafe API calls between backend and frontend code in the same app, but it works great for service-to-service communication, too.

I recently needed to implement service-to-service communication with tRPC (while building Konbini!) on Google Cloud Run, which makes this just a touch harder to do than usual, because unless you configure the service you’re calling to allow unauthenticated requests from anyone on the internet (which, if it’s an internal service, you probably don’t want to do), requests to it need to include authentication headers for a service account that’s allowed to make calls to that service.

(You may be used to solving this problem by just putting your services on a private network that isn’t reachable from the internet, but Cloud Run doesn’t let you do that.)

Thankfully, tRPC lets us set those headers, and the Google Auth Library helps us generate them. (Note that this library only supports Node.js, so if you’re running a different JS runtime on Cloud Run, you’ll need to fetch a token from the metadata server yourself.)

Anyway, here’s the gist of what I ended up doing:

import type { AppRouter } from "../../../other-service/src/trpc";

import { createTRPCClient } from "@trpc/client";
import { auth as googleAuth } from "google-auth-library";

export const client = createTRPCClient<AppRouter>({
  url: `${process.env.OTHER_SERVICE_URL}/trpc`,
  headers: async () => {
    if (process.env.NODE_ENV === `production`) {
      const tokenClient = await googleAuth.getIdTokenClient(
        process.env.OTHER_SERVICE_URL
      );

      return await tokenClient.getRequestHeaders();
    }

    return {};
  },
});

This is very similar to the code you’d normally write to create a tRPC client, but with a bit of additional code to set the headers we need on each request.

If this seems suspiciously simple, it’s because the Google Auth Library automatically pulls the credentials it needs from the Cloud Run environment, so you don’t need to configure anything — it just gives you the headers.

Note that your requests will still fail if the service account assigned to the client service isn’t allowed to call the other service. All Cloud Run services use the default Compute Engine service account by default, though, so it should work out of the box.

Feel free to reach out if you run into problems with this approach, or if you just have other thoughts about it!

E2E Testing an App with Clerk Authentication in Cypress

Thu, 21 Apr 2022 00:00:00 GMT

Background: Clerk is a hosted authentication and user management product.

I recently started writing E2E tests in Cypress for an app that uses Clerk for authentication, and there wasn’t anything out there to guide me, so here’s what I ended up with after fiddling with it for a bit.

(Note: In my case, this is a Next.js app using Clerk’s Next.js SDK, but my understanding is that this code will work everywhere, because their client SDKs all ultimately use ClerkJS under the hood.)

I wrote a custom Cypress command that waits for Clerk to load, signs the user out if they aren’t signed out already, and then signs in with test credentials (see here for how you can set these so that they’re accessible via Cypress.env()).

Cypress.Commands.add(`initializeAuth`, () => {
  cy.log(`Initializing auth state.`);

  cy.visit(`/`);

  cy.window()
    .should((window) => {
      expect(window).to.not.have.property(`Clerk`, undefined);
      expect(window.Clerk.isReady()).to.eq(true);
    })
    .then(async (window) => {
      await window.Clerk.signOut();
      await window.Clerk.client.signIn.create({
        identifier: Cypress.env(`TEST_USER_IDENTIFIER`),
        password: Cypress.env(`TEST_USER_PASSWORD`),
      });
    });
});

If you’re using TypeScript to write your tests (which I recommend!), you’ll also want to add this command to your custom command types.

declare global {
  namespace Cypress {
    interface Chainable {
      /**
       * Initialize auth to a state where you're
       * logged in as the test user.
       *
       * @example cy.initializeAuth()
       */
      initializeAuth(): Chainable<void>;
    }
  }
}

Ultimately, you’ll probably want to use this command in a before or beforeEach hook to reset the auth state before every test, like so:

describe(`Test Page`, () => {
  beforeEach(() => {
    cy.resetDatabase(); // another custom command
    cy.initializeAuth();
  });

  // ... tests go here
});

Happy testing! Please let me know if you run into problems with this approach.

Simple, Privacy-Friendly, and Free Analytics Using Serverless Functions

Thu, 15 Apr 2021 00:00:00 GMT

Whether it’s for work or play, most of us usually want some analytics on our websites, but the available options typically come with one or more pretty major downsides:

Massive JavaScript payload — Most of the big analytics products have this problem. gtag.js is ~35kB minzipped, ~90kB uncompressed (at time of writing); some others are even worse. Yes, these days this is loaded asynchronously and doesn’t directly hold up your page while it gets loaded and parsed, but that’s still an absolutely indefensible amount of overhead just to get some super basic data on how your site is being used.

Privacy implications — Again, this is one you’re almost certainly going to hit if you’re looking at a free, hosted analytics product, like Google Analytics or Yandex.Metrica. Third-party cookies may thankfully be going the way of the dodo, but tools like Google Analytics — if you use the tracking code they give you — will still insist on giving your users at least first-party cookies they can use to identify them across multiple requests (and visits), regardless of whether you want that feature or not.

From a practical perspective, this means you now need to ask your users to consent to these tracking cookies, and from an ethical perspective, you may just want to avoid tracking your users like this altogether, especially if it means doing it through someone like Google.

Inaccuracy due to people blocking analytics — Whether it’s through privacy-enhancing browser features, browser extensions like uBlock Origin or even something like Pi-hole, lots of people these days are running something that blocks obvious analytics requests. Depending on how privacy-conscious your audience is, as many as ~45% of your visitors could be blocking analytics entirely.

Serverside analytics naturally don’t suffer from this problem, and some privacy-focused analytics products like Fathom Analytics and Plausible Analytics give you ways to use your own domain for analytics to avoid being caught by this, but if you’re using a typical big-name analytics product (unless you’re being particularly clever about it, which we’ll get to), this is a huge drawback you’re kind of just stuck with.

Cost — If you’re looking to add analytics to something like a personal site, this is probably your biggest problem. Modern, privacy-focused analytics products like Fathom Analytics, Plausible Analytics and Netlify Analytics give you ways to avoid many or even all of the other problems on this list, but they’re not free — and there’s nothing wrong with that, but I can’t justify paying a monthly subscription just to see how many people read my blog posts, and you probably can’t either.

One way to get around this could be self-hosting an open-source analytics product like Matomo Analytics, but unless you already have somewhere you can put that without incurring any extra costs, that costs money, too.

That being said, if you’re looking for an analytics solution for your business or something else that actually makes money, I think you should strongly consider just paying for one of these products and being done with it.

Enter Serverless

Okay, so what can we do about this? Well, we can be smart about the way we use analytics, and take control of how our analytics are collected while still leaving the bulk of the work to a third-party analytics product (if we want to).

How do we do that? With a serverless function.

More specifically, here’s what this approach entails:

Write a serverless function that takes an analytics event — whatever that means to you — and pipes it wherever you want. This gives you complete control over what and how much data you send, what you do with it before sending it to a third party, and where it ends up.

The most immediately useful thing you could do here is pipe the event to a traditional analytics product like Google Analytics (giving you the benefits of a proper analytics product without many of the downsides we talked about), and I imagine this is what most people will want to do, but the place you store your events can be literally anything — a log file, a database you can later build your own analytics tools on top of, hell, it could even be a spreadsheet.

You could also do stuff like pipe events to your own database while still sending them to Google Analytics, so you still own all of your data if you ever want to do something with it that Google Analytics doesn’t let you do.
Write a tiny bit of clientside code (which, again, you now have complete control over) to send events to your analytics function.

If we go over those downsides we mentioned at the start of this post again, we can see that this approach allows us to pretty much completely sidestep them:

Massive JavaScript payload — There is none, our clientside code can be as simple as a single fetch() call.
Privacy implications — More-or-less none, we don’t have to track anything we don’t want to. Google will still have your data if that’s where you choose to pipe it, but there’s not a lot they can do with it if it’s all anonymized and not tied to a specific user in any way (though you can still do that if you want to, of course).
Inaccuracy due to people blocking analytics — Our analytics requests won’t be blocked because they’re not tied to a known analytics product. As long as we don’t do something like use the word analytics in the name of our endpoint, they’ll just look like any other request.
Cost — On the analytics side this allows us to use free products like Google Analytics without many of the downsides they usually come with, and as for our serverless function, chances are you won’t ever have to pay a dime to run it with how generous pretty much every platform’s free tier is. For instance, AWS Lambda gives you one million free requests every month (at time of writing).

What This Looks Like in Practice

I have to stress that this post is much more about the general concept than it is about a specific implementation — you don’t have to do anything here the same way I did it. That being said, I’m including an example of how I implemented this pattern for my own website (veselin.dev) because I think it’ll be helpful, and because I think my usecase is common enough that many people should be able to take what I did here and carry it over to their own website(s) with few or no changes.

Here’s my analytics function. It’s a Netlify Functions function that mostly just proxies events to Google Analytics using the Measurement Protocol (note: I’m using Universal Analytics here because the Measurement Protocol for GA4 is still in alpha). It anonymizes IP addresses before sending them to Google (though it also enables Google Analytics’ IP anonymization feature that does the same thing, because why not), and sidesteps Google’s requirement of a user identifier (a cid or uid) by just generating a new one for every event.

import { Handler, APIGatewayEvent } from "aws-lambda";
import axios from "axios";
import { URLSearchParams } from "url";
import * as uuid from "uuid";
import anonymizeIP from "ip-anonymize";

interface RequestBody {
  type: "pageview" | "event";
  params: any;
}

interface Response {
  statusCode: number;
}

const handler: Handler<APIGatewayEvent, Response> = async (event) => {
  const body: RequestBody = JSON.parse(event.body);

  const analyticsRequestBody = new URLSearchParams();
  analyticsRequestBody.append("v", "1");

  // enable IP anonymization, even though we're doing it here anyway
  analyticsRequestBody.append("aip", "1");

  // GA makes us send a cid parameter, so we send a new UUID every time
  // because we don't actually want to track users across requests
  analyticsRequestBody.append("cid", uuid.v4());

  // Override user agent
  analyticsRequestBody.append("ua", event.headers["user-agent"]);

  // Override user IP (but anonymize it first)
  analyticsRequestBody.append("uip", anonymizeIP(event.headers["client-ip"]));

  // Set event data

  analyticsRequestBody.append("tid", process.env.GOOGLE_ANALYTICS_TRACKING_ID);
  analyticsRequestBody.append("t", body.type);

  for (const paramName in body.params) {
    analyticsRequestBody.append(paramName, body.params[paramName]);
  }

  // Send event to Google Analytics

  await axios.post(
    "https://www.google-analytics.com/collect",
    analyticsRequestBody.toString()
  );

  return { statusCode: 200 };
};

export { handler };

For the time being I’m happy with this solution, but you could expand on this by implementing something like Plausible’s daily identifier method (described here) to anonymously identify unique users, and sending that to Google as your uid. (Update: I’ve since published anonymous-user-id, an npm package that helps you do this!)

My clientside code is basically a single fetch() call that sends the same data Google Analytics usually sends for a pageview event. You’ll want to put this code somewhere it’ll be called both on the initial pageload and whenever the user navigates to a different page — in my case, that’s Gatsby’s onRouteUpdate hook.

if (process.env.NODE_ENV !== `production`) {
  return null;
}

const sendPageView = () => {
  const pagePath = location
    ? location.pathname + location.search + location.hash
    : undefined;

  fetch("/.netlify/functions/event", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      type: "pageview",
      params: {
        dh: window.location.hostname,
        dp: pagePath,
        dt: document.title,
        dr: document.referrer,
      },
    }),
  });
};

// wrap inside a rAF to make sure react-helmet is done with its changes
// (https://github.com/gatsbyjs/gatsby/issues/11592)

if (`requestAnimationFrame` in window) {
  requestAnimationFrame(() => {
    requestAnimationFrame(sendPageView);
  });
} else {
  // simulate 2 rAF calls
  setTimeout(sendPageView, 32);
}

I’m not sending any custom events on my site right now, but those would be handled similarly. If you use analytics heavily on your site, you’ll probably want to extract this code into a utility function rather than duplicating it every time you want to send an event.

Why Serverless?

You wouldn’t be wrong to notice that everything we’re doing here could be implemented just fine in a traditional backend architecture. So, why am I focusing on serverless specifically? Well, there’s a few reasons:

More and more sites (like the one I talk about in my example!) are being built on Jamstack or serverless architectures without a traditional backend. Using a serverless function for this works equally well regardless of what your existing stack is.
Even if you have an existing backend you could integrate this functionality into, using a serverless function has the benefit of ensuring that you don’t have to worry about the performance impact of handling all these extra requests that would typically be handled directly by a third-party service.
While this isn’t a usecase I talk about a lot in this post, a serverless function not tied to an existing backend is easier to use to collect events from a bunch of different places at once.
Circling back to the Jamstack point again, serverless functions are inherently infinitely scalable. If you have a site deployed to something like a CDN — where your site will never get hugged to death — the last thing you want to have to worry about is whether your analytics can keep up with your traffic.

That being said, of course you could apply all of these ideas to a traditional backend, too. I just focus on serverless as a solution in this post because I think it’s a great fit for this usecase.

The Ethics of Circumventing Blockers

To address a bit of an elephant in the room, yes, a significant part of why this pattern is useful is that it allows you to do analytics for users running browser extensions or other tools that would typically block them.

My opinion on this overlaps significantly with Fathom’s answer to this question in their post announcing their custom domain feature (which is explicitly designed to do this) — I don’t believe this is unethical because the reason people block analytics is that they’re traditionally extremely invasive, and if you’re doing what I’ve described here, you’re doing everything possible to protect your users’ privacy.

Or, to put it a different way — you’re not collecting any more data than serverside analytics would be, and nobody thinks serverside analytics are unethical because they can’t be blocked by a browser extension, right?

That being said, it’s possible to use the techniques I’ve talked about here in an unethical way, for example by proxying requests to Google Analytics through your own domain, but not doing anything to mitigate the privacy concerns of using Google Analytics. All I can do there is ask that you please don’t do that.

Conclusion

That’s pretty much it! I’m very curious to see how many other people have stumbled their way into similar solutions before (I know I’ve seen stuff like people proxy_passing requests to Google Analytics with nginx to make them less obvious), and I’m also curious to see if anyone comes up with more creative ways of building on top of this concept than just piping events to Google Analytics.

PHP Debugging With Xdebug 3 Inside a Docker Container

Tue, 13 Apr 2021 00:00:00 GMT

I recently spent a good few hours getting Xdebug to work with my development setup (which is PhpStorm running inside WSL 2 on Windows 10, and PHP/Xdebug running inside a Docker container, inside WSL 2, with Docker Desktop), so here I am writing up the surprisingly simple solution I ended up with — partially for my own future reference, but also to help out anyone who finds themselves in a similar situation.

(Don’t get discouraged by my ultra-convoluted setup — this configuration should actually theoretically work for pretty much any environment, for reasons I’ll get into a little later on.)

Preamble

Xdebug can be tricky to configure, because it works in reverse from the way you’re probably used to interacting with your PHP application — instead of sending requests to your PHP code, Xdebug needs to know how to send requests from where your code is running to your client application (i.e. probably your editor or IDE), in order to establish a debugging connection.

This is fine if your setup is relatively simple (i.e. Xdebug can just go to localhost:9003 and hit your client), but if your setup is a little more complicated than that — i.e. if PHP/Xdebug is running on a different physical machine, in a VM, in a container, or similar — it won’t work out of the box.

Xdebug 3 makes this whole exercise a little easier for us by generally overhauling the way Xdebug is configured to make it simpler, but if your setup is a little convoluted (like mine), it can still be finicky to get right, hence this post.

The Meat and Potatoes

Okay, so here’s what I ended up with.

This is the bit you need to put somewhere in your PHP configuration (i.e. your php.ini, or wherever you usually configure your PHP extensions):

[Xdebug]
xdebug.mode=debug
xdebug.start_with_request=yes
xdebug.discover_client_host=true
xdebug.client_host=host.docker.internal

Let’s go over this line-by-line:

xdebug.mode=debug enables step debugging (which is probably what you want to use Xdebug for.)

xdebug.start_with_request=yes tells Xdebug that we want to activate step debugging at the start of every request, for simplicity’s sake. I recommend setting this to yes and forgetting about it, but if you’ve done this before and you know you prefer to activate step debugging only for specific requests, leave this line out and Xdebug will default to trigger.

The last two lines are where it gets interesting. xdebug.discover_client_host=true tells Xdebug to attempt to extract the IP of the client from the HTTP request, and xdebug.client_host tells it what to try if that doesn’t work.

With all of these set the way we’ve set them, here’s what happens every time you send a request to your PHP application:

Xdebug activates step debugging (because our xdebug.mode is debug, and xdebug.start_with_request is set to yes).
Xdebug attempts to automatically connect to the host the HTTP request came from (using $_SERVER['REMOTE_ADDR'], $_SERVER['HTTP_X_FORWARDED_FOR'], or a custom HTTP header you’ve configured).
If the above didn’t work, it falls back to trying to connect to our xdebug.client_host, which is host.docker.internal, a DNS name for the host helpfully set for us by Docker Desktop on Windows or macOS (though unfortunately not on Linux).

The fact that Xdebug lets us set both xdebug.discover_client_host and xdebug.client_host as a fallback is the key to making this configuration work for just about any setup, whether you’re on Linux, macOS or Windows, and even whether you’re using Docker or not.

If you’re using Docker Desktop, having host.docker.internal set as a fallback host means Xdebug will always be able to find its way to the host, and on most other setups, client discovery (via xdebug.discover_client_host) should just work. Theoretically, the only scenario this config should break in is if you’re using something that breaks client discovery that isn’t Docker (e.g. a Vagrant virtual machine), in which case there’s probably something else you can set xdebug.client_host to to fix it.

Installing Xdebug in Your Docker Image

If you’re using Docker (and if you’re not, I highly recommend thinking about it!), the below is an example of what I do to install and enable Xdebug in development without carrying it over to the production image, using multi-stage builds:

# (or your base image)
FROM php:8.0-fpm-alpine as base

# whatever you do to set up your image
# for example:
#   - copying your source code
#   - installing extensions
#   - doing a `composer install`

FROM base as development

RUN pecl install xdebug && docker-php-ext-enable xdebug

FROM base as production

# whatever you do to prepare your image for production
# for example: removing packages you don't need in production

Conclusion

That’s it! You should now have an Xdebug config that works for everything from Docker Desktop to just running PHP on the same Linux machine as your editor.

If you’re using this config and you’ve found a setup it doesn’t work for, please yell at me and I’ll probably add a note to this post!

On Docs, DX, and Developer Happiness

Sun, 01 Sep 2019 00:00:00 GMT

One thing I’ve increasingly taken notice of recently is that a lot of developers, especially in corporate environments, seem to tend to put a very low priority on the DX of their projects. Documentation is poor or non-existent; setting up a working development environment is like pulling teeth. You’ve got tickets to close and you’re sure they’ll figure it out, so who cares, right?

To be clear — I totally understand why this happens, and I’m not here to judge anyone who might’ve recognized some of their own projects in that description. I do, however, strongly believe that you should care, and I’m here to tell you why.

A Little Work Now = A Lot Less Work Later

Think about how many times in your life you’re going to find yourself setting up a local dev environment for a given project. At the very least, you’re probably going to do it every time you get a new dev machine and git clone all the stuff you’re working on, right? You’re also going to do it every time you help someone do the same, e.g. a new person on your team.

If your project is a pain to set up, that quickly adds up to a lot of unnecessary frustration and wasted time. Doing this by hand is also bound to breed subtle inconsistencies in each environment, potentially leading to bugs.

On top of that, if running your app involves more than a single command (e.g. you need to run a web server and something like a module bundler), you have to remember to do this every single time you work on it.

Automate, Automate, Automate

Okay, so we’re aware that some of our processes may be inefficient. How do we fix it?

The absolute best way you can choose to deal with this issue is by codifying your app’s environment and other setup into some sort of automated tooling that can do all of this for you. Luckily, we live in the age of Docker and containerization, which means the tools for this we have at our disposal today are immensely more powerful than they were barely a few years ago.

This post isn’t really the place for a deep dive into using Docker as a development environment, but the gist of what you want to do is:

Use a Dockerfile to codify as much as you can about the immediate environment your app needs to run — this should include things like your language runtime, any binaries it needs (e.g. imagemagick), config files, and so on.
Use a docker-compose.yml to codify as much as you can about your app’s external dependencies, such as cache servers, database servers, mail servers, and so on.

This will undoubtedly take some time, especially so if you first have to take the time to understand what this Docker thing is all about. Once everything’s said and done, however, if you did everything right, you’ll have reduced the process to set up a working instance of your app to this:

git clone <your-repo> .
docker-compose up

On top of that, if you’ve already done this once, it’ll take you a lot less time to do it in the future — especially if your apps share a common stack. It takes me maybe an hour to set all this stuff up for a typical new project these days.

Learning by Example

If the above seems a little too abstract, let’s look at a simple example of a docker-compose.yml you could use for a WordPress project.

version: "3"

services:
  wp:
    image: wordpress:latest
    ports:
      - 8080:80
    volumes:
      - ./wordpress:/var/www/html:rw
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_NAME: wordpress
      WORDPRESS_DB_USER: root
      WORDPRESS_DB_PASSWORD: password
    depends_on:
      - db
    links:
      - db

  db:
    image: mysql:latest
    ports:
      - 3306:3306
    command:
      [
        "--default_authentication_plugin=mysql_native_password",
        "--character-set-server=utf8mb4",
        "--collation-server=utf8mb4_unicode_ci",
      ]
    environment:
      MYSQL_DATABASE: wordpress
      MYSQL_ROOT_PASSWORD: password

It’s self-explanatory enough — The wp service is the actual WordPress site, and the db service is a MySQL database (what we called an “external” dependency above). For the purposes of this example, this is all we need to get us to a point where you can just do a docker-compose up to run the project.

Now, you’ll notice that I didn’t provide a Dockerfile — That’s because, in this instance, the official wordpress image does all we need. For a more complex project, however (or if you wanted to deploy a Docker image to production, too), you’d probably want to write a Dockerfile that extends the official image.

A Word of Warning

This is a bit of a tangent in the context of this article, but one thing about Docker that I really need to mention, but nobody ever seems to talk about, is that the way you want to use Docker differs pretty significantly depending on whether you’re using it for development, or deploying Docker images to production.

The main thing it comes down to is where your code is. Take the volume mounted onto the WordPress container in the example above:

volumes:
  - ./wordpress:/var/www/html:rw

The WordPress code on your host machine is mounted onto the container — Any changes you make to it will immediately be reflected in your running app.

On the other hand, if you’re building an image to deploy to production, you have to copy your code into your image at build time, so that it’s self-contained, and so that it can do anything it needs to do with your code at build time (building binaries, module bundling, etc.).

I find it really important to point out this distinction whenever I get the chance to, because for some reason, a lot of high-profile tutorials out there will have you make a “development” Docker setup that has you rebuild your image every time you want it to reflect the changes in your code, even though this is totally impractical if you’re actually using Docker as your dev environment. I dread to think how many people have given up on using Docker for development because they found the idea of constantly rebuilding your app ridiculous (and rightfully so).

Another thing worth mentioning is that, depending on your stack, you may need to have separate images for development and production (you can use multi-stage builds to do this without having to maintain multiple Dockerfiles). PHP, for instance, doesn’t care — you’re using the exact same code and runtime in every environment. On the other hand, if you’re using something like a static site generator, or a language like Go that doesn’t have an external runtime, your production image is going to need to be much more lean than the one you use for development.

Automation Isn’t Everything

We’ve just spent a bunch of time talking about Docker and Docker Compose, and while containerizing your app absolutely makes a huge difference and is probably the single most helpful thing you can do to improve the onboarding experience, we should take a moment to remember that reducing your setup to a docker-compose up isn’t the be-all and end-all of making your app friendly to work with.

Here are a few other things to keep in mind.

If your app or library has any sort of public API, you must make it easy for people to see what it actually is; expecting people to dive into your code doesn’t count. For HTTP APIs, tools like the OpenAPI spec and Swagger UI make it easy to automatically generate API docs and expose them in a friendly way; use them.
Even if you’ve simplified your dev environment setup and documented any public APIs you may have, there’s probably still a bunch of stuff you could put in your README. What is this app? Who’s it for? What standards/protocols does it implement? Is it a monorepo, or are there other projects one should look at to get the full picture? Hell, take a screenshot (if applicable) and put it at the top of the README. The more context people can glean from your docs, the better.
If you can’t go the full-on containerization route for whatever reason (please make the effort to try, though!), make sure all the steps one needs to go through to set up a dev environment for your app are well-documented. Don’t assume people will just know what to do with your code, even if your app uses a similar stack to other projects in your team or company.

Think Like a FOSS Maintainer

To conclude this post, I think the single most important piece of advice I can give on this topic is — Think like a FOSS maintainer. Pretend for a second that you’re not developing an internal project that people are obligated to use, but a personal open-source project nobody will ever use if the DX is poor and the extent of your docs is a near-blank README.

To an extent, you don’t even have to pretend — Yeah, of course you don’t have to sell people on your project if they’re obligated to work with it, but that doesn’t mean the atittude you take towards the stuff in this post doesn’t matter. Making the effort to make your coworkers’ jobs easier makes a huge difference in how they see you, your project, your team, and even affects the culture of the company as a whole.

To be blunt, animosity between teams can grow from the most trivial of things; we have to make the extra effort to extend that olive branch when we can.

The way I personally see this (with the disclaimer that this is absolutely my personal opinion — I can hardly pretend to speak for everyone) is that making sure your projects are well-documented is just a sign of basic respect towards your fellow devs. I wouldn’t dream of making someone bang their head against a wall trying to work on a project of mine just because I couldn’t be bothered to write proper docs.

Recap

Let’s go over the important stuff one more time:

Do your best to automate your dev environments. This makes as much of a difference for you as it does for everyone else, especially if you switch between projects a lot.
Document your APIs. If you maintain an API other people are supposed to use, you can’t expect them to dive into your code to understand it.
Put everything in your READMEs. The more comprehensive your docs are, the less often you’ll have people tapping you on your shoulder to ask you things.
Think like a FOSS maintainer. Sell people on using your project, even if you don’t have to.

And, of course: Be excellent to each other.

Haxe + Flambe: An adventure

Fri, 27 Mar 2015 00:00:00 GMT

I recently built and released a small project, a web-based (HTML5 + Flash) game built using Haxe and Flambe.

With the project as complete as it’ll ever be, I feel like sharing my experiences with the framework. For many reasons, it was an interesting framework to work with, and it brings some really fresh ideas to the table, but I don’t think I’ll be starting up another Flambe project soon. I outline why below.

What is Flambe?

Flambe is a cross-platform (but mainly web focused) gamedev framework built on top of Haxe, a language that compiles to everything under the sun.

Now, Flambe is somewhat different in the world of Haxe-based frameworks in that it doesn’t use OpenFL (a Haxe framework that allows you to build Flash applications in Haxe and deploy them as native builds to pretty much anything) as part of the build process — rather, it rolls its own solution, limiting itself to only Flash and HTML5 as export options.

I didn’t particularly care about this limitation, as I was building a web game anyway. However, those wanting to build a desktop or mobile game using the framework will probably want to look elsewhere, as you’ll be limited to AIR’s relatively poor performance on mobile (using the Flash target), as opposed to full native performance using OpenFL (which generates C++/SDL code).

Haxe: The first hurdle

I have a love-hate relationship with Haxe. On one hand, I love that it lets me use a language structure I’m fairly familiar with (as a C#/Java dev) to write software for more-or-less anything with a processor. The whole concept is very intriguing to me. On the other hand, Haxe has one big problem: A lack of ecosystem, brought on by not being a very well known language.

You see this problem manifest itself in many ways. Probably the biggest is the lack of a truly good development environment. The only ‘real’ IDE Haxe has is FlashDevelop, which is Windows-only and simply isn’t very good. Contrast this to the IDEs available for C# (Visual Studio), Java (IntelliJ IDEA), Python (a number of IDEs, including Visual Studio), JavaScript/CoffeeScript/TypeScript (WebStorm, among many others), and you see why I feel like a proper IDE is kind of a must in 2015.

Now, this is a bit unfair to the Haxe mantainers — it isn’t their fault nobody has properly integrated a modern IDE with Haxe. But I do think it’s something people ought to consider, especially going into remotely complex projects.

The second problem Haxe has is a lack of large community, and as such a lack of general-purpose libraries and ‘knowledge base’. Libraries I can do without, since I’m only doing gamedev anyway, and the basics are there — but if you find yourself stuck on some language quirk, the Haxe documentation itself is very barebones, and there’s no guarantee you’ll find your answer somewhere on the Web, like there usually is with popular languages. You may simply be left hanging to figure it out on your own, even if the problem is extremely simple.

Someone once said that the only way to truly learn Haxe is to find someone who knows Haxe well and pester them until they’ve told you everything they know. I understand now how true this is.

What Flambe does well

There are a few features in Flambe that truly surprised me, especially in the context of an otherwise fairly barebones framework.

Live asset reloading. Change a file in an external editor and it updates in-game as soon as you hit save. This is incredibly useful for art, as you can see how it looks in the game without pressing a single button — It just works.

Animations/tweening as a function of the framework itself. Flambe makes a big deal out of most values being instances of Flambe’s ‘AnimatedFloat’ class, and it should. Any value that’s an AnimatedFloat (most properties of the Sprite class, for instance) can be tweened — with easing — by calling a single function.

On the other hand, this isn’t that big of a deal. You can get more or less the same functionality in other frameworks by using a third party tweening library, like LeanTween for Unity.

**Scenes are incredibly versatile and are very easy to transition between. **Flambe’s scene transition framework blows other systems, like Unity’s, out of the water. In Unity, you might be able to replicate what Flambe does now that asynchronous loading no longer requires Unity Pro, but it would be a large undertaking to achieve a relatively minor effect.

Want to make a scene that overlays the scene below it, and slide it in from off-screen? Flambe can do that in two lines of code. Want to keep the state of the last scene saved so you can transition back seamlessly? Flambe does that by default.

Where Flambe falls short

Despite being an interesting framework, I probably won’t be using it again in the foreseeable future, and I’d find it difficult to recommend to someone else — For a serious project, anyway. Here’s why.

**Apparent disinterest in following industry standards. **Flambe packages both Flash and HTML5 builds as thin clients that grab assets from your server at runtime. This is great if you plan on hosting the games yourself, but the issue arises when you realize that it doesn’t let you package a Flash build as a single SWF, which is how every single Flash portal expects you to package your game.

I managed to work around this by using a fork of an older version of Flambe (4.0 if I remember correctly) that supports exporting a single SWF with an included preloader. But this obviously isn’t an ideal solution, and I’d be high and dry if that developer hadn’t been generous enough to release his fork.

**Lack of support for fairly common features, while supporting more advanced features in their stead. **For instance, Flambe comes with Flump support out of the box, which is great! However, it leaves out support for traditional spritesheet and frame-based animations, which are much more standard and used by many, many more developers.

Granted, frame-based animations are fairly trivial to implement — I implemented them into my game in less than an hour. But if anything, this only raises the question of why Flambe hasn’t done them from the beginning, let alone now.

Conclusion

So, is Flambe an objectively bad framework? Of course not, no. There’s a reason it’s been used by companies like Nickelodeon and Disney, even over much more popular Haxe/web frameworks.

However, should I find myself doing a similar project in the future, I’ll likely be using an OpenFL-based framework with a longer history and larger community like HaxeFlixel, or simply OpenFL itself (the API is fairly simple to work with, or so I’ve heard).

If you’re planning on building a game for the purposes of a gamejam, or to publish on your own website, or for a client that doesn’t mind the limitations, I still encourage you to give Flambe a whirl. You may find that it’s exactly what you’re looking for.

lynn.zone (blog)

You Probably Don't Need a VPN

The problem

What can we do here?

So, should you use a VPN?

Further reading

"Opting Out" of Tracing Headers on Google Cloud Platform

Service-to-Service Communication on Google Cloud Run with tRPC

E2E Testing an App with Clerk Authentication in Cypress

Simple, Privacy-Friendly, and Free Analytics Using Serverless Functions

Enter Serverless

What This Looks Like in Practice

Why Serverless?

The Ethics of Circumventing Blockers

Conclusion

PHP Debugging With Xdebug 3 Inside a Docker Container

Preamble

The Meat and Potatoes

Installing Xdebug in Your Docker Image

Conclusion

On Docs, DX, and Developer Happiness

A Little Work Now = A Lot Less Work Later

Automate, Automate, Automate

Learning by Example

A Word of Warning

Automation Isn’t Everything

Think Like a FOSS Maintainer

Recap

Haxe + Flambe: An adventure

What is Flambe?

Haxe: The first hurdle

What Flambe does well

Where Flambe falls short

Conclusion