Colan Schwartz on Cloud Architecture, Security, Privacy & Startups

Forfeiting the Frontier: How Washington Is Ceding Its Own AI Advantage

Sat, 13 Jun 2026 00:00:00 +0000

Forfeiting the Frontier: How Washington Is Ceding Its Own AI Advantage

On 12 June 2026, Commerce Secretary Howard Lutnick ordered Anthropic to suspend its two most capable models, Fable 5 and Mythos 5, for every foreign national. The ban covered foreign nationals abroad and inside the United States alike, including the company’s own non-citizen employees. No provider can sort a live user base by nationality in real time, so the practical effect was a worldwide shutoff of both models. Anthropic’s statement says the order cited “national security authorities” but gave no written detail, only a verbal account of a narrow technique.

This episode will be litigated and probably resolved; Anthropic expects to restore access. But the important question is not whether one directive survives a court challenge. It is what a pattern of maneuvers like this does to the very thing the United States probably wants to protect: its lead in frontier AI. These maneuvers erode that lead, and that they do so precisely because of what they are: selective, improvised, and aimed at a domestic company rather than a foreign adversary.

How a lead actually erodes

A technological advantage is rarely lost the way people fear it will be, with a rival leaping ahead overnight. It erodes more slowly, from underneath. A frontier lead is not held by any single model or company. It is held by an ecosystem: the researchers willing to build their careers in one country, the investors willing to fund decade-long bets there, and the market on which raw capability turns into products, standards, and dependencies that the rest of the world comes to rely on. Capability itself moves easily from place to place. Ecosystems stay put, right up until they don’t.

What dislodges an ecosystem is not a single shock but a loss of confidence that the place is predictable. Consider the people who sustain a frontier: a founder choosing where to incorporate, an investor pricing a fifteen-year position, a researcher deciding where to spend a career, a finance ministry weighing whether to build national systems on top of an American model. Each of these is a long-term bet. And each depends less on any single government action than on how erratic the government’s behaviour is overall. A country can absorb a great deal of policy that is heavy-handed but predictable and still remain the obvious place to build. What it cannot absorb is the impression that a model serving hundreds of millions of people on a Tuesday can be switched off by letter on a Friday, on contested grounds, against one company. That impression is what dissolves the ecosystem.

These are maneuvers, not policy

It’s true that Mythos is no ordinary chatbot. Anthropic itself describes it as having found previously unknown security flaws across major operating systems and browsers. That is a real capability, and it genuinely alarmed regulators.

But the order fits neither the scope nor the logic of a genuine security measure. The model most users lost was Fable 5, the public and heavily safeguarded one. The stated reason was a “jailbreak” that, by Anthropic’s account, exposed only minor flaws that were already known, the kind of flaws that other public models, OpenAI’s GPT-5.5 among them, reveal even without any such jailbreak. A genuine security measure applies to a whole class of products. This one named a single company while its closest competitor kept operating without restriction.

That selectivity is not an accident; it is the pattern. The dispute goes back to February, when Anthropic refused to let the Pentagon use its models for autonomous weapons or domestic surveillance. The President then ordered every agency to drop the company, and the Pentagon branded it a “supply-chain risk,” a label normally reserved for foreign adversaries. OpenAI secured a Defense Department deal within hours. A federal judge later blocked those moves, finding that they “appear designed to punish Anthropic” and amounted to “classic First Amendment retaliation.”

The clearest sign is what the same government does with the same capability when it serves its own ends. The administration is reportedly running Mythos at the NSA, with Anthropic engineers embedded to support it, and the Pentagon’s own technology chief has called Mythos a “separate national security moment” that the government must harness, even as the supply-chain label remains in place. A capability cannot be too dangerous to show a foreign national in a coffee shop and, at the same time, important enough to run in the nation’s own cyber operations. In fairness, the order followed an ignored request to delay the launch, so the government did have concerns and a channel for raising them.

However, this kind of incoherence is exactly what long-term decisions weigh most heavily. A strict but consistent set of rules is a known cost, one the ecosystem can plan around. A government that bars a domestic company from offering the very capability the government itself runs, and that a rival sells freely, does not impose a known cost. It introduces unpredictability, and unpredictability is what makes a place the wrong one to build in.

Both exits cost the United States

If Anthropic stays put, it operates at the discretion of an administration willing to use its procurement and export powers against a vendor over a contract dispute. That is a precarious position for a company preparing a U.S. listing and financing tens of billions of dollars in computing power. But capability does not stay locked inside one company’s distribution. If it cannot be deployed at scale from within the United States, the frontier simply advances somewhere with fewer constraints. The rational response for any government watching is what is often called “sovereign AI”: the principle that a nation should not build its critical systems on top of technology that a foreign executive can switch off overnight. Every maneuver of this kind strengthens that argument, and funds the non-American alternatives that follow from it.

If Anthropic instead reduced its dependence on the United States, the disruption would be larger still. It would lose the U.S. market the moment a security pretext was invoked. More importantly, the American companies it connects to, the major cloud providers and enterprise channels, would be barred from dealing with a banned foreign entity, and that is a far larger commercial loss than any single contract. The honest counterpoint is that Anthropic’s reliance on U.S. capital and U.S. chips makes a clean exit more theory than plan. But that is exactly the point. A national champion that cannot easily leave is not a reassurance. It is a live demonstration, for everyone still deciding where to build, of how costly it is to be trapped.

What gets priced in

The narrow cost here is the disruption to Anthropic’s customers, which the company says it is working to reverse. The deeper cost falls on the one factor that actually decides who leads a decade from now: the willingness of talent, capital, and allied governments to commit, irreversibly, to the American frontier. That willingness depends on predictability, and predictability is exactly what a series of selective, improvised interventions destroys. Veteran critics have warned of precisely this, about this very action: they expect it to push foreign-born researchers back to their home countries and to make American AI look, to investors, like an unsafe bet. The stated goal of the entire posture is to stay ahead of China. Undermining a domestic frontier lab, while a competitor and the government itself use the same capability, is the surest way to hand that lead away.

Export controls are a tool for denying a capability to an adversary. Aimed instead at a domestic champion, over a capability the same government is running and a rival is selling without restriction, the tool points the wrong way. The capability does not disappear. It moves elsewhere, and the advantage (that was never the capability itself but the conditions that attract it) moves along with it.

Cloud Lock-in is Mostly Self-Inflicted

Thu, 04 Jun 2026 00:00:00 +0000

Cloud lock-in feels like something the market does to you. Mostly, it’s a series of small architecture defaults you accepted without weighing the trade-offs.

Two of them do most of the damage: too much cloud-specific infrastructure as code (IaC), and too many cloud-specific managed services. The providers steer you there by default, the tutorials assume it, and the path of least resistance quietly becomes the path of permanent residence.

Both are choices. Both are fixable, provided you decide they matter early, while fixing them is still cheap.

Problem 1: Cloud-specific IaC

Every cloud ships its own native IaC and its own native CI/CD, and each one is genuinely pleasant to use right up until the day you want to leave: CloudFormation, ARM/Bicep, Deployment Manager, plus the pipeline layer of GitHub Actions wired to one cloud’s identity model, Azure Pipelines tasks, and CodePipeline/CodeBuild.

The pipeline layer is where this gets worse, because pipeline code is sneaky. People think of it as glue rather than infrastructure, so it doesn’t get the same scrutiny, and it metastasizes. My last client had approximately 80% of their automation living in Azure Pipelines: not just deployment steps, but environment logic, secret handling, approval gates, the works. None of that is portable. The day they want a second cloud, or a credible negotiating position with their first one, that 80% is a rewrite, not a migration. That’s nuts, and it’s also completely normal, which is the problem.

The fix is boring and it works: write as much of your IaC as possible in Terraform (or OpenTofu) and Ansible, and treat anything cloud-specific as a liability you have to justify rather than a default you reach for.

Terraform owns the infrastructure: networks, compute, IAM, storage, the lot. One language, one state model, one mental model across every provider.
Ansible owns configuration and the imperative bits Terraform is bad at.
CI/CD becomes a thin orchestration layer whose only job is to call Terraform and Ansible. The actual logic lives in version-controlled scripts and modules, not in proprietary pipeline YAML. If your pipeline definition is more than a few steps long, you’re putting business logic in the one place you can’t take with you.

Now the honest caveat, because anyone who’s actually done this will call me on it otherwise: Terraform is not a magic portability button. The HCL, the workflow, the state management, the module patterns, your team’s muscle memory all transfer. But a Terraform aws_* resource is still an AWS resource. The provider blocks are cloud-specific by definition, and you don’t get to write resource "generic_database" and have it land on three clouds. What you get is a single tool and a single workflow wrapping every provider, so a migration becomes “rewrite the resource definitions” instead of “rewrite the resource definitions and relearn the entire tooling stack and port all the pipeline logic.” That’s a dramatically smaller blast radius, and it’s the difference between a switch you can credibly threaten and one you can’t.

Problem 2: Cloud-specific managed services

This is the deeper hook, and it’s deeper precisely because the services are good. Aurora, DynamoDB, Cosmos DB, BigQuery, SQS, the whole managed-everything catalog: these are genuinely excellent, and they remove real operational pain. That’s exactly what makes them the stickiest form of lock-in. You don’t notice the dependency forming, because every individual decision to use one was the sensible one.

The portable alternative is to run open-source services yourself and let your IaC manage them: PostgreSQL instead of a proprietary managed database, an open message broker instead of the cloud-native queue. Package it as Helm charts, manage those charts with Terraform, and the entire definition of your stateful services becomes provider-agnostic. Kubernetes ends up as the portability substrate: the cluster looks roughly the same whether the nodes underneath it are EC2, Azure VMs, or GCE, so the workload definitions ride along to wherever you point them.

And here’s where I have to argue against my own enthusiasm, because “just self-host Postgres on Kubernetes” is the kind of advice that sounds clean on social media and bites you at 3 a.m.

Managed services exist for a reason. When you self-host PostgreSQL, you now own backups, failover, point-in-time recovery, version upgrades, patching, and the pager. The cloud was charging you for that, and a lot of teams genuinely come out ahead paying the premium rather than staffing the expertise. Portability is a benefit with a recurring operational cost attached, and you should price both sides before deciding.
Kubernetes is overkill for plenty of projects. If you’re running a handful of services, standing up a cluster to win portability you’ll never exercise is a bad trade. The Terraform code stays portable either way, which is the real point, so you can get a lot of the benefit without committing to K8s as your runtime for everything.
Stateful workloads on Kubernetes are their own discipline. Operators have made this far more reasonable than it was five years ago, but a database on K8s is not a fire-and-forget proposition. Go in with eyes open.

So this isn’t “never touch a managed service.” It’s “decide deliberately.” Use a proprietary managed service when the operational savings clearly beat the lock-in cost; just know that you’re making that trade, in writing, rather than discovering it the day you try to leave.

How to actually decide

The useful question isn’t “cloud-specific or portable?” in the abstract. It’s: for this specific component, what does the exit cost, and is the convenience worth that price?

A rough hierarchy that’s served me well:

Default to portable for the foundation: IaC tooling, CI/CD logic, networking and identity patterns, your stateful core. This is the stuff that’s expensive to unwind later and cheap to get right now.
Allow cloud-specific where the value is genuinely differentiated and the alternative would be a heroic amount of undifferentiated heavy lifting. Some managed services really are better than anything you’d run yourself, and dogma here just costs you money.
Refuse to let pipeline code become load-bearing. This is the cheapest win on the list and the one people skip. Keep the proprietary layer thin.

The strategic reason this is worth the discipline ties straight back to the competition story. When a recent report called Canada’s cloud market “broken”, its sharpest recommendation wasn’t about breaking up the incumbents; it was about forcing compatibility so customers can actually switch. Regulators are circling egress fees and data portability for the same reason: lock-in gives hyperscalers pricing power over customers who can’t credibly leave. But you don’t have to wait for a regulator to hand you leverage. An architecture that could move is a negotiating position whether or not you ever pull the trigger, and most of the time you won’t need to, because the provider knows you could. The cheapest exit is the one you designed in from day one. The most expensive one is the rewrite you start the morning you finally decide you’ve had enough.

The goal was never to leave. It’s to stay because you decided to, not because you had no other option. That choice gets made early, in the boring decisions, or it doesn’t get made at all.

Announcing Drubernetes v2: Moving from Bitnami to the Official MariaDB Operator

Wed, 22 Oct 2025 00:00:00 +0000

This article was originally published on the BackUpScale blog.

Why It Matters

For many open-source projects and small teams, Bitnami’s charts were the default starting point for running DBs and applications on Kubernetes. When a large vendor changes course, it sends ripples across the ecosystem; it can suddenly make basic infrastructure harder or more expensive to maintain. Drubernetes v2 ensures that Drupal deployments remain fully open, self-contained, and future-proof, regardless of corporate licensing shifts. Community-driven alternatives are essential to preserve innovation and accessibility.

Background: Why Drubernetes Needed a v2

When we first built Drubernetes, the goal was simple: make it easy for various organizations to deploy Drupal on Kubernetes using Terraform for infrastructure automation. Our stack relied heavily on community-maintained Helm charts (most notably Bitnami’s MariaDB chart) for reliability and ease of integration.

But the open-source ecosystem around Bitnami has shifted dramatically.

Bitnami’s Policy Shift: From Open Access to Paywall

Bitnami historically maintained one of the best collections of open Helm charts in the cloud-native space. These charts were widely used for MySQL, MariaDB, Redis, WordPress, and many others, often forming the foundation of production workloads for startups and open-source projects.

However, following VMware’s 2022 acquisition by Broadcom and its ensuing restructuring, Bitnami’s open chart repositories were deprecated, and support for their community versions effectively ended. As covered in Fastcode’s analysis, Broadcom’s pivot toward expensive subscription-only licensing has created a domino effect, shuttering long-standing open-source pipelines and forcing projects like ours to re-architect.

For open-source maintainers like BackUpScale, continuing to use Bitnami’s images now involves licensing uncertainty, limited updates, instability and the risk of losing upstream security fixes.

Simply put: Bitnami’s stack is no longer a viable base for sustainable open-source projects with limited funding.

What Changed in Drubernetes v2

To keep Drubernetes fully open and future-proof, we replaced our only Bitnami dependency, MariaDB, with the official MariaDB Enterprise Operator.

You can review the full changelog on the 2.0.0 release page and discussion in Issue #3.

Migration Guide: Upgrading from v1.x to v2.0

While this release represents a major step forward, the migration process does require manual intervention due to the difference in architectures.

Please review the complete details in the release notes.

Looking Ahead

Drubernetes v2 isn’t just about keeping up with upstream changes. It’s about reinforcing the open-source foundations we depend on. By moving to the official Helm charts, we gain:

Transparent governance and roadmaps
Consistent upstream support
Easier compliance for enterprise users
Freedom from vendor lock-in

We’ll continue to monitor the health of the operator ecosystem and ensure Drubernetes remains reliable, free from opaque licensing traps.

For more information, visit:

The introducing article: Want to Run Drupal in Kubernetes? Try Our New Terraform Module
The project page: gitlab.com/backupscale/drubernetes
The Terraform registry module: BackUpScale/drupal/kubernetes

Comparing Canada’s Challenger Banks for Business: Loop, Venn, EQ Bank & Wealthsimple

Tue, 26 Aug 2025 00:00:00 +0000

Update 2026-06-28: When I originally published this, Wealthsimple didn’t support business accounts, but they started to on March 19th, 2026, and then introduced a payment card on June 18th. So I thought it was time to include them in the comparison table. Check it out below!

Introduction

Small business owners and startup founders across Canada are exploring modern fintech banking alternatives to escape the frustrations of the Big 5 banks (BMO, Scotiabank, CIBC, RBC, and TD). New challenger banks like Loop, Venn (formerly known as Vault), and EQ Bank are offering digital-first business accounts with low fees and innovative features. In this article, we’ll compare the key differences between these three fintech banking options for Canadian businesses. We’ll also touch on why some other names (like Wealthsimple or Tangerine) aren’t included, and mention how an international option (Wise) fits into the picture.

Why Not Include Wealthsimple or Tangerine?

Two other popular financial brands often come up in conversations about fintech banking: Wealthsimple and Tangerine. However, neither currently provides a full-fledged business operating account for day-to-day transactions, so we haven’t included them in the main comparison:

Wealthsimple

Wealthsimple for Business is essentially a high-interest savings and investment account for corporations, not a business chequing/operating account. In fact, Wealthsimple explicitly states, “We don’t offer business banking at the moment”. Their “Save for Business” account is only for parking surplus cash (it even requires you to have a separate business chequing account elsewhere to use it) and is limited to incorporated businesses (not available for sole proprietors). In short, Wealthsimple can help your company invest or earn interest on idle funds, but you cannot run daily business payments through it.

Tangerine

Tangerine (the online bank backed by Scotiabank, previously ING Direct) only offers business savings accounts and GICs, not a transactional chequing account. The Tangerine Business Savings Account is designed to complement a business chequing at another institution, meaning you must already have a business account elsewhere to use it. It’s great for earning high interest on business cash, but you can’t write cheques, deposit client payments, or pay bills directly from a Tangerine business account. Tangerine’s focus is strictly on savings products for businesses, so it’s outside the scope of comparing operating accounts.

Now, let’s turn our attention to three fintech players that do offer operating business accounts: Loop, Venn, and EQ Bank.

Challenger Bank Comparison Overview

Below is a side-by-side comparison of features for Loop, Venn, and EQ Bank business accounts. We’ll dive into many of these features in the commentary that follows.

Feature	Loop	Venn (formerly Vault)	EQ Bank	Wealthsimple
Business types supported	Not sole proprietorships (corporations only)	Corporations only	Sole proprietors & corporations	Incorporated businesses only
CDIC-insured accounts	Planned (via partner bank)	Yes (via partner bank)	Sole proprietorships only	Yes
Additional accounts (multiple accounts or sub-accounts under one business)	No (one account per currency)	Yes: 5 in free plan (sub-accounts)	Yes: Up to 10 in total	Yes: Up to 8
Non-“prepaid” payment cards (corporate debit/credit card availability)	Yes: Visa debit/credit card	Yes: Mastercard debit/credit card	No, prepaid MastCard only	No, prepaid Visa only
Add funds via Interac e-Transfer	Planned (not yet supported)	Yes: Via sending e-Transfer to yourself	Yes: Supports incoming e-Transfers	Yes: after setting up autodeposit
Get paid via Interac e-Transfer (receive from others)	Planned (not yet)	Yes: Supports incoming e-Transfers	Yes: Supports incoming e-Transfers	Yes: after setting up autodeposit
Send Interac payments (send e-Transfers out)	No (use EFT/wire instead)	Yes: Can send Interac e-Transfers	Yes: Can send Interac e-Transfers	Yes: Can send Interac e-Transfers
Helpful chatbot (in-app support chat)	Yes	Yes	Yes	Yes
Proof of account ownership (signed/stamped document)	Yes: Provides official letter	Yes: Provides official letter	Possibly on request, void cheque only by default	Possibly on request, void cheque only by default
Secure 2FA login (TOTP, passkeys support)	Yes: Supports authenticator app	Yes: Supports authenticator app	No: Only basic 2FA (email/SMS)	Yes: Passkeys and authenticator apps
Full-year statements (annual statement or summary)	No (monthly statements only)	No (monthly statements only)	Yes: Print 12 months on Transactions page	No (monthly statements only)
Free transactions on free plan	Yes: Unlimited free transactions	No: Pay per transaction on free tier	Yes: Most free (limits on free Interac)	Yes: Unlimited free transactions
Cashback on card spending	0% (point-based rewards only)	1% cashback on all spend	1% cash back with $10k monthly spend	1% cashback on all spend
Automated recurring payments (scheduled payments)	Planned (not yet)	Yes: Can schedule recurring payments	Yes: Can schedule recurring payments	Yes: With Interac e-Transfers & bill payments
Automatic balance top-ups (auto-transfer to maintain balance)	Planned (not yet)	Yes: Auto top-ups supported	No	No, but when deposits come in, a percentage can be auto-saved elsewhere
Pay bills through account	No	No	Yes: Can pay billers through the Canadian bill payment system	Yes: Can pay billers through the Canadian bill payment system
Pay business taxes through account	No	No	Yes: Can pay Canada Revenue Agency (CRA) / business taxes	Yes: Can pay Canada Revenue Agency (CRA) / business taxes
Secure document upload (for onboarding or support)	Yes: Upload documents in portal	No (use email for documents)	No (use email for documents)	Yes: Upload documents in portal
Mobile wallet support (Apple/Google/Samsung)	Yes: Apple Pay & Google Pay only	Google Pay only, so far	N/A (no card)	Apply & Google, but not Samsung
Foreign currency accounts (operating accounts in other currencies)	Yes: USD, EUR, GBP accounts	Yes: USD, EUR, GBP accounts	None (CAD only)	No, but USD accounts are planned
Foreign exchange (FX) conversion fee (currency conversion markup)	0.50% (Free plan), 0.25% (Plus@$49/mo), 0.10% (Power@$199/mo)	0.45% (Essentials@$0/mo), 0.35% (Plus@$40/mo), 0.25% (Pro@$100/mo)	N/A (no FX service for business; only on personal side)	N/A until USD accounts are available
Team access (multi-user support)	Yes: Included on free plan (invite team/users)	Limited: Paid plan required to add non-owner users	None: No multi-user login support	Co-owners only
Referral program (rewards for referrals)	Yes: Offers referral bonuses (Author’s Loop referral code)	Yes: Offers referral bonuses, both parties (Author’s Venn referral link)	No (business accounts have no referral program)	Yes: Offers referral bonuses, both parties (Author’s Wealthsimple referral link)

Table: Feature comparison of Loop vs. Venn vs. EQ Bank vs. Wealthsimple business accounts.

Below we discuss many of these items in detail.

Busines Types

One fundamental difference is which types of businesses each platform will accept. Both Loop and Venn currently cater exclusively to registered corporations (incorporated companies). Venn’s help center makes it clear that they only support corporations, and Loop’s onboarding is similarly geared toward corporations (and potentially other formal business entities), rather than simple sole proprietorships. In contrast, EQ Bank is more inclusive; its business account is open to both corporations and sole proprietors (as long as the sole proprietorship is a registered business under your own name).

Geography

Notably, neither Venn nor EQ Bank’s business accounts are available to Quebec businesses at the time of writing. Venn explicitly does not support businesses in Quebec yet, and EQ Bank’s sign-up page likewise notes Quebec is excluded for now. Loop is based in Toronto and does not list the same regional restriction so Quebec companies should be able to use Loop, which could be a deciding factor if you operate there.

Deposit Insurance

When it comes to the safety of your deposits, EQ Bank has the clear edge by virtue of being a bank. EQ Bank is a trademark of Equitable Bank, which is a CDIC member institution. This means funds in an EQ Bank business account are eligible for CDIC deposit insurance, up to $100,000 per depositor (in the “business accounts” category). However, there’s an important nuance: EQ clarifies that sole proprietorship deposits are insured as part of your personal coverage (since a sole proprietorship isn’t a separate legal entity). Corporate accounts at EQ Bank may not receive separate CDIC coverage beyond that (corporate entities can be considered distinct depositors, but CDIC insurance for corporate deposits can depend on account structuring; EQ’s documentation emphasizes coverage for sole proprietors). In summary, EQ Bank offers the reassurance of CDIC insurance, particularly for sole proprietors’ funds.

Neither Loop nor Venn is a bank themselves, so deposits with Loop or Venn are not directly CDIC-insured (if either company were to fail, the funds aren’t guaranteed by the government). That said, both use partner financial institutions to hold client funds. Loop has indicated that its Canadian dollar accounts are held with a CDIC member bank (so effectively the funds are insured via that partner). Venn likewise holds funds with “Tier 1” partner banks, and while they offer CDIC-insured products like GICs through a partner, the operating balances in Venn are not themselves CDIC-insured as of now (Venn’s terms describe your balance as an “unsecured claim” against Venn, not covered by deposit insurance). In the future, both Loop and Venn may obtain their own CDIC membership or otherwise make insurance more explicit (“Planned”), but currently only EQ Bank provides clear CDIC protection for your business deposits.

Account Security

On security features, all three platforms use basic two-factor authentication (2FA) for logins, but Loop and Venn support more secure methods via authenticator apps. EQ Bank’s 2FA for business accounts relies on sending a code via email, which is considered less secure than app-based 2FA. Loop and Venn, being newer platforms, allow users to set up TOTP (time-based one-time password) authenticator apps. This is a plus for those concerned about account security. You might prefer Loop or Venn if you want the option to secure your login with something like Proton Authenticator (or any other password manager / authenticator that suppports TOTP). Regardless, all three take security seriously (encryption, fraud monitoring, etc.), but the extra 2FA methods on Loop and Venn give them a slight edge for the security-conscious.

Account Structure: Single vs Multiple Accounts

If your business likes to use multiple accounts for budgeting or fund separation, note the differences in how each platform handles this.

Loop

Loop provides one primary account in each supported currency (CAD by default, and USD/EUR/GBP if you activate global accounts). You cannot open additional “sub-accounts” in the same currency for free at this time; essentially, Loop gives you one account per currency. For example, you might have a CAD account and also a USD account with Loop, but you can’t have two separate CAD accounts under the same business profile. This may change in the future (Loop has hinted at adding multiple sub-accounts), but currently it’s one account per currency on the platform.

Venn

Venn allows multiple accounts through a “sub-accounts” feature, which is great for organizing funds (e.g., setting aside money for taxes, payroll, etc. in separate buckets). On Venn’s free Essentials plan, you can create up to 5 sub-accounts. If you upgrade to the paid Plus or Pro plans, that limit increases (10 sub-accounts on Plus, and unlimited on Pro). All these accounts can be in any of the supported currencies. This flexibility is a notable advantage of Venn for businesses that want envelope budgeting or just separate ledgers for different purposes without opening entirely new bank accounts.

EQ Bank

EQ Bank also supports multiple accounts; you can open up to 10 separate EQ Bank Business Accounts for one business. Each account earns interest individually and can be named for its purpose. Since EQ doesn’t offer sub-account “buckets” within one login, those 10 are essentially 10 distinct bank accounts (but all under your business profile and accessible in one dashboard). That’s plenty for most small businesses’ needs. Keep in mind all those accounts will be CAD only (no foreign currency), but you could, for example, have an Operating Account, a Savings Account, and other specific accounts and transfer between them easily, all with no fees.

Interest on balances is another consideration here. EQ Bank stands out for offering a high interest rate (currently 2.50% interest on the full balance of its business accounts). This effectively turns every dollar in your EQ accounts into interest-earning savings until you need to spend it. Neither Loop nor Venn pay interest on ordinary balances (in fact, both explicitly state no interest on balance as a trade-off). Venn’s workaround is that it offers GICs (Guaranteed Investment Certificates) within the platform if you want to park money for interest. Those GIC funds are CDIC-insured through Peoples Trust and currently yield around 5%, but they aren’t accessible for daily banking until maturity. Loop does not offer any interest-bearing products at the moment (Loop’s focus is more on saving you fees rather than paying interest). So, if maximizing interest on idle cash is a priority, EQ Bank’s 2.50% yield on everyday business balances is very attractive, whereas Loop/Venn give 0% on deposits (you’d have to move money out to an investment to earn interest).

Payments and Money Movement (Interac, Wires, Deposits)

A critical aspect of any operating account is how you can move money in and out. Here’s how our three contenders compare on payments and transfers.

Interac e-Transfers

EQ Bank and Venn both fully support Interac e-Transfer for Canadian dollar payments, while Loop currently does not (as of this writing). With EQ Bank, you can send up to 50 e-Transfers per month for free (and 100 incoming e-Transfers for free). That’s usually plenty for a small business; if you exceed 50 outgoing e-Transfers in a month, fees would apply ($0.50 each beyond the free limit). Venn allows sending and receiving Interac e-Transfers as well. On Venn’s free plan, outgoing Interac transfers are actually one of the few things that might incur a fee (Venn’s pricing mentions a $2 fee for local transfers on the free tier), but the speed and convenience are there. Loop, on the other hand, does not support sending Interac e-Transfers yet, and you can’t have clients send money to your Loop account via Interac either (Loop lacks an email or phone number for auto-deposit). In fact, a noted drawback is you “can’t receive CAD payments from customers” via e-Transfer into Loop. Loop has this feature on the roadmap (marked as “Planned”), but for now, moving Canadian dollars to/from Loop requires linking an external bank (ACH/EFT) or using wire transfers. If your business uses a lot of e-Transfers with Canadian vendors or clients, this is a significant point in favor of Venn or EQ Bank.

Direct deposits and EFTs

All three platforms let you do traditional EFT or ACH transfers. Loop and Venn actually provide you with real bank account numbers (Loop gives you a Canadian transit and account number for CAD, plus US ACH details for USD, etc.; Venn does similarly) so you can set up direct deposit or pre-authorized debits. EQ Bank is a real bank account as well (transit/account under Equitable Bank). So you can, for example, have your payment processor deposit funds directly, or link PayPal/Stripe, etc., in all cases. Incoming and outgoing EFTs are free for all three. Loop and Venn both emphasize free domestic transfers (Loop even markets that it has unlimited free electronic payments, no fees). EQ Bank too charges no fees for EFT transfers. There may be timing differences (Loop and Venn might process EFTs as electronic debits that take 1-2 business days; EQ does overnight batch processing). Overall, routine bank transfers (EFT/ACH) are a strength of all three; you aren’t nickel-and-dimed for electronic transfers as you would be at some big banks.

Wire transfers and international payments

This is where Loop and Venn truly shine compared to both EQ and traditional banks. Loop allows you to send wire payments globally for free (no wire fees at all); you only pay the currency conversion fee if it involves FX. Loop supports sending to 180+ countries in 37 currencies, and even lets you send money to yourself in another bank account abroad for free. Venn also supports a wide range of international transfers (200+ countries). On the free plan Venn does charge a $10 fee to send an international wire, which is still far cheaper than a typical bank ($30–$50), but not zero. On Venn’s paid plans, the international wire fee drops ($8 on Plus, $6 on Pro). Receiving international wires in Venn is free. EQ Bank, unfortunately, does not currently support sending international wires or foreign currency transfers from the business account at all; you can only send/receive Canadian payments (EFT, Interac) in EQ’s business offering. So if you need to pay overseas suppliers or receive USD/EUR wires regularly, EQ Bank alone won’t suffice (you’d need a separate solution, possibly why you’d consider Loop or Venn in the first place).

Deposit methods (adding money)

With EQ Bank, you can deposit via EFT (link an external account and “pull” funds in), mobile cheque deposit (yes, EQ offers cheque deposit via its app, which Loop and Venn do not), or receive Interac e-Transfers. EQ doesn’t accept cash or in-branch deposits (no branches), so you can’t deposit physical cash or drafts. Loop allows deposits by linking an external bank for ACH/EFT pulls (which take a few days). Loop is working on enabling Interac e-Transfer deposits to your Loop CAD account (planned). Venn similarly can link external accounts for EFT, and it has a clever workaround to fund your account instantly: you can send yourself an Interac e-Transfer to your Venn account (Venn provides you a unique email address to auto-deposit, essentially). This way, you could e-Transfer money from, say, your personal bank or another bank, into Venn within minutes. It’s a nice convenience Loop lacks at the moment.

Bill payments and CRA taxes

Only EQ Bank allows outbound bill payments (paying Canadian billers through the Canadian bill payment system). This is useful for things like paying credit card bills, suppliers who have banking set up with billing codes, etc. They even go a step further by enabling CRA business tax payments directly from the account interface. For example, you can make GST/HST remittances or payroll source deduction payments via EQ Bank’s online banking (similar to how one would with a big bank’s business account). Loop and Venn do not have integrated CRA tax payment functionality; you’d have to pay the CRA directly via their Web site, via a third-party or by mailing payments if using those platforms. If you frequently remit taxes, EQ’s built-in support is a time-saver.

Recurring and automated payments

Both Venn and EQ Bank support scheduling recurring payments or transfers. You can set up an automatic weekly payment or monthly rent, etc., easily. Loop currently does not have a recurring/scheduled transfer feature (marked as “Planned”). Venn even offers more sophisticated automation on paid plans, like multi-step approvals for payments (useful in larger companies). It also has an “auto top-up” feature where you can maintain a target balance in a sub-account, e.g., automatically transfer funds from your main account to a sub-account if it dips below a threshold (helpful for budgeting). Loop intends to add similar balance automation, but it’s not there yet. EQ Bank doesn’t offer automatic top-ups or any advanced payment automation beyond standard recurring bill payments.

In summary, Loop and Venn are far superior for international payments and multi-currency transfers, while EQ Bank covers the basics of domestic payments and adds useful bill/CRA payments. If your business is mostly domestic, EQ’s simplicity and free unlimited Interac/EFT may suffice. But if you deal globally, Loop or Venn will be indispensable.

Card Features: Spending, Rewards, and Wallets

All three platforms take different approaches to payment cards for your business:

Loop

Loop issues the Loop Global Visa corporate card. This can be debit card (paid out of your account balance) or a true credit card (corporate card), meaning you get a credit limit and can carry a balance (Loop extends credit based on your business finances). The standout feature is that it’s a multi-currency Visa card: when you spend in USD, EUR, or GBP, it draws from your respective currency balance (if you have funds in those currencies) or from your credit in those currencies. There are no foreign transaction fees on this card for the supported currencies (CAD, USD, EUR, GBP); you can literally travel or make purchases abroad in those currencies with 0% FX fee (big banks typically charge ~2.5% on card purchases). Loop provides both physical and virtual cards (even on the free plan you can have 20 virtual cards for online spending). You can issue cards to employees as well, with spend controls. However, Loop’s rewards are points-based rather than straight cashback. On the free plan, you earn “1× points” per dollar, which effectively is a rewards program but not cash. The points can be redeemed (Loop hasn’t publicly detailed if it’s for statement credits or perks, but there’s no direct cashback on the base plan). Higher Loop plans earn 2× points. It’s a bit complex compared to a simple cashback system. And notably, Loop’s base plan offers 0% cashback in practical terms; the points have some value, but there’s no immediate cash rebate on purchases. If rewards are a priority, consider that.

Venn

Venn offers a Venn Mastercard Corporate Card, which functions like a debit/charge card linked to your account balance. It is not prepaid in the sense that it’s fully integrated with your Venn account and can be used wherever Mastercard is accepted. Like Loop’s, the Venn card is multi-currency: if you have USD in your account and use the card in the US, it will deduct from your USD balance with no FX fee. If you don’t have that currency, it will convert at Venn’s low FX rate. The big advantage of Venn’s card is its rewards: it gives a flat 1% cashback on all purchases, automatically. This cashback is unlimited and applied as real cash (not points), a strong perk for using Venn. For a fintech business card in Canada, 1% back on everything is excellent (few traditional business cards match that without annual fees). Venn provides unlimited virtual cards and free physical cards (one per user; additional physical cards cost $10). You can also set spending limits and category restrictions on each card, which is great for expense management. One limitation: as of now Venn’s card cannot be added to Apple Pay or Google Pay (no mobile wallet support). So you’ll need the physical card on hand for tap payments, etc., whereas Loop’s card can be added to your phone’s wallet for contactless payments.

EQ Bank

EQ Bank currently does not offer any business payment card. When you open an EQ Bank Business Account, you have no debit card or credit card associated with that account. This is a significant difference from Loop and Venn. It means you cannot directly spend your EQ business funds in a store or online via card; you’d have to transfer money out to another account to spend, or pay bills electronically. EQ has hinted at working on a business card in the future, but nothing yet. For some businesses, this is a deal-breaker: If you need a business debit card for routine purchases or a credit card for employee spending, EQ Bank alone won’t suffice. You might then pair it with a separate card (or simply choose Loop or Venn which have integrated cards).

In summary, if having a versatile business card is important, Loop and Venn are the clear winners. Loop’s Visa is great for travel and global spending (no FX fees), and Venn’s Mastercard gives you cash back on everything. They both offer robust expense controls and multiple cards for your team. EQ Bank’s lack of a card is a notable gap. Though you save on fees elsewhere, you’ll need another solution for card-based spending (which could negate some of the simplicity of banking with EQ only).

Fees and Pricing Plans

All three platforms have no monthly fee for their basic offerings; that’s a core appeal of these challengers compared to traditional banks (which often charge $20+ monthly for a business account). However, Loop and Venn use tiered subscription plans to unlock additional features or lower fees, whereas EQ Bank has only one free tier (no paid upgrades, all features included by default).

Loop’s Plans

Loop has three plan levels: Basic ($0/month), Loop Plus, and Loop Power. The Plus and Power plans carry hefty sticker prices if paid monthly ($79 and $299 per month, respectively). Most small businesses will be fine on the free Basic plan, which already gives you multi-currency accounts and cards. The paid plans mainly reduce your FX fees and increase card perks. Specifically, Loop’s FX conversion fee drops from 0.50% on Basic to 0.25% on Plus, and 0.10% on Power. Those are extremely low FX markups; 0.10% is virtually at the interbank rate (great for large volume currency exchange). By comparison, Wise’s well-known FX fee is around 0.43%, so even Loop’s free tier is a tad higher than Wise, but Loop’s top tier undercuts Wise significantly. Loop’s paid plans also increase your card rewards (2× points vs 1×) and come with a higher number of free physical cards and some premium services (like instant deposit of credit card payments). Important: You don’t have to upgrade unless your business would save more in fees than the subscription cost. Many growing businesses might find the $49/month Loop Plus worth it if they do a lot of FX volume, since it halves the FX fee from 0.5% to 0.25%. But if you only occasionally need FX or international payments, you can comfortably stay on free. Loop does not charge transaction fees for day-to-day operations even on the free plan; you get unlimited free transactions (EFTs, wires, etc.), which is very generous.

Venn’s Plans

Venn also has three tiers: Essentials (Free), Plus ($40/month), and Pro ($100/month). The Essentials (free) plan has no monthly fee, but unlike Loop, certain transactions have fees on this tier, notably a $2 fee per outgoing EFT/ACH and $10 per international wire sent. For a business with low volume of payments, that might be fine, but if you do many transfers, those $2 fees add up. The paid Plus plan at $40/mo makes all your domestic transfers free (and lowers the international wire fee to $8). It also lowers the FX conversion fee from 0.45% to 0.35%. The Pro at $100/mo lowers FX further to 0.25%, and further reduces wire fees ($6). Additionally, certain features are gated: for example, sub-accounts: On Essentials you get 5, on Plus you get 10, on Pro unlimited. User access controls (roles/permissions for team members) are only enabled on the paid plans; the free plan allows you to add other owners only. Plus/Pro also add priority support and some advanced automation (rules for accounting, multi-step approvals, etc.). In short, Venn’s free account is great for trying out or for a very small operation, but many businesses will likely need to upgrade to Plus ($40) to avoid nickel-and-dime transaction fees and get multi-user support. The good news is you can start free and only upgrade as needed, and $40/month is still far less than many legacy banks (and could easily pay for itself if you send a lot of EFTs or FX).

EQ Bank’s Pricing

EQ Bank keeps it simple; there is no monthly fee, period, and most transactions are free. You get unlimited free EFTs, free bill payments, and free deposits. The only fees you’d encounter are if you exceed the free Interac e-Transfer limits (more than 50 sent per month) or certain less common actions (like a paper statement by mail, etc., which are optional). There are no premium tiers to pay for as you get everything (that they offer) on the standard account. This simplicity is a big plus for cost-conscious entrepreneurs. However, keep in mind EQ Bank doesn’t offer some of the more advanced or expensive-to-provide features (no foreign wires, no multi-currency, no included international transfer service); those are exactly the things Loop and Venn monetize via their paid plans. So with EQ, it’s “you get what you get for free,” and if you need more exotic services, EQ might not have them at all.

In summary, all three are very affordable compared to traditional banks. Loop and Venn can be used entirely for free, but they also provide upgrade paths for businesses that need higher tiers of service; you’ll have to weigh the benefits of lower FX fees or more features against the subscription cost. EQ Bank is wonderfully straightforward on pricing (free means free), but it doesn’t have premium add-ons. If you outgrow what EQ offers, you might find yourself looking at the others anyway.

Venn’s Own “Venn vs Loop” Comparison (Bias Check)

It’s worth noting that Venn itself publishes a “Venn vs. Loop” comparison on its Web site. This can be a useful read for a quick overview, but keep in mind it is marketing material created by Venn, so it naturally highlights Venn’s advantages and downplays its weaknesses relative to Loop. For example, Venn’s site claims “Venn delivers faster onboarding and more transparent, cost-effective pricing, particularly in FX rates and subscription plans. Additionally, Venn offers actual cashback instead of points…”. It emphasizes that Venn has a 1% cashback reward versus Loop’s points-based rewards. These points are valid: the cashback on Venn’s card is a clear perk over Loop’s point system, and Venn’s transparency about fees is commendable (they openly list that Loop’s FX is 0.5% vs their 0.45% on free tier, etc.).

However, the Venn comparison is of course biased in Venn’s favor. For instance, it calls Venn’s FX rates “industry-leading” while characterizing Loop’s as “markups”, even though in reality Loop’s top-tier 0.10% rate is lower than Venn’s best 0.25%. The Venn write-up also touts features like accounting integrations and automation that Venn has and Loop purportedly lacks or has “limited”. In short, Venn’s comparison is a helpful marketing snapshot, but take it with a grain of salt.

The truth is both Loop and Venn are strong platforms with slightly different focuses: Loop is very focused on cross-border banking, FX, and credit (e.g., offering working capital up to $1M and no-fee international payments), whereas Venn pitches itself as an all-in-one financial platform for businesses, with features like expense management, accounting sync, invoicing, and even investment products. Depending on your business’s needs, one may suit you better. If you value cashback and an integrated bookkeeping/payment stack, Venn is appealing. If you value lowest possible FX fees and free unlimited global transfers, Loop is hard to beat. And Venn’s own comparison, while informative, understandably underscores Venn’s strengths so also look at independent reviews or firsthand trials.

Other Alternatives: A Note on Wise (TransferWise)

You might be wondering about Wise (formerly TransferWise), since it’s popular for international business banking. Wise isn’t included in our main comparison because it’s not a Canadian bank or credit union; it’s a London-based financial tech company. However, they do offer a Business account that many Canadian businesses use alongside their local bank. With Wise, you can hold and manage funds in dozens of currencies, and you get local bank details in several countries (US, UK, Eurozone, etc.), somewhat like Loop and Venn provide. Wise’s currency conversion fees are very competitive, about 0.43% on conversions, which often beats regular banks. They also provide a debit card (Mastercard) that lets you spend from your Wise balances and should work globally anywhere Mastercard is accepted. (However, I have found that it doesn’t work everywhere, which is why I started looking for other options.) In fact, many entrepreneurs use Wise’s card when traveling or paying overseas, because it automatically uses the local currency balance or converts at the low Wise rate with no extra foreign transaction markup.

That said, Wise is not a full replacement for a Canadian business bank account. You can’t do Interac e-Transfers with Wise, you can’t pay Canadian bills or taxes through it, the payment card doesn’t actually work everywhere, and your Wise balance isn’t CDIC-insured (though Wise keeps client funds in safeguarded accounts). Typically, a Canadian business might use Wise in addition to one of the accounts above. For example, the might use Wise for a specific international project or vendor payments, but still maintain a Canadian-dollar account like Loop/Venn/EQ for domestic needs. Wise is an excellent specialist tool for FX and global transactions (in fact, Loop and Venn sometimes compare themselves to Wise on rates), but since Wise isn’t Canadian-owned or focused, we centered our comparison on Loop, Venn, and EQ Bank, which are all Canadian-focused solutions.

Conclusion: Which Challenger Bank to Choose?

Each of these modern business banking options brings something different to the table, and the “best” choice really depends on your company’s priorities.

EQ Bank

EQ Bank is ideal for simple, fee-free banking if your needs are mostly domestic. You’ll earn a high interest rate on your balance, pay no fees, and enjoy unlimited transactions within Canada. It’s perfect for sole proprietors or small businesses that just need a basic account to send/receive money in CAD, pay bills, and maybe stash some savings, all while avoiding bank fees. The downsides are the lack of a card and no multi-currency or advanced features. EQ Bank is like a high-interest business chequing/savings hybrid that’s super easy to use and saves you money, as long as you don’t need bells and whistles.

Loop

Loop shines if your business has a global footprint or significant foreign currency needs. It truly enables a small business to bank like a local abroad by giving you local USD, EUR, GBP accounts and a multi-currency card. The absence of wire fees and ultra-low FX rates on higher plans can save tens of thousands of dollars for businesses that do large international transactions. Loop is also great if you want a credit card for your business, especially one with no FX fees on spend. Keep in mind, Loop is best for corporations (their onboarding might be slower if you’re a very new or small sole prop) and it currently lacks Interac e-Transfers which are ubiquitous in Canada. If you can live without Interac for now (or use EQ/another account as a workaround for e-Transfers), Loop offers tremendous value for cross-border finance. It’s like a modern alternative to having a USD account at RBC plus a EUR account at HSBC, etc., all consolidated in one platform. Many tech startups and e-commerce companies love Loop for these reasons.

Venn

Venn is a compelling choice for a business that wants an all-in-one financial platform and plans to grow into it. It has a broad feature set (global accounts, corporate card, invoicing, even investments) and is building a full ecosystem (expense management, accounting automation, etc.). The 1% cashback on the Venn card is a steady perk that essentially gives you back some money on every business purchase, essentially helping offset the subscription if you do enough volume. Venn’s slightly higher FX fees than Loop are still way better than banks, and everything is transparent. One thing to consider is cost: if you need multiple users or lots of transactions, you’ll likely be on the $40/month plan at least. But even at $40 or $100 a month, Venn might replace several other tools (it could handle international payments, corporate cards, and some accounting tasks in one). Venn is a strong pick for a growing startup or SMB that wants to streamline finances digitally and is okay with a monthly software-as-a-service (SaaS) fee for a better experience. It’s also an obvious alternative if you find Loop’s points/rewards or lack of interest a turn-off; with Venn you get cash back and can even earn interest through GICs on spare cash.

In many cases, a combination might serve you best. Since there’s no cost to open these accounts, some businesses use EQ Bank to park cash (earning interest) and handle Canadian transactions, while also using Loop or Venn for their card and global banking needs. That way you get the best of both: no-fee domestic banking plus advanced FX and card features. Just be mindful of keeping bookkeeping straight if using multiple accounts.

The emergence of Loop, Venn, EQ Bank (and others like Wealthsimple, Tangerine, Neo, etc.) signals a new era of banking in Canada, which is now helpful for businesses. The common theme is zero fees, flexibility, and fintech innovation, which is a refreshing change from the old-school big banks with their paperwork and charges. By evaluating what matters most for your business, be it saving on foreign exchange, earning interest, getting cashback, or simply avoiding fees, you can choose the platform that aligns best. And since all have free tiers, you can even try them to see which interface and service you prefer. One thing is certain: Canadian business owners finally have viable alternatives to the Big 5, and that competition can only be a good thing for entrepreneurs.

Take Them for a Spin

If you’re ready to experiment and set up one or more of these accounts yourself, you can use my referral codes to get started. We’ll both get something back for doing so. However, these are only available for Loop and Venn because EQ Bank doesn’t offer such a program. You can find links for these on the last row of the comparison table above.

Corrections and Updates

I’d like to keep this comparison as accurate and current as possible, but fintech tools evolve quickly. If you spot any errors or want to share updates (e.g. new features, pricing changes, eligibility details) I welcome your corrections. Feel free to reach out and help keep this resource reliable. Thanks for reading!

Want to Run Drupal in Kubernetes? Try Our New Terraform Module

Sun, 13 Jul 2025 00:00:00 +0000

This article was originally published on the BackUpScale blog.

Background

Our customer dashboard, which will soon be used for managing subscriptions to our backup service (and not just newsletters and our contact form, as we’re doing now), is built on the Drupal data management framework. Until now, we’ve been hosting it with a company that specializes in hosting very specific types of applications, like Drupal. This wasn’t working for us because our service is running in our Kubernetes cluster at a cloud service provider that specializes in managed Kubernetes hosting, which let’s us run whatever applications we want, and configure them however we need. The challenge was getting the dashboard to communicate securely with our other applications.

It needs to communicate with our back-end systems in the Kubernetes cluster to:

send requests from customers to provision services,
configure customer accounts, and
receive status information from back-end services to create log entries that users can see in their accounts.

Why the old approach broke down

In order for things to work with the old set up, we’d have to:

expose internal applications to the Internet (so the dashboard site could access them), and
add additional layers of security to the communications to ensure privacy.

We didn’t feel as confident with this set-up as moving everything into our private Kubernetes network, which protects all of our services with a single firewall. Keeping non-public facing services within that network ensures that they’re not accessible by anyone on the greater Internet (except our staff using the company VPN), which ensures greater security and privacy for our users.

In order to make the change, we needed to be able to run a Drupal site within Kubernetes. Given that Drupal is a popular framework, and Kubernetes is a popular container orchestration system, we assumed that there would be good options for putting them together using open-source infrastructure as code (IaC) to handle the automated provisioning (we automate everything here). However, we weren’t able to find anything that could help us.

Evaluated options

We explored the following options:

The Bitnami Helm chart did at least one very strange thing: It was placing the Drupal code files on the persistent volume instead of placing them in the container image. We wanted the Drupal code (or at least the Composer files that build it along with any custom code) to be version controlled with Git. When we tried to work around this, they made it very difficult to make these changes.

Jeff Geerling simply stopped recommending his earlier approach (except for potentially hosting many sites on a hosting platform), and said that he currently uses his own Kubernetes primitives. So we took that idea, and expanded on it to build a fairly complete solution. Once we had something that worked for us, we believed we could make it generic enough to make it available to everyone else. So that’s what we did.

This move eliminated two Internet-facing endpoints and let us apply a single network-policy layer to all microservices. Additionally, running inside the cluster removes a public load balancer, and shrinks latency.

Meet “Drubernetes”

Because we automate all of our infrastructure with Terraform, we just released Drubernetes, a new module in the Terraform Registry, which provisions Drupal onto a generic Kubernetes cluster. It shouldn’t matter where your cluster is, who’s managing it for you, or if you’re managing it yourself on your own hardware. We wanted to provide something standardized that everyone can use and build from.

Contribute

Contributions are welcome! Please try it, and provide any feedback that you may have. The project is hosted on Gitlab.com, and any issues can be opened from the board.

As always, if you have questions or feedback, feel free to reach out. We appreciate your support and can’t wait to bring you the next chapter of BackupScale.

From DevOps Headaches to Seamless Onboarding: How Dropping Chocolatey Made DDEV the Perfect Fit for a Client's Drupal Team

Thu, 03 Jul 2025 00:00:00 +0000

TL;DR

After watching my enterprise client’s Drupal developers lose hours every week wrestling with raw Docker Compose for local development, I championed a switch to DDEV, the open‑source tool that gives “container superpowers with zero required Docker skills” (ddev.com). One of the snags on their Windows laptops was the Chocolatey package manager, whose install path clashed with locked‑down corporate security policies. Working with DDEV maintainer Randy Fay, I removed the Chocolatey dependency, paving the way for a leaner installer that shipped in version 1.24.5. Development team members now onboard much quicker, and leadership can point to measurable productivity gains.

The Starting Point: Docker Compose Drag

My client’s legacy workflow relied on a bespoke Docker Compose stack. Developers routinely diverted time to babysit containers as well as copying their work in and out, instead of writing code. This echeos industry findings that more than 58% of engineers lose 5‑plus hours per week to “unproductive work”. This DevOps overhead impacts their developer experience (DX), which is a distraction from their actual work.

Enter DDEV

DDEV abstracts all that Docker plumbing with simple commands (e.g. ddev start, ddev stop) while still running everything locally. Its promise, “environments in minutes, multiple concurrent projects, and less time to deployment”, resonated immediately.

Why It Mattered for Drupal

A Drupal codebase is never just PHP; it drags along Composer, Drush, front‑end toolchains, and database snapshots. DDEV’s predefined Drupal preset provides a reproducible stack with Nginx/Apache, MariaDB, and Mailhog out of the box.

A Windows Speed‑Bump Called Chocolatey

The developers work on locked‑down Windows laptops. DDEV’s install script used the Chocolatey package manager, which corporate policies block from writing to its default location. Workarounds involved various hurdles, exactly the sort of DevOps toil they wanted to eliminate.

Collaborating Upstream: Issue #6636 → PR #7049

While in research mode, I discovered Issue #6636, maintainers themselves wanted to drop Chocolatey but hadn’t had the bandwidth to do so. I volunteered a pull request that:

removed Chocolatey from the installation process, and
no longer required an Administrator Powershell; an unprivileged user could install it in a terminal.

The patch was merged on April 10, 2025.

Release v1.24.5: Goodbye Chocolatey

A month later, the change landed in v1.24.5 with a shout‑out in the release notes: “Chocolatey removed from automated Windows installation scripts. Thanks to @colans.” It’s now simpler for Windows developers to install DDEV, and they don’t have to be an administrator.

Business Case (as pitched to leadership)

“Container superpowers with zero required Docker skills” isn’t just marketing. Here’s the quantified rationale I presented:

Zero Docker config: ddev start replaces hand‑rolled Compose files.
No central registry maintenance: Images build locally.
Host‑level commands: ddev drush status or ddev composer install without docker exec.
Unified file system: The code lives on the host, eliminating copy‑in/out cycles.
Safe rebuilds: Deleting containers never loses work.
Always‑on Git: blame, diff, and branch with the active code without stepping into the container.
First‑class Composer: Composer‑managed Drupal is just a ddev composer require away.

Outcomes

Onboarding time dropped from half a day (or more) to an hour (or less).
Additonal support for broken sandboxes fell to nearly zero.
Developers now focus on development, not troubleshooting containers, or moving code in and out of them.

Gratitude

I’d like to thank Randy Fay for prompt code reviews, patient feedback, and for shepherding the change into a release.

Ready to Try?

Head to the official DDEV installation documentation and give it a spin. And if you’re stuck on Windows, it’s now much easier to install.

Got Automated Tests? Stick with Container-Friendly Software

Wed, 19 Feb 2025 00:00:00 +0000

Modern DevOps pipelines rely heavily on automated testing. Ideally, each new code commit kicks off a sequence of tests in an environment that mirrors production—ensuring that issues are caught early, and deployment remains reliable. With container technology, it’s easier than ever to spin up temporary test environments that replicate production exactly, run your tests in isolation, then tear them down once done.

However, when you are forced to integrate closed-source, SaaS-based software that cannot be containerized, the smoothness of your CI/CD pipeline can suffer. Below, we’ll explore the pros of container-based testing, the complexities of closed-source SaaS, and a range of strategies—from spinning up new SaaS instances on demand to using mock services—to handle these challenges effectively.

Why Container-Friendly Dependencies Are Ideal

Ephemeral Environments
Containers allow you to spin up a clean environment for each test. You install your application stack, clone the branch under test, run tests, and then the containers disappear—ensuring no test data contaminates subsequent runs.
Consistency and Reproducibility
Container images can be versioned and shared, ensuring that every pipeline (and every developer) is using the exact same setup.
Scalability
Container orchestration systems (like Kubernetes or Docker Swarm) can spin up and tear down test environments at scale, handling parallel test executions efficiently.
Reduced “Works on My Machine” Issues
By relying on Docker images that define system-level dependencies, you remove guesswork from platform variations (e.g., different OS versions, missing libraries, etc.).

The Problem: Closed-Source SaaS You Can’t Containerize

If your organization relies on SaaS or closed-source tools that cannot be run inside a container, it complicates your testing pipeline:

No direct control over the external environment.
Licensing or security constraints may restrict how often you can spin up new instances.
Potential concurrency conflicts if multiple test runs share the same persistent SaaS instance, leading to data contamination or mismatched configurations.

Nonetheless, businesses often can’t simply ditch such tooling overnight. Below are strategies to accommodate closed-source SaaS within otherwise container-based CI/CD pipelines.

Three Core Strategies for Dealing with Non-Containerizable SaaS

Ephemeral SaaS Instances Per Test Run
- What It Is: Your CI pipeline dynamically creates a new SaaS instance for every test run, configures it, runs tests, then destroys it.
- Pros:
  - Guaranteed isolation across test runs (no data collision).
  - Reflects the ephemeral nature of containerized open-source solutions.
- Cons:
  - Potentially higher cost if the vendor charges per instance or usage.
  - Slower pipeline if instance provisioning is lengthy.
A Dedicated (Shared) SaaS Instance, Reset Between Runs
- What It Is: You keep one or a few permanent instances of the SaaS for testing. After each run, the environment is reset to a baseline.
- Pros:
  - Avoids spinning up new SaaS instances each time.
  - Potentially lower cost and faster test startup.
- Cons:
  - Risk of data contamination if resets aren’t perfect.
  - If tests can’t run in parallel, queued runs might cause a bottleneck.
A Dedicated SaaS Instance Using Unique Identifiers
- What It Is: Each test run or source (e.g., branch, pull request) is assigned a unique ID. All data in the SaaS is tagged with that ID, preventing collisions among simultaneous tests.
- Pros:
  - Enables parallel test runs without interfering data.
  - No need to spin up new SaaS instances if the vendor charges per environment.
- Cons:
  - Requires robust tagging logic in your tests and application code.
  - Configuration differences between test runs become complicated—different versions might need separate instances anyway.

Additional/Advanced Approaches

1. Use Stubs or Mocks for Partial Testing

What It Is: Instead of hitting the actual SaaS, you create a mocked or stubbed version of the external service’s API. Your application interacts with this mock during tests, which returns predictable responses.
Pros:
- Much faster and cheaper test cycles—mocks don’t require an external service.
- You avoid concurrency issues, since the mock is purely local to the test environment.
Cons:
- You lose true end-to-end coverage. While you can unit test and integration test the part you’re mocking separately, you don’t see how everything works together in a real-world environment.
- Requires diligence to keep your mock’s behavior in sync with the vendor’s real API changes.

Key Takeaway: Mocks and stubs are great to cover the bulk of your test scenarios quickly (especially early in your pipeline). However, you still need to run some end-to-end tests against the real SaaS instance for final validation.

2. Vendor-Provided Sandbox or Testing Environments

What It Is: Some SaaS vendors offer specialized “sandbox” instances that mimic production but have no direct effect on live data. They may offer test licenses or allow ephemeral environments.
Pros:
- Isolated from your actual production instance—reduces risk of messing up real data.
- May include features for easier resets or data seeding.
Cons:
- Not all vendors provide this, or it may cost extra.
- You might still face concurrency or versioning limitations.

Key Takeaway: If your vendor supports a sandbox environment, explore whether you can automate the creation/reset of these sandboxes within your pipeline.

3. Hybrid Approach: Containers for Your Stack + Shared SaaS

What It Is: You containerize everything else (databases, applications, etc.) and rely on a single shared SaaS instance for the parts you can’t containerize.
Pros:
- Retains most benefits of ephemeral containers for your primary app components.
- Less overhead than spinning up multiple SaaS instances.
Cons:
- You still have to manage concurrency or data collision issues on the SaaS side.
- Potential version/config mismatches if your code requires different SaaS configurations in different branches.

Key Takeaway: This approach is common because it strikes a balance between ephemeral containers and the practicality of a single external SaaS environment.

4. Multi-Stage (Layered) Testing

What It Is: Break down tests into stages, focusing first on unit and integration tests (often using mocks or local containers) and then following up with a smaller set of full end-to-end tests that hit the actual SaaS.
Pros:
- Fast feedback loop for the majority of tests.
- Minimizes expensive calls to SaaS or the complexity of ephemeral SaaS instances.
Cons:
- Requires more complex CI/CD orchestration: you need a pipeline that handles multiple stages and merges results.

Key Takeaway: A layered test approach can speed up development while still ensuring that at least some tests verify the real SaaS integration.

5. Negotiate Better Container Support or Testing Licenses from Your Vendor

What It Is: Push your SaaS provider for a container-friendly version or a specialized testing license that allows ephemeral usage.
Pros:
- Could achieve a setup very close to fully containerized environments.
- Reduces friction and complexities in your DevOps pipeline.
Cons:
- Some vendors may be unwilling or unable to provide containerized distributions.
- Negotiations might require more expense or long-term commitments.

Key Takeaway: If your application depends heavily on a particular SaaS, it might be worth investing in a conversation with the vendor about containerization or improved testing provisions.

Final Thoughts

Prefer Container-Friendly Software
Whenever possible, choose tools that can be run in containers. This keeps your CI/CD process straightforward, consistent, and fully ephemeral.
Assess SaaS Vendors Carefully
If you can’t containerize a critical dependency, at least check whether the vendor supports sandbox environments, easy resets, or ephemeral provisioning.
Combine Approaches
In practice, most teams use a combination of mocking/stubbing, ephemeral containerized components, and partial end-to-end SaaS integration tests. This layered approach often balances speed, cost, and thorough coverage.
Manage Data Collisions
If sharing a single SaaS instance, ensure you reset appropriately or use unique identifiers to segregate test data.
Consider Long-Term Roadmaps
Over time, aim to reduce dependencies on tools that cannot be containerized. Even if it’s not immediate, planning for more portable and container-compatible solutions in the future can pay off in smoother, faster pipelines and fewer operational headaches.

Ultimately, the moral of the story remains: avoid using software you can’t run in containers whenever possible. If you’re stuck with non-containerizable SaaS, invest in thoughtful workarounds—whether that’s mocking, ephemeral instances, dedicated instances with resets, or unique IDs—to keep your CI/CD pipelines efficient and reliable.

A Comparison of Managed Kubernetes Providers Without Egress Fees

Sun, 27 Oct 2024 00:00:00 +0000

Introduction

When it comes to deploying production-ready Kubernetes clusters, egress fees can be a hidden cost that quickly adds up. Major cloud providers like AWS, Google Cloud, and Azure are known for charging these fees, making it harder to predict monthly expenses, send data to other service providers, and ultimately migrate your data elsewhere. Some managed Kubernetes providers, however, offer a more transparent pricing model by eliminating egress fees altogether. In this article, we’ll compare four such providers: Scaleway, Upcloud, Civo, and Server Space.

All four providers offer managed Kubernetes, which simplifies cluster maintenance and scalability. However, there are differences in their feature sets and usability, which we’ll explore in more detail in the cost comparison section.

Criteria for Comparison

Let’s take a look at the data, and then dig into each feature below the table.

Feature Comparison

Feature	Scaleway	Upcloud	Civo	Server Space
Data centre locations	France, Netherlands, Poland	Various US, Various Europe, Singapore, Australia	Germany, UK, US	Canada, US, Brazil, Netherlands, Turkey, Kazakhstan, UAE
ReadWriteMany?	No	No	No	No
Terraform support?	Yes	Yes	Yes	Incomplete
Cluster provisioning speed	7 minutes	?	2 minutes	?
Years in business	25	12	6	5
Service level agreement (SLA)	99.50%	100.00%	99.95%	99.90%
Headquarters	Paris, France	Helsinki, Finland	London, UK	Amsterdam, Netherlands

Data Centre Locations

Each provider has different data centre locations that could impact your choice based on the proximity to your target audience. Scaleway is only in Europe, Civo is on both sides of the pond, Server Space is in seven countries worldwide, and Upcloud has multiple regions in several countries, providing flexibility for diverse deployment needs.

ReadWriteMany Support

ReadWriteMany (RWX) is a crucial feature for Kubernetes clusters that allows multiple pods to read from and write to the same persistent volume simultaneously. This capability is particularly important for workloads requiring shared access, such as content management systems, logging, and certain types of databases.

Currently, the major cloud providers (hyperscalers like AWS, Google Cloud, and Azure) are the only ones that support RWX natively in their managed Kubernetes offerings. As such, the managed Kubernetes providers we discuss here do not yet offer built-in RWX support. When asked about this limitation, they indicated that support for this is on their roadmaps, but it is at least six months away.

However, this isn’t a deal breaker. For those who require RWX functionality today, there are several workaround options you can explore:

Install Longhorn: The easiest option for setting up block storage in your cluster. It provides a simple way to add RWX support to your pods.
Rook or OpenEBS: These alternatives are more flexible and powerful, allowing you to build a highly customized storage solution for your cluster.

Using these self-managed storage solutions can also lead to cost savings. They utilize the unused storage already available on your nodes, avoiding the need to purchase additional storage volumes from the cloud provider. This cost-effective approach means that as long as your node storage is sufficient, you won’t incur extra charges.

Official Terraform Support

Official Terraform support is vital for those looking to adopt infrastructure-as-code practices. While Scaleway, Upcloud, and Civo offer full Terraform integration, Server Space’s support is currently incomplete, potentially complicating automated deployments.

Cluster provisioning speed

Another technical feature to consider is the cluster provisioning speed, although this is less of an issue in production. During development, however, you’re going to be spinning up and tearing down your clusters quite often. It’s important to factor in how much time your developers are spending waiting on these operations.

Civo is very fast: They claim one and a half minutes, and while it sometimes goes over, it’s always less than two. This is still the fastest I’ve seen anywhere. Scaleway is quite slow, relatively. I haven’t had a chance to test this on Upcloud or Server Space yet so if anyone out there has data on this, please share it with me, and I’ll update this article.

To see how these compare with other providers, take a look at the article Comparing the Top Eight Managed Kubernetes Providers. It also has some great analysis. The only problem with that article is that it doesn’t review outside of the top eight. Scaleway is included, which is nice.

Years in Business

The experience of each provider varies significantly, which can influence the maturity and reliability of their services. Scaleway, established 25 years ago, has a long-standing presence in the European cloud market, offering a range of services beyond Kubernetes. Upcloud, with 12 years in operation, also brings a well-established infrastructure and a solid reputation for reliability. Civo, though relatively newer with 6 years in business, has quickly made a name for itself as a specialized Kubernetes provider, particularly valued by developers for its simplicity and speed. Server Space, the newest provider on this list with 5 years in business, offers a streamlined approach with competitive pricing and growing popularity in the European market.

Service Level Agreements (SLAs)

The SLAs offered by each provider reflect their commitment to uptime and service reliability, which are essential considerations for production workloads. Upcloud leads with a 100% SLA, providing the highest guarantee of availability among the providers reviewed. Civo, with a 99.95% SLA, balances reliability with cost-efficiency, making it suitable for various production applications. Scaleway’s 99.5% SLA may appeal to users prioritizing budget-conscious solutions with some tolerance for minor downtime. Server Space, with a 99.9% SLA, offers solid reliability while remaining competitively priced. These SLAs give users a range of options to match their reliability needs and budget.

Headquarters

Each provider’s headquarters location offers insight into their operational focus and areas of influence. Scaleway is headquartered in Paris, France, and emphasizes data sovereignty and compliance with European regulations, which can be beneficial for EU-based companies. Upcloud, based in Helsinki, Finland, operates with a global outlook, providing services across many regions and maintaining a strong presence in Europe. Civo, headquartered in London, UK, has a focus on simplicity and developer-centric solutions, particularly attractive to startups and small to medium-sized enterprises. Server Space, based in Amsterdam, Netherlands, also emphasizes European data regulations and affordability, appealing to businesses across the EU and beyond.

One notable aspect of the providers compared here is that none of them are headquartered in the United States. This offers a potential advantage in terms of data privacy, as they are not directly subject to US laws that can sometimes compel US-based companies to turn over data stored in other countries.

A prominent case exemplifying this challenge is the Microsoft Corp. v. United States case. In 2013, the U.S. Department of Justice sought to access emails stored in Microsoft’s Ireland data center as part of a criminal investigation. Microsoft argued that the US warrant should not apply extraterritorially to data stored outside the US. Ultimately, the US Court of Appeals ruled in favor of Microsoft, setting a precedent that US-issued warrants do not reach overseas data centers by default. In 2018, however, the passage of the CLOUD Act (Clarifying Lawful Overseas Use of Data Act) revised US policy, allowing authorities to access foreign-stored data held by US-based companies through new, standardized agreements between nations.

This shift in legislation means that US-based companies may still face complex requirements when holding data abroad. For businesses concerned about extraterritorial data demands, providers headquartered outside the U.S., such as Scaleway in France, Upcloud in Finland, Civo in the UK, and Server Space in the Netherlands, might offer additional peace of mind regarding international data sovereignty.

Cost Breakdown

Here’s a breakdown of the basic cost for running a production-ready cluster with each provider, including managed Kubernetes, nodes, and a load balancer.

Product	Scaleway	Upcloud	Civo	Server Space
Managed Kubernetes	€80	€60	$0.00	€38
4 vCPU 8 GB Nodes (x3)	€77 * 3 = €231	€52 * 3 = €156	$40 * 3 = $120	€91.69
Load Balancer	€11.52	€30	$10.00	€0
Total	€322.52	€246	$130.00	€129.69
Converted to USD	$357.94	$273.02	$130.00	$144.00

The above cost comparison does not include additional services like block storage, object storage, or database services. I’m focusing on the essential resources needed.

Cost Analysis

Scaleway: The most expensive option at $357.94/month, primarily due to higher managed Kubernetes and node costs. It may still be worth it for those needing extensive data center options and a strong European presence.
Upcloud: A mid-tier option at $273.02/month, offering a competitive SLA and global data center coverage.
Civo: The most affordable choice at $130.00/month. Its free managed Kubernetes service significantly lowers the barrier to entry, making it suitable for startups or small-scale production environments.
Server Space: Priced at $144.00/month, it offers a balanced approach with competitive node pricing and a relatively low basic cluster cost. However, incomplete Terraform support might be a consideration.

Additional Considerations

While the above costs cover the basic managed Kubernetes setup, additional services like Block storage, Object storage, and Database services could further influence the overall expenses. Moreover, the lack of ReadWriteMany support across all providers could be a factor in workload planning.

Conclusion

For those looking to avoid the restrictive egress fees of major hyperscalers, these four providers offer a solid alternative. Civo emerges as the most affordable option, while Scaleway offers more features and data center options at a higher cost. Upcloud and Server Space fall somewhere in between, with unique strengths that might appeal to different use cases.

When choosing the right provider, consider factors like data centre location, pricing, SLA, and Terraform support in relation to your specific needs. Each provider has its own strengths, and this comparison aims to help you find the most cost-effective and reliable Kubernetes environment for your workloads.

Load Balancing SSH or Other TCP Connections? Preserve Client IP Addresses with One of These Tools

Sat, 07 Sep 2024 00:00:00 +0000

Introduction

In modern cloud environments, applications often sit behind load balancers or proxies to manage incoming traffic efficiently. While this setup helps distribute connections and scale services, it introduces a challenge: the original client’s IP address gets replaced by the proxy’s IP address. For many applications—particularly those using SSH—this can be problematic, as it obscures the real source of a connection.

Unlike HTTP traffic, which already has mechanisms like the X-Forwarded-For header to pass the client’s IP address to the backend, SSH and other TCP-based services don’t have built-in support for this. This makes it difficult for backend services to log or restrict access based on the original client IP address, which is important for security and monitoring.

To solve this, tools like go-mmproxy, mmproxy, and TPROXY exist to preserve the client’s IP address while forwarding traffic through a proxy. Each tool has unique strengths and is suitable for different environments. In this article, we’ll break down the differences between these solutions and help you choose the right one for handling SSH connections behind a proxy.

Why Preserving Client IP Addresses Matters

When using SSH or similar services to manage servers, it’s common to apply security rules based on the client’s IP address. For example, you might want to allow access only from a specific range of trusted IP addresses or monitor connections based on where they originate. However, when an SSH server sits behind a load balancer or proxy, the server will see the proxy’s IP address instead of the client’s IP address.

This loss of visibility into the real source IP address can create security blind spots. Fortunately, the PROXY protocol solves this issue by passing connection details, like the original client’s IP address, to the backend service. However, many services, including SSHD (the SSH daemon), don’t natively support the PROXY protocol. That’s where go-mmproxy, mmproxy, and TPROXY come into play.

Before diving into the comparison, it’s important to understand how these tools function at a high level. You can use one of them as another proxy that sits betweeen your load balancer and your service. They capture the connection, spoof the original client’s IP address, and then forward the remaining payload to the destination service. This ensures that your backend service can still see and act upon the original client’s IP address while handling the connection.

The request flow looks like this:

Client request > Load balancer > go-mmproxy/mmproxy/TPROXY > Your service (e.g. SSH)

Comparing the Tools: go-mmproxy vs mmproxy vs TPROXY

So how do you decide which one to use? Each tool has its own strengths, weaknesses, and ideal use cases. Choosing the right one depends on factors like performance, ease of setup, resource efficiency, and the complexity of your environment. Below is a comparison table that breaks down the key features and considerations for each tool, helping you make an informed decision based on your specific needs for handling these types of connections.

Feature/Aspect	go-mmproxy	mmproxy	TPROXY
Language	Go	C	Kernel-level feature (uses iptables)
Performance	Slightly higher overhead due to Go’s runtime	Lower overhead (written in C)	High performance, minimal overhead (kernel)
Ease of Use	Easy to set up, Go binaries are simple to compile or download	Requires compilation or binaries	More complex to configure (requires iptables and kernel support)
Configuration	Simple CLI options with flags like `--allowed-subnets`	Simple CLI, but requires C and networking knowledge	Complex `iptables` rules and kernel setup
Routing	Requires Linux routing (ip rules and iptables)	Also requires routing setup	Handles routing natively within the kernel
IPv4/IPv6 Support	Full support for IPv4 and IPv6 traffic	Full support for IPv4 and IPv6	Full support for IPv4 and IPv6
Scalability	Scales well with Go’s concurrency, but with a slightly higher memory footprint	Lightweight and highly scalable	Extremely scalable, as it’s part of the kernel
Resource Efficiency	Higher CPU and memory usage due to Go runtime	Extremely lightweight (C-based)	Extremely efficient (minimal CPU and memory usage)
Concurrency	Built-in concurrency via Go goroutines	Relies on system-level processes/threads	Kernel-level concurrency, highly optimized
Installation	Easy with `go get` or precompiled binaries	Requires manual compilation or prebuilt binaries	Built into the Linux kernel (but requires iptables setup)
Use Case Fit	Ideal for those looking for ease of use with Go	Best for lightweight, low-resource environments	Ideal for complex, large-scale deployments with advanced networking needs

Detailed Breakdown

Language and Resource Efficiency

go-mmproxy: Written in Go, it benefits from Go’s built-in concurrency model (goroutines), which makes handling many simultaneous SSH connections efficient. However, Go’s garbage collector introduces slightly more memory overhead compared to mmproxy or TPROXY.
mmproxy: Written in C, mmproxy is extremely lightweight and resource-efficient. Its minimal CPU and memory usage make it ideal for servers where resource efficiency is critical.
TPROXY: As a kernel-level feature, TPROXY is the most resource-efficient of the three. Because it operates entirely within the Linux kernel, it avoids the overhead of user-space processes, making it perfect for environments where performance and scalability are essential.

Performance

go-mmproxy: Go’s runtime adds some overhead due to garbage collection and memory management. While this doesn’t impact moderate-traffic environments significantly, it may become a factor in large-scale, high-performance setups.
mmproxy: mmproxy’s C-based implementation ensures minimal overhead, making it more suitable for environments with high traffic and performance requirements.
TPROXY: As a kernel-native solution, TPROXY is the most performant, with almost no overhead. It can handle large volumes of traffic with minimal CPU and memory usage, making it ideal for high-performance and low-latency SSH traffic scenarios.

Ease of Use and Setup

go-mmproxy: Installation is straightforward, especially if you are familiar with Go. You can either compile it using go get or download a precompiled binary. The configuration is user-friendly, with intuitive CLI options like --allowed-subnets to filter incoming connections by IP address.
mmproxy: Setting up mmproxy can be slightly more involved, particularly if you need to compile it from source. Precompiled binaries are available, but you need to be comfortable working with C-based tools and networking configurations.
TPROXY: Setting up TPROXY is more complex, as it requires configuring iptables rules and custom routing tables in the Linux kernel. While this offers maximum flexibility, it demands more networking expertise and time.

Configuration and Features

go-mmproxy: Offers a wide range of configuration options, including the ability to restrict access to specific IP address ranges using the --allowed-subnets flag. Its features are well-suited for users who need flexibility in filtering and managing connections.
mmproxy: mmproxy provides basic functionality to preserve client IP addresses and forward traffic, without the additional filtering features found in go-mmproxy. It’s designed to be simple, lightweight, and efficient.
TPROXY: TPROXY allows for advanced configuration at the kernel level. You can create complex routing and traffic management rules using iptables, making it highly flexible but also more difficult to set up and maintain.

Routing Setup

go-mmproxy and mmproxy: Both require additional routing setup using iptables and custom routing tables to forward traffic while preserving the original client IP address. This setup involves creating rules to mark packets and routing them correctly to the backend SSH service.
TPROXY: TPROXY natively handles routing within the kernel, eliminating the need for external routing tools. However, the initial setup with iptables can be complex and requires an in-depth understanding of networking.

Scalability and Concurrency

go-mmproxy: Thanks to Go’s concurrency model, go-mmproxy scales well in environments that need to handle many SSH connections at once. However, its Go runtime introduces some resource overhead, which can limit scalability in extremely resource-constrained environments.
mmproxy: mmproxy scales well due to its lightweight, low-overhead design. It relies on system-level concurrency, which can be fine-tuned for the specific environment.
TPROXY: As part of the Linux kernel, TPROXY scales effortlessly, handling large volumes of traffic with minimal overhead. It is the best choice for large-scale, performance-critical deployments where resource efficiency is paramount.

Recommendations

Use go-mmproxy if…

You want a simple, easy-to-use solution for preserving client IP addresses in SSH traffic.
You are familiar with Go or prefer using precompiled binaries.
You need features like IP address filtering (--allowed-subnets) for extra security.
Your environment handles moderate traffic and can tolerate the Go runtime’s resource usage.

Use mmproxy if…

You need a highly efficient and lightweight solution for handling SSH traffic.
Resource constraints are a priority, and you want to minimize CPU and memory usage.
You’re comfortable with C-based tools and networking configurations.
You prefer a minimalistic approach and don’t require advanced filtering features.

Use TPROXY if…

You need the highest possible performance and scalability with minimal resource overhead.
You’re working in a large-scale environment with complex networking requirements.
You have the expertise to configure iptables and manage routing tables in the Linux kernel.
You want full control over how traffic is handled at the network level, with the ability to create custom rules and manage routing efficiently.

Conclusion

When it comes to preserving client IP addresses for TCP connections behind proxies, the choice between go-mmproxy, mmproxy, and TPROXY depends on your specific needs and environment. go-mmproxy offers ease of use and flexibility, while mmproxy is a highly efficient, resource-light option for more constrained setups. TPROXY provides unmatched performance and scalability but requires advanced configuration.

By understanding the strengths of each tool, you can choose the right solution for your traffic needs and ensure your systems maintain the security and visibility needed to manage connections effectively.

Having Trouble Logging into Sites Protected with Cloudflare?

Tue, 02 Jul 2024 00:00:00 +0000

Introduction

Recently, I’ve encountered a recurring problem while trying to log into several sites that use Cloudflare’s Turnstile for bot protection as a replacement for CAPTCHAs. It’s a machine-learning tool they’ve developed instead. Personally, I’ve seen this on major sites like GitLab.com, ChatGPT, and even Cloudflare itself. While Turnstile aims to enhance security, it often produces false positives that can be rather frustrating.

It’s not just me experiencing these issues. I’ve seen other users complain about this in various forums, sharing workarounds that sometimes work for some but not for others. There doesn’t seem to be a universal solution that works for everyone.

The inconsistency of Turnstile is another significant issue. Despite having the same Firefox configuration on both my office desktop and my mobile laptop, which are synchronized via my Firefox account, I can log into GitLab.com on one but not the other. What does Turnstile think is different about them? I’m really not sure.

Potential Solutions

Improve Turnstile Accuracy

Cloudflare needs to continually refine its algorithms to reduce the number of false positives. This involves better distinguishing between legitimate users and bots through more advanced machine learning techniques.

Broader Browser Support

Ensuring that Turnstile operates smoothly across all major browsers and devices is crucial. Regular testing and updates can help achieve this goal. Most of the time, Firefox doesn’t work for me, but does in Chromium-based browsers (e.g. Chrome). Other users are reporting the opposite: Firefox works, but Chromium/Chrome doesn’t.

User Feedback Mechanism

Implementing a more efficient feedback mechanism where users can report issues directly can help Cloudflare identify and address specific problems more quickly. This feedback loop is essential for continuous improvement.

Adding a button for “Hey, I’m a human!” would be nice, but there would have to be a way to prevent bots from using it too. Maybe add some easy out-of-band task that only humans can do?

Streamlined UX Design

Simplifying the verification process can help maintain a positive user experience. What if you only had to do this infrequently? And only once for all of the sites you log into? If I get through, how about issuing me a token (e.g. a passkey) that works for an entire year everywhere? This brings up authentication on the Web generally, but maybe we need to bring these two discussions together.

Transparency

Let us know why we’re being blocked, and what we can do to fix it. Keeping this secret so the bots don’t adapt isn’t helpful here. And we all know that security through obscurity doesn’t work anyway.

Workarounds

What eventually worked for me on most of the sites I use is a recipe from the GitLab.com issue, thanks to Alix Brunet:

Shut down your browser by quitting it.
- Make sure every instance has been exited, and no background processes are remaining.
Launch your browser on the command line with all extensions/plugins disabled.
- On Firefox, this is firefox --safe-mode.
- On Chromium-based browsers, this is with the --disable-extensions switch, e.g. flatpak run org.chromium.Chromium --disable-extensions if you’re using the Flatpak package.
Log in to the site you’re trying to log into.
- It should work this time.
Shut down your broswser again.
- Make sure every instance has been exited, and no background processes are remaining.
Start your browser normally, without disabling extensions.
- This should now work, assuming that you don’t clear your cookies automatically on exit.

However, for ChatGPT, I also had to disable my VPN. The above process didn’t work until I did that. My login cookies weren’t helping on their own; I kept getting the Turnstile checkbox.

Conclusion

Security is good, but if it doesn’t allow authorized parties access, what’s the point of the service? As an extreme example, blocking everything is great security, but users will go elsewhere.

Maybe this is a good example of machine learning that isn’t quite ready to leave the lab just yet.

The Hidden Costs of Cloud Egress Fees: Finding a Better Alternative

Tue, 25 Jun 2024 00:00:00 +0000

Using third-party services with the major cloud providers

Looking for a cloud provider that will maintain the low-cost advantage of working with third-party services? They don’t advertise this, but the major Infrastructure-as-a-service (IaaS) providers charge you exorbitantly to get data out of their networks. Why? You can use them as long as you want, but you can never leave. It’s been referred to as the “Hotel California effect”, and these are called “egress fees”. By the time you realize how much you’re paying for cloud computing at Amazon Web Services (AWS), Microsoft’s Azure, or Google Cloud Platform (GCP), you have all of your data there, and you’re not going to pay the ransom (so you just stick around).

I currently have a need to send data to Wasabi, an independent storage provider, but can’t justify the cost of getting my data over there from the Big 3. Exporting the data alone would cost more than storing my data. So I need a better option.

If you’d like to mix & match services from various providers, it’s not worth it, because the money you’d be saving by going elsewhere for specialized services is lost on egress fees. It’s anti-competitive, and governments are starting to catch on. And while this has been talked about before, it’s not widely recognized as as problem.

Enter the Bandwidth Alliance

One of the big players, Cloudflare, has been working on this issue through its Bandwidth Alliance (see original announcement). It’s a good initiative, but let’s see how much traction it has over time. It looks like the number of members has doubled in 6 years, but some of agreements are simply deals between Cloudflare and the other party, not across the entire alliance. And I haven’t seen more commitments from the companies to drop these fees entirely. Yes, some of them never charged these fees, which is great, but it would be really nice if ones that did stopped doing so.

So what do we do?

Due diligence: Do your research on a cloud provider before you sign up with them initially. What do they charge? Don’t just look at services you want there; look at how much it’s going to cost you to get your data somewhere else. If they don’t list “egress” fees, look for “network” or “transfer out” fees. If they don’t provide it, ask. Don’t sign up for a cloud provider in the first place if you’re not happy with those terms.

If you do get stuck in one, and want to get out, you’d better hope it’s Google Cloud Platform (GCP). They’re the only one that allows you to get out, albeit permanently. For a good analysis of the offering, check out the Register article. At least it’s something.

Holori has done a comparison of these costs, and provides a tool for spend across clouds. They don’t support an exhaustive list of providers, but hopefully they’re working on this.

I looked into Fly.io because they run containers wherever you want, thinking that they’d support this kind of activity. They focus internationally, right? So you’d think they’d support data portability between cloud providers. Well, guess again: “We bill for outbound data transfer from the region a VM is running in, inbound transfer is free.”

Of course there’s Cloudflare; they’re the ones pushing on this the hardest. They have a “serverless” computing offering called Workers. (I’m putting “serverless” in quotation marks because it’s a silly name. It should really be called “server-managed” because there are still servers; you just don’t have to manage them yourself.) I looked into this option, but I need something that’ll respond to possibly long-running SSH connections, and they don’t seem to be designed for that. What I could really use is a similar offering that works with containers, not just code (sorry, but I’m not rewriting SSHD in JavaScript) that can respond to SSH requests. It’s either that or Kubernetes. And while they support services around Kubernetes, they don’t offer it themselves. (Cloudflare: If you’re reading this, please add support for server-managed containers and Kubernetes!)

My last hope was one of the OpenStack providers. If they’re into open infrastructure, they’d certainly be into data portability. No dice. My favourite Canadian OpenStack provider, Vexxhost, charges $0.10/GB (or $100/TB). I looked into the other ones as well, but none of them advertised this feature so I just assumed it wasn’t available. If I were them, I’d be promoting it as a selling point.

Conclusion

In the end, I landed on Scaleway for the project I’m currenlty working on. They’ve never charged for this, and actually promote it in their marketing.

Scaleway believes in unmetered bandwidth and is committed to working with the Bandwidth Alliance on advocating for all-unmetered plans on customer connections (Source).

They’re a good choice in this case because they support the Bandwidth Alliance, offer server-managed containers and Kubernetes (even an option that works with other cloud providers), and have a Terraform provider to automate all infrastructure as code (IaC).

I would have loved working with one of the big three to get deeper experience with them, but the cost, for this project anyway, simply isn’t justifiable.

Update 2024-10-05: I no longer recommend Scaleway because their Kubernetes cluster spin-up time is very long and they’re significantly more expensive than some other good options. See A Comparison of Managed Kubernetes Providers Without Egress Fees for details.

Moving Terraform State from OpenStack Swift to GitLab

Mon, 09 Jan 2023 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

For our cloud computing, we typically use an OpenStack provider because of its open-source nature: There’s no vendor lock-in, and the IaaS code is peer-reviewed unlike providers such as AWS, Azure, GCP, etc. (Shout out to Vexxhost for having great support!) As such, we’ve been using OpenStack’s Swift object storage service for storing Terraform’s state, which allows Terraform to track all of the resources it manages for automating infrastructure.

Recently, however, support for the Swift backend has been removed. If you’re still using Swift for this purpose, you’ll need to migrate your Terraform state files to another backend. Because the official migration documentation is sparse, I’ll describe how to migrate from Swift to GitLab-managed Terraform state. GitLab is a fantastic option because it can be used to manage so many other aspects of your project that you need anyway: Git repository hosting, issue tracking, CI/CD, etc. We use GitLab for all of our projects so it’s a great fit for us.

The actual step of migrating the data is well supported, but there’s some required set-up before and after.

When you change backends, Terraform gives you the option to migrate your state to the new backend. This lets you adopt backends without losing any existing state.

Prerequisities

Downgrade to the latest pre-1.3 Terraform version.
- e.g. sudo apt install terraform=1.2.9
Navigate to your Terraform directory.
- cd /path/to/git/repository/terraform
Move your local state files out of the way as they could be set up for a different environment.
- mv .terraform /tmp
Set up your environment variables to connect to use your existing state backend.
- e.g. source ../openstackrc/vexxhost-...-staging-ca-ymq-1.openrc.sh
Initialize Terraform from the remote state.
- terraform init
Back up your current state.
- cp .terraform/terraform.tfstate terraform.tfstate.backup-staging
In your Terraform code, in your backend stanza, replace swift with http.
Unset the old state environment variables.
- export TF_CLI_ARGS_init=
Fetch one of your Gitlab personal access tokens with the api permission. If you don’t have any that aren’t expired, create a new one in your settings.

Setting variables in your local environment

To actually migrate the data, the GitLab documentation says to set a single environment variable, and then manually run terraform init with many options. Given that this is error-prone and not easily repeatable, I’d recommend using a shell script (or similar) instead.

Create a file named setup-terraform-variables, and populate it like so:

#!/usr/bin/env bash
#############################################################################
## Set up Terraform variables.
#############################################################################

# Set up the Gitlab.com state backend.
export OS_PROJECT_CLOUD="$OS_PROJECT_DESCRIPTION-$OS_PROJECT_ENVIRONMENT-$OS_REGION_NAME"
export TF_GITLAB_PROJECT_ID="<Copy this from your GitLab project page>"
echo "Please enter your gitlab.com username: "
read -r TF_GITLAB_USERNAME
export TF_GITLAB_USERNAME
echo "Please enter your gitlab.com personal access token: "
read -sr TF_GITLAB_PASSWORD
export TF_GITLAB_PASSWORD
export TF_STATE_ADDRESS="https://gitlab.com/api/v4/projects/${TF_GITLAB_PROJECT_ID}/terraform/state/${OS_PROJECT_CLOUD}"
export TF_CLI_ARGS_init="\
-backend-config='address=$TF_STATE_ADDRESS' \
-backend-config='lock_address=$TF_STATE_ADDRESS/lock' \
-backend-config='unlock_address=$TF_STATE_ADDRESS/lock' \
-backend-config='username=$TF_GITLAB_USERNAME' \
-backend-config='password=$TF_GITLAB_PASSWORD' \
-backend-config='lock_method=POST' \
-backend-config='unlock_method=DELETE' \
-backend-config='retry_wait_min=5'"

You can set other Terraform variables in here as well, and include it in other deployment-environment-specific shell scripts that you run to set up each one. For example, if you’re using OpenStack generally, these would be your openstackrc files, which contain your credentials for accessing the API.

For a further optimization, you can write the GitLab credentials to a local file so as not to have to enter them every time, but I’ll leave this as an exercise to the reader. (If I get a chance, I’ll come back here and update it.)

Changing the backend type

In your Terraform configuration files, it’s necessary to change the backend type from Swift to HTTP.

 terraform {
-   backend "swift" {
+   backend "http" {
     # Must be read from environment variable `TF_CLI_ARGS_init` because normal
     # variables cannot be used here.
   }
 }

If you’re wondering why we need to use TF_CLI_ARGS_init, and can’t use Terraform variables in the stanza, see my earlier article Setting Deployment Environments’ Terraform State Backends with Environment Variables .

Migrating the data

You can now run:

terraform init

You should now see something like this, which requires your confirmation part-way through.

Initializing the backend...
Terraform detected that the backend type changed from "swift" to "http".

Acquiring state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "swift" backend to the
  newly configured "http" backend. No existing state was found in the newly
  configured "http" backend. Do you want to copy this state to the new "http"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Releasing state lock. This may take a few moments...

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
[...]

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Resetting your local environment to use the new state storage

This will purge your old state, and set it up to match the new remote state.

Move your local state files out of the way as they reflect the old state backend.
- mv .terraform /tmp
Upgrade Terraform to the latest version.
- e.g. sudo apt install terraform
Set up your environment variables again, using the updated code, to connect to your desired cloud environment. This script must include setup-terraform-variables as discussed above.
- source ../openstackrc/vexxhost-...-staging-ca-ymq-1.openrc.sh
Initialize Terraform from the remote state.
- terraform init

Confirming the presense of the remote state files

You can now see any of your state files in the GitLab Web UI on your Gitlab project’s page. Simply navigate to Infrastructure -> Terraform, and they’ll be listed.

Setting Deployment Environments' Terraform State Backends with Environment Variables

Mon, 12 Dec 2022 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Terraform is an essential tool for automating cloud-computing infrastructure and storing it in code (IaC). While there are several ways to navigate between deployment environments (e.g. Dev, Staging & Prod), I’d like to talk about how this can be done with environment variables, and explain why it can’t be done more naturally with Terraform variables.

Background

I originally wrote this as a comment on the feature request Using variables in terraform backend config block, which explains Terraform’s design limitations preventing it from allowing any variables in backend configurations, which is what Terraform uses to determine where to store its state data files (which track the cloud resources it manages).

To illustrate, this is where variables can’t be used, even though the configuration must change when the environment changes:

terraform {
  backend "whatever" {
    [...]
  }
}

Alternatives to using environment variables

Alternatively, it’s possible to have a single backend state configuration that stores data for multiple environments using Workspaces. You can then use Terraform CLI commands to switch between them. However, this solution is not not appropriate for all use cases, as per the documentation:

Workspaces are not appropriate for system decomposition or deployments requiring separate credentials and access controls.

If this isn’t a problem for you, Workspaces is a good option. Otherwise, please keep reading.

Solution

In demonstrating a solution, I’m going to be working with OpenStack, which is the IaaS run by many cloud providers, but the same concept can be applied to other IaaS, such as AWS, Azure, GCP, etc. Here, the backend state data will be stored in OpenStack’s Swift object storage service.

Update (2023-04-11): While the example below is still valid conceptually, Swift can no longer be used as a Terraform backend because support for it has been removed. If you’re still using Swift for this purpose, the official migration documentation is sparse. However, I provide guidance in a more recent article: Moving Terraform State from OpenStack Swift to GitLab.

Typically, you’d download an OpenStack RC file, which contains the credentials needed to access the API. In each of these (e.g. vexxhost-abc-prod-ca.openrc.sh, where the format is provider-project-environment-region), I append the following lines to the end:

[...]

#############################################################################
## Additional set-up not included with IaaS provider's OpenStack RC files ###
#############################################################################
export OS_PROJECT_ENVIRONMENT="abc-prod-ca"
source $(dirname "$0")/setup-terraform-variables.sh

The sourced (included) file then looks like:

#!/usr/bin/env bash
#############################################################################
## Set up Terraform variables.
#############################################################################

# Set up backend state container names.
export OS_PROJECT_STATE="$OS_PROJECT_ENVIRONMENT-terraform-state"
export OS_PROJECT_STATE_ARCHIVE="$OS_PROJECT_STATE-archive"
export TF_CLI_ARGS_init="\
-backend-config='container=$OS_PROJECT_STATE' \
-backend-config='archive_container=$OS_PROJECT_STATE_ARCHIVE'"

# You can set other TF variables in here as well.
echo "Please enter the outgoing e-mail account's password: "
read -sr TF_VAR_smtp_password_unquoted
export TF_VAR_smtp_password="\"$TF_VAR_smtp_password_unquoted\""

The special environment variable here is TF_CLI_ARGS_init. This is what Terraform uses to configure the backend, if it’s set. So by changing that environment variable, you can change the backend configuration. All that’s needed in the code is the following, basically just the backend type, with details being pulled in from the environment.

In main.tf (or wherever), the backend definition:

terraform {
  backend "swift" {
    # Must be read from environment variable `TF_CLI_ARGS_init` because normal
    # variables cannot be used here.
  }
}

Switching between environments

To switch between environments, I simply source another OpenStack RC file; I have one for every environment that I care about. With the above set-up, it also switches the Swift object storage containers used for backend state. You can then run terraform init.

I’ve found that sometimes it’s necessary to delete your .terraform directory before rerunning terraform init though, or it’ll get confused and you’ll get strange error messages. So as standard practise, I run the following command to remove it first:

`1`	`mv .terraform /tmp`

Summary

You cannot use Terraform variables to vary the backend state configuration.
You can, however, use the environment variable TF_CLI_ARGS_init instead.
Run a script to set this environment variable with the configuration matching the deployment environment you’d like to work with.
Simply specify the backend type in the Terraform code, and TF_CLI_ARGS_init along with your script(s) will take care of the rest.

Protecting your cloud networks with WireGuard VPN and Ansible

Fri, 24 Jul 2020 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Why use a VPN?

Within cloud computing, there are various types of sites and services not meant for public consumption (e.g. analytics software, databases, log servers, etc.). For security reasons, it’s best to keep these accesssible only via the private network, which is behind the firewall.

To provide access to these resources, a virtual private network (VPN) should be used, with network access granted only to trusted individuals within the organization.

Why use WireGuard specifically?

Traditionally, OpenVPN, IPsec and other solutions were the go-to options within the open-source software space. However, these are often complex to set up and have relatively massive code bases making them difficult to maintain.

For example, OpenVPN requires that a certificate authority (CA) be set-up. This is a complex piece of software, which shouldn’t be necessary for running a VPN. WireGuard simply requires the exchange of public keys in order to set up a secure connection, much like SSH and PGP.

There’s been a lot of interest in WireGuard lately, notably because:

It was recently added to the Linux kernel.
NordVPN, one of the major VPN service providers, has started using it.
Mozilla, the company behind the Firefox Web browser, just started offering it as a service.
It’s recommended in The Definitive 2019 Guide to Cryptographic Key Sizes and Algorithm Recommendations

However, in the Why Not WireGuard article, some opposition was raised. Let’s tackle some of the points raised there.

It does not allow using a dynamic IP address on the server side of the tunnel

Fine, but we’re doing the opposite. The clients can have dynamic IP addresses, but the server never will. So it’s irrelevant for this use case.

It’s not easy to use yet

That’s precisely the purpose of this article: To introduce an easy way to set it up and maintain it with Ansible.

It lacks cipher agility and upgrading many clients is difficult

Lacking cipher agility is actually a good thing. A better approach is to use versioned protocols. And it’s actually no more difficult to upgrade WireGuard clients than anything else. Both of these non-issues are discussed very nicely in the article Against Cipher Agility in Cryptography Protocols.

Trouble integrating it into Linux

As it was recently added to the Linux kernel, this is no longer an issue.

It lacks performance

While performance can always be improved, this doesn’t appear to be a critical issue for the application. For most use cases, it’s perfectly usable.

Conclusion

None of the above “issues” are actually a problem here.

Why use Ansible for configuration?

Ansible allows for automated deployment of configuration, which removes the need for manually installing, configuring and maintaining applications. It provides tonnes of modules, including those for files, storage, system, networking and even cloud provisioning (although I would generally recommend Terraform for this purpose).

Its units of work are called “tasks” that are run sequentially (procedural) in “roles” and “playbooks” to perform operations such as installing server software, its configuration, and handling various other types of system administration. Ansible strives for simplicity, resulting in playbooks that are essentially self-documenting. It can safely be run multiple times (as it strives to be idempotent), running tasks only when necessary, leaving already-configured items as-is.

A new Ansible role

While there were several WireGuard roles available for installing and maintaining the application, they either:

didn’t cater to the cloud gateway VPN use case,
lacked documentation, and/or
intentionally omitted critical elements (e.g. packet forwarding to internal hosts) for implementation by the user.

As such, I’ve written a comprehensive one.

It’s packaged as a collection as this is the newer distribution format that doesn’t concern itself with the location of source control repositories. Traditionally, it was necessary for roles to be hosted on GitHub for them to be published on Galaxy, the site for sharing Ansible contributions. As I prefer GitLab for hosting code repositories, this seemed more natural. The project is therefore hosted on GitLab.com.

The collection contains the single WireGuard role, and can be installed with ansible-galaxy (Ansible 2.9+). For older versions of Ansible, simply clone the Git repository and create a symbolic link to roles/wireguard_cloud_gateway within it.

Documentation can be found in the role’s README.

Key features

Support for clients and servers

Some of the other roles I researched didn’t provide much support for configuring the client side of the VPN, meaning the devices which connect to the cloud gateway server to access private network resources.

My role, on the other hand, can be run in either client or server mode: the same role can be used for configuring both. Running it in server mode configures the server (on the gateway VM), and running it in client mode configures the client devices (who connect to the server to gain access to the private network).

Support for configured security groups

For cloud security akin to traditional firewalls, security groups are essential for protecting virtual-machine (VM) compute instances. While these can be configured manually, ideally such configuration would be infrastructure as code (IaC) implemented via a tool such as Terraform, stored in a version control system (VCS) such as Git.

In a typical VPN-server set-up, the incoming (“ingress”) rules for such a “VPN” security group would block access to all ports except the one upon which the VPN communicates. This configuration should be applied to the VM that will be running WireGuard. However, if this is the case, using Ansible to install it won’t work because Ansible uses SSH to connect to the VM, and the SSH port is blocked.

In order to allow for such a secure set-up, a security group ID (e.g. “public_ssh”) can be provided to the role as a variable, which will be used to temporarily allow SSH access. Once the installation is complete, this temporary access will be revoked.

Support for multiple networks

For those of us that rely on VPN technology, it’s often necessary to connect to multiple VPNs at the same time, or at least prevent network resource IDs from overlapping. For example, you want to avoid having two VMs on different networks from having the same IP address.

WireGuard supports this by allowing multiple interfaces. By default, wg0 is used as the first one, but wg1, wg2, etc. can all coexist. If each remote network can exist on a different subnet, there’s no conflict from the client perspective. For example:

wg0 can be used to access network A, with subnet 10.1.0.0/16
wg1 can be used to access network B, with subnet 10.2.0.0/16

To set this up in your Ansible playbook when calling the role, set the service_interface variable. By default, it’s wg0. Also, be sure to set the client_accessible_ips properly; this defines the subnet. For details, see the default variables file.

Support

The role was originally written for specific systems, notably OpenStack networks and Ubuntu VMs. However, I’d like to see the role support as many systems as possible (e.g. the IaaS platforms Amazon Web Services (AWS), Microsoft’s Azure, Google Cloud Platform (GCP), Digital Ocean, etc. and non-Debian-based operating systems (OSes) such as CentOS, Red Hat, etc.).

If you work with these systems, I’d be more than happy to accept your supporting code via merge requests. Otherwise, if you’re able to provide funding, I can add support for these systems on your behalf.

To get in touch with me for that, or for any other reason, please use my contact form. I provide consulting in several areas, such as:

Enterprise cloud architecture
Cloud computing infrastructures as a service (IaaS)
Infrastructure automation with Terraform
Automating full-stack configuration with Ansible
OpenStack, Amazon Web Services (AWS), Google Cloud Platform (GCP) & Microsoft Azure consulting
Virtual private networking (VPNs) with OpenVPN and WireGuard

Does your Drupal hosting company lack native Composer support?

Thu, 12 Mar 2020 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Best practices for building Web sites in the Drupal framework (for major versions 8 and above) dictate that codebases should be built with the Composer package manager for PHP. That is, the code repository for any sites relying on it should not contain any upstream code; it should only contain a makefile with instructions for assembing it.

However, there are some prominent Drupal hosting companies that don’t support Composer natively. That is, after receiving updates to Composer-controlled Git repositories, they don’t automatically rebuild the codebase, which should result in changes to the deployed code.

If you’re hosting your site(s) at one of these companies, and you have this problem, why not consider the obvious alternative?

Aegir, the one-and-only open-source hosting system for Drupal that’s been around for over 10 years, has had native Composer support for over 2 years. That is, on each and every platform deployment (“platform” is Aegir-speak for a Drupal codebase), Aegir reassembles the upstream code assets by running the following automatically:

`1`	`composer create-project --no-dev --no-interaction --no-progress`

As a result, any sites created on that platform (or migrated/upgraded to it) will have access to all of the assets built by Composer.

Additionally, Aegir now ships with the Aegir Deploy module, which enhances the platform creation process. It allows for the following types of deployment:

Classic/None/Manual/Unmanaged
Drush Makefile deployment
Pure Git
Composer deployment from a Git repository
Composer deployment from a Packagist repository

For more information, please read the Deployment Strategies section of the documentation.

If you’d like to get started with Aegir, the best option would be to spin up an Aegir Development VM, which allows you to run it easily, play with it, and get familiar with the concepts. Naturally, reading the documentation helps with this too.

Afterwards, review the installation guide for more permanent options, and take advantage of our Ansible roles. We have a policy role that configures the main role using our favoured approach.

For help, contact the community, or get in touch with me directly. I provide the following Aegir services:

Installation & maintenance in corporate/enterprise (or other) environments
Architectural and technical support
Hosting guidance
Coaching
Audits
Upgrades
Conversion to best practices

Aegir 5 is coming, and not just for Drupal!

Tue, 10 Mar 2020 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Aegir is the one-and-only FLOSS hosting system for Drupal sites that’s been around for over 10 years, a rock in the community. While Drupal hosting companies have come and gone, Aegir’s always been there for folks who want to host Drupal sites themselves. According to recent data at the time of this writing, there are 567 instances (that we know about).

It’s used by organizations worldwide such as the US National Democratic Institute, NASA, and the European Commission.

While Aegir 3 is the currently stable recommended major release, we’ve started working on Aegir 5, which is a complete rewrite. It has notable differences such as:

Drush, traditionally used as the provisioner, has been replaced by Ansible, which allows for the hosting of any type of site or service, not just Drupal.
The front-end, formerly Drupal 7, has been replaced by Drupal 8, which allows us to take advantage of all of the newer features it provides.
Components are now best-of-breed open-source tools such as Celery, for the task queue. When Aegir was originally written, tools such as Ansible and Celery didn’t exist so all of the functionality was written as Aegir-specific code. We can now get off that island.
The entire project is now maintained in a single code repository, unlike the traditional four (Hosting, Provision, Hostmaster and Eldir) that have been maintained historically.
An out-of-the-box framework for automatic site updates via the Distributions concept. This was experimental in Aegir 3.

While still maintaining Aegir 3, we’d like to direct any new development initiatives towards the more modern Aegir 5.

The initial focus is on supporting Drupal. We then intend to add support for Matomo, Hugo, and other applications we use. However, documentation has been started on how to add support for anything else so merge requests or funding for new apps are greatly encouraged.

Along with the main documentation site, which includes the architecture, there is also an issues board for tracking tickets.

While we’re working on it as quickly as we can, we sometimes get delayed by other priorities. As such, we’re actively looking for sponsors to help us prioritize development. Please get in touch if you’re interested in partnering, collaborating, providing funding, or anything else.

Installing the OpenStack CLI on Ubuntu 19.10 (Eoan), 20.04 (Focal) or later

Mon, 09 Mar 2020 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

When working with OpenStack as an infrastructure-as-a-service (IaaS) cloud-computing platform, it’s rather convenient to be able to interface with it via its command-line interface (CLI).

While the service is typically installed on the Ubuntu Long-Term Support (LTS) operating system (OS), which has releases every two years, running the CLI from other OSes, such as interim Ubuntu releases, is often necessary. However, it is currently not possible to install the command-line client with supported Debian packages on Ubuntu 19.10.

Update (2020-05-30): The simplest way to install the OpenStack CLI nowadays, at least on Ubuntu 20.04 (Focal) and later, is via the Snap package. Debian packages are no longer made available:

`1`	`sudo snap install openstackclients --classic`

This should be all you need to do; there’s no reason to follow the instructions in the remaining portion of this article.

Update (2021-08-18): You may run into the following error on Ubuntu 21.04:

% openstack versions show
Traceback (most recent call last):
  File "/usr/local/bin/openstack", line 5, in <module>
    from openstackclient.shell import main
ModuleNotFoundError: No module named 'openstackclient'

If you do, run the following commands:

sudo apt install pip
sudo pip install python_openstackclient

Openstack commands should start working again.

Attempting to follow the instructions in the documentation for the two listed OpenStack releases, which require an additional software archive, results in the message:

cloud-archive for Rocky only supported on bionic

cloud-archive for Stein only supported on bionic

In addition, attempting to use the native software repository is impossible because the package isn’t available.

As such, I’ve written a short script to fetch and install all of the necessary Debian packages from another release.

#!/bin/sh

DEST_DIR=/tmp/openstack-cli-debs
mkdir $DEST_DIR

wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-netaddr/python-netaddr_0.7.19-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-iso8601/python-iso8601_0.1.11-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/pyinotify/python-pyinotify_0.9.6-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-monotonic/python-monotonic_1.5-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.context/python-oslo.context_2.22.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-warlock/python-warlock_1.2.0-2_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/s/stevedore/python-stevedore_1.30.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-os-service-types/python-os-service-types_1.6.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.config/python-oslo.config_6.8.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-debtcollector/python-debtcollector_1.20.0-2_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.log/python-oslo.log_3.42.3-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-os-client-config/python-os-client-config_1.31.2-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-requestsexceptions/python-requestsexceptions_1.4.0-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-deprecation/python-deprecation_2.0.6-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-dogpile.cache/python-dogpile.cache_0.6.2-6_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-json-patch/python-jsonpatch_1.21-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-munch/python-munch_2.3.2-1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-openstackclient/python-openstackclient_3.18.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-cinderclient/python-cinderclient_4.1.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-cliff/python-cliff_2.14.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-glanceclient/python-glanceclient_2.16.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-keystoneauth1/python-keystoneauth1_3.13.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-keystoneclient/python-keystoneclient_3.19.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-neutronclient/python-neutronclient_6.11.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-novaclient/python-novaclient_13.0.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-openstacksdk/python-openstacksdk_0.26.0-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/universe/p/python-osc-lib/python-osc-lib_1.12.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.i18n/python-oslo.i18n_3.23.1-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.serialization/python-oslo.serialization_2.28.2-0ubuntu1_all.deb
wget -cP $DEST_DIR http://mirrors.kernel.org/ubuntu/pool/main/p/python-oslo.utils/python-oslo.utils_3.40.3-0ubuntu1_all.deb

sudo dpkg -i $DEST_DIR/*

sudo apt --fix-broken install

Hopefully this helps other folks who have also run into this issue.

Drupal North 2019: Drupal SaaS: Building software as a service on Drupal

Fri, 15 Nov 2019 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

On Friday, June 14th, I presented this session at Drupal North 2019. That’s the annual gathering of the Drupal community in Ontario and Quebec, in Canada.

As I realized I hadn’t yet posted this information yet, I’m doing so now.

Session information:

Are you (considering) building a SaaS product on Drupal or running a Drupal hosting company? Have you done it already? Come share your experiences and learn from others.

Among other things, we’ll be discussing:

Project vs. product business

Installation profiles / distributions

Customer service (e.g. GitLab’s Service Desk)

Hosting architecture (Drupal hosting companies vs. Aegir)

Infrastructure (IaaS hosting providers: OpenStack vs. AWS, GCS, Azure, etc.)

E-commerce, recurring billing and subscription provider integration

Aegir Site Subscriptions

Others?

Resource quotas

Site admin access permissions for clients

…and any other related topics that come up.

A video recording of my presentation is available on:

My slides (with clickable links) are available on our presentations site.

Exposing Drupal's Taxonomy Data on the Semantic Web

Thu, 24 Oct 2019 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

As a content management framework, Drupal provides strong support for its taxonomical subsystem for classifying data. It would be great if such data could be exposed via the Simple Knowledge Organization System (SKOS) standard for publishing vocabularies as linked data. As Drupal becomes used more and more as a back-end data store (due to features such as built-in support for JSON:API), presenting this data in standard ways becomes especially important.

So is this actually possible now? If not, what remains to be done?

Drupal’s history

First, let’s explore some of Drupal core’s history as it relates to the Semantic Web and Web services formats, also useful for future reference. This is basically the backstory that makes all of this possible.

REST support was added to Views

This was implemented in the (now closed) issues:

Non-Schema.org namespace mappings were removed (including contrib’s UI support) in Drupal 8

Here’s the change notice:

Use schema.org types and properties in RDF mappings

And a follow-up issue requesting support for additional namespaces:

Allow usage of any namespace in RDF mapping UI

The community chose to replace JSON-LD with HAL in Drupal 8

Here’s an article with the details:

Decoupling Drupal 8 Core: Web Services in Core and the Serialization Module

Multiple Components

As this is really a two-part issue, adding machine-readable metadata and then making machine-readable data available, I’ll split the discussion into two sections.

Adding machine-readable metadata

While there’s an RDF UI module that enables one to specify mappings between Drupal entities and their fields with RDF types and properties, it only supports Schema.org via RDFa (not JSON-LD).

As explained very well in Create SEO Juice From JSON LD Structured Data in Drupal, a better solution is to use the framework provided by the Metatag module (used by modules such as AGLS Metadata). The article introduces the Schema.org Metatag module, which uses the Metatag UI to allow users to map Drupal data to Schema.org, and exposes it via JSON-LD.

So one solution would be to:

Clone Schema.org Metatag, calling the new module SKOS Metatag.
Replace all of the Schema.org specifics with SKOS.
Rejoice.

But after taking some time to process all of the above information, I believe we should be able to use the knowledge of the vocabulary hierarchy to add the SKOS metadata. We probably don’t need any admin UI at all for configuring mappings.

Assuming that’s true, we can instead create a SKOS module that doesn’t depend on Metatag, but Metatag may still be useful given that it already supports Views.

Making the machine-readable data available

Exposing the site’s data can be done best though Views. I wouldn’t recommend doing this any other way, e.g. accessing nodes (Drupal-speak for records) directly, or through any default taxonomy links for listing all of a vocabulary’s terms. (These actually are Views, but their default set-ups are missing configuration.) A good recipe for getting this up & running, for both the list and individual items, is available at Your First RESTful View in Drupal 8.

To actually access the data from elsewhere, you need to be aware of the recent API change To access REST export views, one now MUST specify a ?_format=… query string, which explains why some consumers broke.

The JSON-LD format is, however, not supported in Core by default. There is some code in a couple of sandboxes, which may or may not work, that will need to be ported to the official module, brought up-to-date, and have a release (ideally stable) cut. See the issue JSON-LD REST Services: Port to Drupal 8 for details.

Now, the Metatag solution I proposed in the previous section may work with Views natively, already exposing data as JSON-LD. If that’s the case, this JSON-LD port may not be necessary, but this remains to be seen. Also, accessing the records directly (without Views) may work as well, but this also remains to be seen after that solution is developed.

Conclusion

Clearly, there’s more work to be done. While the ultimate goal hasn’t been achieved yet, at least we have a couple of paths forward.

That’s as far as I got with pure research. Due to priorities shifting on the client project, I didn’t get a chance to learn more by reviewing the code and testing it to see what does and doesn’t work, which would be the next logical step.

If you’ve got a project that could make use of any of this, please reach out. We’d love to help move this technology further along and get it implemented.

References

General information

Contributed modules that probably aren’t helpful (but could be)

Structured Data (JSON+LD Rich Snippets)
JSON LD Schema API
WissKI
Ontology (OWL)
DCAT
RDF entity
Drupal2RDF (This is brand new, but at the time of this writing, there’s not enough info/code yet to figure out what it’s trying to do.)
There’s a module called Smart Glossary which allows you to have multilingual SKOS Thesauri on your Drupal site, but I don’t think it’s useful at all as it’s part of a suite of modules maintained by PoolParty, where they expect you to use their data store:

Questions about importing SKOS data (not exporting it)

Aegir DevOps: Deployment Workflows for Drupal Sites

Tue, 24 Sep 2019 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Aegir is often seen as a stand-alone application lifecycle management (ALM) system for hosting and managing Drupal sites. In the enterprise context, however, it’s necessary to provide mutiple deployment environments for quality assurance (QA), development or other purposes. Aegir trivializes this process by allowing sites to easily be copied from one environment to another in a point-and-click fashion from the Web front-end, eliminating the need for command-line DevOps tasks, which it was designed to do.

Setting up the environments

An Aegir instance needs to be installed in each environment. We would typically have three (3) of them:

Development (Dev): While generally reserved for integration testing, it is sometimes also used for development (e.g. when local environments cannot be used by developers or there are a small number of them).
Staging: Used for QA purposes. Designed to be a virtual clone of Production to ensure that tagged releases operate the same way as they would there, before being made live.
Production (Prod): The live environment visible to the public or the target audience, and the authoritative source for data.

(While outside the scope of this article, local development environments can be set up as well. See Try Aegir now with the new Dev VM for details.)

To install Aegir in each of these, follow the installation instructions. For larger deployments, common architectures for Staging and Prod would include features such as:

Separate Web and database servers
Multiple Web and database servers
Load balancers
Caching/HTTPS proxies
Separate partitions for (external) storage of:
- The Aegir file system (/var/aegir)
- Site backups (/var/aegir/backups)
- Database storage (/var/lib/mysql)
etc.

As these are all out of scope for the purposes of this article, I’ll save these discussions for the future, and assume we’re working with default installations.

Allowing the environments to communicate

To enable inter-environment communication, we must perform the following series of tasks on each Aegir VM as part of the initial set-up, which only needs to be done once.

Back-end set-up

The back-ends of each instance must be able to communicate. For that we use the secure SSH protocol. As stated on Wikipedia:

SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine.

Steps to enable SSH communication:

SSH into the VM.
- ssh ENVIRONMENT.aegir.example.com
Become the Aegir user.
- sudo -sHu aegir
Generate an SSH key. (If you’ve done this already to access a private Git repository, you can skip this step.)
- ssh-keygen -t rsa -b 4096 -C "ORGANIZATION Aegir ENVIRONMENT"
For every other environment from where you’d like to fetch sites:
1. Add the generated public key (~/.ssh/id_rsa.pub) to the whitelist for the Aegir user on the other VM so that the original instance can connect to this target.
  - ssh OTHER_ENVIRONMENT.aegir.example.com
  - sudo -sHu aegir
  - vi ~/.ssh/authorized_keys
  - exit
2. Back on the original VM, allow connections to the target VM.
  - sudo -sHu aegir
  - ssh OTHER_ENVIRONMENT.aegir.example.com
  - Answer affirmatively when asked to confirm the host (after verifying the fingerprint, etc.).

Front-end set-up

These steps will tell Aegir about the other Aegir servers whose sites can be imported.

On Aegir’s front-end Web UI, the “hostmaster” site, enable remote site imports by navigating to Administration » Hosting » Advanced, and check the Remote import box. Save the form. (This enables the Aegir Hosting Remote Import module.)
For every other server you’d like to add, do the following:
1. Navigate to the Servers tab, and click on the Add server link.
2. For the Server hostname, enter the hostname of the other Aegir server (e.g. staging.aegir.example.com)
3. Click the Remote import vertical tab, check Remote hostmaster, and then enter aegir for the Remote user.
4. For the Human-readable name, you can enter something like Foo's Staging Aegir (assuming the Staging instance).
5. You can generally ignore the IP addresses section.
6. Hit the Save button.
7. Wait for the server verification to complete successfully.

All of the one-time command-line tasks are now done. You or your users can now use the Web UI to shuffle site data between environments.

Deploying sites from one environment to another

Whenever necessary, this point-and-click process can be used to deploy sites from one Aegir environment to another. It’s actually a pull method as the destination Aegir instance imports a site from the source.

Reasons to do this include:

The initial deployment of a development site from Dev to Prod.
Refreshing Dev and Staging sites from Prod.

Steps:

If you’d like to install the site onto a new platform that’s not yet available, create the platform first.
Navigate to the Servers tab.
Click on the server hosting the site you’d like to import.
Click on the Import remote sites link.
Follow the prompts.
Wait for the batch job, Import and Verify tasks to complete.
Enable the imported site by hitting the Run button on the Enable task.
The imported site is now ready for use!

Try Aegir now with the new Dev VM

Mon, 09 Sep 2019 00:00:00 +0000

Originally published on the Consensus Enterprises blog.

Have you been looking for a self-hosted solution for hosting and managing Drupal sites? Would you like be able able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites had free automatically-updating HTTPS certificates? You probably know that Aegir can do all of this, but it’s now trivial to set up a temporary trial instance to see how it works.

The new Aegir Development VM makes this possible.

History

Throughout Aegir’s history, we’ve had several projects striving to achieve the same goal. They’re listed in the Contributed Projects section of the documentation.

Aegir Up

Aegir Up was based on a VirtualBox virtual machine (VM), managed by Vagrant and provisioned with Puppet. It was superseded by Valkyrie (see below).

Aegir Development Environment

Aegir Development Environment took a completely different approach using Docker. It assembles all of the services (each one in a container, e.g. the MySQL database) into a system managed by Docker Compose. While this is a novel approach, it’s not necessary to have multiple containers to get a basic Aegir instance up and running.

Valkyrie

Valkyrie was similar to Aegir Up, but provisioning moved from Puppet to Ansible. Valkyrie also made extensive use of custom Drush commands to simplify development.

Its focus was more on developing Drupal sites than on developing Aegir. Now that we have Lando, it’s no longer necessary to include this type of functionality.

It was superseded by the now current Aegir Development VM.

Present

Like Valkyrie, the Aegir Development VM is based on a VirtualBox VM (but that’s not the only option; see below) managed with Vagrant and provisioned with Ansible. However, it doesn’t rely on custom Drush commands.

Features

Customizable configuration

The Aegir Development VM configuration is very easy to customize as Ansible variables are used throughout.

For example, if you’d like to use Nginx instead of Apache, simply replace:

`1`	`aegir_http_service_type: apache`

…with:

`1`	`aegir_http_service_type: nginx`

…or override using the command line.

You can also install and enable additional Aegir modules from the available set.

Support for remote VMs

For those folks with older hardware who are unable to spare extra gigabytes (GB) for VMs, it’s possible to set up the VM remotely.

While the default amount of RAM necessary is 1 GB, 2 GB would be better for any serious work, and 4 GB is necessary if creating platforms directly from Packagist.

Support for DigitalOcean is included, but other IaaS providers (e.g. OpenStack) can be added later. Patches welcome!

Fully qualified domain name (FQDN) not required

While Aegir can quickly be installed with a small number of commands in the Quick Start Guide, that process requires an FQDN, usually something like aegir.example.com (which requires global DNS configuration). That is not the case with the Dev VM, which assumes aegir.local by default.

Simplified development

You can use it for Aegir development as well as trying Aegir!

Unlike the default set-up provisioned by the Quick Start Guide, which would require additional configuration, the individual components (e.g. Hosting, Provision, etc.) are cloned repositories making it easy to create patches (and for module maintainers: push changes upstream).

Conclusion

We’ve recently updated the project so that an up-to-date VM is being used, and it’s now ready for general use. Please go ahead and try it.

If you run into any problems, feel free to create issues on the issue board and/or submit merge requests.

DrupalCamp Ottawa 2018: Drupal SaaS: Building software as a service on Drupal

Fri, 26 Oct 2018 00:00:00 +0000

On Friday, October 26th, I presented at DrupalCamp Ottawa 2018, the annual gathering of the Drupal community in Ottawa, Ontario, Canada.

Session Information

Are you (considering) building a SaaS product on Drupal or running a Drupal hosting company? Have you done it already? Come share your experiences and learn from others.

Among other things, we’ll be discussing:

Project vs. product business

Installation profiles / distributions

Customer service (e.g. GitLab’s Service Desk)

Hosting architecture (Drupal hosting companies vs. Aegir)

Infrastructure (IaaS hosting providers: OpenStack vs. AWS, GCS, Azure, etc.)

E-commerce, recurring billing and subscription provider integration > - Aegir Site Subscriptions

Others?

Resource quotas

Site admin access permissions for clients

…and any other related topics that come up.

Resources

A video recording of our presentation is available on YouTube (with the audio track missing, unfortunately), and my slides (with clickable links) are available at talks.consensus.enterprises/drupal-saas.

Drupal North Toronto 2018: Hosting Drupal Sites? You need Aegir

Sat, 18 Aug 2018 00:00:00 +0000

On Saturday, August 11th, Christopher Gervais and I presented at Drupal North 2018, the annual gathering of the Drupal community in southern Ontario and Quebec, Canada.

Session Information

Do you need a self-hosted solution for hosting and managing Drupal sites? Would you like to be able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites could automatically get free HTTPS certificates?

If you said yes to any of the above questions, there’s only one option: the Aegir Hosting System. While it’s possible to find a company that will host Drupal sites for you, Aegir helps you maintain control whether you want to use your own infrastructure or manage your own software-as-a-service (SaaS) product. Plus, you get all the benefits of open source.

We’ll cover:

History

Architecture

Basic features

Advanced features

Development workflows

Recent additions

Future

Questions & discussion

Resources

A one-hour video recording of our presentation is available on YouTube, and our slides (with clickable links) are attached here.

Slides

DrupalCamp Montreal 2018: Hosting Drupal Sites? You need Aegir

Fri, 15 Jun 2018 00:00:00 +0000

On Friday, June 15th, Christopher Gervais and I presented at DrupalCamp Montreal 2018, the annual gathering of the Drupal community in Montreal, Canada.

Session Information

Do you need a self-hosted solution for hosting and managing Drupal sites? Would you like to be able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites could automatically get free HTTPS certificates?

If you said yes to any of the above questions, there’s only one option: the Aegir Hosting System. While it’s possible to find a company that will host Drupal sites for you, Aegir helps you maintain control whether you want to use your own infrastructure or manage your own software-as-a-service (SaaS) product. Plus, you get all the benefits of open source.

We’ll cover:

History

Architecture

Basic features

Advanced features

Development workflows

Recent additions

Future

Questions & discussion

Resources

A one-hour video recording of our presentation is available on YouTube, and our slides (with clickable links) are attached here.

Slides

DrupalTO Aegir Presentation

Fri, 01 Jun 2018 00:00:00 +0000

On Tuesday, May 29th, I presented at a DrupalTO meetup, the monthly gathering of the Toronto Drupal users’ group.

Session Information

The presentation focused on the Aegir Hosting System for managing Drupal sites. For more information, see the event. My slides are attached.

Resources

Slides

Aegir: Your open-source hosting platform for Drupal sites

Thu, 07 Dec 2017 00:00:00 +0000

If you need an open-source solution for hosting and managing Drupal sites, there’s only one option: the Aegir Hosting System. While it’s possible to find a company that will host Drupal sites for you, Aegir helps you maintain control whether you want to use your own infrastructure or manage your own software-as-a-service (SaaS) product. Plus, you get all the benefits of open source.

Aegir turns ten today. The first commit occurred on December 7th, 2007. We’ve actually produced a timeline including all major historical events. While Aegir had a slow uptake (the usability wasn’t great in the early days), it’s now being used by all kinds of organizations, including NASA.

I got involved in the project a couple of years ago when I needed a hosting solution for a project I was working on. I started by improving the documentation, working on contributed modules, and then eventually the core system. I’ve been using it ever since for all of my SaaS projects and have been taking the lead on Drupal 8 e-commerce integration. I became a core maintainer of the project about a year and a half ago.

So what’s new with the project? We’ve got several initiatives on the go. While Aegir 3 is stable and usable now (Download it!), we’ve started moving away from Drush, which traditionally handles the heavy lifting (see Provision: Drupal 8.4 support for details), and into a couple of different directions. We’ve got an Aegir 4 branch based on Symfony, which is also included in Drupal core. This is intended to be a medium-term solution until Aegir 5 (codenamed AegirNG), a complete rewrite for hosting any application, is ready. Neither of these initiatives is stable yet, but development is ongoing. Feel free to peruse the AegirNG architecture document, which is publicly available.

Please watch this space for future articles on the subject. I plan on writing about the following Aegir-related topics:

Managing your development workflow across Aegir environments
Automatic HTTPS-enabled sites with Aegir
Remote site management with Aegir Services
Preventing clients from changing Aegir site configurations

Happy Birthday Aegir! It’s been a great ten years.

Representing Drupal at the GSoC 2017 Mentor Summit

Thu, 26 Oct 2017 00:00:00 +0000

I’ve been mentoring students as part of Drupal’s Google Summer of Code (GSoC) program for the last two years, where we guide students in working on Drupal projects over the summer. (For the projects I’ve been involved in, see User-friendly encryption now in Drupal 8! and Client-side encryption options now available in Drupal.) This year, our organization administrator, Matthew Lechleider, invited me to the Mentor Summit.

The Google-provided summit creates a forum for members of free/libre and open-source software (FLOSS) organizations to come together to discuss GSoC, mentoring, and FLOSS in an unconference format. I met attendees from all over the world, who flew in from far-reaching places to interact as part of a wider community. Generally, two mentors are invited from each organization, but some had more and some had less.

I arrived late Friday night, having missed that day’s introductory sessions due to some trouble at the US border. Historically, in my experience, we Canadians haven’t had too much trouble getting across the border for technology conferences. This has recently changed so it’s now necessary to provide proof of intent for being in the country (a signed invitation from the organizers) as well as proof of business activities (corporate and tax documents). Needless to say, all of this took a significant amount of time to prepare. Eventually though, I was allowed through and made my way to Sunnyvale, California.

On Saturday morning, the day started with Lightning Talks, where attendees gave presentations on their student projects having only a few minutes each to speak. There were so many presentations that it was necessary to split the session into two, continuing after dinner that same evening. While there were several interesting projects highlighted, the most interesting to me was Jitsi’s speech-to-text service. Besides making video conferences accessible through textual media, it also allows for automated note-taking. This was one of the truly amazing projects completed by a student over the summer.

In talking about Drupal with other folks, I was surprised to hear that many other delegates do not have paying day jobs associated with their organizations. They work on these projects on the side, and generally don’t get paid for them. For example, nobody in the Kodi contributor community gets paid; it’s all volunteer work. While there are volunteer contributions to Drupal, many of those contributors eventually turn that knowledge into paid work. I suppose we’re a lucky bunch, being able to work on an open-source project and get paid for it. And speaking of Kodi, I’m happy to report that they’re using Drupal for their website!

There were quite a few conversations about messaging applications, with a large XMPP delegation. There were also folks from the Zulip and Rocket.Chat communities. It was interesting to hear from a former XMPP developer who’s shifted completely to Matrix with the Riot client, exactly as I’ve done. I use that client and the federated protocol to bridge with other communications networks such as proprietary closed-source Slack and classic Internet Relay Chat (IRC) whenever possible. Matrix already integrates with these two protocols, and has built-in support. The goal is to eventually use only one messaging client, instead of the many applications we all have installed on all of our devices. Rocket.Chat has already started working on Matrix integration, while Zulip hasn’t. They’re open to it, and may move in this direction eventually, but for now they’re focused on user-experience innovations. In the Drupal community, we’ve had a very long discussion about using Matrix for our communications alongside IRC, and have finally put a plan into place to make this happen. For those eager to jump in, it’s now possible to use Matrix as an always-on IRC bouncer client to connect to Drupal’s IRC channels.

Alongside Drupal, representatives from other content management systems (CMSes) also attended. There were folks from both the Joomla and Plone communities. It would have been great to connect with them, but I didn’t get a chance. I was hoping that Airship CMS would have representation as that crew has been doing a lot of excellent security work with PHP projects (including helping us with Drupal), but they weren’t in attendance.

All in all, it was an excellent conference. In my humble opinion, it’s really important to stay in touch with this greater community, cross-pollinate with folks doing similar work in the public interest, and keep contributing!

Client-side encryption options now available in Drupal

Mon, 18 Sep 2017 00:00:00 +0000

After the success of last year’s GSOC project with Drupal, I thought it would be a great idea to see if we could take what we did there (server-side encryption) and do something similar on the client side. The benefit of this approach is that unencrypted content/data is never seen by the hosting server. So it’s not necessary to trust it to the same degree. This has been a requested feature for some time, and become very popular within the instant-messaging space.

I posted the idea, but wasn’t sure how much traction there would be given the additional complexity. Before long, there were two interested students, Marcin Czarnecki and Tameesh Biswas, who were interested in the project given their interest in cryptography. They both wrote very good proposals, which we in the Drupal community accepted.

With the help of Adam Bergstein (my co-mentor from last year) and Talha Paracha (last year’s student), we were able to mentor both students in working towards completing their projects, even with the added complexity. Unlike last year, users’ passwords couldn’t be used to encrypt anything because the site has access to these. An out-of-band mechanism was necessary to perform the encryption, public-key cryptography. It needed to be in the hands of users themselves instead of being handled implicitly by the server.

I’m delighted to report that both students passed. The community can now take their projects and build upon them. Please review the new Drupal modules at Client-side content encryption (overview) and Client Side File Crypto (overview). If there are any issues, please open tickets in the respective queues.

User-friendly encryption now in Drupal 8!

Mon, 12 Sep 2016 00:00:00 +0000

The problem with most encryption strategies nowadays is that they require third-party software and/or services, require maintenance of additional keys and/or secrets, and provide an awful user experience.

Earlier this year, I started wondering why we couldn’t simply encrypt data with pre-existing secrets, the passwords users already have for logging into their Drupal sites. They shouldn’t have to deal with public and private keys and other cryptographic details. So I did some research, and was happy to discover that the security model is already in existence. The folks at ownCloud have not only published it (Data Encryption Model 1.1 and 2.2); they’ve already implemented it in their product. What’s even better is that the product is also written in PHP like Drupal, and has an open-source license. So the ideas and code can be reused.

Not too long after I made this discovery, the Drupal community was looking for project ideas for Google’s Summer of Code (GSOC). So I added mine to the list. There were several students interested in the topic, and wrote proposals to match. Talha Paracha’s excellent proposal was accepted, and he began in earnest. With Adam Bergstein (nerdstein) and I mentoring him, Talha successfully worked though all phases of the project. For details, please see his blog posts.

Now that GSOC 2016 has come to a close, we have a full project release for the Pubkey Encrypt module. It’s currently in beta, awaiting community review before we publish a production-ready version. We’ve included an architecture document, user stories, and usage documentation. There’s also a video! Please take the time to experiment with the module, and create tickets for any issues that you find.

At the time of this writing, only field data can be encrypted via the Field Encryption module. The File Encryption module is still in development, but as soon as it’s released, it should work with Pubkey Encrypt as well.

Authenticating Drupal users via OAuth2

Sun, 06 Sep 2015 00:00:00 +0000

I recently had a client that began delegating access to all of its data assets across the enterprise network via OAuth, specifically the OAuth 2.0 protocol. While I was there architecting a Drupal solution as their new Web platform, they wanted me to hook into this system to authenticate their Drupal users. Although there have been some modules available in the ecosystem to support OAuth2, there weren’t any available to provide this functionality. So I created the OAuth2 Authentication module.

This module allows users to log into a Drupal site authenticating against a remote identity provider (IDP) via OAuth2. That is, if a user’s credentials can be used to retrieve a valid access token, he/she will be logged into the site with those credentials and the token will be added to his/her session. If the user account doesn’t exist yet, it will be created.

In doing this, we’re making the assumption that resource requesters are actually resource owners. Generally, one shouldn’t make that assumption as OAuth2 is an authorization mechanism, not an authentication mechanism. Ideally, logging in users via OAuth2 should be done with OpenID Connect. It provides a proper identity layer on top of OAuth2. It’s essentially the evolution of SAML; see my answer to Can OAuth 2 be used for SSO? Or do I need a more sophisticated authentication? for details. In situations where one doesn’t have access to an OpenID Connect server, but does have access to an IDP that speaks OAuth2 and can trust the environment in which all of it operates, this module is sufficient.

The security implications of using this module should be well understood. If one doesn’t control the environment in which it’s running, then it shouldn’t be used. For example, I don’t recommend running with this concept in a mobile environment as it can’t be trusted to the same extent as a Drupal site behind a corporate firewall. The following articles are on the subject are noteworthy:

It also wouldn’t hurt to study the official OAuth 2.0 Threat Model and Security Considerations.

Initial Set-Up

Install and enable the OAuth2 Client and OAuth2 Authentication modules as you would any other.
If you wish to override any of the methods in the OAuth2AuthenticationClient class to change the module’s behaviour, create another class that extends it and implement the desired methods. This is best done in a custom module for your site, something like Sitename Authentication (sitename_authentication) where S/sitename is the name of your site.
Surf to the configuration page over at Home » Administration » Configuration » Web services » OAuth2 Authentication to configure your token endpoint. This section is mandatory while the others are optional. They contain sane defaults, but look over all of it to make sure it’s what you need for your set-up.
If you subclassed OAuth2AuthenticationClient, replace the default class name in Miscellaneous Settings » Client Class with the name of your new class.
Hit the Save configuration button to save your settings.
Enjoy!

Notes

Once you’ve got this set up, you’ll have to ensure that the Web-services client module you’re using supports the OAuth2 protocol (i.e., token access to resources). If you’re already using one that doesn’t, you’ll have to add that support. Otherwise, go with one that supports this already.
When an existing local user logs in, the module will attempt to get an access token for him/her. On success, the token will be added to the user’s session. On failure, the user will still be logged in, but will not get a token. Whenever a request to get a token is made, the results are reported in the log.
If an existing user whose password has changed on the IDP, but not Drupal yet, logs in, the password hash stored locally will be updated. This is attempted after a local login failure: If the user can authenticate remotely, the account is updated locally, and the user is logged in normally. There are no noticeable differences to the end user.

Issues

Token Expiration

If the total expiration time for your tokens, including successive tokens returned by your token server through refresh tokens (RTs), is less than the maximum time a user can be logged in (see Session Expire for details), users will still be logged in when their final tokens expire.

As this module doesn’t (yet) deal with that situation, you’ll need to come up with a solution that meets your requirements. Some background information on this can be found over at Best-Practice for dealing with OAuth 2.0 Token expiration at the Consumer.

Options

Automatically log out each user after being logged in for the token expiry time.
Extend the token expiration time to the maximum amount of time a user can be logged in.
Add support for refresh tokens (RTs) that can keep working until a user’s login session expires.
Have the token server issue tokens that don’t expire.
Some combination of the above.

Real-World Solutions

Helpful Drupal Modules

Session expire (also explains the default login session length)
Automated Logout
Ejector Seat

Similar Modules

OAuth2 Login redirects users to another Drupal site for authentication, and then sends them back logged in once they’re authenticated. This module doesn’t do any redirection; everything is done behind the scenes. Users logging in won’t even know that they’re authenticating against another system. They simply log in using the normal Drupal login process, but get an access token on top of that (if granted). Users that don’t exist locally will be created during the login process.

In conclusion, although this solution isn’t the most appropriate given the technology that’s now available, it does fit a lot of real-world use cases. At the time of this writing, there are 130 sites using it. This is quite impressive given that I recommend against it on the project page!

Responding to Drupal's Highly Critical SQL Injection Vulnerability

Thu, 03 Sep 2015 00:00:00 +0000

On October 15th, 2014, the highly critical SA-CORE-2014-005 - Drupal core - SQL injection vulnerability was announced. Shortly afterwards, research showed that sites not patched that same day could very well be compromised. Two weeks later, a public service announcement was released explaining the gravity of the situation. There was also a FAQ, a flowchart for dealing with it and a module that could potentially confirm a compromised site. Needless to say, it was a challenging time for the community.

At the time, I was asked by a client of mine to analyze a site to determine risk.

Some common attack vectors associated with the vulnerability were:

Changing the superuser’s (user ID 1) username, password or e-mail address.
Adding new users to the user table with the administrator role (usually ID 3).
Adding entries to the menu_router table.
Adding PHP files to the code base or in the sites/all/files directory.
Adding nodes (pages) or blocks with executable PHP.
Downloading the list of user passwords
Determining hackability through the version listed in CHANGELOG.txt
Spawned processes run by the Web server
Adding new roles
Permission changes

So based on the above, and some other sources, it was possible to produce a list of things one could look for to determine if a site had been compromised. As has been discussed elsewhere, failure to confirm any of these did not mean the site was not compromised, but it did provide some indication of risk.

Rerun the Security Review module.
Perform checks with the Drupalgeddon & Site Audit modules.
Check for changes to user 1’s username, password or e-mail address.
Check all users in roles other than “anonymous” and “authenticated”.
Check for strange entries in the menu_router table.
Check for code files outside of version control.
Check for code files inside sites/all/files.
Check for new nodes or blocks in the DB.
Check for strange processes spawned by the Web server user.
Check for any new roles.
Check for any permission changes.
Check the mail logs for anything suspicious being sent out.
Scan site with the Sucuri tools Free Website Malware and Security Scanner and Unmask Parasites.

There are some worthwhile things to note with respect to the checklist above:

Some of the clues (such as 5 and 6 above) would have been long gone as they’re rebuilt during cache clears and deployments.
It’s theoretically possible that malicious processes could have been spawned by the Web servers. The names could have been renamed to look non-malicious, but unless something was set up to make these persist across power cycles, they would be wiped on a system reboot.
It’s also theoretically possible that user passwords could have been compromised. In Drupal 7, hashed passwords are salted, but the random string used for the salt could have been read as it’s in sites/colan.consulting/settings.php. With that in mind, the site would be susceptible to brute force, dictionary and rainbow table attacks. If the site isn’t going to be rebuilt, it would be a good idea to expire all passwords and forcing users to reset them. If another system is handling authentication, this isn’t an issue.

And of course there could be any number of other things. The best course of action would be to rebuild the site. If that’s a challenge, always consider the level of risk before deciding not to. Hopefully we won’t have to make too many of these determinations in the future.

References

Drupal Helpers: Tools for DevOps and Deployment

Mon, 10 Aug 2015 00:00:00 +0000

As I’ve been architecting Drupal solutions for almost ten years now, I’ve accumulated quite a bit of knowledge on devops best practices, which constitutes a sizeable amount of the consulting that I do. This includes documentation, configuration management, development processes and deployment processes. In this article, I’ll be introducing Drupal Helpers, a collection of standard scripts and configurations that I use on all of my client projects (where applicable).

At the time of the writing, the repository provides support for the following operations and Drush set-up. Additional tools are always welcome.

Refreshing a development site’s database and files

The resync-drupal-db-for-dev script essentially deploys the database and files from a staging site (Staging) or production site (Prod) onto a development site, but it also does many other things that should be done as part of that process, devifying it.

It takes source and destination Drush aliases as arguments and the third one, a list of modules to disable, is optional. Here is the usage example:

resync-drupal-db-for-dev <SOURCE_DRUSH_ALIAS> <DESTINATION_DRUSH_ALIAS> [<MODULES_DISABLE>]

It performs the following tasks:

Saves a cache-cleared dump of the destination database (DB) as a backup.
Overwrites the destination DB with the source’s.
Rebuilds the registry in case PHP file locations have changed.
Updates the DB schema.
Reverts all features to the code in Features modules.
Reverts all views to those defined in code.
Disables modules that shouldn’t be enabled during development.
Enables modules that are helpful for development.
Disables CSS and JavaScript caching.
Sets the files and temporary directories to standard locations.
Enables on-screen error reporting.
Disables user-initiated cron runs. This should really be disabled everywhere for performance reasons.
Clears all caches.
Overwrites the destination’s files directory with the source’s.
Runs cron.

Developers should be doing all of this every time they refresh their DBs. Because it’s tricky and time-consuming to do all of these manually, some of the steps are often missed. This leads to configuration mismanagement issues between development sandboxes and other environments. I’d recommend that this script, or another one like it, be run frequently on the authoritative development/integration site (Dev) and local development sites to prevent such mishaps.

The script tries to be as versatile as possible, working in a variety of GNU/Linux environments. If it doesn’t work for yours, please submit a merge request so that we can get support added.

Deploying code to a development/integration site

The deploy-drupal-code-dev script is useful for deploying the latest development code that’s been merged to the development branch to Dev. As part of that process, it does everything else that’s necessary after a code deployment, including clearing external Varnish caches. It doesn’t produce output to the screen when running it directly, as it’s assumed to be run as a cron job. All output gets redirected to a log file. A best practice is to have it run nightly so that Dev is kept up-to-date.

Deploying code to staging or production sites

The two (2) scripts deploy-drupal-code-qa and deploy-drupal-code-prod basically follow the same idea as the Dev script above, except that:

They require a Git release tag as an argument, as only tagged releases should be deployed to Staging and Prod.
They produce output directly to the screen, as they are intended to be run manually.

PHP configuration for Drush

The drush.ini configuration file (see Configuring php.ini for details) adds Drush-specific configuration to PHP, differentiated by its being run from the command line; Drush’s PHP doesn’t go through a Web server.

As Drush is often called upon for batch processing, it requires more resources than Web-server PHP. Also, there aren’t usually multiple instances of it running so we don’t need be as concerned about overflowing resource limits. With Web servers, there could be a huge number of PHP processes running on public-facing sites.

The configuration file does the following:

Increases the memory limit.
Ensures all errors are shown.
Sets the time zone.
Increases the maximum execution time. You may need to do this for MySQL/variants as well.

Make sure this is set up by placing the file (or a symlink to it) in your /etc/php5/cli/conf.d/ directory (or the equivalent for different systems).

Default Drush alias configuration

The default.alias.drushrc.php Drush alias file is a standard location for storing all common Drush configuration. It needs to be set in each of your site alias’ configuration stanzas to use it.

The line is:

`1`	`<?php$aliases['dev'] = array( 'parent' => '@default', ...?>`

It could actually use an update now that Better handling of structure-tables and skip-tables options (including cache_* support!) is done. We no longer need to explicitly state which cache tables we need to skip during certain operations.

Backing up databases

The backup-drupal-db script will back up a Drupal DB whenever it’s run. It can be run stand-alone or as a Cron job. To save storage space, cache tables will not be included in the compressed dump files. Each is timestamped, and the “LATEST” file will always be a symbolic link to the most recent backup. Old backups are automatically deleted after a set number of days; the default is 60 (~2 months).

Rebuilding a site with its latest Drush makefile

The drupal-remake script will rebuild a Drupal site’s document root to reflect recent changes in its Drush makefile. It performs the following tasks:

Takes the site off-line.
Backs up the existing database.
Updates the Git repository with the latest code.
Purges the old code base.
Rebuilds it.
Re-installs the previous sites directory (with site-specific configuration).
Properly sets all file permissions.
Updates the DB schema.
Turns the site back on-line.
Clears all caches.

Deploying Solr onto the GlassFish application server

The deploy-solr-on-glassfish script sets up the Solr search engine to run on the GlassFish application server.

With Solr 5, it’s no longer necessary to run the search engine in an application server. It can run as a stand-alone application. See Install Apache Solr 5 and Drupal Search API on your laptop in minutes for an example of how to do this. For earlier versions though, this is helpful in getting the Java stack set up to run alongside the PHP one.

A special thanks goes out to Jamon Camisso for posting the original version of this on GitHub earlier.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

To reach me, simply send an e-mail to:

colan.relive907@passmail.net

Clicking on the link should start composing a new message in your mail client. If not, cut and paste into a new message yourself.

Contact forms are normally used to hide e-mail addresses from spam harvesters, but I’m using an e-mail aliasing service called SimpleLogin (via Proton Mail). I can block senders or disable the alias altogether, and then generate a new one whenever this particular address gets overly misused.

Links

Mon, 01 Jan 0001 00:00:00 +0000

Search

Mon, 01 Jan 0001 00:00:00 +0000

Colan Schwartz on Cloud Architecture, Security, Privacy & Startups

Forfeiting the Frontier: How Washington Is Ceding Its Own AI Advantage

Forfeiting the Frontier: How Washington Is Ceding Its Own AI Advantage

How a lead actually erodes

These are maneuvers, not policy

Both exits cost the United States

What gets priced in

Cloud Lock-in is Mostly Self-Inflicted

Problem 1: Cloud-specific IaC

Problem 2: Cloud-specific managed services

How to actually decide

Announcing Drubernetes v2: Moving from Bitnami to the Official MariaDB Operator

Why It Matters

Background: Why Drubernetes Needed a v2

Bitnami’s Policy Shift: From Open Access to Paywall

What Changed in Drubernetes v2

Migration Guide: Upgrading from v1.x to v2.0

Looking Ahead

Comparing Canada’s Challenger Banks for Business: Loop, Venn, EQ Bank & Wealthsimple

Introduction

Why Not Include Wealthsimple or Tangerine?

Wealthsimple

Tangerine

Challenger Bank Comparison Overview

Business Account Eligibility (Who Can Sign Up)

Busines Types

Geography

Deposit Insurance

Account Security

Account Structure: Single vs Multiple Accounts

Loop

Venn

EQ Bank

Payments and Money Movement (Interac, Wires, Deposits)

Interac e-Transfers

Direct deposits and EFTs

Wire transfers and international payments

Deposit methods (adding money)

Bill payments and CRA taxes

Recurring and automated payments

Card Features: Spending, Rewards, and Wallets

Loop

Venn

EQ Bank

Fees and Pricing Plans

Loop’s Plans

Venn’s Plans

EQ Bank’s Pricing

Venn’s Own “Venn vs Loop” Comparison (Bias Check)

Other Alternatives: A Note on Wise (TransferWise)

Conclusion: Which Challenger Bank to Choose?

EQ Bank

Loop

Venn

Take Them for a Spin

Corrections and Updates

Want to Run Drupal in Kubernetes? Try Our New Terraform Module

Background

Why the old approach broke down

Evaluated options

Meet “Drubernetes”

Contribute

From DevOps Headaches to Seamless Onboarding: How Dropping Chocolatey Made DDEV the Perfect Fit for a Client's Drupal Team

TL;DR

The Starting Point: Docker Compose Drag

Enter DDEV

Why It Mattered for Drupal

A Windows Speed‑Bump Called Chocolatey

Collaborating Upstream: Issue #6636 → PR #7049

Release v1.24.5: Goodbye Chocolatey

Business Case (as pitched to leadership)

Outcomes

Gratitude

Ready to Try?

Got Automated Tests? Stick with Container-Friendly Software

Why Container-Friendly Dependencies Are Ideal

The Problem: Closed-Source SaaS You Can’t Containerize

Three Core Strategies for Dealing with Non-Containerizable SaaS

Additional/Advanced Approaches

1. Use Stubs or Mocks for Partial Testing