The engineers who thrive in an AI-augmented world won’t be fighting against the technology or ignoring it. They’ll understand how to amplify their strengths through intelligent collaboration with AI systems. Instead of asking “Will AI take my job?” they’re asking “How can AI make me 10x more effective at the work that matters most?”

What’s fascinating is that the skills that make you great at working with AI are remarkably similar to the skills that make you great at working with other engineers. Clear communication, structured thinking, and productive division of labor are fundamentals that remain constant whether you’re pair programming with a colleague or collaborating with an AI model.

Here’s what that collaboration looks like in practice, and how to position yourself to lead in an AI-first world.

AI Amplifies Systems Thinking Through Better Collaboration

The biggest opportunity comes from using AI to think through complex systems more thoroughly. AI excels at analyzing patterns, suggesting edge cases, and helping you reason through architectural decisions. Engineers who learn to collaborate effectively with AI on design and planning create better systems than either could build alone.

This mirrors how the best engineering teams work together. When you’re designing a system with a colleague, you externalize your thinking, challenge each other’s assumptions, and explore alternatives. Working with AI requires the same discipline. You articulate problems clearly, make your constraints explicit, and iterate on solutions collaboratively.

The practical skill here involves having productive conversations with AI about system design in the same way you would with a colleague.

Start by clearly defining the problem space: What are the constraints? What are the non-obvious requirements? What could go wrong? AI can help you explore these questions more comprehensively before you commit to solutions.

This resembles how senior engineers mentor junior team members—by asking good questions and helping them think through problems systematically. The difference is that AI can process vast amounts of information quickly and suggest patterns you might not have considered.

Start practicing this now by using AI to review your design documents, challenge your assumptions, and suggest alternatives. The goal is ensuring you consider angles you might have missed, just like getting a thorough code review from a thoughtful colleague.

Human Skills Become Your Competitive Advantage

As AI handles more routine implementation work, the uniquely human aspects of engineering become increasingly important. Understanding and communicating business context, navigating organizational complexity, and making judgment calls under uncertainty—these skills differentiate great engineers from good ones.

Product intuition becomes especially critical. AI can generate code, but it can’t determine whether you’re building the right thing for solving your customers’ problems. Engineers who understand user needs, translate business requirements into technical solutions, and make trade-offs based on strategic priorities remain indispensable.

These are the same skills that make you valuable on any engineering team. The ability to see the bigger picture, understand stakeholder needs, and make technical decisions that serve business objectives has always been what separates senior engineers from code writers.

The ability to work across disciplines becomes more valuable as well. The best AI implementations often require understanding domain expertise, user experience implications, and business impact. Engineers who can bridge these contexts design better AI integrations, just as they design better systems when working with product managers, designers, and other stakeholders.

Communication skills get amplified too. Evaluating options, explaining trade-offs, and building consensus around technical decisions becomes crucial when AI can generate multiple potential solutions quickly. You’re curating and contextualizing solutions rather than just implementing them—much like how lead engineers guide technical discussions and help teams make good decisions collectively.

Building AI-Native Systems From the Ground Up

The most significant opportunities lie in designing systems built around AI capabilities from the beginning, rather than retrofitting AI into existing architectures. This requires thinking differently about how software systems work and collaborating effectively during the design process.

AI-native systems often need different patterns for data flow, error handling, and user interaction. They might handle probabilistic outcomes rather than deterministic ones, incorporate continuous learning loops, and provide transparency into decision-making processes. Engineers who understand these patterns early will have a significant advantage.

This resembles the transition any engineering team makes when adopting new paradigms. The teams that succeed are those that collaborate well during the learning process, share knowledge effectively, and iterate toward better patterns together.

Working with AI also means getting comfortable with a different development workflow. Instead of writing every function from scratch, you might orchestrate AI services, design feedback loops for model improvement, and build systems that get smarter about your application over time. The engineering challenge shifts from pure implementation toward integration and optimization.

Starting small with AI integrations in your current projects is a practical approach for seeing how AI systems can help. Add intelligent features to existing applications. Experiment with AI APIs and services. Build systems that can incorporate AI capabilities without requiring complete rewrites. Each project teaches you more about AI-native patterns, similar to how you’d gradually adopt any new technology stack.

Developing AI Collaboration Skills

Learning to work effectively with AI as a thinking partner goes beyond using AI tools. You’re developing a collaborative workflow where AI augments your problem-solving process rather than just automating tasks.

This means getting good at prompt engineering, but more importantly, learning to structure problems and code in ways that AI can help with effectively. Some problems benefit from AI’s pattern recognition capabilities. Others need AI’s ability to generate and evaluate multiple approaches quickly. Understanding when and how to use these capabilities makes you more effective.

Good engineers know when to ask colleagues for help, how to frame problems clearly, and which team members bring the right expertise to different challenges. Working with AI requires similar social and communication skills.

It’s also critical to develop good judgment about AI outputs. AI can generate impressive solutions that miss important constraints or edge cases. Engineers who can quickly evaluate AI suggestions, identify potential issues, and iterate toward better solutions will consistently outperform those who either avoid AI entirely or accept its outputs uncritically.

This mirrors how you’d work with any collaborator—trusting their expertise while applying your own judgment, asking clarifying questions, and building on their contributions with your own insights and context.

Positioning for Long-Term Success

Engineers who thrive long-term will view AI as a force multiplier for their existing strengths rather than a replacement for their role. If you’re great at system design, AI can help you explore more architectural options. If you excel at debugging, AI can help you identify patterns across larger codebases. If you’re skilled at optimization, AI can help you analyze performance bottlenecks more comprehensively.

The strategic approach involves doubling down on your strengths while developing AI collaboration skills that amplify them. You don’t need to become an AI researcher unless that’s your passion. Instead, become expert at applying AI to the problems you already enjoy solving.

This also means staying close to the business impact of your work. Engineers who understand how their technical decisions affect user experience, business metrics, and organizational goals will always be valuable, regardless of how AI capabilities evolve. The technology might change, but the need for good judgment about what to build and how to build it remains constant.

The Compound Advantage of Early Adoption

Engineers who start developing AI collaboration skills now will have years of experience when these capabilities become standard across the industry. This includes technical knowledge and intuition for when AI helps and when it doesn’t, understanding failure modes, and building robust workflows around AI capabilities.

Start with AI tools that augment your current workflow—code completion, documentation generation, test writing. Gradually expand to more complex collaborations like architectural design, system optimization, and problem analysis. Each interaction teaches you more about effective AI collaboration.

Just like learning to work well with any new team member, the key is consistent practice and honest feedback. Try different approaches, see what works, and gradually build more sophisticated collaborative patterns.

The goal is becoming highly effective at leveraging AI rather than becoming dependent on it. Engineers with this skill set will consistently deliver better results faster than those working without AI augmentation. As AI capabilities improve, this advantage compounds.

The future belongs to engineers who see AI as an opportunity to tackle harder problems, build better systems, and have greater impact. Instead of competing with AI, they’re collaborating with it to push the boundaries of what’s possible in software engineering.

The best engineers have always been force multipliers—they make everyone around them more effective. AI gives these engineers a new kind of leverage. Instead of just amplifying the capabilities of their teams, they can amplify their own problem-solving abilities and tackle challenges that were previously beyond reach.

The practices that make you great at working with AI—clear communication, structured thinking, productive collaboration, and sound judgment—are the same practices that make you great at working with people. Master these fundamentals, and you’ll thrive regardless of how the technology landscape evolves.

For years, my value came from being able to debug the trickiest issues, architect complex systems, and untangle the technical problems that had everyone else stuck. I was fast, thorough, and reliable. But in a leadership role, continuing to be the primary problem solver wasn’t scaling—it was becoming a bottleneck.

I realized that every problem I solved myself, though satisfying, was a missed opportunity to develop someone else’s problem-solving capabilities. Instead of asking “How can I fix this?” I started asking “Who could learn the most from figuring this out, and how can I set them up for success?”

This mindset shift transforms everything about how you approach engineering leadership. Instead of optimizing for immediate solutions, you optimize for building a team that can tackle increasingly complex challenges independently. Here’s what that looks like in practice.

Teaching Through Ownership, Not Tasks

The difference between assigning tasks and developing problem solvers is the difference between “implement this API endpoint” and “figure out how we should handle user authentication for this new feature.” One teaches someone to follow specifications; the other teaches them to think through trade-offs and business impact, research solutions, and make technical decisions.

Strategic delegation becomes about identifying problems that are slightly beyond someone’s current comfort zone—complex enough to require real thinking, but not so complex that they’ll get stuck without making progress. When we needed to optimize our database performance, instead of diving in myself, I paired our most eager junior backend engineer with our database expert and said, “Figure out why our queries are getting slower and what we should do about it.” (They did an excellent job and both learned new things in the process.)

The key is providing enough context for good decision-making while resisting the urge to prescribe the solution.

This means sharing the business constraints, the technical requirements, and the success criteria, then stepping back and letting them work through the problem-solving process. When they hit roadblocks, you guide them toward resources and approaches rather than answers.

What you’re really doing is teaching people to ask the right questions: What are we optimizing for? What are the constraints? What could go wrong? How will we know if it’s working? These thinking patterns transfer to every future problem they encounter.

Building Problem-Solving Muscle Through Learning

The best problem solvers aren’t necessarily the ones who know the most—they’re the ones who are best at learning what they need to know. Creating a culture where continuous learning is expected and supported turns every project into an opportunity to develop new problem-solving capabilities.

This means structuring work so that people regularly encounter unfamiliar challenges with appropriate support systems. When someone expresses curiosity about machine learning, performance optimization, or distributed systems, find ways to connect that interest to real problems your team needs to solve. The developer who wants to understand ML can take point on improving your recommendation algorithm. The engineer curious about performance can lead the investigation into why your app feels sluggish.

Internal knowledge sharing amplifies this effect. Regular deep-dive sessions where team members present problems they’ve solved create a library of problem-solving approaches that everyone can learn from. But more importantly, the act of sharing forces people to articulate their thinking process, which helps them develop more systematic approaches to future problems.

The compound effect is remarkable. Teams that prioritize learning consistently punch above their weight because they’re better at recognizing patterns, adapting to new situations, and breaking down complex problems into manageable pieces.

Communication That Enables Independent Thinking

The goal of communication in leadership isn’t just clarity—it’s creating the conditions where people can make good decisions without constantly checking in with you. This means providing not just what was decided, but the reasoning behind decisions, the factors that were considered, and the principles that guide similar situations.

When you share context richly, you’re teaching people to think through problems the way you would, but with their own insights and perspectives.

Instead of saying “use Redis for caching,” explain why caching is needed, what alternatives were considered, what trade-offs matter, and how to evaluate whether it’s working. Now when similar performance problems arise, they have a framework for thinking through solutions.

One-on-ones become especially valuable for developing problem-solving skills. These conversations are where you can understand how someone approaches challenges, what assumptions they’re making, and where their thinking might benefit from different perspectives. Often, the most helpful thing you can do is ask questions that help them think through problems more systematically.

The ultimate goal is asynchronous problem-solving—people having enough context and judgment to tackle new challenges without waiting for direction. When that happens, your team’s problem-solving capacity isn’t limited by your bandwidth.

Identifying and Developing Natural Problem-Solving Styles

Every engineer has a natural approach to problem-solving, but not everyone has had the opportunity to develop and refine that approach. Part of creating problem solvers is recognizing these natural inclinations and providing opportunities to strengthen them.

Some people are naturally systematic—they break down complex problems into smaller pieces and work through them methodically. Others are more intuitive—they see patterns and connections that aren’t immediately obvious. Some are great at asking the right questions to clarify requirements. Others excel at considering edge cases and potential failures.

The key is matching people with problems that play to their strengths while gradually expanding their toolkit. Let the systematic thinker lead the database migration planning. Give the pattern-recognizer the tricky debugging challenge. Ask the question-asker to work with product managers on requirement gathering.

But also create opportunities for people to develop complementary skills. Pair the intuitive problem solver with someone more methodical. Have the detail-oriented engineer work on a project that requires big-picture thinking. These collaborations teach people new approaches while solving real problems.

Leadership development happens naturally when people get comfortable with their own problem-solving style and learn to facilitate problem-solving in others.

Removing Obstacles to Problem-Solving Growth

The biggest barriers to developing problem solvers are often systemic rather than individual. People can’t develop good judgment if they don’t have access to the information they need to make decisions. They can’t learn from mistakes if the environment punishes experimentation. They can’t tackle complex problems if they’re constantly interrupted by urgent but low-value work.

Your role becomes creating the conditions where problem-solving skills can develop naturally.

This often means advocating upward for better tools, more reasonable deadlines, or clearer priorities. It means protecting your team’s focus time and ensuring they have access to the resources they need to dive deep into problems.

Sometimes it’s about facilitating conversations between teams so your engineers can get the context they need to make good technical decisions. Sometimes it’s about negotiating for technical debt time so people can practice the long-term thinking that prevents problems rather than just solving them reactively.

The most important obstacle to remove is the fear of making mistakes. Problem-solving skills develop through experimentation, and experimentation requires an environment where intelligent failures are treated as learning opportunities rather than performance problems.

The Multiplier Effect

What makes this approach so rewarding is that the impact compounds exponentially. A team of capable problem solvers doesn’t just solve more problems—they solve harder problems, prevent problems through better design, and create solutions that other teams can build on.

When you develop someone’s problem-solving abilities, you’re not just helping them with their current role. You’re giving them tools they’ll use throughout their career, whether they stay individual contributors or move into leadership themselves. The engineer who learns to think systematically about performance problems becomes someone who designs performant systems from the start.

The ripple effects extend beyond your immediate team. Problem solvers become mentors. They raise the bar in technical discussions. They ask better questions in design reviews. They contribute to a culture where good technical decision-making is normal rather than exceptional.

This is the ultimate lever in engineering leadership: instead of solving problems yourself, you create the conditions where great solutions emerge naturally from your team.

Instead of being the bottleneck, you become the catalyst that makes everything else work better.

The transition from solving problems to creating problem solvers is challenging because it requires patience and faith in other people’s potential. But when you see someone tackle a problem that would have stumped them six months ago, or when your team consistently delivers solutions that surprise you with their thoughtfulness, you realize you’ve built something much more valuable than any individual technical contribution: a system that continuously generates great technical work.

While working on a personal project recently, I wanted Cline to process about a hundred files that were each in subdirectories of a project. I fired up Cline and picked Gemini 2.5 Pro (context window FTW) and asked it to recurse through the subdirectories, process the files, and put the results in a new file.

Cline got to work… slowly. I watched as the “API Request…” spinner appeared for each file read and each time it saved the results. About twenty minutes and $26 later, it finished.

Okay, I thought, that’s not great, but not untenable. The cost of convenience, right? I opened up the results file to take a look and.. sigh. Not great work. It was obvious that some files had been skipped despite my very careful instructions to process each and every one.

So, like a glutton for punishment, I made a list of the files Cline had skipped and asked it to try again. Tired of babysitting, I raised the “Maximum Request Auto Approval” limit to more than I thought would be needed to finish processing the files that were left, and went to take a coffee break.

When I came back, Cline was done. The results? Still not great. Files had still been skipped, some files that were processed were missing results, and, oh, my task bill had risen to $78.

Okay, this was untenable. Reading all this data into context was costly and slow.

Then the coffee started to kick in, I guess, because it dawned on me: why in the world was I using expensive API calls to do something a Bash one-liner could do?

“Cline, write a Bash command that will recurse through the data/ directory and obtain the content of all the files and copy it into a single new file.”

Which produced:

find data/ -type f -exec cat {} + > all_data.txt

This command:

• find data/ - searches recursively in the data directory.
• -type f - specifies that we’re looking for files only (not directories, links, etc.).
• -exec cat {} + - for all files found, execute the cat command. The {} is a placeholder for the filename, and the + is a crucial optimization that groups multiple filenames into a single cat command, avoiding the overhead of launching a new process for every single file.
• > all_data.txt - redirects the standard output of the cat command (which is the concatenated content of all the files) into a new file named all_data.txt.

Then I asked Cline to read the resulting all_data.txt file, process it, and output the results.

It took about two minutes.

And it cost me $0.78.

What just happened?

My initial naive approach had accidentally created a perfect storm of computational inefficiency.

When Cline processed each file individually, it was making separate API calls for every single operation - reads, writes, the works. With about 100 files, that meant roughly 200+ API calls, each one spinning up its own network round-trip with all the latency that entails. Every time I saw that “API Request…” spinner, I was watching money float away into the ether.

But here’s the kicker: large language models like Gemini charge based on token consumption.

It’s not just the file content they’re charging for; every single API call also included the entire conversation history, system prompts, and my instructions.

With a stateless API, that context has to be re-transmitted with every single request. If my average context was around 10,000 tokens and I made 200 calls, I burned through 2 million tokens (10,000 * 200) on overhead alone, before even counting the actual data.

Combining all the files with bash flipped this whole equation on its head. Instead of 200 API calls, I made exactly one. Instead of bearing the network latency for every file operation, combining the files locally on my machine meant the filesystem could actually optimize that work. What had taken almost an hour of network round-trips for Gemini to access all the data was reduced to a couple hundred milliseconds of local file operations.

The expensive lesson in algorithmic thinking

This whole debacle reminded me why understanding the cost model of your tools matters just as much as understanding their capabilities. API pricing is designed around per-request and per-token charges, which naturally punishes fine-grained operations. It’s similar to how databases are optimized for bulk operations rather than processing individual rows - the overhead of each transaction quickly becomes the bottleneck.

My first approach had O(n) complexity for API calls, where n equals the number of files. The bash solution reduced that to O(1) by batching everything locally first. That’s the difference between linear scaling and constant cost, and at $78, I felt every bit of that mathematical distinction.

There’s also something to be said about data locality here. My original method couldn’t take advantage of any local caching or filesystem optimizations. Every operation had to go over the network to an API server, get processed, and come back. The bash approach kept everything local until the very end, letting my machine’s filesystem cache work its magic.

The real cost of convenience

I’d fallen into the trap of thinking that because I could use an AI tool for everything, I should use it for everything. But there’s a difference between leveraging AI for tasks that require intelligence and using it as an expensive replacement for basic system utilities.

The irony is that I probably spent more mental energy managing and troubleshooting the AI approach than I would have just thinking through the problem for five minutes and reaching for the right tool from the start. Sometimes the most sophisticated solution is knowing when to employ a basic tool.

My little bit of laziness bought me a $78 lesson that boils down to this: always understand the economic model of your tools, especially when they’re priced per operation. The most elegant and cost-effective solution isn’t always the newest and most technically exciting one.

Conversational AI has changed the game entirely. Instead of starting with a blank page and trying to remember every detail a new team member might need, you can have AI help you think through the process systematically. The result isn’t just better docs—it’s documentation that actually serves your team’s needs as you grow and evolve.

Here’s how to use AI to build documentation that scales with your team and genuinely improves how you work together.

Documentation That Welcomes New Team Members

The best part about using AI for documentation is that it naturally thinks from an outsider’s perspective. While you and your team already understand your system’s quirks and design decisions, AI starts fresh every time—much like a new hire would.

Most conversational AI tools allow you to upload code files or paste code snippets. You can then use prompts that help surface the knowledge your team takes for granted:

Write documentation for a new software engineer joining our team. Assume they’re experienced but know nothing about our specific domain, architecture decisions, or business logic. Include the “why” behind non-obvious technical choices and flag anything that might seem strange or unexpected to an outside developer.

This approach reveals the implicit knowledge that experienced team members forget to document—why certain patterns exist, what alternatives were considered, and where the potential gotchas are. It transforms documentation from a chore into a useful onboarding tool that actually reduces the time senior developers spend answering questions.

To create comprehensive documentation you can use immediately, provide the AI with additional context such as:

• What the application does and who uses it
• Key architectural decisions and their reasoning
• Setup and deployment processes
• Integration points with other systems
• Common troubleshooting scenarios

Your role becomes reviewing and refining rather than writing from scratch—which is often the difference between documentation that gets done and documentation that gets skipped.

Operational Documentation That Actually Helps

One of the most valuable types of documentation is also the most overlooked: information organized for when things go wrong. During incidents, you need answers fast, not comprehensive explanations.

AI excels at creating focused, actionable documentation because you can specify exactly what situation you’re optimizing for:

Create incident response documentation for this codebase. Focus on: 1) How to quickly identify what component is failing, 2) Common failure modes and their symptoms, 3) Step-by-step debugging workflows, 4) Who to contact for different types of issues. Write this as if the person reading it is stressed, tired, and needs answers in under 5 minutes.

This type of documentation serves a completely different purpose than your standard README or API docs. It’s designed for when your most knowledgeable developers aren’t available and someone needs to resolve an issue quickly.

The beauty of AI-generated operational docs is that they’re naturally structured for scan-ability rather than linear reading—exactly what you need during high-pressure situations.

Capturing Institutional Knowledge

Here’s where AI really shines: helping you identify and document the knowledge that exists only in people’s heads. This institutional knowledge is often the difference between a change that takes 30 minutes and one that takes 3 hours of debugging.

You can surface these knowledge gaps by asking AI to analyze your code from a risk perspective:

Analyze this code and identify areas where domain knowledge or business context would be critical for modification. What would a developer need to know about our business, users, or regulatory requirements to safely change this code? What assumptions about data, timing, or external systems are embedded here?

For inline documentation, you can focus on the business logic and integration points that aren’t obvious from the code itself:

Add inline documentation to this code file without changing any of the code. Focus on documenting business logic, data assumptions, and integration points that wouldn’t be obvious to someone unfamiliar with our domain.

This process often improves the code itself—explaining your logic to AI sometimes reveals opportunities for clearer naming, better structure, or simplified approaches.

Making Documentation a Team Superpower

The real opportunity here isn’t just better individual documentation—it’s democratizing the ability to create good documentation across your entire team. Developers who previously avoided writing docs because they didn’t know where to start now have a collaborative partner to help structure their thoughts.

1. Start with high-impact documentation: Focus on onboarding guides and operational runbooks first. These provide immediate value and create positive momentum around documentation practices.
2. Use AI to improve existing docs: You can ask AI to review and improve documentation you already have, suggesting missing information or better organization.
3. Make it iterative: Documentation doesn’t need to be perfect on the first pass. Use AI to create initial drafts that you can refine based on team feedback and real usage patterns.
4. Leverage different formats: AI can help create everything from README files to inline comments to architectural decision records, adapting the style and depth based on the audience and purpose.

Practical Tips for Better Results

When working with AI to create documentation, providing context about the intended audience and use case dramatically improves the output. Explain not just what the code does, but who will be using the documentation and in what situations.

For complex codebases, you might get better results by working with smaller sections and then asking AI to help you organize everything into a coherent structure. Many AI tools can also provide downloadable files if you specify that in your prompt, which saves time on longer documents.

The goal isn’t to replace human judgment in documentation—it’s to remove the barriers that prevent good documentation from getting written in the first place. AI handles the initial structure and comprehensive coverage, while you focus on accuracy, team-specific context, and ensuring the documentation actually serves your workflows.

Good documentation transforms how teams work together. It reduces interruptions, accelerates onboarding, and creates resilience when key team members aren’t available. With AI handling the heavy lifting of initial creation, maintaining comprehensive documentation becomes achievable rather than aspirational.

Your future team members (and your future self during the next production incident) will definitely appreciate the investment.

So I brought together everything that’s great about Hugo plus everything that’s great about sharing your 3AM thoughts with the world from your phone, thanks to Collected Notes. I put it in a new Hugo site template with a fancy new theme I call Quint.

You can deploy the Quint site template with one button (this button):

The Quint template can use the Collected Notes app as a CMS and also saves your posts to the site repository, for redundancy. It fetches new posts each time you build, and if you’re deploying via Netlify or GitHub Actions, you can use a webhook to deploy the site whenever you make a new post with Collected Notes.

To set up your own site:

1. Deploy the Quint template to Netlify with the button above, or clone the repo if you plan to use another deployment solution.
2. Sign up for Collected Notes if you haven’t already (there’s a free plan) and download the Collected Notes app on your iPhone.
3. Update the utils/fetch-posts.js file to use your Collected Notes site name.
4. Allow the GitHub Action to push changes back to your repository to save your posts. Under Settings > Actions > General > Workflow permissions, choose Read and write permissions.

Netlify will trigger a new build each time you push to your site repo, or, if you have a Collected Notes Premium subscription, you can set a Netlify Build Hook URL in your Collected Notes site settings to automatically redeploy the site when you make a post or update an existing post.

I hope you found this post helpful! If you have any suggestions or improvements to add, feel free to contribute to the repository.

There are bound to be situations in which this isn’t enough, such as when you want to read in a large amount of text from a file. Using the OpenAI API allows you to send many more tokens in a messages array, with the maximum number depending on your chosen model. This lets you provide large amounts of text to ChatGPT using chunking. Here’s how.

Chunking your input

The gpt-4 model currently has a maximum content length token limit of 8,192 tokens. (Here are the docs containing current limits for all the models.) Remember that you can first apply text preprocessing techniques to reduce your input size – in my previous post I achieved a 28% size reduction without losing meaning with just a little tokenization and pruning.

When this isn’t enough to fit your message within the maximum message token limit, you can take a general programmatic approach that sends your input in message chunks. The goal is to divide your text into sections that each fit within the model’s token limit. The general idea is to:

1. Tokenize and split text into chunks based on the model’s token limit. It’s better to keep message chunks slightly below the token limit since the token limit is shared between your message and ChatGPT’s response.
2. Maintain context between chunks, e.g. avoid splitting a sentence in the middle.

Each chunk is sent as a separate message in the conversation thread.

Handling responses

You send your chunks to ChatGPT using the OpenAI library’s ChatCompletion. ChatGPT returns individual responses for each message, so you may want to process these by:

1. Concatenating responses in the order you sent them to get a coherent answer.
2. Manage conversation flow by keeping track of which response refers to which chunk.
3. Formatting the response to suit your desired output, e.g. replacing \n with line breaks.

Putting it all together

Using the OpenAI API, you can send multiple messages to ChatGPT and ask it to wait for you to provide all of the data before answering your prompt. Being a language model, you can provide these instructions to ChatGPT in plain language. Here’s a suggested script:

Prompt: Summarize the following text for me

To provide the context for the above prompt, I will send you text in parts. When I am finished, I will tell you “ALL PARTS SENT”. Do not answer until you have received all the parts.

I created a Python module, chatgptmax, that puts all this together. It breaks up a large amount of text by a given maximum token length and sends it in chunks to ChatGPT.

You can install it with pip install chatgptmax, but here’s the juicy part:

import os
import openai
import tiktoken

# Set up your OpenAI API key
# Load your API key from an environment variable or secret management service
openai.api_key = os.getenv("OPENAI_API_KEY")

def send(
    prompt=None,
    text_data=None,
    chat_model="gpt-3.5-turbo",
    model_token_limit=8192,
    max_tokens=2500,
):
    """
    Send the prompt at the start of the conversation and then send chunks of text_data to ChatGPT via the OpenAI API.
    If the text_data is too long, it splits it into chunks and sends each chunk separately.

    Args:
    - prompt (str, optional): The prompt to guide the model's response.
    - text_data (str, optional): Additional text data to be included.
    - max_tokens (int, optional): Maximum tokens for each API call. Default is 2500.

    Returns:
    - list or str: A list of model's responses for each chunk or an error message.
    """

    # Check if the necessary arguments are provided
    if not prompt:
        return "Error: Prompt is missing. Please provide a prompt."
    if not text_data:
        return "Error: Text data is missing. Please provide some text data."

    # Initialize the tokenizer
    tokenizer = tiktoken.encoding_for_model(chat_model)

    # Encode the text_data into token integers
    token_integers = tokenizer.encode(text_data)

    # Split the token integers into chunks based on max_tokens
    chunk_size = max_tokens - len(tokenizer.encode(prompt))
    chunks = [
        token_integers[i : i + chunk_size]
        for i in range(0, len(token_integers), chunk_size)
    ]

    # Decode token chunks back to strings
    chunks = [tokenizer.decode(chunk) for chunk in chunks]

    responses = []
    messages = [
        {"role": "user", "content": prompt},
        {
            "role": "user",
            "content": "To provide the context for the above prompt, I will send you text in parts. When I am finished, I will tell you 'ALL PARTS SENT'. Do not answer until you have received all the parts.",
        },
    ]

    for chunk in chunks:
        messages.append({"role": "user", "content": chunk})

        # Check if total tokens exceed the model's limit and remove oldest chunks if necessary
        while (
            sum(len(tokenizer.encode(msg["content"])) for msg in messages)
            > model_token_limit
        ):
            messages.pop(1)  # Remove the oldest chunk

        response = openai.ChatCompletion.create(model=chat_model, messages=messages)
        chatgpt_response = response.choices[0].message["content"].strip()
        responses.append(chatgpt_response)

    # Add the final "ALL PARTS SENT" message
    messages.append({"role": "user", "content": "ALL PARTS SENT"})
    response = openai.ChatCompletion.create(model=chat_model, messages=messages)
    final_response = response.choices[0].message["content"].strip()
    responses.append(final_response)

    return responses

Here’s an example of how you can use this module with text data read from a file. (chatgptmax also provides a convenience method for getting text from a file.)

# First, import the necessary modules and the function
import os

from chatgptmax import send

# Define a function to read the content of a file
def read_file_content(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        return file.read()

# Use the function
if __name__ == "__main__":
    # Specify the path to your file
    file_path = "path_to_your_file.txt"
    
    # Read the content of the file
    file_content = read_file_content(file_path)
    
    # Define your prompt
    prompt_text = "Summarize the following text for me:"
    
    # Send the file content to ChatGPT
    responses = send(prompt=prompt_text, text_data=file_content)
    
    # Print the responses
    for response in responses:
        print(response)

Error handling

While the module is designed to handle most standard use cases, there are potential pitfalls to be aware of:

• Incomplete sentences: If a chunk ends in the middle of a sentence, it might alter the meaning or context. To mitigate this, consider ensuring that chunks end at full stops or natural breaks in the text. You could do this by separating the text-chunking task into a separate function that:
  1. Splits the text into sentences.
  2. Iterates over the sentences and adds them to a chunk until the chunk reaches the maximum size.
  3. Starts a new chunk when the current chunk reaches the maximum size or when adding another sentence would exceed the maximum size.
• API connectivity issues: There’s always a possibility of timeouts or connectivity problems during API calls. If this is a significant issue for your application, you can include retry logic in your code. If an API call fails, the script could wait for a few seconds and then try again, ensuring that all chunks are processed.
• Rate limits: Be mindful of OpenAI API’s rate limits. If you’re sending many chunks in rapid succession, you might hit these limits. Introducing a slight delay between calls or spreading out requests can help avoid this.

Optimization

As with any process, there’s always room for improvement. Here are a couple of ways you might optimize the module’s chunking and sending process further:

• Parallelizing API calls: If OpenAI API’s rate limits and your infrastructure allow, you could send multiple chunks simultaneously. This parallel processing can speed up the overall time it takes to get responses for all chunks. Unless you have access to OpenAI’s 32k models or need to use small chunk sizes, however, parallelism gains are likely to be minimal.
• Caching mechanisms: If you find yourself sending the same or similar chunks frequently, consider implementing a caching system. By storing ChatGPT’s responses for specific chunks, you can retrieve them instantly from the cache the next time, saving both time and API calls.

Now what

If you found your way here via search, you probably already have a use case in mind. Here are some other (startup) ideas:

• You’re a researcher who wants to save time by getting short summaries of many lengthy articles.
• You’re a legal professional who wants to analyze long contracts by extracting key points or clauses.
• You’re a financial analyst who wants to pull a quick overview of trends from a long report.
• You’re a writer who wants feedback on a new article or chapter… without having to actually show it to anyone yet.

Do you have a use case I didn’t list? Let me know about it! In the meantime, have fun sending lots of text to ChatGPT.

Text preprocessing can help shorten and refine your input, ensuring that ChatGPT can grasp the essence without getting overwhelmed. In this article, we’ll explore these techniques, understand their importance, and see how they make your interactions with tools like ChatGPT more reliable and productive.

Text preprocessing

Text preprocessing prepares raw text data for analysis by NLP models. Generally, it distills everyday text (like full sentences) to make it more manageable or concise and meaningful. Techniques include:

• Tokenization: splitting up text by sentences or paragraphs. For example, you could break down a lengthy legal document into individual clauses or sentences.
• Extractive summarization: selecting key sentences from the text and discarding the rest. Instead of reading an entire 10-page document, extractive summarization could pinpoint the most crucial sentences and give you a concise overview without delving into the details.
• Abstractive summarization: generating a concise representation of the text content, for example, turning a 10-page document into a brief paragraph that captures the document’s essence in new wording.
• Pruning: removing redundant or less relevant parts. For example, in a verbose email thread, pruning can help remove all the greetings, sign-offs, and other repetitive elements, leaving only the core content for analysis.

While all these techniques can help reduce the size of raw text data, some of these techniques are easier to apply to general use cases than others. Let’s examine how text preprocessing can help us send a large amount of text to ChatGPT.

Tokenization and ChatGPT input limits

In the realm of Natural Language Processing (NLP), a token is the basic unit of text that a system reads. At its simplest, you can think of a token as a word, but depending on the language and the specific tokenization method used, a token can represent a word, part of a word, or even multiple words.

While in English we often equate tokens with words, in NLP, the concept is broader. A token can be as short as a single character or as long as a word. For example, with word tokenization, the sentence “Unicode characters such as emojis are not indivisible. ✂️” can be broken down into tokens like this: [“Unicode”, “characters”, “such”, “as”, “emojis”, “are”, “not”, “indivisible”, “.”, “✂️”]

In another form called Byte-Pair Encoding (BPE), the same sentence is tokenized as: [“Un”, “ic”, “ode”, " characters", " such", " as", " em, “oj”, “is”, " are", " not", " ind", “iv”, “isible”, “.”, " �", “�️”]. The emoji itself is split into tokens containing its underlying bytes.

Depending on the ChatGPT model chosen, your text input size is restricted by tokens. Here are the docs containing current limits. BPE is used by ChatGPT to determine token count, and we’ll discuss it more thoroughly later. First, we can programmatically apply some preprocessing techniques to reduce our text input size and use fewer tokens.

A general programmatic approach

For a general approach that can be applied programmatically, pruning is a suitable preprocessing technique. One form is stop word removal, or removing common words that might not add significant meaning in certain contexts. For example, consider the sentence:

“I always enjoy having pizza with my friends on weekends.”

Stop words are often words that don’t carry significant meaning on their own in a given context. In this sentence, words like “I”, “always”, “enjoy”, “having”, “with”, “my”, “on” are considered stop words.

After removing the stop words, the sentence becomes:

“pizza friends weekends.”

Now, the sentence is distilled to its key components, highlighting the main subject (pizza) and the associated context (friends and weekends). If you find yourself wishing you could convince people to do this in real life (coughmeetingscough)… you aren’t alone.

Stop word removal is straightforward to apply programmatically: given a list of stop words, examine some text input to see if it contains any of the stop words on your list. If it does, remove them, then return the altered text.

def clean_stopwords(text: str) -> str:
    stopwords = ["a", "an", "and", "at", "but", "how", "in", "is", "on", "or", "the", "to", "what", "will"]
    tokens = text.split()
    clean_tokens = [t for t in tokens if not t in stopwords]
    return " ".join(clean_tokens)

To see how effective stop word removal can be, I took the entire text of my Tech Leader Docs newsletter (17,230 words consisting of 104,892 characters) and processed it using the above function. How effective was it? The resulting text contained 89,337 characters, which is about a 15% reduction in size.

Other pruning techniques can also be applied programmatically. Removing punctuation, numbers, HTML tags, URLs and email addresses, or non-alphabetical characters are all valid pruning techniques that can be straightforward to apply. Here is a function that does just that:

import re

def clean_text(text):
    # Remove URLs
    text = re.sub(r'http\S+', '', text)
    
    # Remove email addresses
    text = re.sub(r'\S+@\S+', '', text)
    
    # Remove everything that's not a letter (a-z, A-Z)
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    
    # Remove whitespace, tabs, and new lines
    text = ''.join(text.split())

    return text

What measure of length reduction might we be able to get from this additional processing? Applying these techniques to the remaining characters of Tech Leader Docs results in just 75,217 characters; an overall reduction of about 28% from the original text.

More opinionated pruning, such as removing short words or specific words or phrases, can be tailored to a specific use case. These don’t lend themselves well to general functions, however.

Now that you have some text processing techniques in your toolkit, let’s look at how a reduction in characters translates to fewer tokens used when it comes to ChatGPT. To understand this, we’ll examine Byte-Pair Encoding.

Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) is a subword tokenization method. It was originally introduced for data compression but has since been adapted for tokenization in NLP tasks. It allows representing common words as tokens and splits more rare words into subword units. This enables a balance between character-level and word-level tokenization.

Let’s make that more concrete. Imagine you have a big box of LEGO bricks, and each brick represents a single letter or character. You’re tasked with building words using these LEGO bricks. At first, you might start by connecting individual bricks to form words. But over time, you notice that certain combinations of bricks (or characters) keep appearing together frequently, like “th” in “the” or “ing” in “running.”

BPE is like a smart LEGO-building buddy who suggests, “Hey, since ’th’ and ‘ing’ keep appearing together a lot, why don’t we glue them together and treat them as a single piece?” This way, the next time you want to build a word with “the” or “running,” you can use these glued-together pieces, making the process faster and more efficient.

Colloquially, the BPE algorithm looks like this:

1. Start with single characters.
2. Observe which pairs of characters frequently appear together.
3. Merge those frequent pairs together to treat them as one unit.
4. Repeat this process until you have a mix of single characters and frequently occurring character combinations.

BPE is a particularly powerful tokenization method, especially when dealing with diverse and extensive vocabularies. Here’s why:

• Handling rare words: Traditional tokenization methods might stumble upon rare or out-of-vocabulary words. BPE, with its ability to break words down into frequent subword units, can represent these words without needing to have seen them before.
• Efficiency: By representing frequent word parts as single tokens, BPE can compress text more effectively. This is especially useful for models like ChatGPT, where token limits apply.
• Adaptability: BPE is language-agnostic. It doesn’t rely on predefined dictionaries or vocabularies. Instead, it learns from the data, making it adaptable to various languages and contexts.

In essence, BPE strikes a balance, offering the granularity of character-level tokenization and the context-awareness of word-level tokenization. This hybrid approach ensures that NLP models like ChatGPT can understand a wide range of texts while maintaining computational efficiency.

Sending lots of text to ChatGPT

At time of writing, a message to ChatGPT via its web interface has a maximum token length of 4,096 tokens. If we assume the prior mentioned percent reduction as an average, this means you could reduce text of up to 5,712 tokens down to the appropriate size with just text preprocessing.

What about when this isn’t enough? Beyond text preprocessing, larger input can be sent in chunks using the OpenAI API. In my next post, I’ll show you how to build a Python module that does exactly that.

The solution isn’t mastering Git’s most obscure commands or memorizing every branching strategy ever invented. It’s adopting a workflow that prevents the chaos in the first place. Here’s the approach I use personally and recommend for small teams that want to ship code without the drama.

A Protected Main Branch (No Exceptions)

First rule: no human should have direct push permissions to your master branch. Ever. I don’t care if you’re the CTO, the person who started the repository, or the only one who “really understands the codebase.” The moment you start making exceptions is the moment you start breaking things in production.

Your main branch should be your source of truth for what’s currently deployed. When you create a release from the latest tag, that code should work. Period. If you’re not deploying frequently and automatically, you’re missing out on one of the biggest advantages of this approach.

One Issue, One Branch, One PR (Keep It Simple)

Here’s where most teams overcomplicate things. You’ve got your issues tracked somewhere (and if you don’t, we need to have a different conversation). Each issue represents a well-defined piece of work that can be merged and deployed without breaking anything. Maybe it’s a new feature, a component update, or a bug fix. Doesn’t matter—the process stays the same.

The key is keeping branches short-lived. For a small commercial team, we’re talking days, not weeks. Open source projects with volunteer contributors might stretch this to a few weeks or months, but the principle remains: finish the work, get it reviewed, merge it, and move on.

Here’s what this looks like in practice. Say you’re working on (#28) Add user settings page:

# Get all the latest work locally
git checkout master
git pull
# Start your new branch from master
git checkout -b 28/add-settings-page

Work on the issue, and periodically merge master to stay current:

# Commit to your issue branch
git commit ...
# Get the latest work on master
git checkout master
git pull
# Return to your issue branch and merge in master
git checkout 28/add-settings-page
git merge master

I know some of you are thinking “but what about rebasing?” Look, I like rebasing too. A clean, linear history is beautiful. But I’ve seen too many developers get tangled up in interactive rebasing purgatory while accidentally dropping commits or creating conflicts that didn’t need to exist. Merging might create a slightly messier history, but it’s predictable and reversible. When you’re optimizing for team productivity, predictable wins over pretty.

When your work is ready, open a PR against master. Tests run automatically. Your teammates review the code and leave helpful feedback (hopefully). Maybe you deploy a preview version to staging. Once everything looks good, merge it, close the issue, and delete the branch. (Yes. Delete it. It will be okay.)

Avoiding the Common Disasters

Here are the patterns I see that turn this simple workflow into a nightmare:

Branching off feature branches: This is how you end up with dependency chains that make merging feel like getting the Christmas lights out of storage. Someone starts working on feature B before feature A is merged, then feature C depends on both, and suddenly you need a whiteboard and a computer science degree to figure out the merge order. Just branch from the latest master. Always.

Scope creep on branches: You’re implementing the user settings page, but then you notice the button component could use some updates, and hey, while we’re at it, let’s refactor this entire authentication flow. Stop. That’s how a three-day task becomes a three-week (or three-month) PR that nobody wants to review. Stick to the issue at hand.

Keeping dead branches around: Your branch got merged last month, but it’s still sitting there in the repository like a ghost haunting your Git history. Delete merged branches immediately. Future you will thank present you for not having to scroll through fifty old feature branches trying to find the one you’re actually working on.

Why This Actually Works

This workflow works because it aligns with how small teams actually operate. You don’t need the complexity of GitFlow when you’ve got eight developers. You don’t need long-lived release branches when you’re deploying multiple times per week. You need a system that gets out of your way and lets you focus on building software.

The protection on master means your deployable code stays deployable. The one-issue-per-branch rule keeps PRs reviewable and prevents feature creep. The short-lived branches mean conflicts are small and manageable. The regular merging from master means you catch integration issues early when they’re easy to fix.

Most importantly, this workflow is boring in the best possible way. Once your team gets the hang of it, Git becomes background infrastructure instead of a daily source of stress. Developers stop losing work to merge conflicts. Code reviews become focused discussions about functionality rather than archaeology expeditions through weeks of accumulated changes.

The best development workflows are the ones you don’t have to think about. They handle the routine stuff automatically so you can focus on the interesting problems. This Git strategy does exactly that—it gets out of your way and lets you ship code with confidence.

Here are a few things past colleagues have said about my work in software engineering leadership:

"…the level of organization and process we had was amazing and I personally want to duplicate as much of it as possible!"

"…remember how excited I was in my interview with you Victoria about how well-organized [your previous company] was and how well-thought-out your processes were? I dare say 99.99% of startups, and many larger companies aren’t very organized and it becomes a pain point as they grow. Between the docs and the other processes you had in place, I think you could write a really good book that would be a ‘Blueprint’ for forming a technical team."

"@Victoria Drake you publish a book and I will buy at least a dozen copies to hand out."

Instead of waiting to collect all this information in book form, the first post goes out the first week of January. Here’s a preview of some of the skills you’ll learn in future editions:

• How to set up remote and asynchronous work to be super-efficient
• How to remove the right restrictions to make your processes more productive
• Setting up safeguards to ensure progress doesn’t slow down when you take time off
• Hiring for the culture you want
• Creating the number one secret weapon that makes everyone on your team more productive

These insights are for senior engineers and engineering managers, as well as anyone who wants to start establishing yourself as a leader on your team right away.

You can subscribe monthly, or lock in an early-bird New Year Special subscription for 30% off before Jan 1.

My website Victoria.dev isn’t going anywhere, I’ve just decided to put more effort into this new resource. The paid newsletter format strikes a good balance between the immediate delivery of information, skin-in-the-game for you to implement these ideas, and motivation for me to keep sharing how I acquired the skills and knowledge that live in my head, just as I’ve done for years on my blog.

I’ve held all kinds of roles on engineering teams, from contract developer to Director of Engineering. Over the years I’ve heard from past colleagues and peers about confusing processes, time-wasting meetings, and poor leadership in companies of all sizes. With The Tech Leader Docs, I hope to help those in positions to create positive change in their organization (including you!) to turn that feedback around.

Subscribe today and lock in 30% off. You’ll start 2022 with the practical skills it takes to build a successful engineering team: smarter strategies, less toil, and happier and more productive developers.

Here’s my stupid-simple strategy for tracking and checking off my to-do list.

One page at a time

Plenty of methodologies recommend using sections or different pages of your book for monthly, weekly, and daily views; others advocate for creating sections for each category, such as “Home Tasks” and “Work Tasks” and other such time-wasters. All of this is unnecessary.

A to-do list works because it’s in your face and hard to miss. When you write things down on different pages, they become easy to miss. Don’t do that.

Use one page at a time. Write down one task under another. Don’t sort them, prioritize them (yet), or categorize anything. Just write them down on the current page, where you’re guaranteed to look when you lay eyes on your notebook next.

Intuitive notation

I use my notebook for two things: short notes (just a bit of information – nothing to do) and tasks (something to do). This translates to a notation system of three possible states:

• It’s a note, indicated with a bullet point
• It’s a new task, indicated with a checkbox
• It’s a completed task, with the checkbox checked and the line struck out (because strike-throughs are satisfying)

I use a checkbox to distinguish tasks from notes because I’m an old-school HTML fan, but you do you.

You may like to add your own embellishments to this: I sometimes denote an urgent item with an asterisk. You might like to use a color pen or highlighter (avoid the bullet journal rabbit hole – another time-waster). Just keep it simple, repeatable, and intuitive.

When it’s time to turn the page

When life gets busy, you might fill up a page pretty quickly. If one or two tasks haven’t yet been crossed off, they’re liable to be forgotten. You can avoid this by carrying tasks over to the next page.

It’s straightforward: cross out the task on the page that’s filled up. Turn the page and write it down there again.

That’s silly, you might say, that’s a waste of energy! By the time I write it down all over again, I could’ve done half of it already.

…

I’ll wait.

…

The clever bit about carrying a task over is taking the opportunity to evaluate it. If the task is really a five-minute thing, more often than not, I go ahead and take care of it right there and then. If it’s a longer endeavor, the friction of writing it down again gives me the chance to answer the question of whether it’s something I feel strongly about doing (and hence whether it’s really important that I do it at all). It might not be, and that’s fine. I cross it out and don’t do it. If it is an important task, carrying it over means it remains front of mind until I can make the time to get it done.

Time well spent doing

I’ve explored a myriad of task list apps, pre-printed to-do lists and journals, and all kinds of digital notes for tracking work. I consistently keep returning to the feel of pen on paper and an open notebook on my desk. Why? Minimal cognitive load.

No time spent categorizing and labeling tasks in a complicated system. No time spent remembering how to open that app, where you stored that todo.txt file, or deciding whether to write something down under your weekly or daily plan. No tasks lost in an invisible backlog that grows over the years, becoming more and more infeasible.

Just pen and paper, one page at a time, and the satisfaction of getting things done.

This guide will walk you through setting up Pi-hole on an AWS Lightsail instance that acts as your VPN thanks to OpenVPN. It’s a more succinct version of the official Pi-hole docs for OpenVPN, made specifically for Lightsail with a few tips and tricks added in, because you deserve it.

Create and connect to a Lightsail instance

1. Log in or sign up to AWS and create a Lightsail Instance.

2. Under Select a platform, choose Linux/Unix.

3. Under Select a blueprint, choose the OS Only button.

4. Select the latest officially supported Ubuntu server.

5. You can save a tidbit of effort by putting the following into the Launch script box:

   # Update installed packages
   sudo apt-get update
   sudo apt-get upgrade -y
   

6. Create a new SSH key for this server and ensure you download the .pem.

7. Choose your plan. The $3.50 USD instance is sufficient.

8. Give it a name then click Create instance.

9. Stare eagerly at the page until the instance status is Running, then go to the Networking tab.

10. Create a Static IP and attach it to your new instance. Remember that static IP addresses are free only while attached to an instance.

11. Click on your instance name to return to its dashboard. Go back to the Networking tab. It’ll look a bit different now.

12. Under IPv6 networking, click the toggle to turn it off (unless you know what you are doing and you want IPv6 for some reason. Most of y’all don’t need it).

13. Under IPv4 Firewall, delete the rule for HTTP.

14. Click Add rule. In the Application dropdown, choose Custom.

    • For Protocol, choose UDP.
    • In the Port or range input, enter a UDP port for the OpenVPN server to run on. (It’s typically 1194, which you can choose to use, but you might like a different number for security purposes. Port range is 0-65535.)

15. Connect using SSH and your new key pair, either in your terminal or on the Connect tab with the browser-based client.

Install OpenVPN on your server

After connecting to your server using SSH, install OpenVPN on your server.

# Download OpenVPN
wget https://git.io/vpn -O openvpn-install.sh
chmod 755 openvpn-install.sh
sudo ./openvpn-install.sh

You’ll see:

Welcome to this OpenVPN road warrior installer!

This server is behind NAT. What is the public IPv4 address or hostname?
Public IPv4 address / hostname [x.xx.xxx.xxx]:

…where the default option is your static IP that you set up earlier. Hit return to accept this. Then:

Which protocol should OpenVPN use?
    1) UDP (recommended)
    2) TCP
Protocol [1]: 1

Choose 1 or hit return. Then:

What port should OpenVPN listen to?
Port [1194]: #####

Enter the UDP port number you chose earlier. Then:

Select a DNS server for the clients:
    1) Current system resolvers
    2) Google
    3) 1.1.1.1
    4) OpenDNS
    5) Quad9
    6) AdGuard
DNS server [1]: 1

Choose 1 or hit return. Then:

Enter a name for the first client:
Name [client]: pihole

The Pi-hole will be the client. Name it as you like then Press any key to continue...

OpenVPN will set itself up. Confirm that tun0 has the interface address 10.8.0.1/24 with the following command:

ip addr show tun0

This ensures that the Pi-hole will be set up properly. Now, about that:

Install and configure Pi-hole

On your Lightsail instance, install Pi-hole.

# Download and install Pi-hole
curl -sSL https://install.pi-hole.net | bash

This runs the Pi-hole automated installer. You’ll see some prompts which you can answer using the enter key, arrow keys, tab, and space bar for selecting an option.

The important things:

1. When you see Choose An Interface, ensure you pick tun0. It isn’t the default selection.
2. You’ll need to set the IPv4 address to the interface address you viewed previously using the ip addr command: 10.8.0.1/24. This ensures the Pi-hole uses the VPN.

At time of writing, the second item above wasn’t presented as an option in the automated installer. After the Pi-hole installer finishes, manually change the IP address by editing the configuration file:

> sudo vim /etc/pihole/setupVars.conf

Change the IPV4_ADDRESS to 10.8.0.1/24 and save the file. Restart the Pi-hole with: pihole restartdns.

If you mess up, you can redo the configuration with pihole reconfigure.

Finally, you’ll configure the VPN to use the Pi-hole.

Configure OpenVPN

Confirm the address of the tun0 interface with:

ip a | grep -C 1 'tun0'

You should see: inet 10.8.0.1/24 in there.

Edit the OpenVPN config file with:

sudo vim /etc/openvpn/server/server.conf

Change the line that starts with push "dhcp-option… to use the Pi-hole’s IP address that you confirmed above:

push "dhcp-option DNS 10.8.0.1"

If any other lines start with push "dhcp-option…, comment those out.

If you want to log OpenVPN traffic, add these lines to the end of the file:

log /var/log/openvpn.log
verb 3

Save the config. If you forgot to open Vim with sudo, use the tee trick: :w !sudo tee %, then O, then :q!.

Restart OpenVPN with sudo systemctl restart openvpn-server@server.

Configure firewall

Run the following to control traffic to the server as described here.

sudo iptables -I INPUT -i tun0 -j ACCEPT
sudo iptables -A INPUT -i tun0 -p tcp --destination-port 53 -j ACCEPT
sudo iptables -A INPUT -i tun0 -p udp --destination-port 53 -j ACCEPT
sudo iptables -A INPUT -i tun0 -p tcp --destination-port 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --destination-port 22 -j ACCEPT
sudo iptables -A INPUT -p tcp --destination-port 1194 -j ACCEPT
sudo iptables -A INPUT -p udp --destination-port 1194 -j ACCEPT
sudo iptables -I INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -I INPUT -i lo -j ACCEPT
sudo iptables -P INPUT DROP

# Optionally, also block HTTPS advertisements while you're here.
sudo iptables -A INPUT -p udp --dport 80 -j REJECT --reject-with icmp-port-unreachable
sudo iptables -A INPUT -p tcp --dport 443 -j REJECT --reject-with tcp-reset
sudo iptables -A INPUT -p udp --dport 443 -j REJECT --reject-with icmp-port-unreachable

You can review the results with sudo iptables -L --line-numbers.

These are only stored in memory before you save them, so test out your set up on your client now to see if it all works as expected.

Test your client connection

To test your configuration, try adding a client (the phone or computer that will connect to the VPN).

1. Run the OpenVPN script again: sudo ./openvpn-install.sh and choose 1) Add a new client. Give it a name; you may find it helps to name it by the device, e.g. “phone”. This creates a file that ends in .ovpn. You need to place this file on your client to use it.
2. Install the appropriate OpenVPN app for your device.
3. Transfer the .ovpn file you just obtained to the device if you haven’t already. (See future tasks for a way to copy the file to your host machine.) Follow instructions in your app (try under FAQ) for importing the .ovpn file and activating the VPN.
4. Ensure it seems to connect properly. If you go to DuckDuckGo.com and search for “What’s my IP”, you should see the location of your Lightsail instance. For a more in-depth test, check for DNS leaks at BrowserLeaks.com.

Try browsing for a while. You can also view the Pi-hole dashboard by visiting http://pi.hole/admin/ on this device.

If everything seems all right, go on to saving the configuration on your instance.

Save iptables

Save the iptables you created earlier using the tee command to achieve the second permission.

sudo iptables-save | sudo tee /etc/pihole/rules.v4

You’re finished with configuration on your Lightsail instance. If you wish to disconnect now, you can just type exit.

Future tasks

You’re done with the set up! You now have your very own personal VPN with a Pi-hole keeping you safe from nasty trackers. Here are some references for operations you might like to come back to in the future:

• Reconnect to your Lightsail instance with SSH:
  • ssh -i /path/to/private-key.pem ubuntu@public-ip-address
• Set a password for the web interface dashboard:
  • pihole -a -p
• Access the web interface dashboard:
  • Connect to the VPN, then visit http://pi.hole/admin/
• Update the Pi-hole:
  • pihole -up
• Add a new client (for iOS, Linux, or Windows, or for Android)
• Copy the .ovpn file for a client to your host machine (run on the host machine):
  • ssh -i /path/to/private-key.pem ubuntu@public-ip-address 'sudo cat /path/on/lightsail/client.ovpn' > /path/on/host/client.ovpn
• Beef up that block list! Here’s my favorite resource for updating your Pi-hole adlist table: The Big Blocklist Collection

Enjoy your new, more secure and peaceful Internet! If you found this guide helpful, please share it with someone else.

Most organizations want to improve productivity and output, but few technical teams seem to take a data-driven approach to discovering productivity bottlenecks. If you’re looking to improve development velocity, a couple key metrics could help your team get unblocked. Here’s how you can apply a smidge of data science to visualize how your repository is doing, and where improvements can be made.

Getting quality data

The first and most difficult part, as any data scientist would likely tell you, is ensuring the quality of your data. It’s especially important to consider consistency: are dates throughout the dataset presented in a consistent format? Have tags or labels been applied under consistent rules? Does the dataset contain repeated values, empty values, or unmatched types?

If your repository has previously changed up processes or standards, consider the timeframe of the data you collect. If labeling issues is done arbitrarily, those may not be a useful feature. While cleaning data is outside the scope of this article, I can, at least, help you painlessly collect it.

I wrote a straightforward Python utility that uses the GitHub API to pull data for any repository. You can use this on the command line and output the data to a file. It uses the list repository issues endpoint (docs), which, perhaps confusingly, includes both issues and pull requests (PRs) for the repository. I get my data like this:

$ python fetch.py -h
usage: fetch.py [-h] [--token TOKEN] repository months
$ python fetch.py OWASP/wstg 24 > data.json

Using the GitHub API means less worry about standardization, for example, all the dates are expressed as ISO 8601. Now that you have some data to process, it’s time to play with Pandas.

Plotting with Pandas

You can use a Jupyter Notebook to do some simple calculations and data visualization.

First, create the Notebook file:

touch stats.ipynb

Open the file in your favorite IDE, or in your browser by running jupyter notebook.

In the first code cell, import Pandas and load your data:

import pandas as pd

data = pd.read_json("data.json")
data

You can then run that cell to see a preview of the data you collected.

Pandas is a well-documented data analysis library. With a little imagination and a few keyword searches, you can begin to measure all kinds of repository metrics. For this walk-through, here’s how you can calculate and create a graph that shows the number of days an issue or PR remains open in your repository.

Create a new code cell and, for each item in your Series, subtract the date it was closed from the date it was created:

duration = pd.Series(data.closed_at - data.created_at)
duration.describe()

Series.describe() will give you some summary statistics that look something like these (from mypy on GitHub):

count                           514
mean      5 days 08:04:17.239299610
std      14 days 12:04:22.979308668
min                 0 days 00:00:09
25%          0 days 00:47:46.250000
50%                 0 days 06:18:47
75%          2 days 20:22:49.250000
max               102 days 20:56:30

Series.plot() uses a specified plotting backend (matplotlib by default) to visualize your data. A histogram can be a helpful way to examine issue duration:

duration.apply(lambda x: x.days).plot(kind="hist")

This will plot a histogram that represents the frequency distribution of issues over days, which is one way you can tell how long most issues take to close. For example, mypy seems to handle the majority of issues and PRs within 10 days, with some outliers taking more than three months:

It would be interesting to visualize other repository data, such as its most frequent contributors, or most often used labels. Does a relationship exist between the author or reviewers of an issue and how quickly it is resolved? Does the presence of particular labels predict anything about the duration of the issue?

You aim for what you measure

Now that you have some data-driven superpowers, remember that it comes with great responsibility. Deciding what to measure is just as, if not more, important than measuring it.

Consider how to translate the numbers you gather into productivity improvements. For example, if your metric is closing issues and PRs faster, what actions can you take to encourage the right behavior in your teams? I’d suggest encouraging issues to be clearly defined, and pull requests to be small and have a well-contained scope, making them easier to understand and review.

To prepare to accurately take measurements for your repository, establish consistent standards for labels, tags, milestones, and other features you might want to examine. Remember that meaningful results are more easily gleaned from higher quality data.

Finally, have fun exercising your data science skills. Who knows what you can discover and improve upon next!

If a move towards privacy is what we’re after, we know a new off-the-shelf Google phone isn’t a better answer – but there are more options.

If you don’t want the details, jump straight to The TL;DR at the end.

Linux phones (sort of)

Unless you’re a rather tolerant tech-savvy tinkerer, a Linux phone isn’t one of these options… yet. I’ve personally been very excited about the bevy of emerging options in this space, from freedom-oriented hardware to fully open source, crowd-developed operating systems.

The current state of these efforts is that this magical mashup just isn’t ready yet. Most Linux phone OS such as Ubuntu Touch, Mobian, Pure OS, etc, are in a “mostly working” state, with the missing features ranging from “lack of reliable push notifications” to “intermittent Bluetooth connectivity” to “camera.”

If all you need is text messaging and a web browser, yes, you can probably go this route. For most users however, this isn’t going to make daily-driver status.

If a Linux phone would suit you, I recommend getting your hands on a PinePhone and running Arch Linux ARM (releases on GitHub) with Plasma Mobile.

De-googled Android

For a daily-driver, “de-googled” Android is your best bet. Android itself (specifically, the Android Open Source Project source code) is based on a modified Linux kernel and is free and open source software. When we typically think of “Android phones,” we refer to Android devices with Google’s proprietary software added to the mix, including Google Play Services. A “de-googled” Android phone is essentially the Android OS without Google’s spyware services included by default.

Keep in mind that this route still involves some DIY. You’ll need to install an OS on a device yourself. Don’t worry, there are step-by-step guides available – the most technical thing you’ll likely have to do is copy and paste some commands into your terminal.

Free and open source Android OS comes in multiple flavors, and the choice isn’t arbitrary. Your selection of a “de-googled” phone is going to be determined by a couple factors: the hardware device you have or that you want to use, and the apps (software) you want to run on it.

Hardware

The phone you may already have (or the one you’re willing to purchase) will influence your choice of operating system (OS).

LineageOS

At the time I’m writing this, if you have an older Pixel or another model of Android phone, your best bet for a hassle-free OS with A-class support will be Lineage. Here’s a link to the LineageOS list of supported devices. Clicking on your device here will get you to some installation instructions for your phone.

GrapheneOS

If you have a newer Pixel (generation 3 up to the newer 5) then GrapheneOS could be the way to go. Here are the devices officially supported by GrapheneOS. They also have easy-to-follow installation instructions and help via chat. It is possible to run GrapheneOS on other phones, but not without substantial DIY for which technical knowledge would help.

Generally speaking, GrapheneOS is intended to be a security-hardened operating system targeted at individuals who won’t be miffed if there are tradeoffs for mitigating vulnerabilities. If you don’t have those requirements or intend to use Google Apps on your phone (see Software), then LineageOS will likely suit you better.

New phone, who dis?

If you’re looking to purchase a new phone, you have some flexibility. My general recommendation is to pick up last-season’s version of the model you want. Not only will this likely be cheaper (and often a great deal if you buy refurbished) but the open source community that develops these operating systems will have had more time to work with the device itself, which could help ensure better compatibility and a smoother set up.

Consider buying a refurbished phone (sometimes called “renewed”) locally when you can. This can help fund the small businesses that offer them.

Software

What do you need to do on your phone? Privacy and convenience are typically at odds (a far larger topic I won’t dig into right now) so it can help to narrow down the functionality you need. If your needs look something like:

• Calls and texts
• Web browser
• Web-based email via browser

Then you’re good to go, right out of the box, with either LineageOS or GrapheneOS. They’ll both include free and open source apps that let you do all these things.

If you want a particular application that doesn’t come pre-installed, here’s where we get into some nuance. Your choices depend on the level of privacy you’d like to maintain. Here are your avenues for installing apps, listed in order of preference.

1. Official APKs

Some particularly privacy-focused applications offer an Android Package Kit (APK) that you can download directly in order to install the app. You should only download these when you’ve navigated directly to a domain that the organization owns. Here are my favorites:

• Signal
• ProtonMail

You can download and install APKs whether you choose LineageOS or GrapheneOS.

2. Use F-Droid

If you can’t find an APK for something you want, search for it on F-Droid.

The F-Droid software repository allows you to download and install apps in much the same way that the Google Play store does, with a couple notable differences. All the apps here are free and open source, and no account or profile is required to download them. The F-Droid APK itself can be downloaded and installed from f-droid.org directly on either LineageOS or GrapheneOS.

Just like any open source software, it’s up to the user (you) to ensure that you’re downloading and installing software you trust. If you want help or advice, F-Droid has a healthy community that you can interact with in lots of ways, including via IRC, Matrix, and the Fediverse.

You can find an app for pretty much anything here: from your general-store type functions such as to-do lists, music players, and maps; to specific niche security applications, and even a tea timer. Here are some well-known choices I can easily recommend:

• Standard Notes (https://standardnotes.com)
• Element for Matrix (https://element.io/)
• Tutanota (https://tutanota.com)

3. Aurora Store

If you need an app that isn’t available on F-Droid, your next stop is the Aurora Store. This is an unofficial client for the Google Play Store that lets you download free applications anonymously, without signing into a Google account. Most applications found in the larger stores can be downloaded this way, without requiring Google’s proprietary stuff on your phone.

When loading Aurora Store for the first time, be sure to choose the “Anonymous” option instead of signing in.

The Aurora Store itself can be installed via F-Droid or auroraoss.com. It works on either LineageOS or GrapheneOS – however, apps that require less private permissions or access will probably work better on LineageOS.

Keep in mind that your phone OS in no way supports these apps directly, or knows what’s in them, or what sort of tracking and information exchange they may be up to. It’s a slight privacy downgrade, but still better than a fully Google-ified OS.

4. If you need Google Apps

If this will be your only phone and you simply must have Google Apps on it (think Google Play Store, Gmail, Calendar, Photos, etc) then go with LineageOS. You can choose to try emulating Google Play Services using LineageOS for microG, or install the Google Apps add-on when you install LineageOS.

The TL;DR

Here’s the “Internet personality quiz” version of everything above. You are…

1. Knowledgeable about Linux; mostly use a phone for text, calls, and web browser; and potentially want to help develop Linux phone software.
   • Try a Linux phone such as the PinePhone, but consider one of the other options as a back up for when you just need stuff to work.
2. Security or privacy inclined, happy to use FOSS apps, or do most things via web browser anyway.
   • Get your hands on a Pixel 3{XL, a, a XL}, Pixel 4{XL, a, a 5G}, or Pixel 5, and use GrapheneOS. Installation instructions here.
   • Optionally, download the F-Droid or Aurora Store APKs for apps.
3. Someone who needs Google Apps to work, or you want a phone that isn’t a Pixel, or you’re setting up a device for someone who’s fine using Android but needs it to look familiar.
   • Use LineageOS with any of its supported devices. Click on the device name for installation instructions.
   • If you must have Google Apps and need Google Play Services to work, install the add-on at the same time you install LineageOS.
   • Optionally, download the F-Droid or Aurora Store for installing apps.

Whichever route you choose, my advice is to treat this like a learning experiment. You’re sort of building your own phone, after all, and gaining all the technological independence that comes with that knowledge. If possible, don’t ditch your current phone until you try out one (two?) of these paths. The one you end up liking most could surprise you! It’s great to have options.

In a frenzy of accomplishment you drag it into the house—only to discover that your dining room doorway is several inches too small. It doesn’t fit.

You might say this comedic example is unrealistic. Of course an experienced DIY-er would have measured the doorway first. But in real life, unforeseen problems rarely come solo. Once you finally get the table through the door (after removing the legs and reassembling it inside), you discover the floor’s uneven. The chairs you chose are a few inches too short. The ceiling light hangs too low. Each solution creates new problems you never anticipated.

I’ve seen this exact pattern play out dozens of times in software development, just with different furniture. Teams spend months building features in isolation, only to discover they don’t fit through the “doorways” of real user workflows, existing infrastructure, or business constraints. The solution isn’t better planning—it’s building in context from the start.

The Planning Fallacy (Or: Why We’re All Terrible at This)

Few software developers are accurate when it comes to time and cost estimates. This isn’t a failing of engineers specifically—it’s a deeply human tendency toward optimism when predicting our own future. First proposed by Daniel Kahneman and Amos Tversky in 1979, the planning fallacy explains why our estimates are consistently wrong.

In one study, students were asked to estimate how long they’d take to finish their senior theses. The estimates averaged 27.4 days at the optimistic end and 48.6 days at the pessimistic end. The actual completion time? 55.5 days. Even the pessimistic estimates were too optimistic.

The researchers proposed two main reasons: first, people focus on their future plans rather than their past experiences; second, people don’t think past experiences matter much to the future anyway.

You can probably find examples of this in your own recent project history. Sure, that last “two-day feature” turned into a two-week affair, but that was only because the API documentation was wrong. Or maybe you didn’t finish that database migration when planned, but that was only because you discovered the staging environment was configured differently than production. You’re absolutely, positively, definitely certain that next time will be different.

The reality is that we’re terrible at factoring in the unexpected daily demands of building software.

Legacy code behaves mysteriously. Third-party services have undocumented quirks. Staging environments don’t match production. Users do things we never anticipated. Some measure of ignorance about these complications probably keeps us sane enough to start new projects.

But some measure of accurate planning is also necessary for success. The solution is working in context as much as possible, rather than trying to plan for every contingency.

Context Is Your Reality Check

Let’s reconsider the dining room table story. Instead of spending months out in the garage, what would you do differently to build in context?

You might say, “Build it in the dining room!” While that would be ideal for context, it’s rarely possible in homes or software development. Instead, you do the next best thing: start building, and make frequent visits to context.

Having decided you want to build a table, one of the first questions is “How big will it be?” You’ll have requirements to fulfill (must seat six, must match other furniture, must hold the weight of your annual twenty-eight-course Christmas feast) that lead you to a rough decision.

With a size in mind, you build a mock-up. At this point, the specific materials, style, and color don’t matter—only the three dimensions. Once you have your mock table, you can make your first trip to the context where it will ultimately live. Attempting to carry your foam/wood/cardboard/balloon animal mock-up into the dining room will reveal issues you never considered, and possibly new opportunities as well. Perhaps, though you’d never have thought it, a modern abstractly-shaped dining table would better complement the space. You can take this into account in your next higher-fidelity iteration.

This translates directly to software development, minus the Christmas feast. You may recognize this as the MVP approach, but even here, putting the MVP in context is a step that’s frequently omitted.

I’ve seen teams spend months building a “simple” user authentication system, only to discover that their company’s SSO provider doesn’t support the OAuth flow they built around. Or teams that create beautiful interfaces that completely break when real user data (with its inconsistent formats and edge cases) gets loaded. Where will your product ultimately live? How will it be accessed? What does real data look like?

Building your MVP and attempting to deploy it with realistic constraints will uncover these issues when they’re still manageable.

Even when teams have prior experience with technologies, remember the planning fallacy. People naturally discount past evidence to the point of forgetting. It’s also unlikely that the same exact team is building the same exact product as last time. The language, technology, framework, and infrastructure have likely changed—as have the capabilities and bandwidth of the engineers. Frequent visits to context help you run into issues early, adapt to them, and create short feedback loops.

Go for Good Enough (Then Iterate)

The specific meaning of putting something in context varies from project to project. It might mean deploying to cloud infrastructure, running on a new server, or testing whether your remote office can access the same resources you use. In all cases, keep those short iterations going. Don’t wait to get a version to 100% before finding out if it works in context. Ship it at 80%, see how close you got, then iterate.

This approach feels risky if you’re used to planning everything upfront. But the alternative—discovering fundamental incompatibilities after months of work—is much riskier. Better to learn that your table won’t fit through the door when it’s still made of cardboard than when it’s solid oak.

The best software gets built by teams that understand the difference between the theoretical problem they’re solving and the real environment where their solution needs to work. Context is messy, unpredictable, and full of constraints you never anticipated. That’s exactly why you need to visit it early and often.

Your garage is perfect for focused work, but your dining room is where people actually eat dinner. Build for where your software will really live, not where it’s convenient to develop it.

I saw this firsthand while working with a team where questioning assumptions became a regular part of our code review process. We’d look at every new feature and ask “How might someone abuse this?” I developed a particular talent for finding injection attacks on forms—apparently I have a knack for thinking of creative ways to sneak SQL queries into text fields. After the third or fourth time I caught these vulnerabilities during review, we added validation middleware to eliminate that entire class of problems.

But the real breakthrough was watching how the team’s thinking evolved. Once developers got used to questioning their assumptions about user behavior, they started writing more robust solutions from the start. Security thinking became a starting point rather than something bolted on afterward.

Designing for Reality, Not Just Intent

One of the most effective practices we developed was specifying both the “happy path” and the “unhappy path” during our design process. The happy path was straightforward—everything happens in the way and sequence we intended. But the unhappy paths were where we learned the most: what happens when steps occur out of order? When data is missing or provided in an unexpected format? When external systems fail at exactly the wrong moment?

This dual-path thinking transformed how we approached every feature. Instead of just asking “How should this work?” we started asking “How will this actually be used?” and “What should happen when reality doesn’t match our expectations?” It sounds pessimistic, but it actually made development more fun. It caused us to think about our application from all angles rather than just implementing obvious functionality.

The unhappy path exercise revealed assumptions we didn’t even know we were making. We’d design a user registration flow assuming people would fill out forms completely and submit them once. Then we’d consider reality: What if someone submits the form multiple times? What if they navigate away and come back? What if they fill out the form, wait an hour, then submit it after their session expires?

Each unhappy path scenario led to better design decisions. Race condition handling. Idempotent endpoints. Graceful degradation when external services are unavailable. The code that protected against malicious users also handled legitimate users experiencing network glitches or browser crashes.

Systematic Questioning as a Superpower

There’s a particular mindset that effective security thinking requires—call it systematic skepticism. It’s the ability to look at any system and ask “What assumptions is this making?” and “What happens when those assumptions are wrong?” This kind of thinking makes your software more robust.

Sometimes this means channeling your inner four-year-old—pushing every button, ignoring all instructions, using things in ways their makers never intended. But rather than random exploration, you develop structured ways of challenging system boundaries, finding edge cases, and being creative about the ways that software can be used beyond its intended purpose.

This systematic questioning makes you better at every aspect of development. When you’re used to thinking about edge cases and unexpected inputs, you write more defensive code naturally. When you habitually consider what could go wrong, you build better (more useful) error handling. When you assume users will do unexpected things, you design more intuitive interfaces.

I’ve noticed that developers who adopt this questioning mindset become significantly better at debugging production issues too. Instead of being surprised when something breaks, they’re already thinking “What unexpected condition triggered this?” They approach problems with methodical curiosity rather than frustrated confusion.

Building a Culture of Constructive Skepticism

The key to building security-conscious teams isn’t teaching people to be afraid of attackers—it’s helping them develop genuine curiosity about system behavior under stress. When questioning assumptions becomes intellectually interesting rather than anxiety-inducing, your team will start doing it automatically.

Code reviews become more engaging when everyone is looking for unspoken assumptions about user behavior. Feature planning gets more thorough when “What are the unhappy paths?” is a standard question alongside “What should it do?” Architecture discussions become more robust when you’re considering not just how systems should work together, but how they should behave when dependencies are slow, unavailable, or returning unexpected data.

The practical implementation is surprisingly straightforward. During development, encourage your team to spend time being deliberately unreasonable with whatever they’re building. During design reviews, spend equal time on happy and unhappy paths. During testing, encourage your team to think like someone who’s never seen your application before and doesn’t understand the rules.

What emerges is a team that builds more resilient systems without extra effort. When you’re accustomed to thinking about failure modes, you naturally design systems that handle them gracefully. When you expect users to ignore instructions, you build interfaces that guide them toward success even when they’re not following the intended flow.

Security as Engineering Excellence

What I’ve learned is that security thinking is really just rigorous engineering thinking with a creative twist. It’s the same mental process you use when debugging complex issues or designing APIs that won’t confuse future developers. You’re considering multiple perspectives, anticipating edge cases, and designing for resilience rather than just functionality.

The most successful security-conscious teams I’ve worked with don’t have dedicated security experts who review everything after the fact—they have developers who think about security implications as naturally as they think about performance or usability. This happens through cultural reinforcement and consistent practice, not through mandates or compliance checklists.

The payoff extends far beyond security. Teams that think about unhappy paths build more reliable software. Developers who consider malicious inputs write better input validation for legitimate users. Engineers who design for system failures create more robust integrations. The skills reinforce each other in ways that make everyone more effective.

Most importantly, this approach makes engineering work more intellectually satisfying. There’s something deeply rewarding about anticipating problems and solving them before they happen. When your team develops the habit of systematically questioning their assumptions, they’ll approach every problem with the kind of methodical curiosity that leads to truly robust solutions.

You can help your team become professionally curious about system boundaries, failure modes, and the gap between how software is supposed to work and how it actually gets used. Once they develop that mindset, they’ll write more secure code naturally, because they’ll view software the same way attackers do—as systems that can fail when someone does something unexpected.

I’m writing this guide for a very specific person – possibly you, or someone you know. I’ll explain how a non-technical business leader can find information and take part in the decisions and questions that happen only on GitHub. You don’t need to know how to use Git. You just need a few minutes to follow along, and a desire to be a resource and servant leader for your teams. Let’s do it!

If you haven’t signed up yet, click below to read the very first steps. Once you’re logged in, read on to join in!

Where the magic happens

On GitHub, work is typically grouped by projects or products into what’s called repositories. Your team or company may have just one of these that they regularly use (they might call it a “monorepo”) or several repositories that represent different technical components of a single product.

Once you log in, you’ll be on the Recent activity page. You can search for the name of the repository you want to visit in the search bar at the top left. If your company’s repositories are private, you may need to be invited by an administrator in order to view it.

When you view a repository, it looks like this. I’ve pointed out some of the important bits.

In any repository, there are two main areas where decisions usually take place. These are in Issues and Pull Requests, and you’ll mainly focus your attention here. Click on the Issues tab to see these.

You’ll be presented with a list of Issues, which you can think of as individual topics. This format is essentially a discussion board. Clicking on any of the Issue titles will take you to its thread.

Here’s where the magic happens! Folks on your team use Issues to discuss all kinds of topics. These may be a very technical and esoteric cost-benefit analysis, or a fundamental customer-facing design decision. A quick read of the first message in the thread is likely to reveal whether it’s a decision that could use your help.

Issues are a starting point for work. Here, team members make decisions about the type and scope of a change they plan to make.

When someone has their changes ready, they’ll open a Pull Request so that other team members can preview and give input on those changes before they become part of the repository. Click the Pull Request tab at the top to view these.

You’re presented with a very similar view on this page – yes, it’s another discussion board! You can click on any Pull Request to view its thread as well.

Pull Requests have some additional tabs that team members use for code reviews. All the conversation will show up in the Conversation tab.

Sorting out what’s relevant

A lot of discussion happens in Issues and Pull Request threads, and not all of it may be relevant for you to look at. Thankfully, GitHub has some excellent collaboration tools that can help your team direct your attention to where it’s most needed. These are @mentions and labels.

Ask to be @mentioned

These work the same way on GitHub as they do on Twitter. When someone @mentions you, you’ll receive a notification via email, or you’ll see it in your Notification Center when you’re logged in. This depends on your notification settings, which you should adjust to your liking. If you only want to be notified when someone @mentions you or replies to you, you should uncheck everything on the Notification Settings page except for your preferred options under Participating.

Now when someone references you in a discussion on GitHub, you’ll be notified and you’ll have the chance to respond!

Using labels

Another less direct way to see where you can effectively contribute is to ask your team to use labels. You may recall seeing these in the right sidebar of Issue and Pull Request threads:

You can create different labels to categorize a discussion, and you can apply as many labels to a discussion as you like. In order to have your team draw your attention to threads that might benefit from your input or guidance, ask folks to use a label to point these out. This could be the question label, or any new label of your choosing.

Clicking on a label in the sidebar will take you to a page that shows all the Issues or Pull Requests with that label. The URL will look something like:

https://github.com/<organization name>/<repository name>/labels/question

You can also bookmark this page to easily check it on a regular schedule. This is a great, low-friction way for your team to indicate areas that could use your input.

Collaborating on files

Similar to tracking changes in Google Docs or Word documents, you can edit documents in GitHub in a way that lets your team see the changes you’ve made. This is a fantastic method for collaborating with your team where they work, and avoids the hassle of emailing attachments around.

Text files in repositories have extensions such as .md for Markdown, or .txt for plain text. The majority of documentation on GitHub is in Markdown format. Clicking on any file in the repository file list will open it on the GitHub website. At the top right of the document, look for these buttons:

Clicking on the pencil icon will let you edit the file right there in your web browser. You may not see anything special as you’re typing, but once you commit (like saving) your file, all your changes are tracked with Git! For a step-by-step guide to editing, see Editing files in your repository.

Here are some helpful articles for formatting text with Markdown on GitHub.

Proactive participation

GitHub is well-structured as a collaboration platform. That’s why people of all professions use it not just for software development, but also for networking, getting jobs and sponsorships, and even for hosting simple no-code websites. My own company uses GitHub for everything from collaborating on company documentation to automated Change Control Board processes for FedRAMP.

At your leisure, I encourage you to chase your curiosity and explore. Don’t be shy about asking questions, or asking technical folks on your team to explain something if you think it will enable you to be a bigger help to them. With so much of the world building software on GitHub, there’s a lot you can contribute when you’re where the work happens.

Signal is experiencing technical difficulties. We are working hard to restore service as quickly as possible.

— Signal (@signalapp) January 15, 2021

The small team responded impressively quickly, especially given that a 4,200% spike in new users was utterly implausible before it occurred.

The downside of so many people moving onto this fantastic application is that it caused a brief outage. If you rely solely on a certain application for your communications, brief outages can be debilitating. Even when it seems implausible that your favorite chat, email, or website service could just – poof – vanish overnight, recent events have proved it isn’t impossible.

Have a backup plan. Have several. Here’s how you can improve your digital resiliency for things like websites, messaging, and email.

Messaging

I recommend Signal because it is open source, end-to-end encrypted, cross-platform, and offers text, voice, video, and group chat. It’s usually very reliable; however, strange things can happen.

It’s important to set up a backup plan ahead of any service outages with the people you communicate with the most. Have an agreement for a secondary method of messaging – ideally another end-to-end encrypted service. Avoid falling back on insecure communications like SMS and social media messaging. Here’s a short list for you to explore:

• Signal
• Wire
• Session

If you’re particularly technically inclined, you can set up your own self-hosted chat service with Matrix.

Having a go-to plan B can help bring peace of mind and ensure you’re still able to communicate when strange things happen.

Cloud contacts

Do you know the phone numbers of your closest contacts? While memorizing them might not be practical, storing them solely online is an unnecessary risk. Most services allow you to export your contacts to vCard or CSV format.

I recommend keeping your contacts locally on your device whenever you can. This ensures you still know how to contact people if your cloud provider is unavailable, or if you don’t have Internet access.

Full analog redundancy is also possible here. Remember that paper stuff? Write down the phone numbers of your most important contacts so you can access them if your devices run out of battery or otherwise can’t turn on (drop your phone much?).

Local email synchronization

If your email service exists solely online, there’s a big email-shaped hole in your life. If you can’t log in to your email for any reason – an outage on their end, a billing error, or your Internet is down – you’ll have no way to access your messages for however long your exile lasts. If you think about all the things you do via email in a day, I think the appropriate reaction to not having local copies is 🤦.

Download an open source email client like Thunderbird. Follow instructions to install Thunderbird and set it up with your existing online email service. Your online service provider may have a help document that shows you how to set up Thunderbird.

You can maximize your privacy by turning off Thunderbird’s telemetry.

To ensure that Thunderbird downloads your email messages and stores them locally on your machine:

1. Click the “hamburger” overflow menu and go to Account Settings
2. Choose Synchronization & Storage in the sidebar
3. Ensure that under Message Synchronizing, the checkbox for Keep messages in all folders for this account on this computer is checked.

You may need to visit each of your folders in order to trigger the initial download.

Some other settings you may want to update:

1. Choose Composition & Addressing and uncheck the box next to Compose messages in HTML format to send plaintext emails instead.
2. Under Return Receipts choose Global Preferences. Select the radio button for Never send a return receipt.

You don’t need to start using Thunderbird for all your email tasks. Just make sure you open it up regularly so that your messages sync and download to your machine.

Websites

I strongly believe you should have your own independent website for reasons that go beyond redundancy. To truly make your site resilient, it’s important to have your own domain.

If you know that my website is at the address victoria.dev, for example, it doesn’t matter whether I’m hosting it on GitHub Pages, AWS, Wordpress, or from a server in my basement. If my hosting provider becomes unavailable, my website won’t go down with it. Getting back up and running would be as simple as updating my DNS configuration to point to a new host.

Price is hardly an excuse, either. You can buy a domain for less than a cup of coffee with my Namecheap affiliate link (thanks!). Namecheap also handles your DNS settings, so it’s a one-stop shop.

With your own domain, you can build resiliency for your email address as well. Learn how to set up your custom domain with your email provider. If you need to switch providers in the future, your email address ports to the new service with you. Here are a few quick links for providers I’d recommend:

• ProtonMail: How to use a custom domain with Proton Mail
• Tutanota: Adding of custom email domains
• Fastmail: Custom Domains with Fastmail

Build your digital resiliency

I hope you’ve found this article useful on your path to building digital resiliency. If you’re interested in more privacy topics, you might like to learn about great apps for outsourcing security.

If your threat model includes anonymity or censorship, building digital resiliency is just a first step. The rest is outside the scope of my blog, but here are a few great resources I’ve come across:

• Tor Browser
• IntelTechniques
• Can’t Cancel Me
• Tails portable OS

You can create your own self-hosted Matrix chat for as little as $3.50 USD per month on an AWS Lightsail instance. Your homeserver can federate with other Matrix servers, giving you a reliable and fault-tolerant means of communication.

Matrix is most widely installed via its Synapse homeserver implementation written in Python 3. Dendrite, its second-generation homeserver implementation written in Go, is currently released in beta. Dendrite will provide more memory efficiency and reliability out-of-the-box, making it an excellent choice for running on a virtual instance.

Here’s how to set up your own homeserver on AWS Lightsail with Dendrite. You can also contribute to Dendrite today.

Create a Lightsail instance

Spin up a new Lightsail instance on AWS with Debian as your operating system. It’s a good idea to create a new per-instance key for use with SSH. You can do this by with the SSH key pair manager on the instance creation page. Don’t forget to download your private key and .gitignore your secrets.

Click Create Instance. Wait for the status of your instance to change from Pending to Running, then click its name to see further information. You’ll need the Public IP address.

To enable people including yourself to connect to the instance, go to the Networking tab and add a firewall rule for HTTPS. This will open 443 so you can connect over IPv4. You can also do this for IPv6.

Connect DNS

Give your instance a catchier address by buying a domain at Namecheap and setting up DNS records.

1. On your domain management page in the Nameservers section, choose Namecheap BasicDNS.
2. On the Advanced DNS tab, click Add New Record.

Add an A Record to your Lightsail Public IP. You can use a subdomain if you want one, for example,

• Type: A Record
• Host: matrix
• Value: 13.59.251.229

This points matrix.example.org to your Lightsail instance.

Set up your Matrix homeserver

Change permissions on the private key you downloaded:

chmod 600 <path/to/key>

Then SSH to your Public IP:

ssh -i <path/to/key> admin@<public ip>

Welcome to your instance! You can make it more interesting by downloading some packages you’ll need for Dendrite. It’s a good idea to use apt for this, but first you’ll want to make sure you’re getting the latest stuff.

Dec 2021 update: As the good people of Mastodon point out, you might like to ensure you’re choosing the stable version for Debian. For instance, replace buster below with what’s “stable” at the moment.

Change your sources list in order to get the newest version of Go:

sudo vim /etc/apt/sources.list

Delete everything except these two lines:

deb http://cdn-aws.deb.debian.org/debian buster main
deb-src http://cdn-aws.deb.debian.org/debian buster main

Then replace the distributions:

:%s/buster main/testing main contrib non-free/g

Run sudo apt dist-upgrade. If you’re asked about modified configuration files, choose the option to “keep the local version currently installed.”

Once the upgrade is finished, restart your instance with sudo shutdown -r now.

Go make some coffee, then SSH back in. Get the packages you’ll need with:

sudo apt update
sudo apt upgrade
sudo apt install -y git golang nginx python3-certbot-nginx

You’re ready to get Dendrite.

Get Dendrite

Clone Dendrite and follow the README instructions to get started. You’ll need to choose whether you want your Matrix instance to be federating. For simplicity, here’s how to set up a non-federating deployment to start:

git clone https://github.com/matrix-org/dendrite
cd dendrite
./build.sh

# Generate a Matrix signing key for federation (required)
./bin/generate-keys --private-key matrix_key.pem

# Generate a self-signed certificate (optional, but a valid TLS certificate is normally
# needed for Matrix federation/clients to work properly!)
./bin/generate-keys --tls-cert server.crt --tls-key server.key

# Copy and modify the config file - you'll need to set a server name and paths to the keys
# at the very least, along with setting up the database connection strings.
cp dendrite-config.yaml dendrite.yaml

Configure Dendrite

Modify the configuration file you just copied:

sudo vim dendrite.yaml

At minimum, set:

• server name to your shiny new domain name, e.g. matrix.example.org
• disable_federation to true or false
• registration_disabled to true or false

You might like to read the Dendrite FAQ.

Configure nginx

Get the required packages if you didn’t already install them above:

sudo apt install nginx python3-certbot-nginx

Create your site’s configuration file under sites-available with:

cd /etc/nginx/sites-available
ln -s /etc/nginx/sites-available/<sitename> /etc/nginx/sites-enabled/<sitename>
sudo cp default <sitename>

Edit your site configuration. Delete the root and index lines if you don’t need them, and input your server name.

Your location block should look like:

location / {
    proxy_pass https://localhost:8448;
}

Remove the default with: sudo rm /etc/nginx/sites-enabled/default.

Create self-signed certificates

You can use Certbot to generate self-signed certificates with Let’s Encrypt.

sudo certbot --nginx -d <your.site.address>

If you don’t want to give an email, add the --register-unsafely-without-email flag.

Test your configuration and restart nginx with:

sudo nginx -t
sudo systemctl restart nginx

Then start up your Matrix server.

# Build and run the server:
./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml

Your Matrix server is up and running at your web address! If you disabled registration in your configuration, you may need to create a user. You can do this by running the included dendrite/bin/createuser.

You can log on to your new homeserver with any Matrix client, or Matrix-capable applications like Pidgin with the Matrix plugin.

Other troubleshooting

Log files

If you get an error such as:

... [github.com/matrix-org/dendrite/internal/log.go:155] setupFileHook
  Couldn't create directory /var/log/dendrite: "mkdir /var/log/dendrite: permission denied"

You’ll need to create a spot for your log files. Avoid the bad practice of running stuff with sudo whenever you can. Instead, create the necessary file with the right permissions:

sudo mkdir /var/log/dendrite
sudo chown admin:admin /var/log/dendrite

# Build and run the server:
./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml

Unable to decrypt

If you see: Unable to decrypt: The sender's device has not sent us the keys for this message. you may need to verify a user (sometimes yourself).

1. In your client, open the user’s profile. Click the lock icon if there is one, or otherwise look for a way to verify them.
2. You may be asked to see if some emojis presented to both users match if you’re using certain clients like Element.
3. You can then re-request encryption keys for any sent messages.

Set up your own Matrix server today

I hope you found this introduction to setting up your own Matrix homeserver to be helpful!

The Real Question Behind the Question

When your function discovers something’s wrong, you’re not only choosing between raise and return. You’re making a decision about how your entire application will handle failure, how readable your code will be for the next person, and how many 3 AM prod debugging sessions you’re setting up for your future self and team.

Here’s how I think about this choice, because the right one for your application affects everything from your error logs to your team’s velocity.

When I Reach for Exceptions

I raise exceptions when something genuinely unexpected happens—when the assumptions my function was built on just got violated. If I’m writing a function to parse a config file and the file doesn’t exist, that’s exceptional. The caller expected a valid config, and I can’t deliver on that contract.

def load_config(filepath):
    if not os.path.exists(filepath):
        raise FileNotFoundError(f"Config file not found: {filepath}")
    
    try:
        with open(filepath, 'r') as f:
            return json.load(f)
    except json.JSONDecodeError as e:
        raise ConfigurationError(f"Invalid JSON in config file: {e}")

Here’s why exceptions work well here: the calling code doesn’t need to check every single operation. If any step fails, the exception bubbles up to whoever can actually handle it. Your main application logic stays clean, and error handling happens at the right level.

The business impact here is huge. When your core logic isn’t cluttered with error checking, you can focus on the actual problem you’re solving. Your functions do one thing well, and your error handling is centralized where it belongs.

When I Return Error Values

But sometimes the “error” isn’t really an error—it’s just one of several possible outcomes. When I’m building a user search function, finding zero results isn’t exceptional. It’s totally normal behavior that the caller needs to handle anyway.

def search_users(query):
    results = database.search(query)
    if not results:
        return []  # Empty list, not an exception
    return results

# Calling code feels natural
users = search_users("john")
if users:
    display_users(users)
else:
    show_no_results_message()

This approach shines when you have multiple valid outcomes and the caller needs to make decisions based on which one occurred. It also works well for performance-critical code where exception handling overhead matters.

The Type Safety Angle

Here’s something I’ve started caring about more as codebases grow: how well does your choice play with static type checking? Modern Python with type hints changes the game significantly.

With exceptions, your function signature stays clean:

def parse_user_id(user_input: str) -> int:
    try:
        return int(user_input)
    except ValueError:
        raise InvalidUserIdError("User ID must be a number")

But with return values, you’re often dealing with unions:

def parse_user_id(user_input: str) -> int | None:
    try:
        return int(user_input)
    except ValueError:
        return None

That | None propagates through your entire codebase. Every function that calls this one now has to handle the None case, and mypy will remind you of that fact. Sometimes that’s exactly what you want—explicit error handling at every level. Other times, it creates unnecessary complexity.

The Performance Reality Check

What about performance? Yes, exceptions are slower than returning values, but context matters enormously here.

In tight loops processing thousands of items per second, that overhead can add up. Profiling code where you’ve switched from exceptions to return values might show improved performance of 20-30%. But in typical web application code where you’re dealing with database calls and network requests, exception overhead is noise compared to everything else.

The more important performance consideration is often developer performance. How quickly can someone understand your code? How easily can they modify it without introducing bugs? I’ve seen teams spend weeks debugging subtle issues that wouldn’t have existed with clearer (documented!) error handling patterns.

Patterns That Actually Work in Production

After working on systems that handle millions of requests, here are the patterns I keep coming back to:

For library code: Raise exceptions. Libraries don’t know how their callers want to handle errors, so push that decision up the stack. Custom exception types help callers decide what to catch and what to let bubble up.

For user input validation: Usually return structured error information. Users make mistakes constantly, and that’s normal behavior, not exceptional.

def validate_email(email: str) -> ValidationResult:
    if not email:
        return ValidationResult(valid=False, error="Email is required")
    if "@" not in email:
        return ValidationResult(valid=False, error="Invalid email format")
    return ValidationResult(valid=True)

For external service calls: This is tricky. Network timeouts and service errors happen, but they’re not exactly exceptional in a distributed system. I often use exceptions for the truly unexpected (DNS resolution failures) and return values for the predictable failures (rate limiting, temporary service unavailability).

The 3AM System Down Test

Here’s my ultimate thought experiment test: if something broke in production and you had to debug it at 3 AM, bleary-eyed and chugging coffee, which approach helps you understand what went wrong faster?

Good exceptions with detailed error messages and proper stack traces are incredible for this. You can see exactly where things went wrong and why. But exceptions that get swallowed or re-raised without context are debugging nightmares.

Return values with proper logging can also be great for debugging, especially when you need to understand the sequence of events that led to a problem. But they require more discipline—you need to actually check and log those return values.

Making the Choice

When I’m looking at a specific function, I ask myself:

• Is this condition truly unexpected given the function’s contract?
• Do callers need to make immediate decisions based on this failure?
• How will this pattern scale across my team and codebase?
• What will debugging look like when this inevitably breaks?

There’s no universal right answer, but there are patterns that work well for different situations. The key is being intentional about your choice and consistent within your codebase.

Your error handling strategy affects how quickly new team members can contribute, how easy it is to track down production issues, and how confident you can be when making changes. Choose patterns that serve your team’s long-term productivity, not just today’s immediate problem.

The best choice for error handling is the one that helps you sleep better at night, knowing that when something goes wrong, you’ll be able to figure out what happened and fix it quickly.

If you found some value in this post, there’s more! I write about high-output development processes and building maintainable systems in the AI age. You can get my posts in your inbox by subscribing below.

We spent those two weeks in a weird organizational limbo, working on whatever seemed important while bigger decisions piled up. Upon returning, the CEO was frustrated that so little had been accomplished, and the team was frustrated that they’d been left without clear direction. It was a perfect example of how taking time off as a leader requires completely different preparation than taking time off as an individual contributor.

The reality is that leadership vacation planning isn’t about finishing your own work—it’s about ensuring your team can function effectively without you. Done well, it’s actually a powerful way to develop your team’s autonomy and decision-making capabilities. Done poorly, it creates exactly the kind of organizational dysfunction I witnessed firsthand.

The Information Bottleneck Problem

Here’s what most leaders don’t realize: you’re probably a bigger bottleneck than you think. Not because you’re micromanaging, but because critical context lives in your head that your team needs access to in order to make good decisions. The challenge isn’t documenting everything you know—that’s impossible. The challenge is identifying what your team will actually need while you’re gone.

I’ve learned to approach this systematically. Instead of trying to dump all my knowledge, I focus on the specific work my team will be doing during my absence. What decisions might come up that I can provide context for? What blockers could they encounter and who could help in my absence? Who will take the lead on making decisions to help keep projects moving forward?

This exercise often reveals gaps in team communication that extend beyond vacation planning.

If people don’t know how to prioritize work when you’re gone for a week, they probably struggle with prioritization day-to-day more than you realize.

Vacation prep becomes a forcing function for better ongoing delegation.

The practical approach is straightforward: review your priority list and write down the context and contacts that your team will need to get work done while you’re away. But the deeper value is discovering where your team needs more autonomy and decision-making authority in general.

Decision-Making Without You

The most common mistake I see leaders make is trying to pre-decide everything that might come up while they’re away. This is both impossible and counterproductive. Instead, the goal should be empowering your team to make good decisions using the same framework you would use.

Before any significant time off, I have explicit conversations with my team about what kinds of decisions they can make independently and what should wait for my return. More importantly, I explain the reasoning behind those boundaries so they understand when to escalate and when to proceed.

More than just being on the same page, these boundaries help to build your team’s confidence in their own judgment.

When people understand your decision-making criteria and feel trusted to apply them, they’ll make better choices whether you’re away on vacation or away in a meeting.

The key is being specific about decision authority rather than vague about “checking with me first.” Instead of saying “let me know if anything important comes up,” try “you can approve any engineering changes that don’t affect the database schema, but flag anything that requires downtime for discussion when I’m back.”

Creating Clarity, Not Chaos

The difference between teams that thrive when their leader is away and teams that stagnate comes down to clarity of expectations. Your team needs to know not just what to work on, but how to make trade-offs when priorities conflict, who to go to for different types of help, and what success looks like in your absence.

I’ve found that internal communication about your time off is just as important as external auto-responders.

A quick message to your team explaining where to find information, who’s covering what responsibilities, and how to handle common scenarios prevents a lot of confusion and hesitation.

But the real test is whether your team feels empowered to act or feels like they’re in caretaker mode until you return. The goal is maintaining momentum, not just maintaining the status quo. This requires trusting your team with meaningful work and giving them the context they need to handle unexpected situations.

The Leadership Development Opportunity

Your vacation is actually a development opportunity for your team if you set it up intentionally.

When you step back temporarily, you create space for other people to step up, make decisions, and take on leadership responsibilities.

Instead of just hoping things will be fine while you’re gone, use your absence as a chance to test and develop your team’s capabilities. Give someone the opportunity to run meetings, handle stakeholder communication, or make technical decisions that they’re ready for but haven’t had the chance to practice.

The preparation for this kind of delegation is more involved than just finishing your own work, but the payoff is enormous. You return to a team that’s more capable and confident, and you’ve identified who’s ready for additional responsibilities. Plus, you’ve stress-tested your team’s ability to function without you, which is valuable information for organizational resilience.

Making Time Off Actually Restful

The irony of leadership is that taking time off can be stressful if you’re worried about what’s happening while you’re away. The best vacation preparation eliminates that anxiety by ensuring your team has everything they need to succeed without you.

This means being honest about your availability expectations and sticking to them. If you tell your team you’ll be completely offline, don’t check Slack “just once” and end up getting pulled into work discussions. If you’re going to check in periodically, be specific about when and how, so people know what to expect.

The teams that handle leadership time off best are the ones where this kind of preparation is routine, not exceptional. When delegation, clear communication, and decision-making authority are part of your regular management practice, preparing for vacation becomes straightforward rather than stressful.

Your time off should leave your team more capable, not less. When you return from vacation to find that your team tackled challenges, made good decisions, and maintained momentum without you, you’ll know you’ve built something sustainable. That’s not just good vacation planning—it’s good leadership.

A number of my readers have been kind enough to tell me that you find my blog useful, but there’s something that you don’t know. Up until I recently implemented a search feature on victoria.dev, I had been my own unhappiest user.

My blog exists for all to read, but it’s also my own personal Internet brain. I frequently pull up a post I’ve written when trying to re-discover some bit of knowledge that I may have had the foresight to record. Without a search, finding it again took a few clicks and more than a few guesses. Now, all my previous discoveries are conveniently at my fingertips, ready to be rolled into even more future work.

If you’d like to make your own personal Internet brain more useful, here’s how you can implement your own search feature on your static Hugo site.

Get Lunr

While you can install lunr.js via npm or include it from a CDN, I chose to vendorize it to minimize network impact. This means I host it from my own site files by placing the library in Hugo’s static directory.

You can save your visitors some bandwidth by minifying lunr.js, which I did just by downloading lunr.js from source and using the JS & CSS Minifier Visual Studio Code extension on the file. That brought the size down roughly 60% from 97.5 KB to 39.35 KB.

Save this as static/js/lunr.min.js.

Create a search form partial

To easily place your search form wherever you like on your site, create the form as a partial template at layouts/partials/search-form.html

<form id="search"
    action='{{ with .GetPage "/search" }}{{.Permalink}}{{end}}' method="get">
    <label hidden for="search-input">Search site</label>
    <input type="text" id="search-input" name="query"
    placeholder="Type here to search">
    <input type="submit" value="search">
</form>

Include your search form in other templates with:

{{ partial "search-form.html" . }}

Create a search page

For your search to be useful, you’ll need a way to trigger one. You can create a (static!) /search page that responds to a GET request, runs your search, and displays results.

Here’s how to create a Hugo template file for a search page and get it to render.

Create layouts/search/list.html with the following minimum markup, assuming you’re inheriting from a base template:

{{ define "main" }}
{{ partial "search-form.html" . }}

<ul id="results">
    <li>
        Enter a keyword above to search this site.
    </li>
</ul>
{{ end }}

In order to get Hugo to render the template, a matching content file must be available. Create content/search/_index.md to satisfy this requirement. The file just needs minimal front matter to render:

---
title: Search me!
---

You can run hugo serve and navigate to /search to see if everything builds as expected.

A few libraries exist to help you build a search index and implement Lunr. You can find them here on the Hugo site. If you want to fully understand the process, however, you’ll find it’s not complicated do this without additional dependencies, thanks to the power of Hugo’s static site processing.

Build your search index

Here’s how to build an index for Lunr to search using Hugo’s template rendering power. Use range to loop over the pages you want to make searchable, and capture your desired parameters in an array of documents. One way to do this is to create layouts/partials/search-index.html with:

<script>
window.store = {
    // You can specify your blog section only:
    {{ range where .Site.Pages "Section" "posts" }}
    // For all pages in your site, use "range .Site.Pages"
    // You can use any unique identifier here
    "{{ .Permalink }}": {
        // You can customize your searchable fields using any .Page parameters
        "title": "{{ .Title  }}",
        "tags": [{{ range .Params.Tags }}"{{ . }}",{{ end }}],
        "content": {{ .Content | plainify }}, // Strip out HTML tags
        "url": "{{ .Permalink }}"
    },
    {{ end }}
}
</script>
<!-- Include Lunr and code for your search function,
which you'll write in the next section -->
<script src="/js/lunr.min.js"></script>
<script src="/js/search.js"></script>

When Hugo renders your site, it will build your search index in much the same way as a List page is built, creating a document for each page with its parameters.

The last piece of the puzzle is the code to handle the search process: taking the search query, getting Lunr to perform the search, and displaying the results.

Perform the search and show results

Create static/js/search.js to hold the JavaScript that ties it all together. This file has three main tasks: get the search query, perform the search with Lunr, and display the results.

Get query parameters with JavaScript

This part’s straightforward thanks to URLSearchParams:

const params = new URLSearchParams(window.location.search)
const query = params.get('q')

Search for the query with Lunr

Define and configure an index for Lunr. This tells Lunr what you’d like to search with, and you can optionally boost elements that are more important.

const idx = lunr(function () {
    // Search these fields
    this.ref('id')
    this.field('title', {
        boost: 15
    })
    this.field('tags')
    this.field('content', {
        boost: 10
    })

    // Add the documents from your search index to
    // provide the data to idx
    for (const key in window.store) {
        this.add({
        id: key,
        title: window.store[key].title,
        tags: window.store[key].category,
        content: window.store[key].content
        })
    }
})

You can then execute the search and store results with:

const results = idx.search(query)

Display results

You’ll need a function that builds a list of results and displays them on your search page. Recall the id you gave your ul element in layouts/search/list.html and store it as a variable:

const searchResults = document.getElementById('results')

If a search results in some results (🥁), you can iterate over them and build a <li> element for each one.

if (results.length) { // Length greater than 0 is truthy
    let resultList = ''
    for (const n in results) {
      // Use the unique ref from the results list to get the full item
      // so you can build its <li>
      const item = store[results[n].ref]
      resultList += '<li><p><a href="' + item.url + '">' + item.title + '</a></p>'
      // Add a short clip of the content
      resultList += '<p>' + item.content.substring(0, 150) + '...</p></li>'
    }
    searchResults.innerHTML = resultList
}

For each of your results, this produces a list item similar to:

<li>
    <p>
        <a href=".../blog/add-search-to-hugo-with-lunr/">
        Add search to Hugo static sites with Lunr
        </a>
    </p>
    <p>Yes, you can have an interactive search feature on your static site!...</p>
</li>

If there are no results, ham-handedly insert a message instead.

else {
    searchResults.innerHTML = 'No results found.'
}

Full code for search.js

Here’s what static/js/search.js could look like in full.

Make it go

You now have Lunr, the search index, and the code that displays results. Since these are all included in layouts/partials/search-index.html, add this partial on all pages with a search form. In your page footer, place:

{{ partial "search-index.html" . }}

You can see what this looks like when it’s all put together by trying it out on my blog.

Make it go faster

Since your site is static, it’s possible to pre-build your search index as a JSON data file for Lunr to load. This is where those aforementioned libraries may be helpful, since a JSON-formatted search index would need to be built outside of running hugo to generate your site.

You can maximize your search speed by minifying assets, and minimizing computationally expensive or blocking JavaScript in your code.

Static sites get search, too!

I hope this helps you make your Internet brain more useful for yourself and others, too! Don’t worry if you haven’t got the time to implement a search feature today – you can find this tutorial again when you visit victoria.dev and search for this post! 🥁

The beauty of not completely knowing what you’re doing is a lack of premature judgement. Without a standard to rise to, you’re free to go sideways. Explore. Try things that don’t work, without any expectation they will work. An open world with a beginner’s mindset.

The web that raised me was a little broken. Things didn’t always display the way they were supposed to. That too is part of the beauty. It was just broken enough to make you think for yourself.

1991 was the year of the individual on the web, the first year any layperson could open a web browser and access the new hypermedia dimension. There were no go-to, search-suggested, centralized websites. There were newsgroups. You had what you made and what your meatspace contacts sent you. In 2021, I think we need a return to that level of individualism. We need to make 2021 the year of the independent web.

That’s not to say I think the massive monopolistic platforms are going anywhere. Twitter, Facebook, mainstream “news” media sites – they’re all a kind of utility now, like plumbing and electricity. They’ll find their place in regulation and history. But they are not your website.

Your website is the one you create. Where the content, top-to-bottom, is yours alone to shape and present as you please. Your website is your place of self-expression, without follower counts or statistics to game. Your website is for creation, not reaction.

It’s all yours, but it doesn’t have to seem lonely. Your site can interact with the entire online world through syndication and protocols made possible by this thing we call the Internet. See:

• IndieWeb for POSSE, an abbreviation for Publish (on your) Own Site, Syndicate Elsewhere
• Webmention and an easy way to implement them
• twtxt instances for a decentralized timeline experience
• Neofeed , my personal timeline project made for Neocities . (It’s open source and you can help me extend it! )

Your website is your beginning point. The one source of truth for your identity online, from which you can generate and distribute disposable copies to any platform you please. This is what it means to truly own your content. And on the Internet, your content is you.

This is my website. When I first created it, I did so for myself. I had no expectation of visitors. I just knew I’d rather have these thoughts and things I’ve learned here, out here, made indelible in the folds of the public Internet, instead of on some dark corner of my machine, to be lost forever once I am.

Make your own website. You’ll grow your own sense of well-deserved accomplishment and contribute to your independence on the web. You’ll learn by doing, by scratching your own itch.

Learn about web technologies. Use them as you would if you were a child holding a pencil or paintbrush for the first time. Experiment, with no expectations other than discovering what you can do to make it delight you.

These sites and articles inspired this post and helped me implement webmentions!

• Why the Indie Web movement is so important, Dan Gillmor
• Jamie Tanna
• Max Böck
• Paul Robert Lloyd
• Zachary Dunn
• Adding Webmentions to My Static Hugo Site, Ana Ulin
• Adding Webmention Support to a Static Site, Keith J Grant
• Webmention.rocks

These candidates don’t lack technical ability. The problem is that traditional hiring processes don’t predict who will actually succeed and stay on your team. After years of hiring engineers and watching some thrive while others struggle, I’ve learned that the best predictors of long-term success are often the things most interviews completely miss.

Here’s what I actually look for when hiring engineers, and why these signals matter more than most technical assessments.

Look for Builders, Not Just Coders

The question that matters most isn’t “Can they solve algorithm problems?” It’s “Can they build things that solve problems?” There’s a fundamental difference between someone who can write code and someone who can deliver working software that serves a purpose.

When I review candidates, I’m looking for evidence that they’ve built complete projects from start to finish. Not just coding exercises or tutorial follow-alongs, but actual working software that solves real problems. This could be command-line utilities, web applications, automation tools, or contributions to open source projects—the complexity matters less than the completeness.

What I’m really evaluating is their ability to navigate the full software development lifecycle. Can they scope a problem, make technical decisions, handle edge cases, write documentation, and ship something that actually works? These are the skills that translate directly to success on your team, regardless of whether they learned them in a computer science program or taught themselves on weekends.

The best candidates can walk you through their projects and explain not just how they built something, but why they made specific technical choices. They understand the trade-offs they made and can articulate what they learned from the experience. This kind of thinking is what distinguishes engineers who will contribute meaningfully to your team from those who will struggle to move beyond assigned tasks.

Evaluate Systems Thinking Over Syntax Knowledge

Most technical interviews focus on whether someone knows specific syntax or can solve isolated problems. But the engineers who succeed on teams are the ones who understand how their code fits into larger systems and affects other people’s work.

I look for candidates who demonstrate awareness of follow-on effects. When they describe a project, do they consider performance implications? Do they think about maintainability? Can they explain how their technical decisions might impact other developers or users?

Understanding concepts like mutability, thread safety, and code reusability shows technical competence as well as thinking systematically about software as something that exists in a larger context. Engineers who grasp these concepts naturally write code that’s easier to debug, extend, and maintain. They consider the total cost of ownership, not just the immediate implementation.

During interviews, I ask candidates to explain technical trade-offs they’ve made in their projects. The specific technologies matter less than their ability to reason about complexity, performance, and maintainability. Engineers who think this way will continue learning and adapting as your company’s tech stack evolves.

Assess Communication Skills Through Real Examples

Communication skills aren’t just a “nice to have” for engineers—they’re essential for team effectiveness. But most hiring processes assess communication through artificial interview scenarios rather than looking at how candidates actually communicate about technical topics.

I spend significant time reviewing candidates’ written communication. How do they explain their projects in README files? How do they participate in open source discussions? Can they write clear, helpful documentation? These examples reveal how they’ll communicate with your team when explaining technical decisions, documenting systems, or participating in code reviews.

Pay attention to how candidates describe complex technical concepts during interviews. Can they adjust their explanation based on their audience’s technical background? Do they provide context and examples? Can they acknowledge when they don’t know something without becoming defensive?

The engineers who succeed long-term are those who can collaborate effectively across different skill levels and backgrounds. They can explain technical concepts to non-technical stakeholders, provide helpful code review feedback, and contribute to architectural discussions. These collaborative skills are often better predictors of success than pure technical ability.

Identify Team Players Through Contribution Patterns

The best predictor of how someone will behave on your team is how they’ve behaved on other teams. Rather than asking hypothetical questions about teamwork, look at concrete examples of how candidates have collaborated with others.

Open source contributions provide excellent insight into someone’s collaborative style. How do they handle feedback on their code? Do they contribute thoughtfully to discussions? Can they work within existing conventions and standards? Do they help other contributors or just focus on their own work?

For candidates without extensive open source history, look at how they talk about past team experiences. Do they credit others for successes? Can they describe situations where they helped colleagues or learned from feedback? How do they handle disagreement or conflict?

I’m particularly interested in candidates who show evidence of helping others grow. Engineers who mentor junior developers, contribute to team documentation, or improve development processes tend to have a positive impact that extends far beyond their individual contributions.

Evaluate Learning Ability Over Current Knowledge

Technology changes rapidly, which means the specific skills someone has today matter less than their ability to acquire new skills as needed. The engineers who thrive long-term are those who stay curious and adapt effectively to new challenges.

During interviews, I ask candidates about times they had to learn something completely new for a project. How did they approach unfamiliar technologies? What resources did they use? How did they validate their understanding? The process they describe reveals more about their potential than any specific technology they currently know.

I also look for evidence of intellectual humility. Can candidates acknowledge the limits of their knowledge? Do they ask thoughtful questions? Are they excited about learning from more experienced team members? Engineers who combine confidence in their abilities with openness to learning tend to grow quickly and integrate well with existing teams.

What This Means for Your Hiring Process

Identifying these qualities requires a different approach than traditional technical interviews. Instead of algorithm problems, focus on discussing real projects and technical decisions. Instead of whiteboard coding, review actual code they’ve written and ask them to explain their thinking.

Spend time on behavioral questions that reveal collaborative patterns and learning ability. Make time for informal conversation about what kind of work environment they thrive in and what they’re excited to learn next.

Most importantly, involve your team in the hiring process. The people who will work directly with your new hire are often better at assessing team fit than individual interviewers making isolated decisions.

Remember that hiring is ultimately about predicting future success, not just evaluating current abilities. The candidates who can build complete projects, think systematically about technical decisions, communicate effectively, and continue learning will contribute more to your team’s long-term success than those who simply perform well on coding tests.

Your perfect candidate isn’t necessarily the most technically skilled or the most knowledgeable about your domain. It’s the person who will grow with your team and contribute to the kind of collaborative, effective engineering culture that retains great people and delivers great software.

• How do I become a software developer?
• What language or framework should I learn first?
• Where do I start?

While I’m certain there’s no one right answer for everyone, I’m also certain that the world needs more software developers and systems thinkers.

The best thing I can do to help you lead yourself, learn to code, and become a software developer is to share the most efficient parts of how I did it myself. This is the article I wish I had read when I started coding.

Depth matters

Software is exceedingly complex. Like a good novel that you wish you’d never finish reading, there’s always more to discover and learn. If you don’t want to miss the best parts, don’t be satisfied with surface-level explanations. Always go deeper! Ask why, why, and why again until you get to the fundamentals. Soon enough, you’ll start to see patterns.

By digging deeper, you’ll begin to understand the fundamentals of how things connect, what makes things “fast,” and facets of software operation that you probably can’t even imagine exist. It’s like peeking behind the curtain and seeing a whole world of systems and processes that most people are never aware of.

Going in-depth can expand your mind and your capacity for learning. Keep asking why. Follow every link. Let your curiosity guide you.

Hard stuff matters

Giving yourself the chance to be delighted through discovery doesn’t come for free. It takes a lot of hard work to read and compress complicated ideas into your meat brain.

It’s important not to gloss over the hard stuff. In fact, if something seems too hard to understand, you might benefit from doing it first. You might have to get creative to find ways to explain things to yourself, but when you succeed, it makes everything else easier later on.

Analogies are helpful for understanding hard concepts, but they’ll only help you start to understand concepts at a surface level. Remember to go in-depth. Don’t stop at the analogy.

Writing matters

Write right away. Create a habit of explaining everything you learn to yourself in long-form writing. Better than bullet points, writing with a conversational tone engages parts of your brain that help you to process and remember new information. It’s why humans like and remember stories, and it’s a superpower you get for free.

Start by writing for yourself. Write about what interests you. Try something new, even if it seems rudimentary, and write in-depth about what you learn. (One of my most popular posts is about iteration in Python. When I first wrote it, I considered myself a complete beginner.)

If you want to go a step further, share your writing with the world!. Learn in public, like I do by writing on this site. I often get questions like, “how do I choose a theme for my blog?” or “what platform should I use?” or “what popular language/framework/topic should I focus on?” My answer is: don’t worry about it.

Don’t fret too much about your blog theme or platform. Pick the easiest option for you to get started with for now. All of that will change and improve as you learn, practice, and find your focus. Just start writing, ideally, yesterday.

Write for yourself by explaining what you’re doing, as if it were past-you teaching future-you — because it is. You will be your first reader, and the first judge of how useful your blog can be. Seek to impress yourself!

The language, framework, or version doesn’t matter

Why pigeonhole your abilities before you even start? Pick any software language, framework, or technology that seems to make sense to you when you first read it. Start there.

Remember that it’s important to dig deep and understand the fundamentals. Basic concepts of software transcend languages. Whichever first language you choose, understand functions, variables, return values, iteration, and how immutability works. You’ll find that learning these concepts will make it easier to recognize them in your second language, and learn that too.

Your portfolio doesn’t matter

If your first objective is to build a portfolio, you may be trying to run before you walk. Building a portfolio to showcase to potential employers is a great goal, but a terrible first step.

If you think of creating a polished portfolio as a first step, you’re liable to spend too much time making it pretty and presentable before focusing on the content. As someone who hires software developers, I can tell you wholeheartedly that I’d rather see clean and well-written code than a flashy front page.

Don’t confuse building a portfolio with building projects. Absolutely build projects, right from the beginning. There’s no better way to see the practical application of what you’re learning. Just treat them as first drafts, as training ground, and don’t worry about packaging them up for professional consumption.

By allowing yourself to build some draft projects first, you allow yourself the breathing room to learn from them. Focus on iteration, on making one small thing better each time, and you’ll build a portfolio without even realizing it.

Focus on what matters

Don’t follow this advice blindly; rather, incorporate it into your own systems. Experiment, make it work better than when you found it, then pay it forward by writing down what you’ve learned for someone else to read!

Here are my favorite books for reading or listening to if you want to cultivate a learning mindset. See non-coding books for coders.

If this article benefits you in some way, I encourage you to write about it! The process of learning how to learn is never finished. You can be the next iteration.

Unfortunately, Christmas only comes once a year. After years in the tech and cybersecurity world, my perspective has changed.

I’ve found that people, including myself, value receiving small, constant, incremental improvements far more than big changes once or a few times a year. It makes sense if you think about it. The former constantly delights in small, unexpected ways that make the user experience better. The latter is invisible, except for a few times a year.

There are occasions when big changes make sense. Say you’re re-launching functionality, or coinciding with an event that deserves all the fanfare of an unveiling.

Other than that, and for most of us, holding back doesn’t serve us at all. It may even come from something far more insidious: fear of judgement.

Being brave

Thinking such as “I’ll show it to the world when it’s ready,” always leaves out the most important detail. What does “ready” mean?

If you haven’t written down your definition of “ready,” consider that you may be holding back for no good reason. What’s the worst that could happen, anyway, if you make your work public when it’s less than perfect?

I decided to find out when I started to build in public. Instead of holding back work, I released a first version as soon as it functioned as intended. I leaned on v0.0.* tags as a way to say, “This is available, but still in progress.” Or, I’d say so outright, in the README.

In the world of open source, building in public can be scary stuff. It feels like making yourself vulnerable. It’s opening up part of yourself, a creative part, for scrutiny and nitpicking – by strangers. Of course it’s not comfortable.

Once I overcame the discomfort, once I decided to be brave and appreciate even possibly negative feedback, something amazing happened.

I suddenly had help.

Yes, there was scrutiny and nitpicking – but I don’t think any of it was ill-intentioned. I found that there existed whole communities of people who wanted to help me build a project that they thought was interesting. In some cases, I was utterly amazed when people submitted pull requests for issues I’d opened on my own projects describing enhancements I’d like to have.

I’ve been fortunate to have wonderful experiences with open source so far. Based on these experiences, I’d like to share with you what I’ve discovered to be the most effective ways to be brave and generous when it comes to open source.

‘Tis the season to share generously

When unprompted strangers submit helpful comments, issues, and pull requests on your projects, it feels like Christmas. You can give the gift of helpfulness in your contributions as well.

Treat comments like a face-to-face conversation. Greet the person you’re addressing. Use full sentences. Think about whether what you’re writing will make someone’s day better or worse, and be nice.

When writing issues, include as much technical detail as possible. Screenshots, console logs, screenshots of console logs, your operating system, browser, screen resolution – all these can help maintainers quickly diagnose a root cause.

Pull requests are the best presents ever. They make maintainers happy when well done, and it’s a gift that gives back when your contribution gets merged! 🎉 Give your PR the best chance of getting accepted by looking for and following any project contribution guidelines.

Recognize the human

Our brains are slightly lacking in an evolutionary sense when it comes to interacting with other humans through tiny screens. It can be easy to forget that the actions you put out there will eventually reach one or more other people.

You can help to maintain a great open source community by remembering the humans that make it exist. When commenting, take the time to do it well (see below) and recognize the time that someone else has put in. When closing a thread or merging a contribution, remember to say thank you to the people who pitched in to help. I try to use first names instead of screen names, whenever possible.

You can build personal relationships, too. If you’re a project maintainer, you may choose to give people a way to contact you directly to ask questions or hash out complicated plans. Establishing one-to-one communications with regular contributors is also a great way to build a community around your project.

Recognizing the humans behind the open source community is a simple and meaningful way to give back.

Don’t rush

The vast majority of open source participants are volunteers, which means they don’t get paid for the time they spend building up projects. That sometimes means that other work takes priority. It’s okay if this describes you, too.

It’s important to remember that in most cases, a well-done contribution later is preferred over a half-done contribution sooner. If you’re too short on time now to write a thoughtful comment – don’t! Either draft a quick note and set it aside for later, or comment something along the lines of:

Hi there! Just wanted to let you know that I’ve seen this and I plan to help! I’ll respond in full as soon as I have the time to write a thoughtful comment.

Showing that you think a comment is worth the time to do well is something that open source contributors and repository maintainers both appreciate.

Build generously

When open source participants act with conscientiousness, every day feels like Christmas. Regardless of your type of contribution, you can help build this generous global community year-round.

The humans of open source, by self selection, mostly consist of good people who want to help. If you build openly, share feedback generously, and try to do good in general, I think you fit in here.

I hope you have a very happy holiday season and give many gifts that keep on giving!

For those of you seeing relatives this season, chances are that you’re the designated family tech support. If part of your time home for the holidays is spent on software updates and troubleshooting WiFi, here are a few other quick wins to help boost your family’s online privacy and security.

1. Set up a VPN

Using a VPN is Online Safety 101. Choose a reputable provider with a strict no-logging policy, or if you’re up for it, roll your own.

2. Introduce a password manager

If your family member uses the same password everywhere (<petname>+<house number>, same as last year) because passwords are hard to remember, introduce them to their new best friend, 1Password. Help your family get set up with secure passwords they don’t have to write down on Post-It notes – just one master pass(phrase) is all you need.

When choosing a passphrase, avoid using information easily found on social media accounts, like pet names, favorite sports teams, favorite brands, or birthdays.

3. Switch to DuckDuckGo

Help fight the Internet search monopoly by getting your family to use a search engine that respects their privacy. Go to your browser Settings and set your Default Search Engine (that uses the URL bar) to DuckDuckGo. Break the ice with an instant answer feature, like searching “calendar” so you can count down to Christmas.

(You might want to search for “classic cocktails cheat sheet” after all this.)

4. Install a better browser and blocker

While I prefer a Pi-hole, setting one up can be complex. Instead, help set up a privacy-preserving browser like Firefox or a wide-spectrum blocking extension like uBlock Origin (GitHub source).

Your family will get faster page load times, less advertisements interrupting articles and videos, and fewer sneaky trackers leaking browsing habits to big tech, all with near-zero maintenance.

Be a home-for-the-holidays hero!

Help improve your family’s security posture this holiday season. A little beefed-up cybersecurity may be one of the best gifts you can give!

I’m keeping it short-and-sweet this week. My annual Christmas post drops on December 24, full of warm fuzzy goodness and a tech tip or two. Thank you for being a subscriber – stay tuned!

In the technology teams I lead, we make a constant effort to document all the things. Documentation lives alongside the code as an equal player. This helps ensure that no one needs to make assumptions about how something works, or is calling lengthy meetings to gain working knowledge of a feature. Good documentation saves us a lot of time and hassle.

That said, and contrary to popular belief, the most valuable software documentation is not primarily written for other people. As I said in this well-received tweet:

The secret to good documentation is to write it while you're writing the code. You are your first audience. Explain what you're doing to yourself. Future you will thank you!

— Victoria Drake November 24, 2020

Here are three concrete steps you can take to write good documentation before it’s too late.

1. Start with accurate notes

As you work out ideas in code, ensure you don’t soon forget important details by starting with accurate notes. While you will want to explain things to yourself in long-form later, short-form notes will suffice to capture details without interrupting your coding session flow.

Don’t rely on inline comments that often fail to make sense once you’ve forgotten the context. Keep a document open alongside your code and write down things like commands, decisions, and sources you use. This can include:

• Prompts or shell commands you used
• Why you chose a particular method over another
• Links you visited for help or coughcopy-pastecough inspiration
• The order in which you did things

Don’t worry about full sentences at this point. Just ensure you accurately capture context, relevant code snippets, and helpful URLs. It can also be helpful to turn on any auto-save option available.

2. Explain decisions in long form

The ideal time to tackle this step is when you take a break from coding, but before you completely go out to lunch on whatever it is you’re working on at the moment. You want to ensure that context, ideas, and decisions are all still fresh in your mind when you explain them to yourself.

Go over the short-form notes you took and start expanding them into conversational writing. Be your own rubber duck. Describe what you’re doing as if you were teaching it to someone else. You might cover topics such as:

• Quirky-looking decisions: “I would normally do it this way, but I chose to do something different because…”
• Challenges you ran into and how you overcame them
• Architectural decisions that support your project goals

Stick to the main points. Long-form writing doesn’t mean you’ll be paid by the word! Just use full sentences, and write as if explaining your project to a colleague. You’re explaining to future you, after all.

3. Don’t neglect prerequisite knowledge

This step is best done after a long lunch break, or even the next day (but probably not two). Re-read your document and fill in any blanks that become apparent after putting some distance between yourself and the project.

Take extra care to fill in or at least link to prerequisite knowledge, especially if you frequently use different languages or tools. Even an action as small as pasting in a link to the API documentation you used can save hours of future searching.

Write down or link to READMEs, installation steps, and relevant support issues. For frequently performed command-line actions, you can use a self-documenting Makefile to avoid having to man common tasks each time you come back to a project.

It’s easy to forget supporting details after even just a short break from your project. Capture anything you found helpful this time around.

Document all the things

The next time you catch yourself thinking, “I’m sure I’ll remember this part, no need to write it down,” just recall this emoji: 🤦‍♀️

Software projects are made up of a lot more than just their code. To best set up your future self for success, document all the things! Whether it’s a process you’ve established, Infrastructure as Code, or a fleeting future roadmap idea — write it down! Future you will thank you for it.

If you enjoyed this post, there’s a lot more where that came from! I write about developer ergonomics for high-performing teams and building beautiful, maintainable software in the age of AI. You can subscribe below to see new posts first.

I discovered this the hard way while working with a team that was constantly context-switching between “urgent” projects. Everyone was busy, morale was decent, but we weren’t actually shipping much of value. During one particularly frustrating week, I counted seventeen different tasks that had been labeled as “high priority” by various stakeholders. Our standups felt like disaster reports, and I realized we’d created a system where being busy had become more important than being effective.

The solution turned out to be surprisingly simple, though not easy to implement: put everything into a single, ordered list where only one thing can be most important at any given time.

The Radical Transparency of a Central List

Most teams I’ve encountered operate like a collection of individual to-do lists with some coordination meetings sprinkled on top. Engineering works on technical debt, product pushes for new features, leadership wants infrastructure improvements, and everyone optimizes their own piece of the puzzle. The result is a lot of activity that doesn’t add up to meaningful progress.

A single, centralized, prioritized list changes the entire dynamic. Everyone can see what’s actually being worked on, what’s coming next, and most importantly, what’s not getting done and why. This visibility creates natural conversations about trade-offs that simply don’t happen when work is siloed.

I’ve watched teams discover they were working on competing solutions to the same problem, simply because no one had a complete view of active work. Others realized they were delaying important projects because someone assumed “someone else” was handling the dependency. When everything is visible and ordered, these coordination problems become obvious and fixable.

The transparency also creates a different kind of accountability. When priorities are public and explicit, it becomes much harder to justify working on pet projects or avoiding difficult tasks. The list becomes a shared source of truth that guides decisions rather than each person interpreting priorities through their own lens.

Autonomy Within Structure

One concern I hear frequently is that a single priority list will turn people into order-takers rather than creative problem-solvers. In practice, I’ve found exactly the opposite happens when you implement it correctly.

The key is encouraging people to choose the highest-priority task they can effectively tackle rather than assigning specific tasks to specific people. Someone might skip over the absolute top item because it requires domain knowledge they don’t have, but they can pick up the second or third item that lets them contribute meaningfully while learning something new.

This approach leverages the fact that your team members understand their own capabilities and growth goals better than you do. A senior engineer might choose to mentor a junior developer on a complex task. A frontend specialist might want to tackle a backend task to broaden their skills. These decisions create better outcomes in the long term than top-down task assignment while still maintaining focus on organizational priorities.

The autonomy comes from trusting people to make good decisions about how to contribute most effectively, while the structure comes from ensuring those contributions align with actual business needs.

The Art of Making Yourself Redundant

If your team frequently asks you what they should work on next, you’ve accidentally created a bottleneck—and it’s you. This is one of the most common scaling problems I see with engineering leaders who transition from individual contributor roles.

The goal is building a system where intelligent people can make good decisions without constant input from leadership. This requires making context painfully available—team goals, product strategy, architectural decisions, customer feedback, and anything else that influences prioritization should be accessible and current.

I’ve found that the difference between teams that scale smoothly and teams that hit velocity walls usually comes down to how well they’ve documented the reasoning behind decisions. When someone can understand not just what to build but why it matters and how it fits into the larger strategy, they can make smart trade-offs independently.

This redundancy becomes especially critical during high-pressure situations. When systems are down or deadlines are looming, you don’t want your team waiting for permission to take action. Teams that have practiced autonomous decision-making within clear constraints can respond quickly and effectively without requiring heroic coordination efforts.

The Cultural Transformation

What surprises most leaders is how much this simple change affects team culture. When priorities are clear and transparent, several things happen that go far beyond improved task management.

First, political conversations about priority disappear. There’s no point in lobbying for your favorite project when the criteria for prioritization are explicit and the current order is visible to everyone. Energy that was spent on organizational maneuvering gets redirected toward actual work.

Second, people start thinking about their contributions differently. Instead of optimizing for individual productivity, they begin considering how their work fits into team objectives. This naturally leads to better collaboration and knowledge sharing.

Third, the team develops a shared sense of progress and momentum. When everyone can see important work getting completed in priority order, it creates a satisfying rhythm that isolated individual achievements can’t match.

Implementation Reality

The biggest challenge isn’t creating the list—it’s maintaining the discipline to use it consistently. Teams often start strong but gradually drift back to multiple priority tracks when pressure increases or when compelling new opportunities arise.

I’ve learned to treat priority discipline like any other technical practice that requires ongoing attention. Schedule regular review sessions to reorder the list, have explicit discussions about what we’re choosing not to do, and consistently communicate why keeping a single-priority focus helps maintain development velocity.

The payoff: teams that ship more valuable work with less stress and confusion. When everyone understands what matters most and feels empowered to contribute effectively, both productivity and job satisfaction improve dramatically.

Most importantly, single-priority focus creates sustainable high performance rather than the boom-and-bust cycles that come from constantly shifting between competing urgent demands. Teams learn to work steadily toward important goals rather than reacting to whatever feels most pressing in the moment.

Here’s a reprint of the announcement I wrote for owasp.org. If you’re interested in security testing for web applications and APIs, this is an update you’ll definitely want to check out!

You can become a contributor yourself by joining us on GitHub!

Web Security Testing Guide v4.2 Released

Thursday, December 3, 2020

The OWASP Web Security Testing Guide team is proud to announce version 4.2 of the Web Security Testing Guide (WSTG)! In keeping with a continuous delivery mindset, this new minor version adds content as well as improves the existing tests.

In recent years, the Web Security Testing Guide has sought to remain your foremost open source resource for web application testing. Our previous release marked a move from a cumbersome wiki platform to the highly collaborative world of GitHub. Since then, over 61 new contributors pushing over 600 commits have helped to make the WSTG better than ever.

Version 4.2 of the Web Security Testing Guide introduces new testing scenarios, updates existing chapters, and offers an improved reading experience with a clearer writing style and chapter layout. Readers will enjoy easier navigation and consistent testing instructions.

With new improvements to our development workflow, new contributors will find it easier than ever to help build future versions of the WSTG. A clear and concise contributor’s guide and style guide can help you write new tests or ensure existing scenarios stay current. Core maintainers Rick Mitchell, Elie Saad, Rejah Rehim, and Victoria Drake have implemented modern processes like continuous integration with GitHub Actions. New workflows help to build PDFs and make reviewing new additions and updates easier.

We couldn’t be happier to share this new version with you, and we don’t plan to slow down anytime soon. The dedicated volunteers who’ve made this release possible are already hard at work on the next major version of the WSTG. Come join us and become a contributor!

You can read the Web Security Testing Guide v4.2 online or download a PDF on our project page. We greatly appreciate all the authors, editors, reviewers, and readers who make this open source security endeavor worthwhile.

Thank you for being a part of the WSTG!

To make the Internet possible, two things that needed imagining are layers and protocols. Layers are conceptual divides that group similar functions together. The word “protocol,” means “the way we’ve agreed to do things around here,” more or less. In short, both layers and protocols can be explained to a five-year-old as “ideas that people agreed sounded good, and then they wrote them down so that other people could do things with the same ideas.”

The Internet Protocol Suite is described in terms of layers and protocols. Collectively, the suite refers to the communication protocols that enable our endless scrolling. It’s often called by its foundational protocols: the Transmission Control Protocol (TCP) and the Internet Protocol (IP). Lumped together as TCP/IP, these protocols describe how data on the Internet is packaged, addressed, sent, and received.

Here’s why the Internet Protocol Suite, or TCP/IP, is an imaginary rainbow layer cake.

Layers are imaginary

If you consider the general nature of a rainbow layer sponge cake, it’s mostly made up of soft, melt-in-your mouth vanilla-y goodness. This goodness is in itself comprised of something along the lines of eggs, butter, flour, and sweetener.

There isn’t much to distinguish one layer of a rainbow sponge cake from another. Often, the only difference between layers is the food-coloring and a bit of frosting. When you think about it, it’s all cake from top to bottom. The rainbow layers are only there because the baker thought they ought to be.

Similar to cake ingredients, layers in the context of computer networking are mostly composed of protocols, algorithms, and configurations, with some data sprinkled in. It can be easier to talk about computer networking if its many functions are split up into groups, so certain people came up with descriptions of layers, which we call network models. TCP/IP is just one network model among others. In this sense, layers are concepts, not things.

Some of the people in question are part of the Internet Engineering Task Force (IETF). They created the RFC-1122 publication, discussing the Internet’s communications layers. Half of a whole, the standard:

…covers the communications protocol layers: link layer, IP layer, and transport layer; its companion RFC-1123 covers the application and support protocols.

The layers described by RFC-1122 and RFC-1123 each encapsulate protocols that satisfy the layer’s functionality. Let’s look at each of these communications layers and see how TCP and IP stack up in this model of the Internet layer cake.

Link layer protocols

The link layer is the most basic, or lowest-level, classification of communication protocol. It deals with sending information between hosts on the same local network, and translating data from the higher layers to the physical layer. Protocols in the link layer describe how data interacts with the transmission medium, such as electronic signals sent over specific hardware. Unlike other layers, link layer protocols are dependent on the hardware being used.

Internet layer protocols

Protocols in the Internet layer describe how data is sent and received over the Internet. The process involves packaging data into packets, addressing and transmitting packets, and receiving incoming packets of data.

The most widely known protocol in this layer gives TCP/IP its last two letters. IP is a connectionless protocol, meaning that it provides no guarantee that packets are sent or received in the right order, along the same path, or even in their entirety. Reliability is handled by other protocols in the suite, such as in the transport layer.

There are currently two versions of IP in use: IPv4, and IPv6. Both versions describe how devices on the Internet are assigned IP addresses, which are used when navigating to cat memes. IPv4 is more widely used, but has only 32 bits for addressing, allowing for about 4.3 billion (ca. 4.3×10⁹) possible addresses. These are running out, and IPv4 and will eventually suffer from address exhaustion as more and more people use more devices on the Internet.

The successor version IPv6 aims to solve address exhaustion by using 128 bits for addresses. This provides, um, a lot more address possibilities (ca. 3.4×10³⁸).

Transport layer protocols

In May 1974, Vint Cerf and Bob Kahn (collectively often called “the fathers of the Internet”) published a paper entitled A Protocol for Packet Network Intercommunication. This paper contained the first description of a Transmission Control Program, a concept encompassing what would eventually be known as the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). (I had the pleasure of meeting Vint and can personally confirm that yes, he does look exactly like The Architect in the Matrix movies.)

The transport layer presently encapsulates TCP and UDP. Like IP, UDP is connectionless and can be used to prioritize time over reliability. TCP, on the other hand, is a connection-oriented transport layer protocol that prioritizes reliability over latency, or time. TCP describes transferring data in the same order as it was sent, retransmitting lost packets, and controls affecting the rate of data transmission.

Application layer protocols

The application layer describes the protocols that software applications interact with most often. The specification includes descriptions of the remote login protocol Telnet, the File Transfer Protocol (FTP), and the Simple Mail Transfer Protocol (SMTP).

Also included in the application layer are the Hypertext Transfer Protocol (HTTP) and its successor, Hypertext Transfer Protocol Secure (HTTPS). HTTPS is secured by Transport Layer Security, or TLS, which can be said to be the top-most layer of the networking model described by the Internet protocol suite. If you’d like to further understand TLS and how this protocol secures your cat meme viewing, I invite you read my article about TLS and cryptography.

The Internet cake is still baking

Like a still-rising sponge cake, descriptions of layers, better protocols, and new models are being developed every day. The Internet, or whatever it will become in the future, is still in the process of being imagined.

If you enjoyed learning from this post, there’s a lot more where this came from! I write about computing, cybersecurity, and building great technical teams. You can subscribe below to see new posts first.

Most modern devices and web browsers allow users to choose either a light or dark theme for the user interface. With CSS media queries, you can have your own website’s styles change to match this user setting!

Media queries are also a common way to have elements on web pages change to suit different screen sizes. This is an especially powerful tool when combined with custom properties set on the root element.

Here’s how to use CSS media queries and custom properties to improve your visitor’s browsing experience with just a few lines of CSS.

Catering to color preferences

The prefers-color-scheme media feature can be queried to serve up your user’s color scheme of choice. The light option is the go-to version if no active preference is set, and it has decent support across modern browsers.

Additionally, users reading on certain devices can also set light and dark color themes based on a schedule. For example, my phone uses light colors throughout its UI during the daytime, and dark colors at night. You can make your website follow suit!

Avoid repeating a lot of CSS by setting custom properties for your color themes on your :root pseudo-class. You can specify the themes available with the color-scheme property (currently part of a draft specification, but I like to write my articles to age well). Create a version for each theme you wish to support. Here’s a quick example you can build on:

:root {
    color-scheme: light dark;
}

@media (prefers-color-scheme: light) {
    :root {
        --text-primary: #24292e;
        --background: white;
        --shadow: rgba(0, 0, 0, 0.15) 0px 2px 5px 0px;
    }
}

@media (prefers-color-scheme: dark) {
    :root {
        --text-primary: white;
        --background: #24292e;
        --shadow: rgba(0, 0, 0, 0.35) 0px 2px 5px 0px;
    }
}

As you can see, you can use custom properties to set all kinds of values. To use these as variables with other CSS elements, use the var() function:

header {
    color: var(--text-primary);
    background-color: var(--background);
    box-shadow: var(--shadow);
}

In this quick example, the header element will now display your user’s preferred colors according to their browser settings!

Preferred color schemes are set by the user in different ways, depending on the browser. Here are a couple examples.

Firefox

You can test out light and dark modes in Firefox by typing about:config into the address bar. Accept the warning if it pops up, then type ui.systemUsesDarkTheme into the search.

Choose a Number value for the setting, then input a 1 for dark or 0 for light.

Brave

If you’re using Brave, find color theme settings in Settings > Appearance > Brave colors.

Variable scaling

You can also use a custom property to effortlessly adjust the size of text or other elements depending on your user’s screen size. The width media feature tests the width of the viewport. While width: _px will match an exact size, you can also use min and max to create ranges.

Query with min-width: _px to match anything over _ pixels, and max-width: _px to match anything up to _ pixels.

Use these queries to set a custom property on the :root to create a ratio:

@media (min-width: 360px) {
    :root {
        --scale: 0.8;
    }
}

@media (min-width: 768px) {
    :root {
        --scale: 1;
    }
}

@media (min-width: 1024px) {
    :root {
        --scale: 1.2;
    }
}

Then make an element responsive by using the calc() function. Here are a few examples:

h1 {
    font-size: calc(42px * var(--scale));
}

h2 {
    font-size: calc(26px * var(--scale));
}

img {
    width: calc(200px * var(--scale));
}

In this example, multiplying an initial value by your --scale custom property allows the size of headings and images to magically adjust to your user’s device width.

The relative unit rem will have a similar effect. You can use it to define sizes for elements relative to the font size declared at the root element.

h1 {
    font-size: calc(5rem * var(--scale));
}

h2 {
    font-size: calc(1.5rem * var(--scale));
}

p {
    font-size: calc(1rem * var(--scale));
}

Of course, you can also multiply two custom properties. For example, setting the --max-img as a custom property on the :root can help to save you time later on by not having to update a pixel value in multiple places:

img {
    max-width: calc(var(--max-img) * var(--scale));
}

Raise your responsiveness game

Try out these easy wins for a website that caters to your visitor’s devices and preferences. I’ve put them to good use now on victoria.dev. I invite you to let me know how you like it!

Introducing Simple Subscribe

If you’re interested in managing your own mailing list or newsletter, you can set up Simple Subscribe on your own AWS resources to collect email addresses. This open source API is written in Go, and runs on AWS Lambda. Visitors to your site can sign up to your list, which is stored in a DynamoDB table, ready to be queried or exported at your leisure.

When someone signs up, they’ll receive an email asking them to confirm their subscription. This is sometimes called “double opt-in,” although I prefer the term “verified.” Simple Subscribe works on serverless infrastructure and uses an AWS Lambda to handle subscription, confirmation, and unsubscribe requests.

You can find the Simple Subscribe project, with its fully open-source code, on GitHub. I encourage you to pull up the code and follow along! In this post I’ll share each build step, the thought process behind the API’s single-responsibility functions, and security considerations for an AWS project like this one.

Building a verified subscription flow

A non-verified email sign up process is straightforward. Someone puts their email into a box on your website, then that email goes into your database. However, if I’ve taught you anything about not trusting user input, the very idea of a non-verified sign up process should raise your hackles. Spam may be great when fried in a sandwich, but no fun when it’s running up your AWS bill.

While you can use a strategy like a CAPTCHA or puzzle for is-it-a-human verification, these can create enough friction to turn away your potential subscribers. Instead, a confirmation email can help to ensure both address correctness and user sentience.

To build a subscription flow with email confirmation, create single-responsibility functions that satisfy each logical step. Those are:

1. Accept an email address and record it.
2. Generate a token associated with that email address and record it.
3. Send a confirmation email to that email address with the token.
4. Accept a verification request that has both the email address and token.

To achieve each of these goals, Simple Subscribe uses the official AWS SDK for Go to interact with DynamoDB and SES.

At each stage, consider what the data looks like and how you store it. This can help to handle conundrums like, “What happens if someone tries to subscribe twice?” or even threat-modeling such as, “What if someone subscribes with an email they don’t own?”

Ready? Let’s break down each step and see how the magic happens.

Subscribing

The subscription process begins with a humble web form, like the one on my site’s main page. A form input with attributes type="email" required helps with validation, thanks to the browser. When submitted, the form sends a GET request to the Simple Subscribe subscription endpoint.

Simple Subscribe receives a GET request to this endpoint with a query string containing the intended subscriber’s email. It then generates an id value and adds both email and id to your DynamoDB table.

The table item now looks like:

The confirm column, which holds a boolean, indicates that the item is a subscription request that has not yet been confirmed. To verify an email address in the database, you’ll need to find the correct item and change confirm to true.

As you work with your data, consider the goal of each manipulation and how you might compare an incoming request to existing data.

For example, if someone made a subsequent subscription request for the same email address, how would you handle it? You might say, “Create a new line item with a new id,” however, this might not be best strategy when your serverless application database is paid for by request volume.

Since DynamoDB Pricing depends on how much data you read and write to your tables, it’s advantageous to avoid piling on excess data.

With that in mind, it would be prudent to handle subscription requests for the same email by performing an update instead of adding a new line. Simple Subscribe actually uses the same function to either add or update a database item. This is typically referred to as, “update or insert.”

In a database like SQLite this is accomplished with the UPSERT syntax. In the case of DynamoDB, you use an update operation. For the Go SDK, its syntax is UpdateItem.

When a duplicate subscription request is received, the database item is matched on the email only. If an existing line item is found, its id and timestamp are overridden, which updates the existing database record and avoids flooding your table with duplicate requests.

Verifying email addresses

After submitting the form, the intended subscriber then receives an email from SES containing a link. This link is built using the email and id from the table, and takes the format:

<BASE_URL><VERIFY_PATH>/?email=subscriber@example.com&id=uuid-xxxxx

In this set up, the id is a UUID that acts as a secret token. It provides an identifier that you can match that is sufficiently complex and hard to guess. This approach deters people from subscribing with email addresses they don’t control.

Visiting the link sends a request to your verification endpoint with the email and id in the query string. This time, it’s important to compare both the incoming email and id values to the database record. This verifies that the recipient of the confirmation email is initiating the request.

The verification endpoint ensures that these values match an item in your database, then performs another update operation to set confirm to true, and update the timestamp. The item now looks like:

Querying for emails

You can now query your table to build your email list. Depending on your email sending solution, you might do this manually, with another Lambda, or even from the command line.

Since data for requested subscriptions (where confirm is false) is stored in the table alongside confirmed subscriptions, it’s important to differentiate this data when querying for email addresses to send to. You’ll want to ensure you only return emails where confirm is true.

Providing unsubscribe links

Similar to verifying an email address, Simple Subscribe uses email and id as arguments to the function that deletes an item from your DynamoDB table in order to unsubscribe an email address. To allow people to remove themselves from your list, you’ll need to provide a URL in each email you send that includes their email and id as a query string to the unsubscribe endpoint. It would look something like:

<BASE_URL><UNSUBSCRIBE_PATH>/?email=subscriber@example.com&id=uuid-xxxxx

When the link is clicked, the query string is passed to the unsubscribe endpoint. If the provided email and id match a database item, that item will be deleted.

Proving a method for your subscribers to automatically remove themselves from your list, without any human intervention necessary, is part of an ethical and respectful philosophy towards handling the data that’s been entrusted to you.

Caring for your data

Once you decide to accept other people’s data, it becomes your responsibility to care for it. This is applicable to everything you build. For Simple Subscribe, it means maintaining the security of your database, and periodically pruning your table.

In order to avoid retaining email addresses where confirm is false past a certain time frame, it would be a good idea to set up a cleaning function that runs on a regular schedule. This can be achieved manually, with an AWS Lambda function, or using the command line.

To clean up, find database items where confirm is false and timestamp is older than a particular point in time. Depending on your use case and request volumes, the frequency at which you choose to clean up will vary.

Also depending on your use case, you may wish to keep backups of your data. If you are particularly concerned about data integrity, you can explore On-Demand Backup or Point-in-Time Recovery for DynamoDB.

Build your independent subscriber base

Building your own subscriber list can be an empowering endeavor! Whether you intend to start a newsletter, send out notifications for new content, or want to create a community around your work, there’s nothing more personal or direct than an email from me to you.

I encourage you to start building your subscriber base with Simple Subscribe today! Like most of my work, it’s open source and free for your personal use. Dive into the code at the GitHub repository.

About 60 seconds to billions of years, as it turns out.

All Wi-Fi encryption is not created equal. Let’s explore what makes these four acronyms so different, and how you can best protect your home and organization Wi-Fi.

Wired Equivalent Privacy (WEP)

In the beginning, there was WEP.

Wired Equivalent Privacy is a deprecated security algorithm from 1997 that was intended to provide equivalent security to a wired connection. “Deprecated” means, “Let’s not do that anymore.”

Even when it was first introduced, it was known not to be as strong as it could have been, for two reasons: one, its underlying encryption mechanism; and two, World War II.

During World War II, the impact of code breaking (or cryptanalysis) was huge. Governments reacted by attempting to keep their best secret-sauce recipes at home. Around the time of WEP, U.S. Government restrictions on the export of cryptographic technology caused access point manufacturers to limit their devices to 64-bit encryption. Though this was later lifted to 128-bit, even this form of encryption offered a very limited possible key size.

This proved problematic for WEP. The small key size resulted in being easier to brute-force, especially when that key doesn’t often change.

WEP’s underlying encryption mechanism is the RC4 stream cipher. This cipher gained popularity due to its speed and simplicity, but that came at a cost. It’s not the most robust algorithm. WEP employs a single shared key among its users that must be manually entered on an access point device. (When’s the last time you changed your Wi-Fi password? Right.) WEP didn’t help matters either by simply concatenating the key with the initialization vector – which is to say, it sort of mashed its secret-sauce bits together and hoped for the best.

Initialization Vector (IV): fixed-size input to a low-level cryptographic algorithm, usually random.

Combined with the use of RC4, this left WEP particularly susceptible to related-key attack. In the case of 128-bit WEP, your Wi-Fi password can be cracked by publicly-available tools in a matter of around 60 seconds to three minutes.

While some devices came to offer 152-bit or 256-bit WEP variants, this failed to solve the fundamental problems of WEP’s underlying encryption mechanism.

So, yeah. Let’s not do that anymore.

Wi-Fi Protected Access (WPA)

A new, interim standard sought to temporarily “patch” the problem of WEP’s (lack of) security. The name Wi-Fi Protected Access (WPA) certainly sounds more secure, so that’s a good start; however, WPA first started out with another, more descriptive name.

Ratified in a 2004 IEEE standard, Temporal Key Integrity Protocol (TKIP) uses a dynamically-generated, per-packet key. Each packet sent has a unique temporal 128-bit key, (See? Descriptive!) that solves the susceptibility to related-key attacks brought on by WEP’s shared key mashing.

TKIP also implements other measures, such as a message authentication code (MAC). Sometimes known as a checksum, a MAC provides a cryptographic way to verify that messages haven’t been changed. In TKIP, an invalid MAC can also trigger rekeying of the session key. If the access point receives an invalid MAC twice within a minute, the attempted intrusion can be countered by changing the key an attacker is trying to crack.

Unfortunately, in order to preserve compatibility with the existing hardware that WPA was meant to “patch,” TKIP retained the use of the same underlying encryption mechanism as WEP – the RC4 stream cipher. While it certainly improved on the weaknesses of WEP, TKIP eventually proved vulnerable to new attacks that extended previous attacks on WEP. These attacks take a little longer to execute by comparison: for example, twelve minutes in the case of one, and 52 hours in another. This is more than sufficient, however, to deem TKIP no longer secure.

WPA, or TKIP, has since been deprecated as well. So let’s also not do that anymore.

Which brings us to…

Wi-Fi Protected Access II (WPA2)

Rather than spend the effort to come up with an entirely new name, the improved Wi-Fi Protected Access II (WPA2) standard instead focuses on using a new underlying cipher. Instead of the RC4 stream cipher, WPA2 employs a block cipher called Advanced Encryption Standard (AES) to form the basis of its encryption protocol. The protocol itself, abbreviated CCMP, draws most of its security from the length of its rather long name (I’m kidding): Counter Mode Cipher Block Chaining Message Authentication Code Protocol, which shortens to Counter Mode CBC-MAC Protocol, or CCM mode Protocol, or CCMP. 🤷

CCM mode is essentially a combination of a few good ideas. It provides data confidentiality through CTR mode, or counter mode. To vastly oversimplify, this adds complexity to plaintext data by encrypting the successive values of a count sequence that does not repeat. CCM also integrates CBC-MAC, a block cipher method for constructing a MAC.

AES itself is on good footing. The AES specification was established in 2001 by the U.S. National Institute of Standards and Technology (NIST) after a five-year competitive selection process during which fifteen proposals for algorithm designs were evaluated. As a result of this process, a family of ciphers called Rijndael (Dutch) was selected, and a subset of these became AES. For the better part of two decades, AES has been used to protect every-day Internet traffic as well as certain levels of classified information in the U.S. Government.

While possible attacks on AES have been described, none have yet been proven to be practical in real-world use. The fastest attack on AES in public knowledge is a key-recovery attack that improved on brute-forcing AES by a factor of about four. How long would it take? Some billions of years.

Wi-Fi Protected Access III (WPA3)

The next installment of the WPA trilogy has been required for new devices since July 1, 2020. Expected to further enhance the security of WPA2, the WPA3 standard seeks to improve password security by being more resilient to word list or dictionary attacks.

Unlike its predecessors, WPA3 will also offer forward secrecy. This adds the considerable benefit of protecting previously exchanged information even if a long-term secret key is compromised. Forward secrecy is already provided by protocols like TLS by using asymmetric keys to establish shared keys. You can learn more about TLS in this post.

As WPA2 has not been deprecated, both WPA2 and WPA3 remain your top choices for Wi-Fi security.

If the other ones suck, why are they still around?

You may be wondering why your access point even allows you to choose an option other than WPA2 or WPA3. The likely reason is that you’re using legacy hardware, which is what tech people call your mom’s router.

Since the deprecation of WEP and WPA occurred (in old-people terms) rather recently, it’s possible in large organizations as well as your parent’s house to find older hardware that still uses these protocols. Even newer hardware may have a business need to support these older protocols.

While I may be able to convince you to invest in a shiny new top-of-the-line Wi-Fi appliance, most organizations are a different story. Unfortunately, many just aren’t yet cognizant of the important role cybersecurity plays in meeting customer needs and boosting that bottom line. Additionally, switching to newer protocols may require new internal hardware or firmware upgrades. Especially on complex systems in large organizations, upgrading devices can be financially or strategically difficult.

Boost your Wi-Fi security

If it’s an option, choose WPA2 or WPA3. Cybersecurity is a field that evolves by the day, and getting stuck in the past can have dire consequences.

If you can’t use WPA2 or WPA3, do the best you can to take additional security measures. The best bang for your buck is to use a Virtual Private Network (VPN). Using a VPN is a good idea no matter which type of Wi-Fi encryption you have. On open Wi-Fi (coffee shops) and using WEP, it’s plain irresponsible to go without a VPN. Kind of like shouting out your bank details as you order your second cappuccino.

When possible, ensure you only connect to known networks that you or your organization control. Many cybersecurity attacks are executed when victims connect to an imitation public Wi-Fi access point, also called an evil twin attack, or Wi-Fi phishing. These fake hotspots are easily created using publicly accessible programs and tools. A reputable VPN can help mitigate damage from these attacks as well, but it’s always better not to take the risk. If you travel often, consider purchasing a portable hotspot that uses a cellular data plan, or using data SIM cards for all your devices.

Much more than just acronyms

WEP, WPA, WPA2, and WPA3 mean a lot more than a bunch of similar letters – in some cases, it’s a difference of billions of years minus about 60 seconds.

On more of a now-ish timescale, I hope I’ve taught you something new about the security of your Wi-Fi and how you can improve it!

If you’ve ever said to yourself:

“There’s no one targeting lil ol’ me.”
“I have nothing to hide, anyway.”
“I’m too busy to learn all this stuff. Why can’t someone just give me a simple summary of best practices that I can skim in approximately seven minutes?”

First of all, you might want to stop talking to yourself in public. Secondly, here is a simple summary of best practices that you can skim in approximately seven minutes.

Introducing your three-step starter pack

While there are many different degrees of security, privacy, and anonymity, these three basics are accessible to all:

1. Use a VPN
2. Use multifactor authentication
3. Develop a healthy sense of skepticism

I’ll discuss each of these and help you get started with your security upgrade. But first…

Why is cybersecurity important?

Would you let just anyone walk into your house, or even look through your open doorway from across the street? If not, you might appreciate that the cybersecurity practices we’ll discuss today are not that different from locking your front door.

Cybersecurity isn’t about finding some magic spell that completely secures your online activities – that would be nice, but it’s unrealistic. Good security practices are about employing some thoughtful habits that make your online activities more secure than the next guy, in much the same way as you learned to lock your front door.

Security breaches and incidents happen every day. Most of them occur because an automated scanner cast a wide net and found a person or company with lax security that a hacker could then exploit. Don’t be that guy.

1. Use a VPN

Let’s say you send a lot of mail, but never bother to put your letters in envelopes or even fold them in half. Anyone who bothers to look can read all your dirty secrets (not that you have any).

When you use a Virtual Private Network, or VPN, especially if you often connect to public WiFi, it’s like putting your letters into cryptographically-sealed envelopes and sending them via a special invisible courier service. No one but the intended recipient can read your letters, and no one but you and the courier know to whom the letters are sent.

VPNs prevent others from reading your communications. This may include opportunistic attackers who scan open WiFi, and even your own internet service provider (ISP) who may sell your usage data for advertising dollars.

Choosing a VPN

A few important differentiating factors can help you choose a VPN provider.

1. Is it free? VPNs cost money to operate; if one is offered for free, consider what they might be doing in order to cover their costs. Generally, I recommend avoiding free VPN apps and services; they’ll typically cost you much more than you’ll know. Expect to pay between $5-$10 USD monthly for the service.

2. Where is it based? Understand where your VPN provider is based, and what that country’s laws allow them to do with your data.

3. Do they keep logs? Part of the philosophy of using a VPN is that no one has any business getting into your business when it comes to online activities. When a VPN provider keeps logs of your usage, that defeats the purpose. Instead of your ISP knowing just what you’re up to online, that knowledge is simply transferred to the logging VPN. Look for VPN providers with a strict no-logging policy, or if you’re up for it, roll your own.

2. Use multifactor authentication

Passwords are dead. Computationally, they are a solved problem. Cracking your password is just a matter of time.

Unfortunately, many people still help to speed up the process by using the same compromised passwords for multiple accounts, putting themselves at further risk.

The answer, at least for now, is multifactor authentication (MFA). MFA is made up of three kinds of authentication factors:

1. Something you know, like a pass phrase;
2. Something you have, like a chip pin card or phone; and
3. Something that you are, like your face or fingerprint.

Two or more of these factors are infinitely better than a password alone, especially if your password is on this list.

Multiple authentication factors are now widely supported by account providers and social media sites. If you have the choice, avoid using text messages, or SMS, as a way of receiving authentication codes. SMS authentication leaves you vulnerable to the SIM swap attack - please direct further questions to Jack Dorsey.

Instead, use a One Time Password (OTP) app such as Authy to generate codes on your device. This ensures that you alone, using that particular device, will have the correct authentication code.

You can also use hardware authentication keys such as the YubiKey, but these aren’t yet as widely supported as OTP apps.

3. Develop a healthy sense of skepticism

Social engineering, sometimes SE, is the use of psychological persuasion to get an unwitting target to give up access or information. This can take the form of phishing emails, letters, or phone calls (vishing) as well as far more sophisticated spear-phishing attacks of high-value targets, like company executives.

While some attacks are easier to spot, others use cognitive biases very effectively and are difficult even for security professionals to avoid. No human is immune.

Ultimately, the weakest link in your cybersecurity defense is you. All the VPNs and MFA on the Internet won’t protect you if a scam can trick you into opening the front gates. Always look a Trojan gift horse in the mouth.

Develop the habit of second-guessing things delivered to your virtual doorstep. Email, phone, and messaging scams range in sophistication. Even security professionals can fall for a good scam.

One way to protect yourself is to practice a healthy sense of skepticism. Question communications that ask you to click on links or visit a website, even if they come from someone you know or a company you use.

If you’re not certain that your bank or mother sent this email, pick up the phone and call them. Even if you think you are certain, pick up the phone and double check. You don’t call your mother enough, anyway.

Oh, and if the person on the phone is from your local tax office or the IRS or the CRA and they’re about to freeze your accounts because a case of mistaken identity has resulted in you being criminally charged for not repaying a loan on a 600-foot yacht in Malibu, just hang up. You know better than that. Tax agencies don’t have phones.

A safer Internet

Congratulations! You now have three tools to make your personal cybersecurity better than the next guy’s. If enough people do that, the whole neighborhood (or in this case, the Internet) will benefit as a result.

If this article piqued your interest, you can go further and outsource your security with a password manager and temporary virtual credit cards.

Cheat sheets and other resources

I’ll leave you with a few resources that I’ve enjoyed:

• The Electronic Frontier Foundation website Surveillance Self Defense offers many great guides and how-to’s, such as setting up the encrypted messaging app Signal on your mobile device, and protecting yourself on social media.
• The Cybersecurity and Infrastructure Security Agency (CISA) offers many shareable starter resources.
• Working from home? The National Security Agency Central Security Service has Telework and Mobile Security Guides that discuss best practices for an unprecedented era of remote work.

The Django framework in particular offers your team the opportunity to create an efficient testing practice. Based on the Python standard library unittest, proper tests in Django are fast to write, faster to run, and can offer you a seamless continuous integration solution for taking the pulse of your developing application.

With comprehensive tests, developers have higher confidence when pushing changes. I’ve seen firsthand in my own teams that good tests can boost development velocity as a direct result of a better developer experience.

In this article, I’ll share my own experiences in building useful tests for Django applications, from the basics to the best possible execution.

What to test

Tests are extremely important. Far beyond simply letting you know if a function works, tests can form the basis of your team’s understanding of how your application is intended to work.

Here’s the main goal: if you hit your head and forgot everything about how your application works tomorrow, you should be able to regain most of your understanding by reading and running the tests you write today.

Here are some questions that may be helpful to ask as you decide what to test:

• What is our customer supposed to be able to do?
• What is our customer not supposed to be able to do?
• What should this method, view, or logical flow achieve?
• When, how, or where is this feature supposed to execute?

Tests that make sense for your application can help build developer confidence. With these sensible safeguards in place, developers make improvements more readily, and feel confident introducing innovative solutions to product needs. The result is an application that comes together faster, and features that are shipped often and with confidence.

Where to put tests

If you only have a few tests, you may organize your test files similarly to Django’s default app template by putting them all in a file called tests.py. This straightforward approach is best for smaller applications.

As your application grows, you may like to split your tests into different files, or test modules. One method is to use a directory to organize your files, such as projectroot/app/tests/. The name of each test file within that directory should begin with test, for example, test_models.py.

Besides being aptly named, Django will find these files using built-in test discovery based on the unittest module. All files in your application with names that begin with test will be collected into a test suite.

This convenient test discovery allows you to place test files anywhere that makes sense for your application. As long as they’re correctly named, Django’s test utility can find and run them.

How to document a test

Use docstrings to explain what a test is intended to verify at a high level. For example:

def test_create_user(self):
    """Creating a new user object should also create an associated profile object"""
    # ...

These docstrings help you quickly understand what a test is supposed to be doing. Besides navigating the codebase, this helps to make it obvious when a test doesn’t verify what the docstring says it should.

Docstrings are also shown when the tests are being run, which can be helpful for logging and debugging.

What a test needs to work

Django tests can be quickly set up using data created in the setUpTestData() method. You can use various approaches to create your test data, such as utilizing external files, or even hard-coding silly phrases or the names of your staff. Personally, I much prefer to use a fake-data-generation library, such as faker.

The proper set up of arbitrary testing data can help you ensure that you’re testing your application functionality instead of accidentally testing test data. Because generators like faker add some degree of unexpectedness to your inputs, it can be more representative of real-world use.

Here is an example set up for a test:

from django.test import TestCase
from faker import Faker

from app.models import MyModel, AnotherModel

fake = Faker()


class MyModelTest(TestCase):
    def setUpTestData(cls):
        """Quickly set up data for the whole TestCase"""
        cls.user_first = fake.first_name()
        cls.user_last = fake.last_name()

    def test_create_models(self):
        """Creating a MyModel object should also create AnotherModel object"""
        # In test methods, use the variables created above
        test_object = MyModel.objects.create(
            first_name=self.user_first,
            last_name=self.user_last,
            # ...
        )
        another_model = AnotherModel.objects.get(my_model=test_object)
        self.assertEqual(another_model.first_name, self.user_first)
        # ...

Tests pass or fail based on the outcome of the assertion methods. You can use Python’s unittest methods, and Django’s assertion methods.

For further guidance on writing tests, see Testing in Django.

Best possible execution for running your tests

Django’s test suite is manually run with:

./manage.py test

I rarely run my Django tests this way.

The best, or most efficient, testing practice is one that occurs without you or your developers ever thinking, “I need to run the tests first.” The beauty of Django’s near-effortless test suite set up is that it can be seamlessly run as a part of regular developer activities. This could be in a pre-commit hook, or in a continuous integration or deployment workflow.

I’ve previously written about how to use pre-commit hooks to improve your developer ergonomics and save your team some brainpower. Django’s speedy tests can be run this way, and they become especially efficient if you can run tests in parallel.

Tests that run as part of a CI/CD workflow, for example, on pull requests with GitHub Actions, require no regular effort from your developers to remember to run tests at all. I’m not sure how plainly I can put it – this one’s literally a no-brainer.

Testing your way to a great Django application

Tests are extremely important, and underappreciated. They can catch logical errors in your application. They can help explain and validate how concepts and features of your product actually function. Best of all, tests can boost developer confidence and development velocity as a result.

The best tests are ones that are relevant, help to explain and define your application, and are run continuously without a second thought. I hope I’ve now shown you how testing in Django can help you to achieve these goals for your team!

I’ve been developing with Django for years, and I’ve never been happier with my Django project set up than I am right now. Here’s how I’m making a day of developing with Django the most relaxing and enjoyable development experience possible for myself and my engineering team.

A custom CLI tool for your Django project

Instead of typing:

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
python3 manage.py makemigrations
python3 manage.py migrate
python3 manage.py collectstatic
python3 manage.py runserver

Wouldn’t it be much nicer to type:

make start

…and have all that happen for you? I think so!

We can do that with a self-documenting Makefile! Here’s one I frequently use when developing my Django applications, like ApplyByAPI.com:

VENV := env
BIN := $(VENV)/bin
PYTHON := $(BIN)/python
SHELL := /bin/bash

include .env

.PHONY: help
help: ## Show this help
    @egrep -h '\s##\s' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

.PHONY: venv
venv: ## Make a new virtual environment
    python3 -m venv $(VENV) && source $(BIN)/activate

.PHONY: install
install: venv ## Make venv and install requirements
    $(BIN)/pip install --upgrade -r requirements.txt

freeze: ## Pin current dependencies
    $(BIN)/pip freeze > requirements.txt

migrate: ## Make and run migrations
    $(PYTHON) manage.py makemigrations
    $(PYTHON) manage.py migrate

db-up: ## Pull and start the Docker Postgres container in the background
    docker pull postgres
    docker-compose up -d

db-shell: ## Access the Postgres Docker database interactively with psql. Pass in DBNAME=<name>.
    docker exec -it container_name psql -d $(DBNAME)

.PHONY: test
test: ## Run tests
    $(PYTHON) manage.py test application --verbosity=0 --parallel --failfast

.PHONY: run
run: ## Run the Django server
    $(PYTHON) manage.py runserver

start: install migrate run ## Install requirements, apply migrations, then start development server

You’ll notice the presence of the line include .env above. This ensures make has access to environment variables stored in a file called .env. This allows Make to utilize these variables in its commands, for example, the name of my virtual environment, or to pass in $(DBNAME) to psql.

What’s with that weird “##” comment syntax? A Makefile like this gives you a handy suite of command-line aliases you can check in to your Django project. It’s very useful so long as you’re able to remember what all those aliases are.

The help command above, which runs by default, prints a helpful list of available commands when you run make or make help:

help                 Show this help
venv                 Make a new virtual environment
install              Make venv and install requirements
migrate              Make and run migrations
db-up                Pull and start the Docker Postgres container in the background
db-shell             Access the Postgres Docker database interactively with psql
test                 Run tests
run                  Run the Django server
start                Install requirements, apply migrations, then start development server

All the usual Django commands are covered, and we’ve got a test command that runs our tests with the options we prefer. Brilliant.

You can read my full post about self-documenting Makefiles here, which also includes an example Makefile using pipenv.

Save your brainpower with pre-commit hooks

I previously wrote about some technical ergonomics that can make it a lot easier for teams to develop great software.

One area that’s a no-brainer is using pre-commit hooks to lint code prior to checking it in. This helps to ensure the quality of the code your developers check in, but most importantly, ensures that no one on your team is spending time trying to remember if it should be single or double quotes or where to put a line break.

The confusingly-named pre-commit framework is an otherwise fantastic way to keep hooks (which are not included in cloned repositories) consistent across local environments.

Here is my configuration file, .pre-commit-config.yaml, for my Django projects:

fail_fast: true
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.1.0
    hooks:
      - id: detect-aws-credentials
  - repo: https://github.com/psf/black
    rev: 19.3b0
    hooks:
      - id: black
  - repo: https://github.com/asottile/blacken-docs
    rev: v1.7.0
    hooks:
      - id: blacken-docs
        additional_dependencies: [black==19.3b0]
  - repo: local
    hooks:
      - id: markdownlint
        name: markdownlint
        description: "Lint Markdown files"
        entry: markdownlint '**/*.md' --fix --ignore node_modules --config "./.markdownlint.json"
        language: node
        types: [markdown]

These hooks check for accidental secret commits, format Python files using Black, format Python snippets in Markdown files using blacken-docs, and lint Markdown files as well. To install them, just type pre-commit install.

There are likely even more useful hooks available for your particular use case: see supported hooks to explore.

Useful gitignores

An underappreciated way to improve your team’s daily development experience is to make sure your project uses a well-rounded .gitignore file. It can help prevent files containing secrets from being committed, and can additionally save developers hours of tedium by ensuring you’re never sifting through a git diff of generated files.

To efficiently create a gitignore for Python and Django projects, Toptal’s gitignore.io can be a nice resource for generating a robust .gitignore file.

I still recommend examining the generated results yourself to ensure that ignored files suit your use case, and that nothing you want ignored is commented out.

Continuous testing with GitHub Actions

If your team works on GitHub, setting up a testing process with Actions is low-hanging fruit.

Tests that run in a consistent environment on every pull request can help eliminate “works on my machine” conundrums, as well as ensure no one’s sitting around waiting for a test to run locally.

A hosted CI environment like GitHub Actions can also help when running integration tests that require using managed services resources. You can use encrypted secrets in a repository to grant the Actions runner access to resources in a testing environment, without worrying about creating testing resources and access keys for each of your developers to use.

I’ve written on many occasions about setting up Actions workflows, including using one to run your Makefile, and how to integrate GitHub event data. GitHub even interviewed me about Actions once.

For Django projects, here’s a GitHub Actions workflow that runs tests with a consistent Python version whenever someone opens a pull request in the repository.

name: Run Django tests

on: pull_request

jobs:
  test:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install dependencies
        run: make install
      - name: Run tests
        run: make test

For the installation and test commands, I’ve simply utilized the Makefile that’s been checked in to the repository. A benefit of using your Makefile commands in your CI test workflows is that you only need to keep them updated in one place – your Makefile! No more “why is this working locally but not in CI??!?” headaches.

If you want to step up your security game, you can add Django Security Check as an Action too.

Set up your Django project for success

Want to help keep your development team happy? Set them up for success with these best practices for Django development. Remember, an ounce of brainpower is worth a pound of software!

Django models help humans work with data in a way that makes sense to our brains, and the framework offers plenty of classes you can inherit to help you rapidly develop a robust application from scratch. As for developing on existing Django applications, there’s a feature for that, too. In this article, we’ll cover how to use Django migrations to update your existing models and database.

What’s under the hood

Django migrations are Python files that help you add and change things in your database tables to reflect changes in your Django models. To understand how Django migrations help you work with data, it may be helpful to understand the underlying structures we’re working with.

What’s a database table

If you’ve laid eyes on a spreadsheet before, you’re already most of the way to understanding a database table. In a relational database, for example, a PostgreSQL database, you can expect to see data organized into columns and rows. A relational database table may have a set number of columns and any number of rows.

In Django, each model is its own table. For example, here’s a Django model:

from django.db import models


class Lunch(models.Model):
    left_side = models.CharField(max_length=100, null=True)
    center = models.CharField(max_length=100, null=True)
    right_side = models.CharField(max_length=100, null=True)

Each field is a column, and each row is a Django object instance of that model. Here’s a representation of a database table for the Django model “Lunch” above. In the database, its name would be lunch_table.

The model Lunch has three fields: left_side, center, and right-side. One instance of a Lunch object would have “Fork” for the left_side, a “Plate” for the center, and “Spoon” for the right_side. Django automatically adds an id field if you don’t specify a primary key.

If you wanted to change the name of your Lunch model, you would do so in your models.py code. For example, change “Lunch” to “Dinner,” then run python manage.py makemigrations. You’ll see:

python manage.py makemigrations
Did you rename the backend.Lunch model to Dinner? [y/N] y
Migrations for 'backend':
  backend/migrations/0003_auto_20200922_2331.py
    - Rename model Lunch to Dinner

Django automatically generates the appropriate migration files. The relevant line of the generated migrations file in this case would look like:

migrations.RenameModel(old_name="Lunch", new_name="Dinner"),

This operation would rename our “Lunch” model to “Dinner” while keeping everything else the same. But what if you also wanted to change the structure of the database table itself, its schema, as well as make sure that existing data ends up in the right place on your Dinner table?

Let’s explore how to turn our Lunch model into a Dinner model that looks like this:

from django.db import models


class Dinner(models.Model):
    top_left = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)
    top_right = models.CharField(max_length=100, null=True)
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    bottom_right = models.CharField(max_length=100, null=True)

…with a database table that would look like this:

Manipulating data with Django migrations

Before you begin to manipulate your data, it’s always a good idea to create a backup of your database that you can restore in case something goes wrong. There are various ways to do this depending on the database you’re using. You can typically find instructions by searching for <your database name> and keywords like backup, recovery, or snapshot.

In order to design your migration, it’s helpful to become familiar with the available migration operations. Migrations are run step-by-step, and each operation is some flavor of adding, removing, or altering data. Like a strategic puzzle, it’s important to make model changes one step at a time so that the generated migrations have the correct result.

We’ve already renamed our model successfully. Now, we’ll rename the fields that hold the data we want to retain:

class Dinner(models.Model):
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)

Django is sometimes smart enough to determine the old and new field names correctly. You’ll be asked for confirmation:

python manage.py makemigrations
Did you rename dinner.center to dinner.bottom_center (a CharField)? [y/N] y
Did you rename dinner.left_side to dinner.bottom_left (a CharField)? [y/N] y
Did you rename dinner.right_side to dinner.top_center (a CharField)? [y/N] y
Migrations for 'backend':
  backend/migrations/0004_auto_20200914_2345.py
    - Rename field center on dinner to bottom_center
    - Rename field left_side on dinner to bottom_left
    - Rename field right_side on dinner to top_center

In some cases, you’ll want to try renaming the field and running makemigrations one at a time.

Now that the existing fields have been migrated to their new names, add the remaining fields to the model:

class Dinner(models.Model):
    top_left = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)
    top_right = models.CharField(max_length=100, null=True)
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    bottom_right = models.CharField(max_length=100, null=True)

Running makemigrations again now gives us:

python manage.py makemigrations
Migrations for 'backend':
  backend/migrations/0005_auto_20200914_2351.py
    - Add field bottom_right to dinner
    - Add field top_left to dinner
    - Add field top_right to dinner

You’re done! By generating Django migrations, you’ve successfully set up your dinner_table and moved existing data to its new spot.

Additional complexity

You’ll notice that our Lunch and Dinner models are not very complex. Out of Django’s many model field options, we’re just using CharField. We also set null=True to let Django store empty values as NULL in the database.

Django migrations can handle additional complexity, such as changing field types, and whether a blank or null value is permitted. I keep Django’s model field reference handy as I work with varying types of data and different use cases.

De-mystified migrations

I hope this article has helped you better understand Django migrations and how they work!

Now that you can change models and manipulate existing data in your Django application, be sure to use your powers wisely! Backup your database, research and plan your migrations, and always run tests before working with customer data. By doing so, you have the potential to enable your application to grow – with manageable levels of complexity.

TLS, or Transport Layer Security, refers to a protocol. “Protocol” is a word that means, “the way we’ve agreed to do things around here,” more or less. The “transport layer” part of TLS simply refers to host-to-host communication, such as how a client and a server interact, in the Internet protocol suite model.

The TLS protocol attempts to solve these fundamental problems:

• How do I know you are who you say you are?
• How do I know this message from you hasn’t been tampered with?
• How can we communicate securely?

Here’s how TLS works, explained in plain English. As with many successful interactions, it begins with a handshake.

Getting to know you

The basic process of a TLS handshake involves a client, such as your web browser, and a server, such as one hosting a website, establishing some ground rules for communication. It begins with the client saying hello. Literally. It’s called a ClientHello message.

The ClientHello message tells the server which TLS protocol version and cipher suites it supports. While “cipher suite” sounds like a fancy hotel upgrade, it just refers to a set of algorithms that can be used to secure communications. The server, in a similarly named ServerHello message, chooses the protocol version and cipher suite to use from the choices offered. Other data may also be sent, for example, a session ID if the server supports resuming a previous handshake.

Depending on the cipher suite chosen, the client and server exchange further information in order to establish a shared secret. Often, this process moves the exchange from asymmetric cryptography to symmetric cryptography with varying levels of complexity. Let’s explore these concepts at a general level and see why they matter to TLS.

Asymmetric beginnings

This is asymmetry:

Asymmetric cryptography is one method by which you can perform authentication. When you authenticate yourself, you answer the fundamental question, “How do I know you are who you say you are?”

In an asymmetric cryptographic system, you use a pair of keys in order to achieve authentication. These keys are asymmetric. One key is your public key, which, as you would guess, is public. The other is your private key, which – well, you know.

Typically, during the TLS handshake, the server will provide its public key via its digital certificate, sometimes still called its SSL certificate, though TLS replaces the deprecated Secure Sockets Layer (SSL) protocol. Digital certificates are provided and verified by trusted third parties known as Certificate Authorities (CA), which are a whole other article in themselves.

While anyone may encrypt a message using your public key, only your private key can then decrypt that message. The security of asymmetric cryptography relies only on your private key staying private, hence the asymmetry. It’s also asymmetric in the sense that it’s a one-way trip. Alice can send messages encrypted with your public key to you, but neither of your keys will help you send an encrypted message to Alice.

Symmetric secrets

Asymmetric cryptography also requires more computational resources than symmetric cryptography. Thus when a TLS handshake begins with an asymmetric exchange, the client and server will use this initial communication to establish a shared secret, sometimes called a session key. This key is symmetric, meaning that both parties use the same shared secret and must maintain that secrecy for the encryption to be secure.

By using the initial asymmetric communication to establish a session key, the client and server can rely on the session key being known only to them. For the rest of the session, they’ll both use this same shared key to encrypt and decrypt messages, which speeds up communication.

Secure sessions

A TLS handshake may use asymmetric cryptography or other cipher suites to establish the shared session key. Once the session key is established, the handshaking portion is complete and the session begins.

The session is the duration of encrypted communication between the client and server. During this time, messages are encrypted and decrypted using the session key that only the client and server have. This ensures that communication is secure.

The integrity of exchanged information is maintained by using a checksum. Messages exchanged using session keys have a message authentication code (MAC) attached. This is not the same thing as your device’s MAC address. The MAC is generated and verified using the session key. Because of this, either party can detect if a message has been changed before being received. This solves the fundamental question, “How do I know this message from you hasn’t been tampered with?”

Sessions can end deliberately, due to network disconnection, or from the client staying idle for too long. Once a session ends, it must be re-established via a new handshake or through previously established secrets called session IDs that allow resuming a session.

TLS and you

Let’s recap:

• TLS is a cryptographic protocol for providing secure communication.
• The process of creating a secure connection begins with a handshake.
• The handshake establishes a shared session key that is then used to secure messages and provide message integrity.
• Sessions are temporary, and once ended, must be re-established or resumed.

This is just a surface-level skim of the very complex cryptographic systems that help to keep your communications secure. For more depth on the topic, I recommend exploring cipher suites and the various supported algorithms.

The TLS protocol serves a very important purpose in your everyday life. It helps to secure your emails to family, your online banking activities, and the connection by which you’re reading this article. The HTTPS communication protocol is encrypted using TLS. Every time you see that little lock icon in your URL bar, you’re experiencing firsthand all the concepts you’ve just read about in this article. Now you know the answer to the last question: “How can we communicate securely?”

First, you’ll need to find all the files you want to change. Stringing together what are effectively search queries for find is really only limited by your imagination. Here’s a simple example that finds Python files:

find . -name '*.py'

The -name test searches for a pattern, such as all files ending in .py, but find can do a lot more with other test conditions, including -regex tests. Run find --help to see the multitude of options.

Further tune your search by using grep to get only the files that contain the string you want to change, such as by adding:

grep -le '\<a whale\>'

The -l option gives you just the file names for all files containing a pattern (denoted with -e) that match “a whale”.

Using Vim’s impressive :bufdo lets you run the same command across multiple buffers, interactively working with all of these files without the tedium of opening, saving, and closing each file, one at a time.

Let’s plug your powerful find+grep results into Vim with:

vim `find . -name '*.py' \
-exec grep -le '\<a whale\>' {} \;`

Using backtick-expansion to pass our search to Vim opens up multiple buffers ready to go. (Do :h backtick-expansion in Vim for more.) Now you can apply the Vim command :bufdo to all of these files and perform actions such as interactive search-and-replace:

:bufdo %s/a whale/a bowl of petunias/gce

The g for “global” will change occurrences of the pattern on all lines. The e will omit errors if the pattern is not found. The c option makes this interactive; if you’re feeling confident, you can omit it to make the changes without reviewing each one.

If one of the patterns contains a / character, you can substitute the separator in the above command to make it more readable. Vim will assume the character following the %s is the separator, so for example:

:bufdo %s_a whale_a bowl of peonies/petunias_gce

When you’ve finished going through all the buffers, save all the work you’ve completed with:

:bufdo wq!

Then bask in the glory of your saved time and effort.

Now in beta, GitHub Codespaces provide an online, in-the-browser IDE powered by Visual Studio Code. This lets you use this full-featured IDE, complete with extensions, terminal, Git commands, and all the settings you’re accustomed to, on any machine. You can now bring your development workflow anywhere using a tablet or other browser-based device.

Codespaces is great news for open source contributors, too. Adding a codespace configuration to your project is a great way to invite new folks to easily start contributing.

A new open source contributor or new hire at your organization can quickly fire up a codespace and get hacking on a good first issue with no local environment set up or installations necessary!

We’ve added codespace configuration settings over at the OWASP Web Security Testing Guide (WSTG). Want to take it for a spin? See our open issues.

Configuring Codespaces

You can use Visual Studio Code’s .devcontainer folder to configure a development container for your repository as well.

Many pre-built containers are available – just copy the .devcontainer you need to your repository root. If your repository doesn’t have one, a default base Linux image will be used.

Here’s a reason to remove .vscode from your .gitignore file. Any new codespaces created in your repository will now respect settings found at .vscode/settings.json. This means that your online IDE can have the same Workspace configuration as you have on your local machine. Isn’t that useful!

Making Codespaces personal

For next-level dotfiles personalization, consider committing relevant files from your local dotfiles folder as a public GitHub repository at yourusername/dotfiles.

When you create a new codespace, this brings in your configurations, such as shell aliases and preferences, by creating symlinks to dotfiles in your codespace $HOME. This personalizes all the codespaces you create in your account.

Need some inspiration? Browse my dotfiles repository on GitHub.

Developing in a codespace is a familiar experience for Visual Studio Code users, right down to running an application locally.

Thanks to port forwarding, when I run an application in a codespace terminal, clicking on the resulting localhost URL takes me to the appropriate port as output from my codespace.

When I’m working on this website in my codespace, for example, I run hugo serve then click the provided localhost:1313 link to see a preview of my changes in another browser tab.

Want to stay in sync between devices? There’s an extension for that. You can connect to your codespace from Visual Studio Code on your local machine so you can always pick up right where you left off.

Develop anywhere

Codespaces is a super exciting addition to my GitHub workflow. It allows me to access my full development process pretty much anywhere, using devices like my iPad.

It’ll also make it easier for new open source contributors or new hires at your organization to hit the ground running with a set-up IDE. If you have access to the limited beta, I invite you to spin up a codespace and try contributing to the WSTG, or to an issue on one of my open source projects.

I’m looking forward to general availability and seeing what the open source community will dream up for GitHub Codespaces next!

And yes – codespaces support your favorite Visual Studio Code theme. 😈

Can a Makefile improve your DevOps and keep developers happy? How awesome would it be if a new developer working on your project didn’t start out by copying and pasting commands from your README? What if instead of:

pip3 install pipenv
pipenv shell --python 3.8
pipenv install --dev
npm install
pre-commit install --install-hooks
# look up how to install Framework X...
# copy and paste from README...
npm run serve

… you could just type:

make start

…and then start working?

Making a difference

I use make every day to take the tedium out of common development activities like updating programs, installing dependencies, and testing. To do all this with a Makefile (GNU make), we use Makefile rules and recipes. Similar parallels exist for POSIX flavor make, like Target Rules; here’s a great article on POSIX-compatible Makefiles.

Here’s some examples of things we can make easier (sorry):

update: ## Do apt upgrade and autoremove
    sudo apt update && sudo apt upgrade -y
    sudo apt autoremove -y

env:
    pip3 install pipenv
    pipenv shell --python 3.8

install: ## Install or update dependencies
    pipenv install --dev
    npm install
    pre-commit install --install-hooks

serve: ## Run the local development server
    hugo serve --enableGitInfo --disableFastRender --environment development

initial: update env install serve ## Install tools and start development server

Now we have some command-line aliases that you can check in! Great idea! If you’re wondering what’s up with that weird ## comment syntax, it gets better.

A self-documenting Makefile

Aliases are great, if you remember what they all are and what they do without constantly typing cat Makefile. Naturally, you need a help command:

.PHONY: help
help: ## Show this help
    @egrep -h '\s##\s' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

With a little command-line magic, this egrep command takes the output of MAKEFILE_LIST, sorts it, and uses awk to find strings that follow the ## pattern. It then prints a helpful formatted version of the comments.

We’ll put it at the top of the file so it’s the default target. Now to see all our handy shortcuts and what they do, we just run make, or make help:

help                 Show this help
initial              Install tools and start development server
install              Install or update dependencies
serve                Run the local development server
update               Do apt upgrade and autoremove

Now we have our very own personalized and project-specific CLI tool!

The possibilities for improving your DevOps flow with a self-documenting Makefile are almost endless. You can use one to simplify any workflow and produce some very happy developers.

Please enjoy the (live!) Makefile I use to manage and develop this Hugo site. I hope it inspires you!

SHELL := /bin/bash
.POSIX:
.PHONY: help env install upgrade-hugo serve build start initial

help: ## Show this help
	@egrep -h '\s##\s' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

env:
	pip3 install pipenv

shell: ## Enter the virtual environment
	pipenv shell

install: ## Install or update dependencies
	pipenv install --dev
	pre-commit install --install-hooks
	npm install

HUGO_VERSION:=$(shell curl -s https://api.github.com/repos/gohugoio/hugo/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | cut -c 2-)

upgrade-hugo: ## Get the latest Hugo
	mkdir tmp/ && \
	cd tmp/ && \
	curl -sSL https://github.com/gohugoio/hugo/releases/download/v$(HUGO_VERSION)/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz | tar -xvzf- && \
	sudo mv hugo /usr/local/bin/ && \
	cd .. && \
	rm -rf tmp/
	hugo version

dev: ## Run the local development server
	git submodule update --init --recursive
	hugo serve --enableGitInfo --disableFastRender --environment development

TODAY := $(shell date +%Y%m%d)

post: ## Create a new content/posts/ draft with TITLE=
	hugo new content/posts/$(TODAY)-$(TITLE)/index.md

future: ## Run the local development server in the future
	hugo serve --enableGitInfo --buildFuture --disableFastRender --environment development

build: ## Build site
	hugo --minify --cleanDestinationDir

initial: env install upgrade-hugo serve ## Install tools and start development server

Mt. Fuji is, some folks would say, the cakewalk of mountain climbing. Physically, the hardest portions amount to scrambling over some big boulders; most of it is no more taxing than a hike or climbing a set of stairs. For spiritual reasons, some Japanese folks make the climb at ages upwards of 80 years. There are huts to stop at along the way where you can rent a sleeping bag inside, and buy food and water. Naturally, having done this research and deciding it sounded like a fun outing, I arrived at basecamp in sneakers.

Most of the way up was amazing and thoroughly enjoyable. I saw sights I’d never seen before, like the glow of a city under the sun through a break in the clouds, from above. Walking a path through a cloud was like taking a road into nothingness, with blank grey on all sides that weren’t a mountain. Every time we hit a station marker, I felt pride and accomplishment.

Until it was time to summit.

Most of the people who climb Mt. Fuji wish to reach the summit at sunrise. Some for spiritual reasons, others for Instagram, and for those like myself, it just seemed like the thing to do. Regardless, it was because of these other 5,000 average daily climbers that I found myself in an actual queue that snaked the entire path from the last station hut to the summit – in the pitch black pre-dawn cold. It took hours, for most of which, we stood stock-still, going nowhere. I took to doing calisthenics to stave off frostbite from the cold that threatened my sneaker-shod toes.

We did, eventually, reach the summit, and before sunrise. It remains one of the most beautiful sunrises I’ve seen – a pink-gold light that lit up the peak like breathing life into a painting, and that brought, mercifully, a degree of warmth. I was extremely happy, and felt pride and accomplishment.

Until it was time to descend.

There is a Japanese proverb: “A wise man will climb Mt Fuji once; a fool will climb Mt Fuji twice.” It is my own suspicion that this saying is based entirely on the difficulty of climbing down. The descent is essentially a loosely-packed, dirt and gravel road – on a decline. It is not, I imagine, significantly taxing with proper hiking boots, maybe snow tread, and a couple good spiked hiking poles thrown in. Wearing a pair of flat-soled street shoes, however, I fell. I fell often, and hard, about every three steps, for hours. I tried to take larger steps; it didn’t help. I tried to take smaller steps; that didn’t help, either. I tried cunningly to find a way to surf-slide my way down the mountainside and nearly ended up with a mouthful of dirt. As if literally rubbing salt into my wounds, without the gaiters I hadn’t brought, sand found its way into my shoes. It was without a doubt the most stupefyingly discouraging experience of my life.

On several occasions, more seasoned (smarter? well-prepared?) hikers passed me, a good many of them at least twice my age. I’m hard-pressed to remember another time in my life where I have been so thoroughly shown up by someone who might have been my grandmother, plunking hiking poles into the earth and sauntering past at a steady pace while I picked myself up, elbows scratched and covered in dirt, for the umpteenth time.

Eventually, we reached the bottom. At a tiny basecamp gift shop, I ate a delicious bowl of ramen and the tastiest sponge cake in the shape of a mountain that I’ll likely ever have.

The experience drove home two lessons that have gone on to serve me well: one, that all the good research in the world will not guarantee your experience; and two, that even when faced with a discouraging situation that you can’t seem to think yourself out of and thus the only way is “through,” there may still be something to learn from it, and there may be really good cake at the bottom.

Everything I’d read suggested that Mt. Fuji was the “cakewalk of mountain climbing.” Physically, the hardest portions amounted to scrambling over some big boulders. Most of the climb was no more taxing than hiking or climbing stairs. Japanese folks in their eighties made the journey for spiritual reasons. There were huts along the way for rest, food, and water. Based on this research, I concluded that sneakers would be perfectly adequate.

The ascent was everything I’d been promised. I experienced sights I’d never imagined—cities glowing through breaks in clouds from above, walking through paths of grey nothingness where the trail disappeared into cloud cover. Each station marker brought genuine pride and accomplishment. Even the pre-dawn summit queue with 5,000 other climbers, standing in freezing darkness for hours, felt manageable. We reached the summit before sunrise, and it remains one of the most beautiful moments I’ve experienced.

Then came the descent. That’s where I learned that all the research in the world about reaching goals doesn’t prepare you for what comes after you achieve them.

When Success Becomes the Set Up for Failure

The descent from Mt. Fuji is essentially a loosely-packed dirt and gravel road on a steep decline. With proper hiking boots and trekking poles, it’s probably manageable. In flat-soled street shoes, I fell constantly, and fell hard—every three steps, for hours. I tried to take larger steps; it didn’t help. I tried to take smaller steps; that didn’t help, either. I tried cunningly to find a way to surf-slide my way down the mountainside and nearly ended up with a mouthful of dirt. As if literally rubbing salt into my wounds, without the gaiters I hadn’t brought, sand found its way into my shoes. It was without a doubt the most stupefyingly discouraging experience of my life.

As I picked myself up repeatedly, covered in dirt with scratched elbows, seasoned hikers passed me with ease. Many of them could have been my grandparents, using proper equipment and technique to descend at a steady pace while I struggled and stopped to pour tiny rocks out of my sneakers. The contrast was humbling and instructive.

This experience taught me something crucial about leadership that I’ve applied countless times since: the skills and preparation that get you to success are often different from the skills required to maintain or scale that success. The descent is frequently harder than the climb, and most people don’t prepare for it adequately.

The Post-Achievement Challenge

In business and team leadership, I’ve watched this pattern repeat consistently. The energy, skills, and resources required to achieve a goal are usually well-understood and planned for. But the challenges that come after success—maintaining market position, scaling team culture, or managing the operational complexity of growth—often catch leaders unprepared.

I’ve seen teams that executed brilliant product launches struggle with customer support and maintenance. Startups that successfully raised funding stumble when it comes to executing on their promises to investors. Engineering teams that built innovative solutions fail to create sustainable systems for maintaining and scaling those solutions. The problem isn’t lack of capability—it’s that the descent requires different preparation and different skills than the ascent. What gets you to the summit (innovation, speed, breakthrough thinking) often isn’t what gets you safely back to basecamp (consistency, processes, systematic execution).

Learning from Those Who’ve Made the Journey

Watching those experienced hikers pass me on Mt. Fuji was initially frustrating, but it became one of the most valuable parts of the experience. They had proper equipment, understood the terrain, and moved with confidence that came from experience. Most importantly, they had prepared specifically for the descent, not just the climb.

In leadership roles, I’ve learned to actively seek out people who’ve successfully navigated the “descent” phase of challenges I’m facing. Entrepreneurs who’ve managed hypergrowth. Product managers who’ve maintained market leadership over multiple years. Engineering leaders who’ve scaled teams from ten to fifty people, or CEOs who’ve scaled companies from fifty to five hundred.

These conversations can reveal patterns you may not have discovered on your own. Successful scaling requires different organizational structures than startup growth. Maintaining team culture during rapid hiring requires intentional systems that don’t emerge naturally. Sustaining innovation while managing operational complexity demands new kinds of leadership skills.

People who’ve successfully managed the descent often have hard-won wisdom about preparation and technique that isn’t captured in most “how to reach the summit” advice.

Building Skills Before You Need Them

The most effective leaders I know prepare for post-success challenges while they’re still climbing toward their initial goals. They think systematically about what will be required to maintain and scale whatever they’re building, not just achieve it.

This means building operational capabilities alongside product capabilities. Developing team management skills in individual contributors. Creating sustainable processes while you’re still in startup mode. Planning for the maintenance and evolution of systems as part of their initial implementation.

It also means recognizing that the mindset and skills that drive breakthrough achievements—risk-taking, speed, creative problem-solving—need to be balanced with different capabilities like consistency, systematic thinking, and process optimization. I’ve learned to explicitly ask: “What will success look like, and what challenges will that create?” This question reveals preparation gaps that aren’t obvious when you’re focused entirely on reaching your goals.

When You Find Yourself Unprepared

Despite best intentions, you’ll sometimes find yourself in descent mode without proper preparation—leading a team through unexpected growth, managing a product that succeeded beyond projections, or scaling systems that weren’t designed for current loads. The Mt. Fuji experience taught me how to navigate these situations effectively.

First, acknowledge the reality of your situation without wasting energy on regret about preparation gaps. You can’t change what you didn’t know or plan for previously, but you can adapt your approach based on current conditions. Take the time to solidify new goals in writing, then evaluate whether your efforts are serving them effectively.

Second, focus on learning from people who are managing similar challenges successfully. This isn’t the time for pride or trying to figure everything out independently. The hikers who passed me weren’t showing off—they had practical knowledge that could help. Conversations you have with others who came before you can save you from a lot of stumbles.

Third, lift your gaze. While the ascent phase requires day-to-day tactical thinking, the descent phase requires a strategic longer-term outlook. Implementing systems and culture that support continued success will require patience, persistence, and often a completely different pace than what got you to the summit. Expecting it to be as expedient as the climb leads to frustration and poor decision-making.

Finding Meaning in the Difficult Parts

Eventually, I reached the bottom of Mt. Fuji, exhausted and humbled but intact. At a tiny basecamp shop, I ate the most delicious bowl of ramen and the tastiest mountain-shaped sponge cake I’ll likely ever have.

Even when you’re unprepared and struggling, there’s value in the journey itself. The descent taught me lessons about preparation, humility, and persistence that I’ve applied to all sorts of challenges for years since.

Preparing for Your Next Descent

There is a Japanese proverb: “A wise man will climb Mt Fuji once; a fool will climb Mt Fuji twice.” I suspect this wisdom is based entirely on the difficulty of the descent. But in leadership, you don’t get to choose how many times you’ll face descent challenges—they’re inevitable parts of any significant journey.

The key is recognizing that achieving your goals is often just the beginning of a different kind of challenge. Success creates new problems that require different skills, different preparation, and different mindsets than what got you there initially.

Whether you’re building teams, scaling products, or managing organizational growth, prepare for the descent while you’re planning the climb. Study what happens after success. Learn from people who’ve navigated similar transitions. Build operational capabilities alongside innovative ones.

Most importantly, remember that the descent is still part of the journey, not a failure of the ascent. The challenges that come with success are signs that you’ve accomplished something meaningful. Navigate them with patience, preparation, and the understanding that getting back to basecamp safely can be an even more important achievement than reaching the summit.

At front-and-center on your GitHub profile, your README is a great opportunity to let folks know what you’re about, what you find important, and to showcase some highlights of your work. You might like to show off your latest repositories, tweet, or blog post. Keeping it up to date doesn’t have to be a pain either, thanks to continuous delivery tools like GitHub Actions.

My current README refreshes itself daily with a link to my latest blog post. Here’s how I’m creating a self-updating README.md with Go and GitHub actions.

Reading and writing files with Go

I’ve been writing a lot of Python lately, but for some things I really like using Go. You could say it’s my go-to language for just-for-func projects. Sorry. Couldn’t stop myself.

To create my README.md, I’m going to get some static content from an existing file, mash it together with some new dynamic content that we’ll generate with Go, then bake the whole thing at 400 degrees until something awesome comes out.

Here’s how we read in a file called static.md and put it in string form:

// Unwrap Markdown content
content, err := ioutil.ReadFile("static.md")
if err != nil {
    log.Fatalf("cannot read file: %v", err)
    return err
}

// Make it a string
stringyContent := string(content)

The possibilities for your dynamic content are only limited by your imagination! Here, I’ll use the github.com/mmcdole/gofeed package to read the RSS feed from my blog and get the newest post.

fp := gofeed.NewParser()
feed, err := fp.ParseURL("https://victoria.dev/index.xml")
if err != nil {
    log.Fatalf("error getting feed: %v", err)
}
// Get the freshest item
rssItem := feed.Items[0]

To join these bits together and produce stringy goodness, we use fmt.Sprintf() to create a formatted string.

// Whisk together static and dynamic content until stiff peaks form
blog := "Read my latest blog post: **[" + rssItem.Title + "](" + rssItem.Link + ")**"
data := fmt.Sprintf("%s\n%s\n", stringyContent, blog)

Then to create a new file from this mix, we use os.Create(). There are more things to know about deferring file.Close(), but we don’t need to get into those details here. We’ll add file.Sync() to ensure our README gets written.

// Prepare file with a light coating of os
file, err := os.Create("README.md")
if err != nil {
    return err
}
defer file.Close()

// Bake at n bytes per second until golden brown
_, err = io.WriteString(file, data)
if err != nil {
    return err
}
return file.Sync()

View the full code here in my README repository.

Mmmm, doesn’t that smell good? 🍪 Let’s make this happen on the daily with a GitHub Action.

Running your Go program on a schedule with Actions

You can create a GitHub Action workflow that triggers both on a push to your master branch as well as on a daily schedule. Here’s a slice of the .github/workflows/update.yaml that defines this:

on:
  push:
    branches:
      - master
  schedule:
    - cron: '0 11 * * *'

To run the Go program that rebuilds our README, we first need a copy of our files. We use actions/checkout for that:

steps:
    - name: 🍽️ Get working copy
      uses: actions/checkout@master
      with:
        fetch-depth: 1

This step runs our Go program:

- name: 🍳 Shake & bake README
  run: |
    cd ${GITHUB_WORKSPACE}/update/
    go run main.go    

Finally, we push the updated files back to our repository. Learn more about the variables shown at Using variables and secrets in a workflow.

- name: 🚀 Deploy
  run: |
    git config user.name "${GITHUB_ACTOR}"
    git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"
    git add .
    git commit -am "Update dynamic content"
    git push --all -f https://${{ secrets.GITHUB_TOKEN }}@github.com/${GITHUB_REPOSITORY}.git    

View the full code for this Action workflow here in my README repository.

Go forth and auto-update your README

Congratulations and welcome to the cool kids’ club! You now know how to build an auto-updating GitHub profile README. You may now go forth and add all sorts of neat dynamic elements to your page – just go easy on the GIFs, okay?

• Class-based versus function-based views
• Django models
• Retrieving objects with queries

Understanding these main features are the building blocks for maximizing development efficiency with Django. They’ll build the foundation for you to test efficiently and create an awesome development experience for your engineers. Let’s look at how these tools let you create a performant Django application that’s pleasant to build and maintain.

Class-based versus function-based views

Remember that Django is all Python under the hood. When it comes to views, you’ve got two choices: view functions (sometimes called “function-based views”), or class-based views.

Years ago when I first built ApplyByAPI, it was initially composed entirely of function-based views. These offer granular control, and are good for implementing complex logic; just as in a Python function, you have complete control (for better or worse) over what the view does. With great control comes great responsibility, and function-based views can be a little tedious to use. You’re responsible for writing all the necessary methods for the view to work - this is what allows you to completely tailor your application.

In the case of ApplyByAPI, there were only a sparse few places where that level of tailored functionality was really necessary. Everywhere else, function-based views began making my life harder. Writing what is essentially a custom view for run-of-the-mill operations like displaying data on a list page became tedious, repetitive, and error-prone.

With function-based views, you’ll need figure out which Django methods to implement in order to handle requests and pass data to views. Unit testing can take some work to write. In short, the granular control that function-based views offer also requires some granular tedium to properly implement.

I ended up holding back ApplyByAPI while I refactored the majority of the views into class-based views. This was not a small amount of work and refactoring, but when it was done, I had a bunch of tiny views that made a huge difference. I mean, just look at this one:

class ApplicationsList(ListView):
    model = Application
    template_name = "applications.html"

It’s three lines. My developer ergonomics, and my life, got a lot easier.

You may think of class-based views as templates that cover most of the functionality any app needs. There are views for displaying lists of things, for viewing a thing in detail, and editing views for performing CRUD (Create, Read, Update, Delete) operations. Because implementing one of these generic views takes only a few lines of code, my application logic became dramatically succinct. This gave me less repeated code, fewer places for something to go wrong, and a more manageable application in general.

Class-based views are fast to implement and use. The built-in class-based generic views may require less work to test, since you don’t need to write tests for the base view Django provides. (Django does its own tests for that; no need for your app to double-check.) To tweak a generic view to your needs, you can subclass a generic view and override attributes or methods. In my case, since I only needed to write tests for any customizations I added, my test files became dramatically shorter, as did the time and resources it took to run them.

When you’re weighing the choice between function-based or class-based views, consider the amount of customization the view needs, and the future work that will be necessary to test and maintain it. If the logic is common, you may be able to hit the ground running with a generic class-based view. If you need sufficient granularity that re-writing a base view’s methods would make it overly complicated, consider a function-based view instead.

Django models

Models organize your Django application’s central concepts to help make them flexible, robust, and easy to work with. If used wisely, models are a powerful way to collate your data into a definitive source of truth.

Like views, Django provides some built-in model types for the convenience of implementing basic authentication, including the User and Permission models. For everything else, you can create a model that reflects your concept by inheriting from a parent Model class.

class StaffMember(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    company = models.OneToOneField(Company, on_delete=models.CASCADE)

    def __str__(self):
        return self.company.name + " - " + self.user.email

When you create a custom model in Django, you subclass Django’s Model class and take advantage of all its power. Each model you create generally maps to a database table. Each attribute is a database field. This gives you the ability to create objects to work with that humans can better understand.

You can make a model useful to you by defining its fields. Many built-in field types are conveniently provided. These help Django figure out the data type, the HTML widget to use when rendering a form, and even form validation requirements. If you need to, you can write custom model fields.

Database relationships can be defined using a ForeignKey field (many-to-one), or a ManyToManyField (give you three guesses). If those don’t suffice, there’s also a OneToOneField. Together, these allow you to define relations between your models with levels of complexity limited only by your imagination. (Depending on the imagination you have, this may or may not be an advantage.)

Retrieving objects with queries

Use your model’s Manager (objects by default) to construct a QuerySet. This is a representation of objects in your database that you can refine, using methods, to retrieve specific subsets. All available methods are in the QuerySet API and can be chained together for even more fun.

Post.objects.filter(
    type="new"
).exclude(
    title__startswith="Blockchain"
)

Some methods return new QuerySets, such as filter(), or exclude(). Chaining these can give you powerful queries without affecting performance, as QuerySets aren’t fetched from the database until they are evaluated. Methods that evaluate a QuerySet include get(), count(), len(), list(), or bool().

Iterating over a QuerySet also evaluates it, so avoid doing so where possible to improve query performance. For instance, if you just want to know if an object is present, you can use exists() to avoid iterating over database objects.

Use get() in cases where you want to retrieve a specific object. This method raises MultipleObjectsReturned if something unexpected happens, as well as the DoesNotExist exception, if, take a guess.

If you’d like to get an object that may not exist in the context of a user’s request, use the convenient get_object_or_404() or get_list_or_404() which raises Http404 instead of DoesNotExist. These helpful shortcuts are suited to just this purpose. To create an object that doesn’t exist, there’s also the convenient get_or_create().

Efficient essentials

You’ve now got a handle on these three essential tools for building your efficient Django application – congratulations! You can make Django work even better for you by learning about manipulating data with migrations, testing effectively, and setting up your team’s Django development for maximum happiness.

If you’re going to build on GitHub, you may like to set up my django-security-check GitHub Action. In the meantime, you’re well on your way to building a beautiful software project.

What is the name of your Action? Please include a link too.

Among the several Actions I’ve built, I have two current favorites. One is hugo-remote, which lets you continuously deploy a Hugo static site from a private source repository to a public GitHub Pages repository. This keeps the contents of the source repository private, such as your unreleased drafts, while still allowing you to have a public open source site using GitHub Pages.

The second is django-security-check. It’s an effortless way to continuously check that your production Django application is free from a variety of security misconfigurations. You can think of it as your little CI/CD helper for busy projects – a security linter!

Tell us a little bit more about yourself—how did you get started in software tools?

When I was a kid, I spent several summer vacations coding a huge medieval fantasy world MUD (Multi-User Dungeon, like a multiplayer role-playing game) written in LPC, with friends. It was entirely text-based, and built and played via Telnet. I fell in love with the terminal and learned a lot about object-oriented programming and prototype-based programming early on.

I became a freelance developer and had the privilege of working on a wide variety of client projects. Realizing the difficulty that companies have with hiring experienced developers, I built ApplyByAPI.com to help. As you might imagine, it allows candidates to apply for jobs via API, instead of emailing a resume. It’s based on the Django framework, so in the process, I learned even more about building reusable units of software.

When I became a co-author and a core maintainer for the Open Web Application Security Project (OWASP) Web Security Testing Guide (WSTG), I gained an even broader appreciation for how a prototype-based, repeatable approach can help build secure web applications. Organizations worldwide consider the WSTG the foremost open source resource for testing the security of web applications. We’ve applied this thinking via the use of GitHub Actions in our repository – I’ll tell you more about that later.

Whether I’m creating an open source tool or leading a development team, my childhood experience still informs how I think about programming today. I strive to create repeatable units of software like GitHub Actions – only now, I make them for large enterprises in the real world!

What is the story behind your built GitHub Action? (Why did you build this?)

Developers take on a lot of responsibility when it comes to building secure applications these days. I’m a full-time senior software developer at a cybersecurity company. I’ve found that I’m maximally productive when I create systems and processes that help myself and my team make desired outcomes inevitable. So I spend my free time building tools that make it easy for other developers to build secure software as well. My Actions help to automate contained, repeatable units of work that can make a big difference in a developer’s day.

Do you have future plans for this or other Actions?

Yes! I’m always finding ways for tools like GitHub Actions to boost the velocity of technical teams, whether at work or in my open source projects. Remember the Open Web Application Security Project? In the work I’ve lead with OWASP, I’ve championed the effort to increase automation using GitHub Actions to maintain quality, securely deploy new versions to the web, and even build PDFs of the WSTG. We’re constantly looking into new ways that GitHub Actions can make our lives easier and our readers’ projects more secure.

What has been your favorite feature of GitHub Actions?

I like that I can build an Action using familiar and portable technologies, like Docker. Actions are easy for collaborators to work on too, since in the case of a Dockerized Action, you can use any language your team is comfortable with. This is especially useful in large organizations with polyglot teams and environments. There aren’t any complicated dependencies for running these portable tasks, and you don’t need to learn any special frameworks to get started.

One of my first blog posts about GitHub Actions even describes how I used an Action to run a Makefile! This is especially useful for large legacy applications that want to modernize their pipeline by using GitHub Actions.

What are the biggest challenges you’ve faced while building your GitHub Action?

The largest challenge of GitHub Actions isn’t really in GitHub Actions, but in the transition of legacy software and company culture.

Migrating legacy software is always challenging, particularly with large legacy applications. Moving to modern CI/CD processes requires changes at the software level, team level, and even a shift in thinking when it comes to individual developers. It can help to have a tool like GitHub Actions, which is at once seamlessly modern and familiar, when transitioning legacy code to a modern pipeline.

Anything else you would like to share about your experience? Any stories or lessons learned through building your Action?

I’m happiest when I’m solving a challenge that makes developing secure software less challenging in the future, both for myself and for the technology organization I’m leading. With tools like GitHub Actions, a lot of mental overhead can be offloaded to automatic processes – like getting a whole other brain, for free! This can massively help organizations that are ready to scale up their development output.

In the realm of cybersecurity, not only does creating portable and reusable software make developers’ lives easier, it helps to make whole workflows repeatable, which in turn makes software development processes more secure. With smart processes in place, technical teams are happier. As an inevitable result, they’ll build better software for customers, too.

When I first began to program full time, I found myself constantly tired from the mental exertion. Programming is hard! Thankfully, you can take some solace in knowing it gets easier with practice, and with a great supporting cast. Some very nice folks who preceded us both came up with tools to make the difficult bits of communicating with computers much easier on our poor human meatbrains.

I invite you to explore these super helpful technical tools. They’ll improve your development set up and alleviate much of the mental stress of programming. You soon won’t believe you could have done without them.

Not your average syntax highlighting

If you’re still working with syntax highlighting that just picks out variable and class names for you, that’s cute. Time to turn it up a notch.

In all seriousness, syntax highlighting can make it much easier to find what you’re looking for on your screen: the current line, where your current code block starts and ends, or the absolute game-changing which-bracket-set-am-I-in highlight. I primarily use Visual Studio Code, but similar extensions can be found for the major text editors.

The theme pictured in Visual Studio Code above is Kabukichō. I made it.

Use Git hooks

I previously brought you an interactive pre-commit checklist in the style of infomercials that’s both fun and useful for reinforcing the quality of your commits. But that’s not all!

Git hooks are scripts that run automatically at pre-determined points in your workflow. Use them well, and you can save a ton of brainpower. A pre-commit hook remembers to do things like lint and format code, and even runs local tests for you before you indelibly push something embarrassing. Hooks can be a little annoying to share (the .git/hooks directory isn’t tracked and thus omitted when you clone or fork a repository) but there’s a framework for that: the confusingly-named pre-commit framework, which allows you to create a sharable configuration file of Git hook plugins, not just for pre-commit.

I spend a majority of my time these days coding in Python, so here is my current .pre-commit-config.yaml:

fail_fast: true
repos:
  - repo: https://github.com/DavidAnson/markdownlint-cli2
    rev: v0.1.3
    hooks:
    - id: markdownlint-cli2
      name: markdownlint-cli2
      description: "Checks the style of Markdown/CommonMark files."
      entry: markdownlint-cli2
      language: node
      types: [markdown]
      minimum_pre_commit_version: 0.15.0

There are tons of supported hooks to explore.

Use a type system

If you write in languages like Python and JavaScript, get yourself an early birthday present and start using a static type system. Not only will this help improve the way you think about code, it can help make type errors clear before running a single line.

For Python, I like using mypy for static type checking. You can set it up as a pre-commit hook (see above) and it’s supported in Visual Studio Code too.

TypeScript is my preferred way to write JavaScript. You can run the compiler on the command line using Node.js (see instructions in the repo), it works pretty well with Visual Studio Code out of the box, and of course there are multiple options for extension integrations.

Quit unnecessarily beating up your meatbrain

I mean, you wouldn’t stand on your head all day to do your work. It would be rather inconvenient to read things upside down all the time (at least until your brain adjusted), and in any case you’d likely get uncomfortably congested in short order. Working without taking advantage of the technical ergonomic tools I’ve given you today is a little like unnecessary inversion - why would you, if you don’t have to?

Besides existing without you having to lift a finger, open source tools and software have some distinct advantages. Especially in the case of well-established projects, it’s highly likely that someone else has already worked out all the most annoying bugs for you. Thanks to the ease with which users can view and modify source code, it’s also more likely that a program has been tinkered with, improved, and secured over time. When many developers contribute, they bring their own unique expertise and experiences. This can result in a product far more robust and capable than one a single developer can produce.

Of course, being as varied as the people who build them, not all open source projects are created equal, nor maintained to be equally secure. There are many factors that affect a project’s suitability for your use case. Here are a few general considerations that make a good starting point when choosing an open source project.

How to choose an open source project

As its most basic requirements, a good software project is reliable, easy to understand, and has up-to-date components and security. There are several indicators that can help you make an educated guess about whether an open source project satisfies these criteria.

Who’s using it

Taken in context, the number of people already using an open source project may be indicative of how good it is. If a project has a hundred users, for instance, it stands to reason that someone has tried to use it at least a hundred times before you found it. Thus by the ancient customs of “I don’t know what’s in that cave, you go first,” it’s more likely to be fine.

You can draw conclusions about a project’s user base by looking at available statistics. Depending on your platform, these may include the number of downloads, reviews, issues or tickets, comments, contributions, forks, or “stars,” whatever those are.

Evaluate social statistics on platforms like GitHub with a grain of salt. They can help you determine how popular a project may be, but only in the same way that restaurant review apps can help you figure out if you should eat at Foo’s Grill & Bar. Depending on where Foo’s Grill & Bar is, when it opened, and how likely people are to be near it when the invariable steak craving should call, having twenty-six reviews may be a good sign or a terrible one. While you would not expect a project that addresses a very obscure use case or technology to have hundreds of users, having a few active users is, in such a case, just as confidence-inspiring.

External validation can also be useful. For example, packages that are included in a Linux operating system distribution (distro) must conform to stringent standards and undergo vetting. Choosing software that is included in a distro’s default repositories can mean it’s more likely to be secure.

Perhaps one of the best indications to look for is whether a project’s development team is using their own project. Look for issues, discussions, or blog posts that show that the project’s creators and maintainers are using what they’ve built themselves. Commonly referred to as “eating your own dog food,” or “dogfooding,” it’s an indicator that the project is most likely to be well-maintained by its developers.

Who’s building it

The main enemy of good open source software is usually a lack of interest. The parties involved in an open source project can make the difference between a flash-in-the-pan library and a respected long-term utility. Multiple committed maintainers, even making contributions in their spare time, have a much higher success rate of sustaining a project and generating interest.

Projects with healthy interest are usually supported by, and in turn cultivate, a community of contributors and users. New contributors may be actively welcomed, clear guides are available explaining how to help, and project maintainers are available and approachable when people have inevitable questions. Some communities even have chat rooms or forums where people can interact outside of contributions. Active communities help sustain project interest, relevance, and its ensuing quality.

In a less organic fashion, a project can also be sustained through organizations that sponsor it. Governments and companies with financial interest are open source patrons too, and a project that enjoys public sector use or financial backing has added incentive to remain relevant and useful.

How alive is it

The recency and frequency of an open source project’s activity is perhaps the best indicator of how much attention is likely paid to its security. Look at releases, commit history, changelogs, or documentation revisions to determine if a project is active. As projects vary in size and scope, here are some general things to look for.

Maintaining security is an ongoing endeavor that requires regular monitoring and updates, especially for projects with third-party components. These may be libraries or any part of the project that relies on something outside itself, such as a payment gateway integration. An inactive project is more likely to have outdated code or use outdated versions of components. For a more concrete determination, you can research a project’s third-party components and compare their most recent patches or updates with the project’s last updates.

Projects without third-party components may have no outside updates to apply. In these cases, you can use recent activity and release notes to determine how committed a project’s maintainers may be. Generally, active projects should show updates within the last months, with a notable release within the last year. This can be a good indication of whether the project is using an up-to-date version of its language or framework.

You can also judge how active a project may be by looking at the project maintainers themselves. Active maintainers quickly respond to feedback or new issues, even if it’s just to say, “We’re on it.” If the project has a community, its maintainers are a part of it. They may have a dedicated website or write regular blogs. They may offer ways to contact them directly and privately, especially to raise security concerns.

Can you understand it

Having documentation is a baseline requirement for a project that’s intended for anyone but its creator to use. Good open source projects have documentation that is easy to follow, honest, and thorough.

Having well-written documentation is one way a project can stand out and demonstrate the thoughtfulness and dedication of its maintainers. A “Getting Started” section may detail all the requirements and initial set up for running the project. An accurate list of topics in the documentation enables users to quickly find the information they need. A clear license statement leaves no doubt as to how the project can be used, and for what purposes. These are characteristic aspects of documentation that serves its users.

A project that is following sound coding practices likely has code that is as readable as its documentation. Code that is easy to read lends itself to being understood. Generally, it has clearly defined and appropriately-named functions and variables, a logical flow, and apparent purpose. Readable code is easier to fix, secure, and build upon.

How compatible is it

A few factors will determine how compatible a project is with your goals. These are objective qualities, and can be determined by looking at a project’s repository files. They include:

• Code language
• Specific technologies or frameworks
• License compatibility

Compatibility doesn’t necessarily mean a direct match. Different code languages can interact with each other, as can various technologies and frameworks. You should carefully read a project’s license to understand if it permits usage for your goal, or if it is compatible with a license you would like to use.

Ultimately, a project that satisfies all these criteria may still not quite suit your use case. Part of the beauty of open source software, however, is that you may still benefit from it by making alterations that better suit your usage. If those alterations make the project better for everyone, you can pay it back and pay it forward by contributing your work to the project.

Proper care and feeding of an open source project

Once you adopt an open source project, a little attention is required to make sure it continues to be a boon to your goals. While its maintainers will look after the upstream project files, you alone are responsible for your own copy. Like all software, your open source project must be well-maintained in order to remain as secure and useful as possible.

Have a system that provides you with notifications when updates for your software are made available. Update software promptly, treating each patch as if it were vital to security; it may well be. Keep in mind that open source project creators and maintainers are, in most cases, acting only out of the goodness of their own hearts. If you’ve got a particularly awesome one, its developers may make updates and security patches available on a regular basis. It’s up to you to keep tabs on updates and promptly apply them.

As with most things in software, keeping your open source additions modular can come in handy. You might use git submodules, branches, or environments to isolate your additions. This can make it easier to apply updates or pinpoint the source of any bugs that arise.

So although an open source project may cost no money, caveat emptor, which means, “Jimmy, if we get you a puppy, it’s your responsibility to take care of it.”

Of course, spending undue hours building a back end like Fort Knox may not be necessary for your application, either. Being an advocate for security doesn’t mean always wearing your tinfoil hat (although you do look dashing in it) but does mean building in an appropriate amount of security.

How much security is appropriate? The answer, frustratingly, is, “it depends.” The right amount of security for your application depends on who’s using it, what it does, and most importantly, what undesirable things it could be made to do. It takes some analysis to make decisions about the kinds of risks your application faces and how you’ll prepare to handle them. Okay, now’s a good time to don your tinfoil hat. Let’s imagine the worst.

Threat modeling: what’s the worst that could happen

A threat model is a stuffy term for the result of trying to imagine the worst things that could happen to an application. Using your imagination to assess risks (fittingly called risk assessment) is a conveniently non-destructive method for finding ways an application can be attacked. You won’t need any tools; just an understanding of how the application might work, and a little imagination. You’ll want to record your results with pen and paper. For the younger folks, that means the notes app on your phone.

A few different methodologies for application risk assessment can be found in the software world, including the in-depth NIST Special Publication 800-30. Each method’s framework has specific steps and output, and will go into various levels of detail when it comes to defining threats. If following a framework, first choose the one you’re most likely to complete. You can always add more depth and detail from there.

Even informal risk assessments are beneficial. Typically taking the form of a set of questions, they may be oriented around possible threats, the impact to assets, or ways a vulnerability could be exploited. Here are some examples of questions addressing each orientation:

• What kind of adversary would want to break my app? What would they be after?
• If the control of x fell into the wrong hands, what could an attacker do with it?
• Where could a x vulnerability occur in my app?

A basic threat model explains the technical, business, and human considerations for each risk. It will typically detail:

• The vulnerabilities or components that can cause the risk
• The impact that a successful execution of the risk would have on the application
• The consequences for the application’s users or organization

The result of a risk assessment exercise is your threat model; in other words, a list of things you would very much like not to occur. It is usually sorted in a hierarchy of risks, from the worst to the mildest. The worst risks have the most negative impact, and are most important to protect against. The mildest risks are the most acceptable - while still an undesirable outcome, they have the least negative impact on the application and users.

You can use this resulting hierarchy as a guide to determine how much of your cybersecurity efforts to apply to each risk area. An appropriate amount of security for your application will eliminate (where possible) or mitigate the worst risks.

Pushing left

Although it sounds like a dance move meme, pushing left refers instead to building in as much of your planned security as possible in the early stages of software development.

Building software is a lot like building a treehouse, just without the pleasant fresh air. You start with the basic supporting components, such as attaching a platform to a tree. Then comes the framing, walls, and roof, and finally, your rustic-modern Instagram-worthy wall hangings and deer bust.

The further along in the build process you are, the harder and more costly it becomes to make changes to a component that you’ve already installed. If you discover a problem with the walls only after the roof is put in place, you may need to change or remove the roof in order to fix it. Similar parallels can be drawn for software components, only without similar ease in detangling the attached parts.

In the case of a treehouse, it’s rather impossible to start with decorations or even a roof, since you can’t really suspend them in midair. In the case of software development, it is, unfortunately, possible to build many top-layer components and abstractions without a sufficient supporting architecture. A push-left approach views each additional layer as adding cost and complication. Pushing left means attempting to mitigate security risks as much as possible at each development stage before proceeding to the next.

Building bottom-to-top

By considering your threat model in the early stages of developing your application, you reduce the chances of necessitating a costly remodel later on. You can make choices about architecture, components, and code that support the main security goals of your particular application.

While it’s not possible to foresee all the functionality your application may one day need to support, it is possible to prepare a solid foundation that allows additional functionality to be added more securely. Building in appropriate security from the bottom to the top will help make mitigating security risks much easier in the future.

In typical developer rationality, there was clearly only one option. I decided to create the same theme in both frameworks, and to give you, dear reader, a side-by-side comparison.

This post isn’t a comprehensive theme-building guide, but intended to familiarize you with the process of building a theme in either generator. Here’s what we’ll cover:

• How theme files are organized
• Where to put content
• How templating works
• Creating a top-level menu with the pages object
• Creating a menu with nested links from a data list
• Putting the template together
• Create a stylesheet
  • Sass and CSS in Jekyll
  • Sass and Hugo Pipes in Hugo
• Configure and deploy to GitHub Pages
  • Configure Jekyll
  • Configure Hugo
  • Deploy to GitHub Pages
• Showtime
• Wait who won

Here’s a crappy wireframe of the theme I’m going to create.

If you’re planning to build-along, it may be helpful to serve the theme locally as you build it; both generators offer this functionality. For Jekyll, run jekyll serve, and for Hugo, hugo serve.

There are two main elements: the main content area, and the all-important sidebar menu. To create them, you’ll need template files that tell the site generator how to generate the HTML page. To organize theme template files in a sensible way, you first need to know what directory structure the site generator expects.

How theme files are organized

Jekyll supports gem-based themes, which users can install like any other Ruby gems. This method hides theme files in the gem, so for the purposes of this comparison, we aren’t using gem-based themes.

When you run jekyll new-theme <name>, Jekyll will scaffold a new theme for you. Here’s what those files look like:

.
├── assets
├── Gemfile
├── _includes
├── _layouts
│   ├── default.html
│   ├── page.html
│   └── post.html
├── LICENSE.txt
├── README.md
├── _sass
└── <name>.gemspec

The directory names are appropriately descriptive. The _includes directory is for small bits of code that you reuse in different places, in much the same way you’d put butter on everything. (Just me?) The _layouts directory contains templates for different types of pages on your site. The _sass folder is for Sass files used to build your site’s stylesheet.

You can scaffold a new Hugo theme by running hugo new theme <name>. It has these files:

.
├── archetypes
│   └── default.md
├── layouts
│   ├── 404.html
│   ├── _default
│   │   ├── baseof.html
│   │   ├── list.html
│   │   └── single.html
│   ├── index.html
│   └── partials
│       ├── footer.html
│       ├── header.html
│       └── head.html
├── LICENSE
├── static
│   ├── css
│   └── js
└── theme.toml

You can see some similarities. Hugo’s page template files are tucked into layouts/. Note that the _default page type has files for a list.html and a single.html. Unlike Jekyll, Hugo uses these specific file names to distinguish between list pages (like a page with links to all your blog posts on it) and single pages (like one of your blog posts). The layouts/partials/ directory contains the buttery reusable bits, and stylesheet files have a spot picked out in static/css/.

These directory structures aren’t set in stone, as both site generators allow some measure of customization. For example, Jekyll lets you define collections, and Hugo makes use of page bundles. These features let you organize your content multiple ways, but for now, lets look at where to put some simple pages.

Where to put content

To create a site menu that looks like this:

Introduction
    Getting Started
    Configuration
    Deploying
Advanced Usage
    All Configuration Settings
    Customizing
    Help and Support

You’ll need two sections (“Introduction” and “Advanced Usage”) containing their respective subsections.

Jekyll isn’t strict with its content location. It expects pages in the root of your site, and will build whatever’s there. Here’s how you might organize these pages in your Jekyll site root:

.
├── 404.html
├── assets
├── Gemfile
├── _includes
├── index.markdown
├── intro
│   ├── config.md
│   ├── deploy.md
│   ├── index.md
│   └── quickstart.md
├── _layouts
│   ├── default.html
│   ├── page.html
│   └── post.html
├── LICENSE.txt
├── README.md
├── _sass
├── <name>.gemspec
└── usage
    ├── customizing.md
    ├── index.md
    ├── settings.md
    └── support.md

You can change the location of the site source in your Jekyll configuration.

In Hugo, all rendered content is expected in the content/ folder. This prevents Hugo from trying to render pages you don’t want, such as 404.html, as site content. Here’s how you might organize your content/ directory in Hugo:

.
├── _index.md
├── intro
│   ├── config.md
│   ├── deploy.md
│   ├── _index.md
│   └── quickstart.md
└── usage
    ├── customizing.md
    ├── _index.md
    ├── settings.md
    └── support.md

To Hugo, _index.md and index.md mean different things. It can be helpful to know what kind of Page Bundle you want for each section: Leaf, which has no children, or Branch.

Now that you have some idea of where to put things, let’s look at how to build a page template.

How templating works

Jekyll page templates are built with the Liquid templating language. It uses braces to output variable content to a page, such as the page’s title: {{ page.title }}.

Hugo’s templates also use braces, but they’re built with Go Templates. The syntax is similar, but different: {{ .Title }}.

Both Liquid and Go Templates can handle logic. Liquid uses tags syntax to denote logic operations:

{% if user %}
  Hello {{ user.name }}!
{% endif %}

And Go Templates places its functions and arguments in its braces syntax:

{{ if .User }}
    Hello {{ .User }}!
{{ end }}

Templating languages allow you to build one skeleton HTML page, then tell the site generator to put variable content in areas you define. Let’s compare two possible default page templates for Jekyll and Hugo.

Jekyll’s scaffold default theme is bare, so we’ll look at their starter theme Minima. Here’s _layouts/default.html in Jekyll (Liquid):

<!DOCTYPE html>
<html lang="{{ page.lang | default: site.lang | default: "en" }}">

  {%- include head.html -%}

  <body>

    {%- include header.html -%}

    <main class="page-content" aria-label="Content">
      <div class="wrapper">
        {{ content }}
      </div>
    </main>

    {%- include footer.html -%}

  </body>

</html>

Here’s Hugo’s scaffold theme layouts/_default/baseof.html (Go Templates):

<!DOCTYPE html>
<html>
    {{- partial "head.html" . -}}
    <body>
        {{- partial "header.html" . -}}
        <div id="content">
        {{- block "main" . }}{{- end }}
        </div>
        {{- partial "footer.html" . -}}
    </body>
</html>

Different syntax, same idea. Both templates pull in reusable bits for head.html, header.html, and footer.html. These show up on a lot of pages, so it makes sense not to have to repeat yourself. Both templates also have a spot for the main content, though the Jekyll template uses a variable ({{ content }}) while Hugo uses a block ({{- block "main" . }}{{- end }}). Blocks are just another way Hugo lets you define reusable bits.

Now that you know how templating works, you can build the sidebar menu for the theme.

Creating a top-level menu with the pages object

You can programmatically create a top-level menu from your pages. It will look like this:

Introduction
Advanced Usage

Let’s start with Jekyll. You can display links to site pages in your Liquid template by iterating through the site.pages object that Jekyll provides and building a list:

<ul>
    {% for page in site.pages %}
    <li><a href="{{ page.url | absolute_url }}">{{ page.title }}</a></li>
    {% endfor %}
</ul>

This returns all of the site’s pages, including all the ones that you might not want, like 404.html. You can filter for the pages you actually want with a couple more tags, such as conditionally including pages if they have a section: true parameter set:

<ul>
    {% for page in site.pages %}
    {%- if page.section -%}
    <li><a href="{{ page.url | absolute_url }}">{{ page.title }}</a></li>
    {%- endif -%}
    {% endfor %}
</ul>

You can achieve the same effect with slightly less code in Hugo. Loop through Hugo’s .Pages object using Go Template’s range action:

<ul>
{{ range .Pages }}
    <li>
        <a href="{{.Permalink}}">{{.Title}}</a>
    </li>
{{ end }}
</ul>

This template uses the .Pages object to return all the top-level pages in content/ of your Hugo site. Since Hugo uses a specific folder for the site content you want rendered, there’s no additional filtering necessary to build a simple menu of site pages.

Creating a menu with nested links from a data list

Both site generators can use a separately defined data list of links to render a menu in your template. This is more suitable for creating nested links, like this:

Introduction
    Getting Started
    Configuration
    Deploying
Advanced Usage
    All Configuration Settings
    Customizing
    Help and Support

Jekyll supports data files in a few formats, including YAML. Here’s the definition for the menu above in _data/menu.yml:

section:
  - page: Introduction
    url: /intro
    subsection:
      - page: Getting Started
        url: /intro/quickstart
      - page: Configuration
        url: /intro/config
      - page: Deploying
        url: /intro/deploy
  - page: Advanced Usage
    url: /usage
    subsection:
      - page: Customizing
        url: /usage/customizing
      - page: All Configuration Settings
        url: /usage/settings
      - page: Help and Support
        url: /usage/support

Here’s how to render the data in the sidebar template:

{% for a in site.data.menu.section %}
<a href="{{ a.url }}">{{ a.page }}</a>
<ul>
    {% for b in a.subsection %}
    <li><a href="{{ b.url }}">{{ b.page }}</a></li>
    {% endfor %}
</ul>
{% endfor %}

This method allows you to build a custom menu, two nesting levels deep. The nesting levels are limited by the for loops in the template. For a recursive version that handles further levels of nesting, see Nested tree navigation with recursion.

Hugo does something similar with its menu templates. You can define menu links in your Hugo site config, and even add useful properties that Hugo understands, like weighting. Here’s a definition of the menu above in config.yaml:

sectionPagesMenu: main

menu:
  main:
    - identifier: intro
      name: Introduction
      url: /intro/
      weight: 1
    - name: Getting Started
      parent: intro
      url: /intro/quickstart/
      weight: 1
    - name: Configuration
      parent: intro
      url: /intro/config/
      weight: 2
    - name: Deploying
      parent: intro
      url: /intro/deploy/
      weight: 3
    - identifier: usage
      name: Advanced Usage
      url: /usage/
    - name: Customizing
      parent: usage
      url: /usage/customizing/
      weight: 2
    - name: All Configuration Settings
      parent: usage
      url: /usage/settings/
      weight: 1
    - name: Help and Support
      parent: usage
      url: /usage/support/
      weight: 3

Hugo uses the identifier, which must match the section name, along with the parent variable to handle nesting. Here’s how to render the menu in the sidebar template:

<ul>
    {{ range .Site.Menus.main }}
    {{ if .HasChildren }}
    <li>
        <a href="{{ .URL }}">{{ .Name }}</a>
    </li>
    <ul class="sub-menu">
        {{ range .Children }}
        <li>
            <a href="{{ .URL }}">{{ .Name }}</a>
        </li>
        {{ end }}
    </ul>
    {{ else }}
    <li>
        <a href="{{ .URL }}">{{ .Name }}</a>
    </li>
    {{ end }}
    {{ end }}
</ul>

The range function iterates over the menu data, and Hugo’s .Children variable handles nested pages for you.

Putting the template together

With your menu in your reusable sidebar bit (_includes/sidebar.html for Jekyll and partials/sidebar.html for Hugo), you can add it to the default.html template.

In Jekyll:

<!DOCTYPE html>
<html lang="{{ page.lang | default: site.lang | default: "en" }}">

{%- include head.html -%}

<body>
    {%- include sidebar.html -%}

    {%- include header.html -%}

    <div id="content" class="page-content" aria-label="Content">
        {{ content }}
    </div>

    {%- include footer.html -%}

</body>

</html>

In Hugo:

<!DOCTYPE html>
<html>
{{- partial "head.html" . -}}

<body>
    {{- partial "sidebar.html" . -}}

    {{- partial "header.html" . -}}
    <div id="content" class="page-content" aria-label="Content">
        {{- block "main" . }}{{- end }}
    </div>
    {{- partial "footer.html" . -}}
</body>

</html>

When the site is generated, each page will contain all the code from your sidebar.html.

Create a stylesheet

Both site generators accept Sass for creating CSS stylesheets. Jekyll has Sass processing built in, and Hugo uses Hugo Pipes. Both options have some quirks.

Sass and CSS in Jekyll

To process a Sass file in Jekyll, create your style definitions in the _sass directory. For example, in a file at _sass/style-definitions.scss:

$background-color: #eef !default;
$text-color: #111 !default;

body {
  background-color: $background-color;
  color: $text-color;
}

Jekyll won’t generate this file directly, as it only processes files with front matter. To create the end-result filepath for your site’s stylesheet, use a placeholder with empty front matter where you want the .css file to appear. For example, assets/css/style.scss. In this file, simply import your styles:

---
---

@import "style-definitions";

This rather hackish configuration has an upside: you can use Liquid template tags and variables in your placeholder file. This is a nice way to allow users to set variables from the site _config.yml, for example.

The resulting CSS stylesheet in your generated site has the path /assets/css/style.css. You can link to it in your site’s head.html using:

<link rel="stylesheet" href="{{ "/assets/css/style.css" | relative_url }}" media="screen">

Sass and Hugo Pipes in Hugo

Hugo uses Hugo Pipes to process Sass to CSS. You can achieve this by using Hugo’s asset processing function, resources.ToCSS, which expects a source in the assets/ directory. It takes the SCSS file as an argument. With your style definitions in a Sass file at assets/sass/style.scss, here’s how to get, process, and link your Sass in your theme’s head.html:

{{ $style := resources.Get "/sass/style.scss" | resources.ToCSS }}
<link rel="stylesheet" href="{{ $style.RelPermalink }}" media="screen">

Hugo asset processing requires extended Hugo, which you may not have by default. You can get extended Hugo from the releases page.

Configure and deploy to GitHub Pages

Before your site generator can build your site, it needs a configuration file to set some necessary parameters. Configuration files live in the site root directory. Among other settings, you can declare the name of the theme to use when building the site.

Configure Jekyll

Here’s a minimal _config.yml for Jekyll:

title: Your awesome title
description: >- # this means to ignore newlines until "baseurl:"
  Write an awesome description for your new site here. You can edit this
  line in _config.yml. It will appear in your document head meta (for
  Google search results) and in your feed.xml site description.
baseurl: "" # the subpath of your site, e.g. /blog
url: "" # the base hostname & protocol for your site, e.g. http://example.com
theme: # for gem-based themes
remote_theme: # for themes hosted on GitHub, when used with GitHub Pages

With remote_theme, any Jekyll theme hosted on GitHub can be used with sites hosted on GitHub Pages.

Jekyll has a default configuration, so any parameters added to your configuration file will override the defaults. Here are additional configuration settings.

Configure Hugo

Here’s a minimal example of Hugo’s config.yml:

baseURL: https://example.com/ # The full domain your site will live at
languageCode: en-us
title: Hugo Docs Site
theme: # theme name

Hugo makes no assumptions, so if a necessary parameter is missing, you’ll see a warning when building or serving your site. Here are all configuration settings for Hugo.

Deploy to GitHub Pages

Both generators build your site with a command.

For Jekyll, use jekyll build. See further build options here.

For Hugo, use hugo. You can run hugo help or see further build options here.

You’ll have to choose the source for your GitHub Pages site; once done, your site will update each time you push a new build. Of course, you can also automate your GitHub Pages build using GitHub Actions. Here’s one for building and deploying with Hugo, and one for building and deploying Jekyll.

Showtime

All the substantial differences between these two generators are under the hood; all the same, let’s take a look at the finished themes, in two color variations.

Here’s Hugo:

Here’s Jekyll:

Spiffy!

Wait who won

🤷

Both Hugo and Jekyll have their quirks and conveniences.

From this developer’s perspective, Jekyll is a workable choice for simple sites without complicated organizational needs. If you’re looking to render some one-page posts in an available theme and host with GitHub Pages, Jekyll will get you up and running fairly quickly.

Personally, I use Hugo. I like the organizational capabilities of its Page Bundles, and it’s backed by a dedicated and conscientious team that really seems to strive to facilitate convenience for their users. This is evident in Hugo’s many functions, and handy tricks like Image Processing and Shortcodes. They seem to release new fixes and versions about as often as I make a new cup of coffee.

If you still can’t decide, don’t worry. Many themes are available for both Hugo and Jekyll! Start with one, switch later if you want. That’s the benefit of having options.

Cybersecurity can be fiddly and time-consuming. You might need to reset forgotten passwords, transfer multifactor authentication (MFA) codes to different devices, or deal with the fallout of compromised payment details in the event one of your accounts is still breached.

Thankfully, most of the work necessary to keep up our cybersecurity measures can be outsourced.

Here are three changes you can make to significantly reduce the chances of needing to fiddle with any of these things again.

1Password

I’ve historically avoided password managers because of an irrational knee-jerk reaction to putting all my eggs in one basket. You know what’s great for irrational reactions? Education.

To figure out if putting all my passwords into a password manager is more secure than not using one, I set out to see what some smart people wrote about it.

First, we need to know a thing or two about passwords. Troy Hunt figured out almost a decade ago that trying to remember strong passwords doesn’t work. In more recent times, Alex Weinert expanded on this in Your Pa$$word doesn’t matter. TL;DR: our brains aren’t better at passwords than computers, and please use MFA.

So passwords don’t matter, but complicated passwords are still better than memorable and guessable ones. Since I’ve next to no hope of remembering a dozen variations of p/q2-q4! (I’m not a chess player), this is a task I can outsource to 1Password. I’ll still need to remember one, long, complicated master password - 1Password uses this to encrypt my data, so I really can’t lose it - but I can handle just one.

Using 1Password specifically has another, decidedly obvious, advantage. I chose 1Password because of their Watchtower feature. Thanks to Troy Hunt’s Have I Been Pwned, Watchtower will alert you if any of your passwords show up in a breach so you can change them. Passwords still don’t completely work, but this is probably the best band-aid there is.

One last bonus is that using a password manager is a heck of a lot more convenient. I don’t need to take a few tries to type in a complicated password. I don’t end up spending time resetting passwords I’ve forgotten on sites I only rarely use.

When tasked with remembering all their own passwords, people typically create simpler passwords that are easier to remember – and easier to hack. This occurs most frequently on sites that are considered unimportant. Using 1Password and generated passwords, those sites are now also first-class citizens in the land of strong passwords, instead of being half-abandoned and half-open attack vectors.

So, yes, all my eggs are in one basket. A well-protected, complex, and monitored basket.

Authy

Okay - so it’s more like one-and-a-half baskets. 🤷🏻

Authy, from the folks over at Twilio, provides a 2FA solution that’s more secure than SMS. Unlike Google Authenticator, you can choose to back up your 2FA codes in case you lose or change your phone. (1Password offers 2FA functionality as well - but, you know, redundancies.)

With Authy, your back up is encrypted with your password, similarly to how 1Password works. This makes it the second password you can’t forget, if you don’t want to lose access to your codes. If you reset your account, they all go away. I can deal with remembering two passwords; I’ll take that trade.

I’ve tried other methods of MFA, including hardware keys, which can make accessing accounts on your phone more complicated than I care to put up with. I find the combination of 1Password and Authy to be the most practical combination of convenience and security that yet exists to my knowledge.

Privacy.com

Finally, there’s one last line of defense you can put in place in the unfortunate event that one of your accounts is still compromised. All the strong passwords and MFA in the world won’t help if you open the doors yourself, and scams and phishing are a thing.

Since it’s rather impractical to use a different real credit card every place you shop, virtual cards are just a great idea. There’s no good reason to spend an afternoon (or more) resetting your payment information on every account just to thwart a misbehaving merchant or patch up a data breach from that online shop for cute salt shakers you made a purchase at last year (just me?).

As a bonus, a partnership between 1Password and Privacy.com lets you easily create virtual credit cards using the 1Password extension.

By setting up a separate virtual card for each merchant, in the event that one of those merchants is compromised, you can simply pause or delete that card. None of your other accounts or actual bank details are caught up in the process. Cards can have time-based limits or be one-off burner numbers, making them ideal for setting up subscriptions.

This is the sort of basic functionality that I hope, one day, becomes more prevalent from banks and credit cards. In the meantime, I’ll keep using Privacy.com. That’s my referral link; if you’d like to thank me by using it, we’ll both get five bucks as a bonus.

Outsource better security

All together, implementing these changes will probably take up an afternoon, depending on how many accounts you have. It’s worth it for the time you’d otherwise spend resetting passwords, setting up new devices, or (knock on wood) recovering from compromised banking details. Best of all, you’ll have continual protection just running in the background.

• 1Password
• Authy
• Privacy.com

We have the technology. Free up some brain cycles to focus on other things - or simply remove some unnecessary stress from your life by outsourcing the fiddly bits.

Want to give the gift of cybersecurity to someone you know? Get them started with a cybersecurity starter pack.

So you need a database. It’s going to handle a few hundred users and mostly read operations. Time to set up a PostgreSQL cluster, debate connection pooling strategies, configure replication, and design backup procedures… right?

When I say, “What about SQLite?”, the response is usually some variation of “That’s not a real database.”

This reaction reveals something important about how engineering teams make technology decisions. We often choose tools based on what sounds impressive rather than what solves our actual problems. SQLite represents an underappreciated truth in engineering leadership: sometimes the boring, simple solution is exactly what your team needs.

SQLite (“see-quell-lite”) is a lightweight SQL database engine that’s self-contained in a single file. It’s library, database, and data, all in one package. For certain applications, SQLite is a solid choice for a production database. It’s lightweight, ultra-portable, and has no external dependencies.

Matching Tools to Actual Requirements

As an engineering leader, one of your most important responsibilities is helping your team choose appropriate technology rather than impressive technology. SQLite excels in specific scenarios that are more common than most teams realize.

SQLite is best suited for production use in applications that:

• Desire fast and simple setup
• Require high reliability in a small package
• Have, and want to retain, a small footprint
• Are read-heavy but not write-heavy
• Don’t need multiple user accounts or features like multiversion concurrency snapshots

These criteria describe a significant percentage of web applications, internal tools, and even customer-facing products. But teams often dismiss SQLite because it doesn’t match their mental model of what a “serious” database looks like.

Recognizing when your team’s technology choices are driven by resume-driven development rather than problem-solving can save you oodles of time and budget wiggle room. Complex solutions carry hidden costs in deployment complexity, operational overhead, and cognitive load that simple solutions avoid entirely.

Understanding the Technical Trade-offs

To guide these decisions effectively, it helps to understand the technical details well enough to evaluate trade-offs intelligently. In the case of SQLite, you can examine its performance characteristics to make this evaluation concrete:

Database Transaction Modes

POSIX system call fsync() commits buffered data (data saved in the operating system cache) referred to by a specified file descriptor to permanent storage or disk. This is relevant to understanding the difference between SQLite’s two modes, as fsync() will block until the device reports the transfer is complete.

SQLite uses atomic commits to batch database changes into single transactions, enabling apparent simultaneous writing of multiple operations. This is accomplished through one of two modes: rollback journal or write-ahead log (WAL).

Rollback Journal Mode

A rollback journal is essentially a back-up file created by SQLite before write changes occur on a database file. It has the advantage of providing high reliability by helping SQLite restore the database to its original state in case a write operation is compromised during the disk-writing process.

Assuming a cold cache, SQLite first needs to read the relevant pages from a database file before it can write to it. Information is read out into the operating system cache, then transferred into user space. SQLite obtains a reserved lock on the database file, preventing other processes from writing to the database. At this point, other processes may still read from the database.

SQLite creates a separate file, the rollback journal, with the original content of the pages that will be changed. Initially existing in the cache, the rollback journal is written to persistent disk storage with fsync() to enable SQLite to restore the database should its next operations be compromised.

SQLite then obtains an exclusive lock preventing other processes from reading or writing, and writes the page changes to the database file in cache. Since writing to disk is slower than interaction with the cache, writing to disk doesn’t occur immediately. The rollback journal continues to exist until changes are safely written to disk with a second fsync(). From a user-space process point of view, the change to the disk (the COMMIT, or end of the transaction) happens instantaneously once the rollback journal is deleted - hence, atomic commits. However, the two fsync() operations required to complete the COMMIT make this option, from a transactional standpoint, slower than SQLite’s lesser known WAL mode.

Write-ahead logging (WAL)

While the rollback journal method uses a separate file to preserve the original database state, the WAL method uses a separate WAL file to instead record the changes. Instead of a COMMIT depending on writing changes to disk, a COMMIT in WAL mode occurs when a record of one or more commits is appended to the WAL. This has the advantage of not requiring blocking read or write operations to the database file in order to make a COMMIT, so more transactions can happen concurrently.

WAL mode introduces the concept of the checkpoint, which is when the WAL file is synced to persistent storage before all its transactions are transferred to the database file. You can optionally specify when this occurs, but SQLite provides reasonable defaults. The checkpoint is the WAL version of the atomic commit.

In WAL mode, write transactions are performed faster than in the traditional rollback journal mode. Each transaction involves writing the changes only once to the WAL file instead of twice - to the rollback journal, and then to disk - before the COMMIT signals that the transaction is over.

For teams handling moderate write loads, WAL mode often provides the performance characteristics they actually need without the operational complexity of distributed databases.

The Performance Reality

Benchmarks tell a compelling story about SQLite’s practical capabilities. On modest hardware—the smallest EC2 instance with no provisioned IOPS—SQLite with WAL mode handles 400 write transactions per second and thousands of reads. For many applications, this represents more capacity than they need.

These numbers matter because they provide concrete data for technology discussions. Instead of theoretical conversations about “what if we need to scale,” you can evaluate whether 400 writes per second actually meets your requirements. Often, it does—with significant room for growth.

More importantly, SQLite eliminates entire categories of operational complexity: connection pooling, database server maintenance, backup procedures, replication configuration, and deployment coordination. The operational overhead you don’t have to manage often provides more value than the theoretical scalability you might need someday.

Making Strategic Technology Decisions

Engineering teams often equate complexity with sophistication and assume that simple solutions won’t scale or aren’t “enterprise-ready” without considering the actual requirements of the enterprise. The SQLite decision exemplifies a broader principle in engineering leadership: optimizing for actual constraints rather than imaginary ones. This requires understanding both the technical capabilities of your options and the real requirements of your systems.

This means asking your teams to articulate specific performance requirements, operational constraints, and growth projections rather than making technology choices based on industry trends or resume building. It means evaluating the total cost of ownership including deployment complexity, operational overhead, and team cognitive load.

Most importantly, it means recognizing that the best technology choice is often the one that solves your current problems effectively while remaining simple enough to understand, maintain, and evolve as requirements change.

Building a Culture of Appropriate Technology

Teams that consistently make good technology choices develop systematic approaches to evaluation rather than relying on instinct or industry hype. They start with requirements, evaluate options based on total cost of ownership, and choose solutions that match their actual needs rather than their aspirational ones.

This culture emerges when leaders model technical decision-making that prioritizes problem-solving over impressiveness. When you advocate for SQLite over PostgreSQL because it better matches your workload, you’re teaching your team to think critically about technology trade-offs.

The long-term impact is teams that build sustainable systems they can actually maintain and evolve. Simple solutions that solve real problems create more value than complex solutions that solve theoretical ones.

For medium-sized, read-heavy applications, SQLite with WAL mode represents exactly this kind of appropriate technology choice. It provides perfectly adequate capability in a perfectly compact package—which is often exactly what your application needs.

When I developed Hydra, a multithreaded link checker written in Python, the performance requirements were clear. It needed to run as part of CI/CD processes, which meant speed was essential for developer productivity. Nobody wants to wait seventeen minutes to learn whether their build succeeded—that’s long enough to make coffee, check Twitter, question your career choices, and wonder if the process crashed.

The project became a case study in systematic performance optimization and the leadership decisions that guide technical implementation. Unlike many Python site crawlers that rely on external dependencies like BeautifulSoup, Hydra uses only standard libraries. This constraint required thinking carefully about how to achieve optimal performance within Python’s limitations.

Understanding the Performance Landscape

As an engineering leader, one of your most important responsibilities is helping your team understand where performance problems actually occur versus where they assume they occur. Most developers have an intuitive sense that network operations are slower than CPU operations, but the actual magnitude of these differences is staggering.

Here are approximate timings for tasks performed on a typical PC:

Peter Norvig first published these numbers in Teach Yourself Programming in Ten Years. While hardware continues to evolve, the relative relationships remain humbling for anyone who’s ever spent time optimizing the wrong thing.

Notice that sending a simple packet over the Internet is over a million times slower than fetching from RAM. These aren’t small performance differences—they’re fundamental constraints that should guide every optimization decision you make.

For Hydra, parsing response data and assembling results happens on the CPU and is relatively fast. The overwhelming bottleneck—by over six orders of magnitude—is network latency. Any optimization effort that didn’t address network I/O would miss the point.

Working Within Python’s Constraints

Python presents an interesting challenge for performance-critical applications. The Global Interpreter Lock (GIL) prevents multiple threads from executing Python bytecodes simultaneously—each thread must wait for the GIL to be released by the currently executing thread. This eliminates race conditions but also prevents true parallel execution of CPU-bound tasks.

For many engineering teams, this limitation becomes a reason to dismiss Python entirely for performance work. But effective technical leadership involves understanding how to work within constraints rather than avoiding tools with limitations.

The key insight is that Python’s GIL limitation doesn’t apply uniformly. While CPU-bound tasks suffer from the GIL, I/O-bound tasks can benefit from concurrent execution because the GIL is released during I/O operations. For Hydra’s use case—fetching web pages over the network—multithreading in Python can provide significant performance improvements despite the GIL.

This distinction matters for strategic technical decisions. Instead of automatically reaching for Go or Rust when performance requirements emerge, understanding Python’s actual constraints can enable better technology choices based on specific workload characteristics.

Choosing the Right Concurrency Model

Python provides multiple approaches to parallel execution, each suited for different types of bottlenecks. Making the right choice requires understanding both technical trade-offs and your application’s specific performance characteristics.

Multiple Processes

Python’s ProcessPoolExecutor uses worker subprocesses to bypass the GIL entirely. This approach maximizes parallelization for CPU-bound tasks by utilizing multiple processor cores effectively.

For compute-heavy operations—mathematical calculations, data processing, algorithm execution—multiple processes provide genuine parallel execution. However, this carries overhead costs in memory usage and inter-process communication that may not be justified for I/O-bound workloads.

Multiple Threads

Python’s ThreadPoolExecutor uses a pool of threads that can execute I/O operations concurrently. While threads can’t execute Python code in parallel due to the GIL, they can perform I/O operations concurrently because the GIL is released during system calls.

For I/O-bound applications—web scraping, API calls, file operations—threading provides excellent performance improvements with lower overhead than multiprocessing.

Implementation Strategy

Here’s how Hydra uses ThreadPoolExecutor to achieve concurrent link checking:

# Create the Checker class
class Checker:
    # Queue of links to be checked
    TO_PROCESS = Queue()
    # Maximum workers to run
    THREADS = 100
    # Maximum seconds to wait for HTTP response
    TIMEOUT = 60

    def __init__(self, url):
        ...
        # Create the thread pool
        self.pool = futures.ThreadPoolExecutor(max_workers=self.THREADS)


def run(self):
    # Run until the TO_PROCESS queue is empty
    while True:
        try:
            target_url = self.TO_PROCESS.get(block=True, timeout=2)
            # If we haven't already checked this link
            if target_url["url"] not in self.visited:
                # Mark it as visited
                self.visited.add(target_url["url"])
                # Submit the link to the pool
                job = self.pool.submit(self.load_url, target_url, self.TIMEOUT)
                job.add_done_callback(self.handle_future)
        except Empty:
            return
        except Exception as e:
            print(e)

The implementation reflects several engineering leadership principles. The thread pool size (100 workers) was determined through profiling and testing rather than guesswork. The timeout mechanism prevents slow requests from blocking overall progress. The callback pattern enables efficient result processing without blocking the main execution thread.

Measuring Real Impact

Performance optimization discussions often remain theoretical without concrete measurements. For Hydra, the improvement was dramatic. Here’s a comparison between the run times for checking my website with a prototype single-thread program and using Hydra:

time python3 slow-link-check.py https://victoria.dev

real    17m34.084s
user    11m40.761s
sys     0m5.436s


time python3 hydra.py https://victoria.dev

real    0m15.729s
user    0m11.071s
sys     0m2.526s