Devopsian

Understanding Go’s Supercharged Map in v1.24

Tue, 04 Mar 2025 22:50:00 +0200

Go 1.24 introduces a new map implementation, inspired by Google’s Swiss Tables, which brings significant optimizations and performance enhancements to the language’s built-in map type. While Go’s previous map implementation was already efficient, the new design takes it a step further by introducing a clever approach to data organization and access.

To help digest this complex change, let’s use an analogy that illustrates how the new map works and how it differs from the previous implementation. Let’s use a relatable analogy: a library. Just as a library organizes books in a way that makes them easy to find and access, a map organizes data for efficient retrieval.

This analogy will provide a high-level understanding of the key improvements without delving too deeply into technical details. For those interested in a more in-depth exploration, we’ll reference additional resources throughout the explanation.

The Library Analogy

Think of Go’s map as a library designed to store books. Here’s how it works:

1. Tables Are Library Sections

The map starts with one table, which is like a section of the library. If this section gets too crowded, the library adds another section. Each table is divided into smaller units called groups.

2. Groups Are Bookshelves

Each table is made up of multiple groups, which are like bookshelves in the library. A group can hold up to 8 books (key-value pairs). These groups are the fundamental storage units in Go maps.

3. Control Word: The Librarian’s Cheat Sheet

Each bookshelf has a label that summarizes key information taped to it, called the control word. This cheat sheet contains metadata about the books on that shelf:

It stores tiny “fingerprints” of each book’s ID (derived from its hash).
It marks whether slots on the shelf are empty, occupied, or deleted.

This cheat sheet helps librarians quickly locate books without flipping through every slot.

It can be pictured as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


+---------------------+ +---------------------+
| Map | | Library |
+---------------------+ +---------------------+
| Table 0 | | Section 1 |
+---------------------+ +---------------------+
| Control Word (64b) | <----> | Shelf label |
+---------------------+ +---------------------+
| Key 0 | Value 0 | | Book 1 | Location 1 |
| Key 1 | Value 1 | | Book 2 | Location 2 |
| ... | | ... |
| Key 7 | Value 7 | | Book 8 | Location 8 |
+---------------------+ +---------------------+
| Control Word (64b) | <----> | Shelf label |
+---------------------+ +---------------------+
| Key 0 | Value 0 | | Book 1 | Location 1 |
| Key 1 | Value 1 | | Book 2 | Location 2 |
| ... | | ... |
| Key 7 | Value 7 | | Book 8 | Location 8 |
+---------------------+ +---------------------+
| ... | | ... |
+---------------------+ +---------------------+
| Table 1 | | Section 2 |
+---------------------+ +---------------------+
| ... | | ... |

How It Works: Storing and Retrieving Books

Let’s walk through an example of storing and retrieving a book in this library.

Storing a Book

Say you want to store “The Great Gatsby” by F. Scott Fitzgerald in the map.

Generate a Hash The librarian generates a unique ID for "The Great Gatsby" using a hash function, e.g., 0xf83c6f3a3c.
Find the Section and Bookshelf The hash is split into two parts:
- H1: Determines which section (table) and bookshelf (group) the book belongs to. (57 bits)
- H2: A small fingerprint stored in the control word for quick identification. (7 bits)
For example:
- H1 says: “Go to Section 1, Bookshelf 3.”
- H2 says: “Fingerprint is 3c.”
Place the Book The librarian places "The Great Gatsby" into an available slot on Bookshelf 3 and updates the control word with 3c.

Retrieving a Book

Now you want to retrieve the book from the map.

Find the Section and Bookshelf The librarian uses H1 from the hash of "The Great Gatsby" to go directly to Section 1, Bookshelf 3.
Check the Cheat Sheet (Control Word) The librarian looks at the control word (3c) to see if any slots match "The Great Gatsby"’s fingerprint. If the fingerprint does not match, they know that slot doesn’t hold the desired book, saving time.
Confirm and Return If there’s a match, they compare keys directly to confirm it’s "The Great Gatsby". Once confirmed, they return its value (10101..).

This process avoids unnecessary checks and minimizes memory lookups, making retrieval lightning-fast.

Handling Collisions

What happens if multiple books generate the same H1 (i.e., they hash to the same group)? This scenario is known as a collision.

In our library analogy:

If a bookshelf is full, the librarian moves to the next available shelf in the same section.
This is called linear probing, where nearby groups are checked for free slots.

To ensure efficiency, if all shelves in a section are full, a new section (table) is added, and some books are redistributed between sections based on updated hash calculations to maintain optimal access.

Why This Design is Brilliant

The new map implementation in Go 1.24 introduces several optimizations inspired by Swiss Table design:

1. Cache-Friendly Layout

Keys and values are stored together in groups, improving cache locality (storing related items close together takes advantage of how memory access works). When looking for an item, both its key and value are likely loaded into memory at once.

2. Fast Probing with Metadata

The control word allows fast rejection of irrelevant slots using SIMD (Single Instruction, Multiple Data allows processing multiple data points with a single instruction, thus boosting performance during lookups) operations. This means multiple slots can be checked simultaneously, speeding up lookups significantly.

While pre 1.24 use tophash which reminds this strategy, it was still required to pointer chasing when overflow buckets were involved, reducing cache efficiency.

Memory Layout Example

Here’s a simplified version of how this might look in memory for a single group:

Control Word	Key0	Value0	Key1	Value1	…	Key7	Value7
`3c`	`"The Great Gatsby"`	`10101..`	…	…	…	…	…

The control word (3c) stores fingerprints for all 8 slots. (We have only one element in the group)
Keys ("The Great Gatsby") and values (10101..) are stored adjacently within each group for better performance.

Performance Gains

The redesign brings significant improvements over older implementations:

Faster lookups: Metadata allows skipping irrelevant slots quickly.
Reduced memory overhead: Group storage eliminates extra pointers, and has better load-factor.
Better scalability: Incremental resizing avoids performance bottlenecks during growth.

For example:

With the optimizations, lookups are up to ~30% faster compared to previous versions.
Memory usage is reduced by as much as ~28% compared to older versions of Go maps.

Conclusion

Go 1.24’s map is like a librarian who gets smarter with every update—finding books faster, using space more efficiently where:

Sections (tables) expand as needed.
Bookshelves (groups) keep related items close together.
Cheat sheets (control words) help librarians find books faster without flipping through every slot.

This design balances speed, memory efficiency, and scalability beautifully—making Go maps one of the most optimized hash table implementations out there!

Whether you’re building high-performance systems or just curious about how things work under the hood, understanding these concepts can help you appreciate Go’s thoughtful engineering even more.

For more detailed post about this implementation, check the official Go’s blog post Faster Go maps with Swiss Tables or ByteSizeGo Swiss Table Maps.

References to very good explanations of maps prior 1.24

Cover image by Janko Ferlič on Unsplash

Handling Retries When Sending Files in Go: Lessons Learned

Thu, 26 Dec 2024 22:00:00 +0200

If you’ve ever implemented a file upload feature in Go, you might have run into a peculiar issue when retrying HTTP requests. I certainly did. This blog post is a deep dive into the problem I faced, why it happened, and what I learned about the io.Reader interface.

The Problem: Retrying Sends an Empty File

In my Go application, one part of the functionality involves reading a file from disk and sending it to a remote server via an HTTP POST request. Here’s a simplified version of what my initial implementation looked like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


file, err := os.Open("path/to/file.txt")
if err != nil {
 log.Fatal(err)
}
defer file.Close()

req, err := http.NewRequest("POST", "https://example.com/upload", file)
if err != nil {
 log.Fatal(err)
}

client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
 log.Fatal(err)
}
defer resp.Body.Close()

My files are small, so loading the content onto memory is not an issue.

All seemed fine until I added a retry mechanism to handle server failures. With retries, the client would attempt to resend the file if the server responded with an error or was unavailable. This is what the retry logic looked like:

1
2
3
4
5
6
7
8


for i := 0; i < 3; i++ {
 resp, err := client.Do(req)
 if err == nil && resp.StatusCode == http.StatusOK {
 break
 }
 log.Printf("Retrying... (%d/3)", i+1)
 time.Sleep(1 * time.Second)
}

During testing, I simulated a server failure, and on the second attempt, the server received a 0-byte file. What happened?

Understanding `io.Reader` and Why This Happens

The root of the problem lies in how Go’s io.Reader interface works. An io.Reader provides a stream of data that can be read sequentially. When you pass a file to http.NewRequest as the request body, the HTTP client reads from the file until it reaches the end. This process advances an internal pointer in the io.Reader.

Once the io.Reader has been fully read, its internal pointer remains at the end of the stream. Subsequent reads will return no data, effectively making the io.Reader empty unless explicitly reset. When the retry logic kicked in, the second attempt reused the same io.Reader that had already been read. At this point, there was nothing left to read, so an empty file was sent.

The Solution: Using a buffer (`bytes.Buffer`)

To fix this, I needed a data structure that could be read multiple times, e.g.bytes.Buffer. This is a memory-backed buffer that implements the io.Reader interface, making it perfect for this scenario. Here’s how I updated the code:

Read the file’s content into a bytes.Buffer.
Use the buffer as the request body.

Here’s the updated implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


file, err := os.Open("/path/to/file.txt")
if err != nil {
 log.Fatal(err)
}
defer file.Close()

// Read file into a buffer first
var buf bytes.Buffer
if _, err := io.Copy(&buf, file); err != nil {
 log.Fatal(err)
}

// Create a request with the buffer instead of passing the original io.Reader
for i := 0; i < 3; i++ {
 req, err := http.NewRequest("POST", "https://example.com/upload", &buf)
 if err != nil {
 log.Fatal(err)
 }

 client := &http.Client{}
 resp, err := client.Do(req)
 if err == nil && resp.StatusCode == http.StatusOK {
 log.Println("File uploaded successfully.")
 break
 }
 log.Printf("Retrying... (%d/3)", i+1)
 time.Sleep(1 * time.Second)
}

By reading the file into a bytes.Buffer, I ensured that the content remained accessible for each retry. The io.Copy function reads from the file and writes its contents to the buffer. Since bytes.Buffer allows multiple reads, the retry mechanism works flawlessly.

Exploring Alternatives to `bytes.Buffer`

While bytes.Buffer is a great choice for many cases, it may not be ideal for very large files due to memory constraints. Here are some alternatives:

io.ReadSeeker If the file is large but can be stored on disk, using an io.ReadSeeker allows resetting the read pointer to the beginning without needing to load the entire file into memory.
Chunked Uploads: For extremely large files, consider splitting the file into smaller chunks and uploading each chunk separately. This method reduces memory usage and improves reliability for large transfers.

Each approach has trade-offs, so choose based on your specific use case.

Memory Considerations

Using a memory-backed buffer like bytes.Buffer works well for small to moderately sized files. Keep in mind that the entire file content will reside in memory. For large files, this can lead to significant memory consumption and potentially out-of-memory errors.

If handling large files is a requirement, consider using disk-backed solutions like temporary files with os.CreateTemp, which allow you to read the data sequentially from the disk without consuming much memory. Alternatively, implement streaming by breaking the file into manageable chunks and sending each chunk separately, ensuring memory efficiency even for very large files.

Best Practices for Retry Logic

When implementing retry mechanisms, keep the following practices in mind:

Use Exponential Backoff: Avoid flooding the server with retries by adding a delay that increases after each failed attempt.
Set a Retry Limit: Always limit the number of retries to prevent infinite loops and resource exhaustion.
Log Errors: Keep track of what caused retries to occur, as this can help debug issues or provide insight into server-side problems.

Incorporating these strategies ensures that your retry logic is robust and doesn’t inadvertently worsen issues. For my use-case, I chose a simple strategy of a retry limit with a short sleep between intervals.

Key Takeaways

Understand io.Reader: Once an io.Reader is read, it can’t be reused unless reset.
Use a reusable buffer: For retry mechanisms, use a structure like bytes.Buffer or io.ReadSeeker to ensure the data remains accessible.
Think about retries early: If retries are a possibility, design with this in mind from the start - and be sure to test it.

Wrapping Up

This was a great reminder of how important it is to understand the tools we use. Go’s io.Reader is simple yet powerful, but it’s essential to know its limitations. By leveraging bytes.Buffer, I was able to build a robust retry mechanism that handles file upload errors gracefully.

Have you faced a similar issue in Go? Share your experiences or solutions in the comments below!

Why I Switched from Makefile to Taskfile

Thu, 24 Oct 2024 22:00:00 +0300

Introduction

Software projects involve several phases, including building, testing, and deploying code. For instance, compiling Go source code results in an executable, while frontend frameworks compile into HTML, CSS, and JavaScript files. Testing is crucial before merging changes or releasing new versions. Deployment scripts often ship software to production. Each phase requires different tools, typically command-line utilities with various flags and parameters. Automation tools simplify these processes, enhancing efficiency in daily workflows.

Makefile

Makefiles are powerful tools that automate software project workflows. Initially developed for C programs, they now support diverse tasks like website generation and data processing.

A Makefile contains directives for the make utility to build or maintain programs and files. It defines tasks and their dependencies, ensuring efficient and reproducible builds.

I won’t dive into Makefiles in this blog post as I’m assuming the reader is familiar with the concept. If not, there is plenty of information over the internet (like this tutorial for example or it’s wikipedia page).

Advantages of Makefile:

Integrated with the make utility, available on most Linux/MacOS systems.
A well-established tool with nearly 50 years of history.

These are the main advantages I think Makefile have. However, Makefiles have limitations, particularly their syntax, which can be cumbersome for complex tasks.

Why I Switched

In one of my projects, I used a Makefile for tasks like running frontend/backend services and database migrations. Here’s an example of a migration task:

1
2
3


migrate-up:
 GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=app host=localhost port=5432 dbname=my-app sslmode=disable user=app" \
 goose -dir database/migrations up

I wanted to load environment variables from a .env file by default but allow overrides with ENV_FILE=.env.production. After struggling with Makefile syntax and solutions that didn’t work, I sought alternatives.

Introducing Taskfile

Taskfile is a Go-based task runner using YAML syntax for defining tasks. It simplifies project workflows by automating repetitive tasks like building, testing, and deploying code.

Benefits of Taskfile:

Readable YAML Syntax: Easier to understand than Makefiles.
Single Binary: No dependencies beyond the Go runtime.
Cross-Platform Support: Works on Linux, macOS, and Windows.

Here’s how I solved my problem using Taskfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


version: '3'
dotenv:
 - '.env'

tasks:
 migrate-up:
 cmds:
 - goose -dir database/migrations up

 migrate-up-prod:
 dotenv:
 - .env.production
 cmds:
 - echo executing DB migration on PRODUCTION ..
 - sleep 2 # allow time to cancel
 - goose -dir database/migrations up

Taskfile’s intuitive API allowed me to quickly implement a solution that was both functional and readable.

Summary

Choosing the right tool can significantly impact productivity. While Makefile served its purpose initially, Taskfile offered a more elegant solution for my needs. Transitioning took less than 30 minutes and simplified my build process considerably.

If you’re seeking an easy-to-use build tool, consider giving Taskfile a try.

The value of API-First design on side-projects

Fri, 12 Jul 2024 16:00:00 +0300

Intro

Lately, I had a chance to try out the API-First design approach. I had never written an OpenAPI document before, so I had no real knowledge of its benefits. It always seemed like too much prep work.

As developers, we often prefer writing code to writing documentation. We dive straight into coding, eager to see our project in action. However, I recently discovered a game-changing approach that has transformed my development process: API-First design. In this post, I’ll share my experience implementing this method in a full-stack hobby project, highlighting how it streamlined my workflow and why it’s worth considering for your next side project.

tl;dr: It will force you to think about your users and how they use your API before writing any code.

I’ve been working on a full-stack hobby project where my backend and frontend use different languages (Go and SvelteKit). I decided to give this approach a try and had my “aha” moment. I wish I had done it before.

Prioritizing Your Application’s Foundation

The API is how we are going to expose our app functionality. An API-first design approach prioritizes the development of APIs before implementing other parts of a software system (or writing code). This method focuses on creating a well-designed, consistent, and user-friendly API that is the foundation for the entire application.

This methodology places the API at the center of the development process, treating it as a first-class citizen rather than an afterthought. Your API comes first, then the implementation.

With a written API specification, we can leverage code generation tools to create some boilerplate code. By defining objects in the specification, code-gen tools can generate the relevant structs, for both the frontend and backend (yes, even when the language used is different). This is a big time saver and it helps us to be consistent.

What is Open API?

“The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection.”

Simply put, it’s a contract that describes your API types and endpoints. You list all your API endpoints, their HTTP methods, what they possibly return, and some description of what they do.

Now, what if I told you, you can use this document to improve and accelerate your dev experience?

Once I had this document, that describes the contract between my API server and its clients, these are the things I could do:

Generate my backend types (Go)
Generate my frontend types (Typescript)
Generate a client code for my server (also Typescript)
Generate a testing client with Insomnia or Postman

This is a lot of boilerplate code I could save myself from writing. It ensures the frontend and backend types are synchronized since both are generated.

Grab a 🍺, and let’s walk through an example.

The Project Structure

We will be using Go for the backend and some JS framework for the frontend, and a simple structure would look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


app/
├─ api/ -- the place for the Swagger OpenAPI document
├─ client/ -- the client-side code
├─ cmd/
│ ├─ app.go -- thin main func that runs our API server
├─ internal/
│ ├─ api/
│ │ ├─ main.go -- for code-gen
│ ├─ users/
│ │ ├─ handlers.go -- implements the API contract

Generate The APIs

Let’s create our API specification. It includes two endpoints and two structs: User and Error. Place this file under your /api directory

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82


openapi: 3.0.3
info:
 title: Devopsian OpenAPI Example
 version: 0.1.0
 contact:
 name: dev
 url: https://devopsian.net
servers:
 - url: "http://localhost/v1"
components:
 schemas:
 Error:
 type: object
 required:
 - code
 - message
 properties:
 code:
 type: integer
 format: int32
 message:
 type: string
 User:
 type: object
 required:
 - id
 - name
 - email
 properties:
 id:
 type: string
 name:
 type: string
 email:
 type: string
 format: email

paths:
 /user:
 get:
 description: Get the current logged-in user
 responses:
 200:
 description: user response
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/User"
 default:
 description: error
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/Error"
 /signup:
 post:
 description: Creates a new user
 responses:
 200:
 description: Creates a user
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/User"
 default:
 description: error
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/Error"
 requestBody:
 required: true
 content:
 application/json:
 schema:
 type: object
 properties:
 name:
 type: string
 email:
 type: string
 format: email

Generate Server-Side Code

To generate the server-side code, we need some library. I found oapi-codegen for that. It supports many popular HTTP libraries (echo, gin, etc.) At the time of writing, I used oapi-codegen@v2.3.0

Add the following files to your /internal/api directory

1
2
3
4
5
6


// /internal/api/main.go
//go:generate oapi-codegen --config cfg.yaml ../api/openapi3.yaml
package api

// make sure to install:
// go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.3.0

1
2
3
4
5
6


# /internal/api/cfg.yaml
package: api
output: server.gen.go
generate:
 models: true
 echo-server: true

I’m using the echo web framework, you can browse the library documentation to use other frameworks. Now run go generate ./... and it will generate the interfaces (handlers) your web server has to implement to fulfill this contract, including the types.

Interface Implementation

Now it’s time to write the implementation. We create a users package where all the user’s API handlers, business logic, storage, etc. are defined. We will keep it simple and implement the handlers with static content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


type UsersHandler struct {
 DB *sql.DB
}

func (u *UsersHandler) GetUser(ctx echo.Context) error {
 // load the user from the database and return it to the caller
 return ctx.JSON(http.StatusOK, api.User{
 Email: types.Email("demo@devopsian.net"),
 Name: "DemoUser",
 Id: "1",
 })
}

func (u *UsersHandler) PostSignup(ctx echo.Context) error {
 var body api.PostSignupJSONBody
 if err := ctx.Bind(&body); err != nil {
 return ctx.JSON(http.StatusBadRequest, api.Error{Code: http.StatusBadRequest, Message: "invalid request"})
 }
 // save user in database

 return ctx.NoContent(http.StatusOK)
}

func New(db *sql.DB) *UsersHandler {
 return &UsersHandler{DB: db}
}

Next, we need to create our web server entry point, we define that at cmd/server.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


type Server struct {
 users.UsersHandler
}

func main() {
 e := echo.New()
 s := Server{UsersHandler: *users.New(nil)}

 api.RegisterHandlers(e, &s)
 e.Logger.Fatal(e.Start(":8080"))
}

Note I explicitly pass in nil as DB implementation for this example, because we don’t use it.

That’s it. If the code compiles, our server implements the API contract. All the API endpoints are handled by the spec. If I had missed something, it would have broken at compile time.

How nice is that?

Generate a Typescript Client

It’s time to generate a client for our API. We use the same openapi schema file to generate a JS client. I won’t include a full frontend project in this example, but rather show how you can generate a client to an existing one.

In the client/ directory, install the code-generation tool for JS:

npm install openapi-typescript-codegen --save-dev. (This post was tested with v0.29.0)

Create a client/api/ directory, and let’s run the tool:

npx openapi-typescript-codegen --input ../api/openapi3.yaml --output api/ --name ApiClient.

This will generate a bunch of typescript files. To use our client we need to create an instance of it.

1
2
3
4
5
6
7
8


// api.ts
import { ApiClient } from './api/ApiClient'

const client = new ApiClient().default

// client has all the methods of our API:
// - getUser()
// - postSignup(requestBody: {name?: string, email?: string})

That’s it.

Summary

API-First design isn’t just another development buzzword—it’s a powerful approach that can significantly enhance your side projects.

By prioritizing your API design before implementation, you gain clarity, consistency, and efficiency.

The OpenAPI specification is a contract between your frontend and backend, enabling automatic code generation for types, clients, and even testing tools.

This approach not only saves time but also ensures better synchronization between different parts of your application.

While it may seem like extra work upfront, the long-term benefits—including improved development speed, reduced errors, and better API documentation—make it a valuable investment for any side project.

If you haven’t tried it yet, now might be the perfect time to give it a shot and experience these benefits firsthand.

Inside EKS Networking: Decoding the Service IP Journey

Sun, 17 Mar 2024 21:30:00 +0300

Intro

Have you ever found yourself deep in the trenches of Kubernetes networking, only to be surprised by a hidden quirk that challenges your understanding? In this blog post, I unravel the mysteries behind Kubernetes networking in Amazon EKS, shedding light on the intricate journey of a packet from the client through the NLB to an ingress controller pod.

This blog is a story about a change in perception. While I was debugging a problem the other day, it got me to question what I was certain I knew. When you’re sure you understand how Kubernetes works, you encounter another road bump that challenges your understanding and makes you do some research, this article is the output of this research.

The Setup

The minimal setup required to examine what’s presented here. You would need an EKS cluster with NGINX ingress-controller deployed with NLB as the entry to your cluster.

The Problem

I was debugging a service with an ingress resource. I had incoming traffic from NLB to my ingress controller. I was surprised to see my ingress controller wasn’t listening on that port. The AWS console shows those endpoints as ‘healthy’, meaning they respond to health check samples. But how is that even possible if there is no process listening on that port?

I gotta say this drove me nuts. I had looked online for similar issues, but nobody mentioned this problem. So I started doing some research, looking for an answer to the question “How does Kubernetes handle Service IP under the hood, on EKS?”

Exploring Kubernetes Networking Magic

Before I present the network flow of a packet, here are some assumptions I take. Kubernetes is a modular platform; this post is relevant for EKS with the VPC-CNI add-on running on v1.27. It was checked with NLB, other load-balancer types might behave differently.

The next part is low-level. You should be familiar with iptables. Two blog posts that cover this topic greatly are — Laymans iptables 101 and the iptables — a comprehensive guide

When a packet arrives, it first goes through the iptables PREROUTING chain. (That’s a builtin one):

1
2
3
4


[root@...] # iptables -t nat -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 1987 packets, 119K bytes)
 pkts bytes target prot opt in out source destination
 756M 5G KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

This rule captures every incoming packet and forwards it to the “service portals", the KUBE-SERVICES, (that’s a custom chain) where it’s being matched and designated to the relevant Kubernetes Service IP (nginx, in our case).

Incoming packets on a matching port are DNAT’ed to a Service IP. There’s a matching rule for our nginx instance. Incoming packets on its listening port 32443 are sent to its Service IP:

1
2


iptables -t nat -nvL KUBE-SERVICES | grep nginx
0 0 KUBE-SVC-I66WCJWOLI45ORGK tcp -- * * 0.0.0.0/0 172.20.152.36 /* ingress-nginx/ingress-nginx-controller:https-32443 cluster IP */ tcp dpt:32443

Now that our packet has a destination Service IP in the cluster. But service objects in Kubernetes are just a layer of abstraction; there are no actual Pods with such IP. It’s synthetic. Kubernetes networking layer needs to translate this IP to the relevant Pods. This process is done by transforming Service IP to Endpoints. This is also done by iptables.

We have a specific KUBE-SVC-* chain, which is constructed for every service object we have. The purpose of this chain is to translate Service IP to Endpoints (the actual pods behind it). This chain has an entry for every pod alive. This is where the kernel performs ’load balancing’ between the pods.

1
2
3
4
5


iptables -t nat -nvL KUBE-SVC-I66WCJWOLI45ORGK
Chain KUBE-SVC-I66WCJWOLI45ORGK (2 references)
 pkts bytes target prot opt in out source destination
 0 0 KUBE-SEP-TQZ3GYVANOOYIJMO all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https-32443 -> 10.1.21.61:443 */ statistic mode random probability 0.50000000000
 0 0 KUBE-SEP-NEOTPBRCKXF3UWGX all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https-32443 -> 10.1.36.6:443 */

We’re not done with iptables just yet. As you can see from the output, there’s another chain our packet has to go through. We’re getting close. Each chain corresponds to a Pod, with a KUBE-SEP-* chain. If we check the rules of this chain:

1
2
3
4
5


iptables -t nat -nvL KUBE-SEP-TQZ3GYVANOOYIJMO
Chain KUBE-SEP-TQZ3GYVANOOYIJMO (2 references)
 pkts bytes target prot opt in out source destination
 0 0 KUBE-MARK-MASQ all -- * * 10.1.21.61 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https-32443 */
 64 3840 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller:https-32443 */ tcp to:10.1.21.61:443

The first rule is for outgoing packets (SNAT). Incoming traffic matches the second rule. This rule has a DNAT target, which is Destination Network Address Translation. This rule rewrite the TCP packet destination IP to the Pod’s IP: 10.1.21.61:443 Now routing continues as normal, reaching the Pod on the relevant port.

Summary

When a packet arrives, it first goes through iptables PREROUTING, where it’s forwarded to the KUBE-SERVICES chain. This chain directs packets to the relevant Kubernetes Service IP. From there, iptables translates the Service IP to Endpoints using the KUBE-SVC- chains, ultimately reaching the correct pod.

In conclusion, understanding Kubernetes networking in EKS requires delving into iptables manipulations and Kubernetes Service abstractions. I hope this post sheds light on these concepts.

How to Structure a Go Project: Start Simple, Refactor Later

Mon, 05 Feb 2024 22:00:00 +0300

Once upon a codebase, in a kingdom of endless debates, a lone developer pondered the age-old question: “What’s the perfect project structure?” Spoiler alert: there isn’t one. Let’s embark on a quest to discover the charm of simplicity and the art of Go project structuring.

The perennial question about what directory structure should I use echoes across various social platforms now and then. This subject was discussed many times. I’ve been programming in Go for a couple of years now and asked myself this question every time I had to start a new project. If you’re asking it yourself, here are my 2 cents for you.

Why is project structure important?

“A well-organized directory structure is the scaffolding upon which a robust codebase stands tall.”

A project structure is important because it affects readability & maintainability. Think about your future self, six months from now reading your code when it’s not fresh in your memory. If you have structured it right (and there are many ways to do it) it would be easier to jump in. It would also be easier for collaboration as other developers can navigate through it more easily. Most of the time we read and maintain existing code, not writing it from scratch.

What are the common pitfalls of project structure?

There is no such thing as a perfect, one-size-fits-all project structure, so stop looking. It’s like asking for the perfect car. It depends on who you ask, and what are their standards and requirements. You won’t find a single answer, because like many things in computer science — it depends. While there’s no one-size-fits-all solution, some general guidelines can be followed.

Some of the common pitfalls:

Over-engineering: Trying to anticipate all the possible scenarios and use-cases before writing any code. This leads to unnecessary complexity and abstractions, which makes your code harder to read and maintain
Under-engineering: Writing all the code in a single package, without any structure or organization. This leads to spaghetti code base, which makes it harder to test and debug
Copy-pasta: Blindly following the structure of another project, without putting any thoughts or understanding the trade-offs behind it.

How to start simple and refactor later?

My advice is to start simple and let the packages grow organically as you’re writing the code. Don’t worry about creating the perfect packages and abstractions at this point. Just start by writing your code in a single package — main and see how it works. A simple directory tree is easier to read, which reflects that it is easier to maintain and collaborate with a team or individuals.

As you write more code, you’ll notice some patterns emerge and repetitions that you can extract to a different package. This will make your code more readable, maintainable, and testable.

I find it much harder to do the right abstraction before writing code. How do you know which packages do you need? Start writing your business logic. It is easier to refactor code when you see it in front of you. It’s usually easier to abstract some logic from a package, rather than refactoring multiple packages because you got it wrong the first time.

The key is to write code and refactor it later.

The cmd package pattern

A powerful pattern but make sure to ask yourself if you need it

This is a great pattern when necessary. Do you write a library? Or an app with a single binary? Then you probably don’t need to use the cmd package. Ask yourself what this abstraction allows you. Remember — keep it simple. This pattern is useful when your code base is compiled to multiple binaries, for example, if you have a Server and a CLI.

Inside this package, you put the sub-directories that contain your main.go files. Each sub-directory is compiled into its own binary. This is the entry point of your app. You should look into spf13/cobra: A Commander for modern Go CLI interactions project, which is a widely used library for creating powerful modern CLI applications. It blends well with spf13/viper: Go configuration with fangs project, which handles loading configurations nicely.

The internal package

The internal package has a special meaning in Go. Packages that reside under internal/ may not be imported by packages outside the source subtree in which they reside. Therefore, these are said to be internal packages. The code placed here is internal to the project, and can’t be used outside of it.

There used to be times people grouped external packages under pkg/, but I find it meaningless. A directory that exists only to hold other packages is a potential code smell. Instead, give your packages descriptive names. Try to describe what they do, while avoiding ambiguous names like utils or common.

The config package

The config package is responsible for creating and managing the configuration object that your app depends on. It simplifies the process of importing configurations from a single source and accessing them from anywhere in your code base. It also avoids the circular dependency problem that can arise from importing configurations from multiple places.

Let the config package handle your app’s settings, harmonizing inputs from flags, environment variables, or files into a cohesive configuration object.

The API package

The API package defines the interface of your app with the outside world. It contains the schema definition (openapi) and the models (which can be generated from schema) that represent the data types and endpoints of your app. Unless a struct is internal to a specific package and is not used outside of it, it should be placed here. The API package should include any struct that is used by more than one package, or that is exposed to the client or the storage.

The API package is the protocol that enables communication between different components of your app.

Controllers, handlers, and storage

Now, let’s delve into structuring controllers, handlers, and storage. Two approaches stand out:

Approach 1: Storage and Handlers (or controllers) packages

With the first approach, you write your storage and service/controller layers in separate packages where the service imports the storage. This allows you to decouple your business logic from your data access layer and use different storage implementations (postgres, sqlite, redis, etc.) without changing your service code. Here’s a simple view of this approach:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


.
└── project/
 ├── api/
 │ ├── models.go
 │ └── openapi3.yaml
 ├── cmd/
 │ ├── server/
 │ │ └── main.go
 │ └── cli/
 │ └── main.go
 └── internal/
 ├── config/
 │ └── config.go
 ├── storage/
 │ └── postgres/
 │ ├── users.go
 │ └── ...
 ├── handlers/
 │ ├── users.go
 │ ├── users_test.go
 │ ├── storage.go
 │ └── ..
 └── server/
 ├── server.go
 ├── server_test.go
 └── ..

Approach 2: Domain Entity Package

With this approach, you encapsulate logic by functionality so that both your service and storage code reside in the same package. This follows the principle of domain-driven design, where you model your code around the business domain and its entities. Each package represents a domain entity (such as user, post, comment, etc.) and contains all the code related to it (such as internal structs, methods, handlers, queries, etc.)

This is a topic for a separate blog post I might do later on, but in the meantime for more details about this type of architecture check out introduction to DDD or watch Kat Zien great talk from GopherCon 2018.. Here’s a simple view of this approach:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


.
└── project/
 ├── api/
 │ ├── models.go
 │ └── openapi3.yaml
 ├── cmd/
 │ ├── server/
 │ │ └── main.go
 │ └── cli/
 │ └── main.go
 └── internal/
 ├── config/
 │ └── config.go
 ├── users/
 │ ├── postgres/
 │ │ └── db.go
 │ ├── handlers.go
 │ ├── handlers_test.go
 │ ├── storage.go
 │ └── ..
 └── server/
 ├── server.go
 ├── server_test.go
 └── ..

Conclusion

In conclusion, there is no single ‘perfect’ way to structure your Go project. There isn’t a one-size-fits-all. There are multiple ways, and it depends on your use-case, preferences, and the trade-offs you’re willing to take. Experiment based on your needs and requirements.

Embrace simplicity initially. Write code in a single package, observe emerging patterns, and refactor as needed. The key here is to write code first, and ‘perfect’ structure later.

I hope this post gave you some insights and ideas on how to structure your next project. Happy coding!

This is how you want to manage your Terraform modules

Tue, 03 Jan 2023 13:00:00 +0200

What if I told you there is a way to manage all your private terraform modules, in a mono-repo, with independent versioning, without using git tags? After researching for a proper open-source tool, I found the right one for the job.

Intro

A private registry is needed.

I looked for a way to manage private terraform modules like public ones. In my company, we write tailor-maid modules that describe our infrastructure. These modules have to be private. Terraform recommends each module has its own git repository, yet, this has the burden of managing and syncing multiple repositories.

The second option suggested is to use a mono-repo, and reference the module’s version using git tags. At first, this seemed alright, but there was a drawback — you had to give up the terraform version syntax. You no longer can use the convenient version = ~> 1.0.0 module parameter.

It seemed to me that there had to be a better way.

The problem

When it comes down to managing my infrastructure, I prefer the mono-repo approach. Since our terraform modules are small configuration blocks, it makes sense. It is much easier to find a module in a single repository rather than searching multiple repositories for a module. Our module release workflow looked something like this

Our repository was structured with sub-directories per module
We used Git tags for module versioning in the form of moduleName-vX.Y.Z
Updates to module versions were hand-delivered to clients (even minor patches)

Our root modules (where we execute terraform plan and apply) reference modules using git tags:

1
2
3
4
5


terraform {
 source = “https://github.com/my-org/my-repo?ref=s3-v1.0.0"

 .. module inputs ..
}

This has worked well for a while. But once we had released a new module version and wanted to use it, there was no convenient way to apply that.

We had to go through our root modules using this module and update their references. (We could write a simple script, but we decided not to. More on that later)

The problem we had with this approach is, you can’t use the version argument in the terraform block. This argument lets you specify a range of acceptable versions instead of a hard-coded one.

For example, you can provide version constraints such as version = “~> 1.2.0, < 2.0” which allows incrementing the “patch” automatically every time you run terraform init. No need to manually update a patch release, and no need to write a bash script. Terraform can manage that reliably for us, with a better API, if we can only discover a way to use this feature.

Manually managing our tags was chaotic. It is not an easy task to manage independent module versions in a mono-repo using the Git tags approach.

A simpler approach is to release a version for the whole repository. That means every new release includes all the modules together. This couples the modules into a single artifact, which is easier to manage, but at the cost of development velocity. You cannot release minor patches just for your module, no matter how small the code change is.

Towards a better future

The structure and workflows we had in place worked; just as not as well or easily as we wanted to. What we needed was to be able to use terraform versioning syntax, while maintaining our mono-repo. To achieve that, we need to treat our terraform modules as artifacts; something we can archive, version, and release independently.

After doing some research, we found an open-source project called Terralist.

Terralist is a private Terraform registry for providers and modules following the published HashiCorp protocols. It provides a secure way to distribute your confidential modules and providers. That looked like a project that might help us to solve our problem, so we decided to try it out.

So we maintain our mono-repo as it is. We created a job in our CI system that archives a single terraform module and uploads it to Terralist. Since it’s a private registry, clients need to authenticate to be able to download the modules. After authentication (using terraform login <terralist-url>) we could use our private modules just as we use the public modules. Yes, it means we can use the versioning syntax I was talking about.

Here is how we use our modules nowadays

1
2
3
4
5
6


terraform {
 source = “https://my-terralist.com/my-org/s3/aws”
 version = “~> 1.0.0”

 .. module inputs ..
}

This client code will retrieve automatic patch updates on every execution from our CI system (if you execute terraform locally, you would need to re-run terraform init). If we make larger code changes to one of our modules, for example, something that might break the API we would increment the major or minor version, so it wouldn’t impact our clients.

Terralist

Terralist has an API that lets you upload a module from a git repository. It means you don’t need to archive the module yourself, just point Terralist to its location.

Terralist will clone the repository and create the artifact for you. Let’s walk through an example.

I have a demo module, which resides in my terraform.git repository under the modules/demo directory. In the module itself, we keep a version.tf with the major and minor versions of the module. These values change only when we introduce a change that breaks our existing API (e.g, adding a new mandatory parameter without defaults). In such case we update the major or minor manually.

A patch number in the semver represents a safe change, such as a bug fix or non-breaking changes to the module. This value is calculated based on the build number of our CI. We don’t really care about its value, because the version constraint applied is in the form of version = "~> 1.0.0". This automatically updates to the latest patch on every execution.

As part of our module build process, we read the values of version.tf and append the patch. Then we upload the module to Terralist using this API call:

1
2
3


curl -X POST registry.example.com/v1/api/modules/demo/aws/1.0.10/upload \
 -H "Authorization: Bearer x-api-key:$TERRALIST_API_KEY" \
 -d '{ "download_url": "https://github.com/example-org/terraform/archive/refs/heads/master.zip//modules/demo" }'

The module version is composed of major.minor coming from the version.tf file, and the .patch is the build ID, which is incremented with every run and guaranteed to be unique.

Pay close attention to the double // – this instruct Terralist to:

Download an archived repository
Extract it locally
Make an archive only from the path modules/demo
Upload it to the registry (basically upload it to S3 and update the registry database)

It’s an elegant solution: I tell Terralist where my module is, and what version it is tagged with, and it takes care for everything else.

Summary

There are various ways to manage your infrastructure code.

It depends on multiple factors, such as team size, how much you are willing to spend on 3rd party tools, and your company policy, to name a few. As of today, I’ve been using Terraform for more than 3 years, relying solely on the open-source ecosystem. That means my team and I manage everything related to our infrastructure (it might change soon, as our infrastructure size has grown).

Terralist was a helpful addition to our stack; it solved a problem we had in our existing workflow, with minimal effort and no code changes to our existing modules. We just had to upload and version them to the new registry. Now, we can release each module independently and decide if we want our clients to automatically upgrade their module version.

Happy Terraforming.

For further reading, HashiCorp documentation contains a lot of good information. Check out the links below

Go templates: customize your output using templates

Sat, 14 May 2022 11:54:30 +0300

Go templates are a powerful tool to customize output the way you want it. It’s a builtin package implements data-driven templates. Templates are executed by applying them to a data structure.

While there are articles covering the basics, I had a hard time findings material on more advanced use-cases, such as looping over complex structs or using a function in the template. This post aims to distill these advanced use-cases with examples.

If you’re unfamiliar with templates, this blog post covers a great introduction to it.

This is our first use-case: I need to provide configuration to Thanos about the sidecars. It’s basically a list of servers Thanos needs to communicate with. I have hundreds of servers. If you’re not familiar with Thanos, that’s okay. It’s not important for this post. I use it just to show a real-world example.

The template and data

We start by creating a template, then the data it’s gonna execute with. Our output would be a YAML configuration file.

Here is an example of how to iterate a slice and use it in a template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


package main

import (
 "os"
 "text/template"
)

type data struct {
 Locations []string
}

var mydata = &data{
 Locations: []string{"NY", "London", "Tokyo"},
}

var tmplSrc = `---
- targets:
{{- range .Locations }}
 - {{ . }}
{{- end }}
`

func main() {
 tmpl := template.Must(template.New("test").Parse(tmplSrc))
 tmpl.Execute(os.Stdout, mydata)
}

# https://go.dev/play/p/hlZhU40wc0C
# output
# ---
# - targets:
# - NY
# - London
# - Tokyo

Using the range action we iterate an object. In my case, it’s a slice of strings. Inside the {{ range .. }} block we refer to items as .

Passing data to the template

In the previous example, we could simply pass in the slice to the template. But I used a struct. This allows me to extend my template with minimal changes to the code.

At the time of writing, the template engine supports up to 1 argument. If you need to provide multiple values, use a struct.

Another (more complex) alternative is this answer on Stackoverflow.

Let’s say instead of templating the location names, we want their respective IPs. This requires a few changes in our code. We would need to update our data struct to a map, of locations to IPs, and the template code too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


type data struct {
 Locations []string
 IPAddrs map[string]string
}

var mydata = &data{
 Locations: []string{"NY", "London", "Tokyo"},
 IPAddrs: map[string]string{
 "NY": "10.0.0.1",
 "London": "20.0.0.1",
 "Tokyo": "30.0.0.1",
 },
}

var tmplSrc = `---
- targets:
{{- range $location, $ip := .IPAddrs }}
 - {{ $ip }} # {{ $location }}
{{- end }}
`

# output
# ---
# - targets:
# - 20.0.0.1 # London
# - 10.0.0.1 # NY
# - 30.0.0.1 # Tokyo

Here I loop over a map and extract the relevant values. Cool.

Important thing to note is, we define $location and $ip variables in the {{ range .. }} statement. They will be available in the template only in the context of the range block. You can’t use them outside this scope:

A variable’s scope extends to the “end” action of the control structure (“if”, “with”, or “range”) in which it is declared, or to the end of the template if there is no such control structure.

A template invocation does not inherit variables from the point of its invocation.

When execution begins, $ is set to the data argument passed to Execute, that is, to the starting value of dot.

Access multiple fields of a data structure

Our requirements have changed. Now we don’t want to template all the locations of the IPAddrs map, but only the selected ones (enabled).

We need to modify our data. Our data struct contains all the locations (slice) and their relevant IP addresses (map). A better name for our Locations field would be EnabledLocations. We change that accordingly.

Now we want to template an IP address only if the location it resides is enabled. We would need to:

Iterate a slice (EnabledLocations)
Fetch relevant data from the map (IPAddrs)
Template the data in our template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


type data struct {
 EnabledLocations []string
 IPAddrs map[string]string
}

var mydata = &data{
 EnabledLocations: []string{"NY", "Tokyo"},
 IPAddrs: map[string]string{
 "NY": "10.0.0.1",
 "London": "20.0.0.1",
 "Tokyo": "30.0.0.1",
 },
}

var tmplSrc = `---
- targets:
{{- range $location := .EnabledLocations }}
{{- $ip := index $.IPAddrs $location }}
 - {{ $ip }} # {{ $location }}
{{- end }}
`

# https://go.dev/play/p/zOhRXGxoOMV
# output
# ---
# - targets:
# - 10.0.0.1 # NY
# - 30.0.0.1 # Tokyo

I have done a few things here:

Remove London from the EnabledLocations slice
Iterate .EnabledLocations slice with range action and save current element to a variable $location
Define a new variable $ip with the value from a map with a key $location
- Inside the {{ range .. }} clause the scope is changed. . cursor now reference the current item in the loop (in this example, it is equal to $location)
- To access the outer scope, I use $. — that way I can access the IPAddrs map
- index function returns the result of indexing its first argument by the following arguments. In other words, the first argument is the key, and the latter is a map, slice, or array

Let’s complicate things a little more. Let’s say our setup grew with more servers per location. We need our template to support multiple IPs per location. This requires changes to our data structure and template once again.

How can we use a dynamic key to fetch values from our map?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


type data struct {
 EnabledLocations []string
 IPAddrs map[string][]string
}

var mydata = &data{
 EnabledLocations: []string{"NY", "Tokyo"},
 IPAddrs: map[string][]string{
 "NY": []string{"10.0.0.1", "10.0.0.2"},
 "London": []string{"20.0.0.1", "20.0.0.2"},
 "Tokyo": []string{"30.0.0.1", "30.0.0.2"},
 },
}

var tmplSrc = `---
- targets:
{{- range $location := .EnabledLocations }}
{{- $ipList := index $.IPAddrs $location }}
 {{- range $ip := $ipList }}
 - {{ $ip }} # {{ $location }}
 {{- end }}
{{- end }}
`

# output
# ---
# - targets:
# - 10.0.0.1 # NY
# - 10.0.0.2 # NY
# - 30.0.0.1 # Tokyo
# - 30.0.0.2 # Tokyo

Here is what changed:

IPAddrs is a map of string to slice of strings
Define the $ipList variable which contains the relevant string slice (using the index function)
I got 2 range loops now: one loop the locations, the other loop each location’s IP list

You can also call methods inside a template, here’s a clear example from StackOverflow how to do it.

Summary

There are multiple ways to template data. Go provides a rich builtin library worth exploring. Make sure the read the docs before banging your head over syntax, or other builtin capabilities such as index, range, and many more I haven’t covered in this post. Actually, the library is much richer than what I present here. Here are the takeaways:

range action is used to iterate data structures
index function is used to extract data from a map, slice, or array
$var := is used to define a variable and use it inside a template
functions can be executed inside templates

A deep dive to Canary Deployments with Flagger, NGINX and Linkerd on Kubernetes

Sat, 05 Mar 2022 21:45:00 +0200

Intro

Lately, I’ve been checking on progressive delivery tools. The two stars are Argo Rollouts and Flagger. Both projects are pretty mature and widely used.

After researching the two for a few hours, I found out that — like most things in Kubernetes — there is more than one way of doing it.

You can enable it with an ingress controller. Or a ServiceMesh. Or both.

As of the time of writing this blog post, I found all the online tutorials were missing some crucial pieces of information. I’m gonna save you a lot of time here, so bear with me.

Why Flagger

Flagger is a progressive delivery tool that automates the release process for apps on Kubernetes. It can gradually shift traffic to the new version while measuring metrics and running conformance tests.

I prefer flagger because of two main points:

It integrates natively: it watches Deployment resources, while Argo uses its own CRD Rollout
It is highly extendible and comes with batteries included: it provides a load-tester to run basic, or complex scenarios

When you create a deployment, Flagger generates duplicate resources of your app (including configmaps and secrets). It creates Kubernetes objects with <targetRef.name>-primary and a service endpoint to the primary deployment.

Flagger is triggered by changes to the target deployment (including secrets and configmaps) and performs a canary rollout and analysis before promoting the new version as the primary.

Read “How Flagger works” to better understand this flow.

There are multiple techniques of Progressive Delivery:

Canary - is a deployment strategy introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to all
A/B testing - is a deployment strategy that lets you try a new version of the application in a limited way in the production environment
Blue/Green - is a deployment strategy that utilizes two identical environments, a “blue” (aka staging) and a “green” (aka production) environment with different versions of an application or service. The new version can be tested, approved, then supersede the stable (green) production environment

In this blog post, I focus on Canary. Canary covers simple and sophisticated use-cases. I will dive into how this actually works, and fill in the missing pieces I had to solve myself.

Prerequisites

If you wanna try this out, you’ll need:

A Kubernetes cluster v1.19+ and NGINX v1.0.2+ (I use aws EKS here)
Linkerd v2.10+ (optional, only if you go for the service mesh solution)

NGINX (Ingress) Canary Deployment

NGINX provides Canary deployment using annotations. With the proper configuration, you can control and increment the number of requests to a different service than the production one.

Deploy nginx

Deploy NGINX ingress controller if you don’t have one already. It has to be monitored by Promethues, hence the podAnnotations:

1
2
3
4
5
6
7


helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
kubectl create ns ingress-nginx
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254

Install Flagger and set it with nginx provider. Flagger can bring Prometheus with it, if you don’t have one installed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Install w/ Prometheus to collect metrics from the ingress controller
helm upgrade -i flagger flagger/flagger \
--namespace ingress-nginx \
--set crd.create=true \
--set prometheus.install=true \
--set meshProvider=nginx

# Or point Flagger to an existing Prometheus instance
helm upgrade -i flagger flagger/flagger \
--namespace ingress-nginx \
--set crd.create=true \
--set metricsServer=http://prometheus.monitoring:9090 \
--set meshProvider=nginx

Gotcha: If you are using an existing Prometheus instance, and it is running in a different namespace, you can’t use the prebuilt metrics. You’ll encounter “no values found for nginx metric request-success-rate” issue. You need to create your own template, check this issue

Create a test namespace and install load testing tool to generate traffic during canary analysis:

1
2
3
4


kubectl create ns test

# Install the load testing tool
kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Deploy our example app — podinfo. Create an ingress resource too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98


# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: podinfo
 labels:
 app: podinfo
spec:
 minReadySeconds: 5
 revisionHistoryLimit: 5
 progressDeadlineSeconds: 60
 strategy:
 rollingUpdate:
 maxUnavailable: 1
 type: RollingUpdate
 selector:
 matchLabels:
 app: podinfo
 template:
 metadata:
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "9797"
 labels:
 app: podinfo
 spec:
 containers:
 - name: podinfod
 image: stefanprodan/podinfo:3.1.0
 imagePullPolicy: IfNotPresent
 ports:
 - name: http
 containerPort: 9898
 protocol: TCP
 - name: http-metrics
 containerPort: 9797
 protocol: TCP
 - name: grpc
 containerPort: 9999
 protocol: TCP
 command:
 - ./podinfo
 - --port=9898
 - --port-metrics=9797
 - --grpc-port=9999
 - --grpc-service-name=podinfo
 - --level=info
 - --random-delay=false
 - --random-error=false
 env:
 - name: PODINFO_UI_COLOR
 value: "#34577c"
 livenessProbe:
 exec:
 command:
 - podcli
 - check
 - http
 - localhost:9898/healthz
 initialDelaySeconds: 5
 timeoutSeconds: 5
 readinessProbe:
 exec:
 command:
 - podcli
 - check
 - http
 - localhost:9898/readyz
 initialDelaySeconds: 5
 timeoutSeconds: 5
 resources:
 limits:
 cpu: 2000m
 memory: 512Mi
 requests:
 cpu: 100m
 memory: 64Mi
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: podinfo
 namespace: test
 labels:
 app: podinfo
 annotations: {}
spec:
 ingressClassName: "nginx"
 rules:
 - host: podinfo.local
 http:
 paths:
 - backend:
 service:
 name: podinfo
 port:
 number: 9898

Note that I use http://podinfo.local as the URL for this service. Use it or change it. You can’t use the kubectl port-forward **to access it.

Next we create the Canary resource. This defines how we roll out a new version, how Flagger performs its analysis and optionally run tests on the new version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: nginx
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # ingress reference
 ingressRef:
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 80
 # container port number or name
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # NGINX Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary/"

For details on the settings defined here, read this.

In short, during a rollout of a new version, we do acceptance-test and load-test. Based on the metrics, Flagger decides if it should keep rolling out the new version, halt, or rollback.

KubeView is a Kubernetes cluster visualizer. It displays and maps out the API objects and how they are interconnected. This is how our Kubernetes test namespace looks like:

Flagger created the service resources and another ingress — podinfo-canary. The special thing about that ingress is it is annotated with canary properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


➜ kubectl describe ingress/podinfo-canary
Name: podinfo-canary
Namespace: test
Address: xxx
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
 Host Path Backends
 ----  ----  --------
 podinfo.local
 / podinfo-canary:9898 (<none>)
Annotations: nginx.ingress.kubernetes.io/canary: true
 nginx.ingress.kubernetes.io/canary-weight: 0

We have no deployment going on, so the canary-weight is 0.

Accessing our app shows:

OK — Let’s deploy a new version of our app and see how it rolls:

1
2


kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.3

This updates a deployment, which triggers Flagger, which updates our Canary and Ingress resources:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


➜ kubectl get canaries -A
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 0 2022-03-04T16:18:05Z

➜ kubectl describe ingress/podinfo-canary
Name: podinfo-canary
Namespace: test
Address: xxx
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
 Host Path Backends
 ----  ----  --------
 podinfo.local
 / podinfo-canary:9898 ()
Annotations: nginx.ingress.kubernetes.io/canary: true
 nginx.ingress.kubernetes.io/canary-weight: 5 <-- this was changed by Flagger

It brought up a new version of deploy/podinfo with podinfo-canary Ingress that points to a service with the same name. (unfortunately, the podinfo-canary isn’t mapped to the service in the picture). The setup looks like this:

We can see some of our requests being served by the new version:

Flagger slowly shifts more traffic to the Canary, until it reaches the promotion stage

It then updates the deployment/podinfo-primary to mark the Canary as the primary, or stable version:

Once the promote step is done, Flagger scales down podinfo deployment.

For reference, you can read more about NGINX Canary annotations and the queries source code Flagger uses to check the NGINX metrics.

NGINX has advanced configurations for Canary, such as nginx.ingress.kubernetes.io/canary-by-header and nginx.ingress.kubernetes.io/canary-by-cookie annotations for more fine-grained control over the traffic reaches to Canary. Check out the documentation.

Using NGINX for Canary controls only traffic coming from an Ingress (outside your cluster). It means service-to-service communication is never going to reach the Canary version during the rollout. If that’s a requirement, check the Linkerd solution below

Linkerd (ServiceMesh) Canary Deployment with Ingress support

Linkerd provides Canary deployment using ServiceMesh Interface (SMI) TrafficSplit API.

The specification states that:

“It will be used by clients such as ingress controllers or service mesh sidecars to split the outgoing traffic to different destinations.”
“For any clients that are not forwarding their traffic through a proxy that implements this proposal, the standard Kubernetes service configuration would continue to operate.”

— What this means is, for Canary to work the Pods involved have to be meshed.

Linkerd’s traffic split functionality allows you to dynamically shift arbitrary portions of traffic destined for a Kubernetes service to different destination service. But how?

Flagger updates the weights in the TrafficSplit resource and linkerd takes care of the rest. In a meshed pod, linkerd-proxy controls the in and out the traffic of a Pod. It can mutate and re-route traffic. It is sort of the “router” of the Pod*.*

Linkerd is the implementation detail here. It watches the TrafficSplit resource and shapes traffic accordingly.

The main points to note using a Service Mesh for Canary:

It works only for meshed Pods. Non-meshed Pods would forward / receive traffic regularly
If you want ingress traffic to reach the Canary version, your ingress controller has to have meshed

Deploy linkerd

Let’s see an example (based on this one from the official docs). I will use podinfo as our example app.

Install linkerd and flagger in linkerd namespace:

1
2
3


linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -
kubectl apply -k github.com/fluxcd/flagger//kustomize/linkerd

Install or patch NGINX controller:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Install NGINX
helm upgrade -i nginx-ingress ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254 \
--set controller.podAnnotations."linkerd\.io/inject"=enabled

# Or upgrade an existing NGINX
helm upgrade nginx-ingress ingress-nginx/ingress-nginx \
--reuse-values \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254 \
--set controller.podAnnotations."linkerd\.io/inject"=enabled

Create a test namespace, enable Linkerd proxy injection and install load testing tool to generate traffic during canary analysis:

1
2
3
4
5


kubectl create ns test
kubectl annotate namespace test linkerd.io/inject=enabled

# install the load testing tool
kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Before we continue, you need to validate both ingress-nginx and the flagger-loadtester pods are injected with the linkerd-proxy container.

Now to the cool parts. Deploy the app by applying the following yaml files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103


# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: podinfo
 labels:
 app: podinfo
spec:
 minReadySeconds: 5
 revisionHistoryLimit: 5
 progressDeadlineSeconds: 60
 strategy:
 rollingUpdate:
 maxUnavailable: 1
 type: RollingUpdate
 selector:
 matchLabels:
 app: podinfo
 template:
 metadata:
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "9797"
 labels:
 app: podinfo
 spec:
 containers:
 - name: podinfod
 image: stefanprodan/podinfo:3.1.0
 imagePullPolicy: IfNotPresent
 ports:
 - name: http
 containerPort: 9898
 protocol: TCP
 - name: http-metrics
 containerPort: 9797
 protocol: TCP
 - name: grpc
 containerPort: 9999
 protocol: TCP
 command:
 - ./podinfo
 - --port=9898
 - --port-metrics=9797
 - --grpc-port=9999
 - --grpc-service-name=podinfo
 - --level=info
 - --random-delay=false
 - --random-error=false
 env:
 - name: PODINFO_UI_COLOR
 value: "#34577c"
 livenessProbe:
 exec:
 command:
 - podcli
 - check
 - http
 - localhost:9898/healthz
 initialDelaySeconds: 5
 timeoutSeconds: 5
 readinessProbe:
 exec:
 command:
 - podcli
 - check
 - http
 - localhost:9898/readyz
 initialDelaySeconds: 5
 timeoutSeconds: 5
 resources:
 limits:
 cpu: 2000m
 memory: 512Mi
 requests:
 cpu: 100m
 memory: 64Mi
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: podinfo
 namespace: test
 labels:
 app: podinfo
 annotations:
 nginx.ingress.kubernetes.io/service-upstream: "true"
 nginx.ingress.kubernetes.io/configuration-snippet: |
 proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:9898;
 proxy_hide_header l5d-remote-ip;
 proxy_hide_header l5d-server-id;
spec:
 ingressClassName: "nginx"
 rules:
 - host: podinfo.local
 http:
 paths:
 - backend:
 service:
 name: podinfo
 port:
 number: 9898

Gotcha: By default, the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service’s Cluster IP and port.

The nginx.ingress.kubernetes.io/configuration-snippet annotation rewrites the incoming header to the internal service name (required by Linkerd).

Next we enable Canary for our deployment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


# canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 20s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Linkerd Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"

For details on the settings defined here, read this.

In short, during a rollout of a new version, we do acceptance-test and load-test. Based on the metrics, Flagger decides if it should keep rolling out the new version, halt or rollback.

If you got up here, your setup should look like

Accessing our app shows:

OK — We are all set. Let’s roll out a new version.

1
2


kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.1

This updates a deployment, which triggers Flagger, which updates our Canary resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


➜ kubectl -n test describe canary/podinfo
...
Status:
 Canary Weight: 5
 Conditions:
 Last Transition Time: 2022-03-04T22:18:55Z
 Last Update Time: 2022-03-04T22:18:55Z
 Message: New revision detected, progressing canary analysis.
 Reason: Progressing
 Status: Unknown
 Type: Promoted
 Failed Checks: 0
 Iterations: 0
 Last Applied Spec: 6ff7d4d4c
 Last Transition Time: 2022-03-04T22:23:15Z
 Phase: Progressing
 Tracked Configs:
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 4m26s flagger New revision detected! Scaling up podinfo.test
 Warning Synced 36s flagger canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available
 Normal Synced 26s flagger New revision detected! Restarting analysis for podinfo.test

We can see Flagger created a new Deployment, and started pointing traffic to it:

Our Canary deployment starts serving traffic gradually:

If everything goes well, Flagger will promote our new version to become primary. The status looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


➜ kubectl -n test describe canary/podinfo
...
Status:
 Canary Weight: 0
 Conditions:
 Last Transition Time: 2022-03-04T24:20:35Z
 Last Update Time: 2022-03-04T24:20:35Z
 Message: Canary analysis completed successfully, promotion finished.
 Reason: Succeeded
 Status: True
 Type: Promoted
 Failed Checks: 0
 Iterations: 0
 Last Applied Spec: 6ff7d4d4c
 Last Transition Time: 2022-03-04T24:20:35Z
 Phase: Succeeded

Summary

Flagger is a powerful tool. It allows safer software releases by gradually shifting the traffic and measuring metrics like HTTP/gRPC. Besides the built-in metrics analysis, you can extend it with custom webhooks for running acceptance and load tests.

It integrates with multiple Ingress controllers and Service Meshes.

In these modern times where successful teams look to increase software releases velocity, Flagger helps to govern the process and improve its reliability with fewer failures reaching production.

While both NGINX and Linkerd can serve Flagger, these are the tradeoffs I found:

NGINX
- Makes the process simpler with fewer components. While everybody uses Ingress, not all of us needs a Service Mesh
- Service-to-service communication, which bypasses Ingress, won’t be affected and never reach the Canary
Linkerd
- Pretty easy Service Mesh to setup with great Flagger integration
- Controls all traffic reaching to the service, both from Ingress and service-to-service communication
- For Ingress traffic, requires some special annotations

That’s it for today. Hope you had some insights and a better understanding of this problem.

Shout out your thoughts on Twitter (@c0anidam). Stay humble, be kind.

Terraform: why data sources and filters are preferable over remote state

Tue, 01 Feb 2022 09:30:00 +0200

Quite often you need to share data or output resources between your Terraform modules. Fundamental modules that build the infrastructure have no dependencies. As your infrastructure grows, the dependencies are inevitable. A frequently used module is the VPC.

Almost every resource (if not all of them) requires a VPC to be placed in. When you look up how to share resources between your modules, you’re led to articles and examples of terraform_remote_state. This solves the problem, but in my opinion, there is a better alternative. By better, I mean, more stable.

Whenever you need to share the state between modules, your first choice should be terraform data sources. terraform_remote_state should be the alternative when the first is not achievable.

Use the data “aws_*” resource

Data sources allow Terraform to use the information defined outside of Terraform, defined by another separate Terraform configuration, or modified by functions.

Using data sources with Terraform is a good design choice. It uses AWS API to fetch resources based on names, filters (tags), etc. instead of hard-coding them in the module or fetching them from a remote state.

Data sources are more accurate. They are always up to date. The code will always check the active resource and won’t depend on the module which created it.

Let’s look at a couple of use cases.

S3 bucket

If your module requires an S3 bucket name or ID as input, it is as simple as that to provide it using the relevant data source:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


data "aws_s3_bucket" "this" {
 bucket = "my-shiny-bucket"
}

module "app" {
 source = "../modules/app"

 .. removed for brevity ..

 s3_bucket = data.aws_s3_bucket.this.id
}

AWS account ID

How many times have you needed to provide your account ID to configure a policy? instead of copy-pasting this value between your module, simply fetch it using a data source:

1
2
3
4
5
6
7
8
9


data "aws_caller_identity" "this" {}

module "app" {
 source = "../modules/app"

 .. removed for brevity ..

 aws_account = data.aws_caller_identity.this.account_id
}

Time to look for the more advanced features.

Filters

Data sources support the filter block, to fetch specific resources. A filter allows you to filter the results returned from a data source call. The use-case I’m using for an example is updating routing tables.

You have your VPC, which contains multiple subnets. You have a task to add a route to these routing tables in the staging environment.

How would you get the list of the routing tables? What if you would like to update just the ones related to private subnets? Does your solution still work?

Let’s look at some code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# main.tf
# -------
data "aws_route_tables" "rts" {
 vpc_id = var.vpc_id

 dynamic "filter" {
 for_each = var.rts_filters
 content {
 name = filter.value.name
 values = filter.value.values
 }
 }
}

resource "aws_route" "r" {
 count = length(data.aws_route_tables.rts.ids)

 route_table_id = tolist(data.aws_route_tables.rts.ids)[count.index]
 destination_cidr_block = "10.0.1.0/22"
 vpc_peering_connection_id = "pcx-45ff3dc1"
}

# terraform.tfvars
# ----------------
rts_filters = [{
 name = "tag:Environment"
 values = ["staging"]
 },
 {
 name = "tag:Name"
 values = ["*private*"]
 }]

The example assumes your resources are tagged with Environment = staging, and contains the string private in their name (e.g, us-east-private-1a, us-east-private-1b, ..)
A dynamic block acts much like a for expression but produces nested blocks instead of a complex typed value. It iterates over a given complex value and generates a nested block for each element of that complex value.
- Using this block, you can support multiple filters. This is translated to multiple filter blocks
- I’m using the tag:.. key, but there are other filters you can use depending on the resource type
Using filter.value we have access to the value of the current element
In the example, I’m adding a route to 10.0.1.0/22 through VPC peering. For additional config options, check aws_route docs

This syntax feels a bit strange at first until you get used to it, and is very powerful.

Summary

In the world of programming or IaC, there is more than one way to solve a problem. In this post I wrote my preference after adopting the use of terraform data sources, rather than using terraform_remote_state. The latter has its use. I find it more complex and fragile to changes.

The takeaway from this is, the statement at the top of the article:

Whenever you need to share the state between modules, your first choice should be terraform data sources. terraform_remote_state should be the alternative when the first is not achievable.

Any thoughts or comments are welcome.

Practical unit-testing web client in Go - part 2

Sat, 02 Oct 2021 23:01:00 +0300

Testing webserver with TLS

In the previous post, I overviewed how to practically test web client in go.

The next step, I want to test a web server with self-signed certificates. To cover this use-case, we need to:

Create self-signed certificates, with SAN property of 127.0.0.1.
Start the testing webserver using this certificate.
Configure my client to trust the RootCA.

Most of the people, when they reach this stage of testing, usually bend corners. You can simply disable the TLS verification with &tls.Config{InsecureSkipVerify: true} and be done with it.

I can’t. Can’t treat security precautions with no care; this is a package that works with a Secret Manager. In my case, testing the client works with self-signed certificate is significant.

Create self-signed certificates

Luckily, there’s a great github repository that covers the creation of certificates: https://github.com/jcbsmpsn/golang-https-example

The repo contains solutions to common issues raised when trying to achieve this task. If you encounter any errors during this procedure, go check it out. To create certificates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


# Create CA key + certificate
openssl req \
 -newkey rsa:2048 \
 -nodes \
 -days 3650 \
 -x509 \
 -keyout ca.key \
 -out ca.crt \
 -subj "/CN=*"

# Create server certificate request and key
openssl req \
 -newkey rsa:2048 \
 -nodes \
 -keyout server.key \
 -out server.csr \
 -subj "/C=GB/ST=London/L=London/O=libvault consultants/OU=IT Department/CN=*"

# Sign the server certificate request with the CA key
# adding SAN IP
openssl x509 \
 -req \
 -days 365 \
 -sha256 \
 -in server.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out server.crt \
 -extfile <(echo subjectAltName = IP:127.0.0.1)

Running the above code results in 2 sets of certificates: ca and server certificates. Now let’s use them.

Start webserver with TLS

There’s no single API to start the test server with TLS configuration, so we need to do it in few steps. First I’ll create a unstarted test server, load the x509 keypairs and configure the server to use them. Only then I start the web server. In code, it looks like this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


const (
 caCertPath = "testdata/certs/ca.crt"
 serverCertPath = "testdata/certs/server.crt"
 serverKeyPath = "testdata/certs/server.key"
)

func TestClientLogin(t *testing.T*) {
 mux := http.NewServeMux()
 ts := httptest.NewUnstartedServer(mux)

 cert, err := tls.LoadX509KeyPair(serverCertPath, serverKeyPath)
 if err != nil {
 // handle error
 }
 // setp the server's TLS configurations
 ts.TLS = &tls.Config{Certificates: []tls.Certificate{cert}}

 // start the server with TLS support
 ts.StartTLS()
 ...

Configure the http.Client to trust my RootCA

Now we have a webserver that uses our own self-signed certificate. Cool. Remember the CA key and certificate we created earlier? Now we need to configure our client to trust certificates signed by our CA.

My client wraps the http.Client structure. To simplify things, here I’ll work straight with the http.Client struct.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


package main
import (
 "crypto/tls"
 "crypto/x509"
)

var (
 caCertPath = "testdata/certs/ca.crt"
 certPool = x509.NewCertPool()
)

caCert, _ := ioutil.ReadFile(caCertPath)
ok := certPool.AppendCertsFromPEM(caCert)
if !ok {
 // error handling
}


tlsCfg := &tls.Config{
 RootCAs: certPool,
}

client := &http.Client{
 Transport: &http.Transport{
 TLSClientConfig: tlsCfg,
 }
}

...

Now, requests made from my client to my secured test webserver are validated.

Practical unit-testing web client in Go

Fri, 28 May 2021 10:45:00 +0300

I’ve started a Go project, a lightweight Hashicorp Vault¹ client with no dependencies, and a simple API (for the user). Part of the reason I use no 3rd party modules is, I want to better understand Go internals, structure, and improve my skills. In this post, I’ll conduct a practical example of how I ended up testing it.

The project’s name is libvault, and it’s my first open-source project.

Vault is a secret manager service with web API and CLI. Applications can communicate with it through HTTP. I find the CLI easy to use, however, the official Vault library is a bit more complicated; it felt like a swiss-army knife when all I needed was just a simple kitchen knife. I decided to implement a light version, that covers basic functionality while maintaining a simple API.

I’m not going to cover the basics in this post. The intention of it is to be practical and provide real-life, working examples. The opinions here are mine. It worked for me, still, it doesn’t mean it would work for you.

I removed ALL the error handling from the code for brevity. Please make sure you handle your errors.

The Go standard library provides a really good testing package. You can manage without external frameworks or 3rd party packages. In my scenario, I need to test a web client that I wrote. The technique I found useful is mock testing.

Mock testing is an approach to unit testing that lets you make assertions about how the code under test is interacting with other system modules. In mock testing, the dependencies are replaced with objects that simulate the behaviour of the real ones. … Such a service can be replaced with a mock object. ~ Wikipedia

I had to choose which component to mock:

Client
Webserver

At a high level, mocking the client means creating a new struct that implements the interface you are testing (mocking the interface). Then, provide your mocking client to the code under test. A good library with examples is testify.

I didn’t find it useful for my use case, as I need to mock the server-side. I prefer not to modify any code on the client if I can. This would result in more reliable tests for my package. So, I’ve chosen to go with the second option. Read on for how.

The httptest package

Go is very friendly to web services; it has a utility package (httptest) for testing an http server. You can simply start a webserver in your testing code.

First I had to fetch an exact response from a real webserver (Vault server), then I could easily mock it using this package. For example, when querying the /v1/auth/approle/login endpoint, the response looks like this:

1
2
3
4
5
6
7


{
 "request_id": "de7c8097-1a38-50a6-b971-fe1836840e45",
 "lease_id": "",
 "renewable": false,
 ...
 ...
}

Now that I have the content, I find it easier to save it to a file (rather than put it inside the code). Go testing package has another cool feature:

The go tool will ignore a directory named “testdata”, making it available to hold ancillary data needed by the tests.²

I created a directory named testdata inside my project and placed the JSON content in a file - approleExample.json.

Start the mocking server

I’ll start with code, followed by an explanation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


package main

import (
 "fmt"
 "io"
 "net/http"
 "net/http/httptest"
)

func TestClientLogin(t *testing.T*) {
 mux := http.NewServeMux()
 ts := httptest.NewServer(mux)
 defer ts.Close()

 mux.HandleFunc("/v1/auth/approle/login", func(w http.ResponseWriter, r *http.Request) {
 // request validation logic
 ...
 // read json response
 jsonPayload, _ := ioutil.ReadFile("testdata/approleExample.json")
 w.Header().Set("Content-Type", "application/json")
 w.WriteHeader(200)
 w.Write(jsonPayload)
 })

 // initalize new client pointing to the testing server
 client, err := NewClient(ts.URL)
 if err != nil {
 // error handling
 }

 // test logic
}

If you’re familiar with the Go http package, this code is pretty self-explanatory. This is another advantage of using the standard library - you have one less thing to learn when you want to contribute.

mux is of ServeMux type. Which is an HTTP request multiplexer. It matches the URL of each incoming request (/v1/auth/approle/login) against a list of registered patterns and calls the handler for the pattern that most closely matches the URL (our mux.HandleFunc function body).
ts is of Server type. A Server is an HTTP server listening on a system-chosen port on the local loopback interface, for use in end-to-end HTTP tests. Note that I use httptest.NewServer to initialize it.
mux.HandleFunc(..) defines a path and an handler (a function) to call. The content inside describes our server’s response.
client, err := NewClient(ts.URL) creates a new Client (my Vault client), providing it the test server URL to work with.

The elegance of this pattern is, every test case has its test webserver with all the relevant configurations. The mocked content, the test logic, etc. are all implemented inside the test itself. This really makes life easier when debugging, reviewing a test case logic or coverage.

We can improve it further, make the code more clear and concise. This code includes some boilerplate: creating the mux, the server, and reading the json content. We just need to refactor these elements out (to a setup() function, and readJson(path string), for example). Then call this setup() function for every test case. I’ll leave that to the reader to decide.

Summary

There are numerous articles about the importance of software testing. Go makes it easier. You should always write tests for your packages; it has so many advantages to just skip it. However, many people do that and I can assume the reason, which is it doesn’t provide any additional functionality to your code. Don’t be one of these people.

Go standard library provides great tools, and you should use them. Personally, I prefer it over other dependencies.

In this post, I gave a practical example of how you can unit-test a web client by mocking it.

The takeaways:

Always write tests. They are too valuable to give up and very easy to do with Go.
Mock the part your code connects with, don’t mock your code. If you write a server, mock the client, and vice-versa. It makes your tests much more reliable.
1. If you mock a server, get a real server response and save it to a file
2. Do that for every API you would like to mock
Use testdata directory to hold your ancillary data needed by your tests

In the next post, I will cover how to test with a TLS (HTTPS) web server and self-signed certificates.

How to use SED in a Jenkins Pipeline

Sat, 08 May 2021 20:37:30 +0200

Have you ever encountered the need to modify a single line in a file? While there are various utilities for this task, I consider sed to be superior to other alternatives.

sed is a stream editor, with which you can modify files. It’s a great utility in your toolbox. If you’re unfamiliar with it, here’s a link for a quick intro.

I had to use it in one of my Jenkins Pipelines. Since escaping commands in a pipeline is a nightmare, I had to debug it for a while until I got it working. If you have a similar task, this might save you some precious time.

The requirement was to update a line in a large file (couple hundreds of lines), with an arbitrary string provided as an input.

An elegant way of doing it with sed is to capture the known prefix, up to a delimiter and then replace the whole thing comes next (until the end of line). For example, we have the following file:

1
2
3
4


app: myapp
env: dev
version: 1.0
..

Now we want to update the version part from Jenkins. How can we do it? When working with Jenkins, I recommend do it verbosly. Print the stuff before and after you change, it would be much easier to debug later on. This is how I decided implementing it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sh """#!/bin/bash
 # Print current version
 cat app.yml | grep version

 # Update version using sed
 sed -i 's|^version: .*|^version: "${VERSION}"|' app.yml

 # Print updated version
 cat app.yml | grep version
 """

Using sh module, with """ ... """ I can escape the escaping nightmare of groovy. Otherwise I would have to put multiple \\\\ in the command, up to the point it’s barely readable. I also include #!/bin/bash, to make sure I’m using bash shell.

Let’s break down the sed command.

-i means do the replacement in-place; meaning sed will update the file. You can provide a suffix after the parameter, and sed will create a backup version of the file before doing the update.
's|..#1..|..#2..|' - s is for substitute, #1 is the string, or pattern to look for and #2 is the replacement. The ${VERSION} variable is defined by Jenkins.
for #1, I’m looking for the prefix “version:” and then using regex to capture whatever comes next.
for #2, I’m putting the full string I expect to be included in the final file.

There are other ways to achieve the same goal, but I find it a good practice to print stuff before and after an automatic job changes something in a file.

Feel free to integrate and adapt this approach to your projects. It’s a clean and efficient way to handle file updates within a Jenkins pipeline.

How to share persistent storage volumes in Swarm

Mon, 22 Feb 2021 23:47:30 +0200

Docker swarm is an orchestration tool, similar to Kubernetes, but simpler to set up and manage. A swarm consists of multiple Docker hosts which run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services). A given Docker host can be a manager, a worker, or perform both roles.

The docker swarm feature is embedded in the Docker Engine (using swarmkit). This means you don’t need to install extra packages to use it. You just docker. If you decided to put it in place, one of the open problems to address is the persistent storage.

In the old days, many processes could share a local disk on the host. It was a common pattern to have a process that writes to files, and another that consumes them. The two processes were living happily on the same host, having no problems working together. But it’s another story in the containerized world. When containers are managed by an orchestrator, you can’t really tell where your container would be scheduled to run. It won’t have a permanent host, neither you want to lock it to a specific host, as you lose many of the orchestrator benefits (resiliency, fail-over, etc.).

The Problem: Data Persistency on the Swarm cluster

Suppose you have two services that need to share a disk, or a service that requires data persistency such as a Redis. What are your options?

One easy option is to use docker volumes. It’s a good option if you run on a single node because when you create a new docker volume, it resides on the host it was created on.

But what happens when it runs on a cluster? good question. Let’s walk through an example. I’m going to one from the use the Docker Documentation (Get Started with Docker Compose)

Here are the files I’m using for this example:

1
2
3
4
5
6


devopsian: ~/demo
➜ tree
.
 |-app.py
 |-Dockerfile
 |-docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# app.py
import time

import redis
from flask import Flask

app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)

def get_hit_count():
 retries = 5
 while True:
 try:
 return cache.incr('hits')
 except redis.exceptions.ConnectionError as exc:
 if retries == 0:
 raise exc
 retries -= 1
 time.sleep(0.5)

@app.route('/')
def hello():
 count = get_hit_count()
 return 'Hello World! I have been seen {} times.\n'.format(count)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Dockerfile
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
RUN pip install -r flask redis
EXPOSE 5000
COPY . .
CMD ["flask", "run"]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# docker-compose.yml
version: "3.7"
services:
 app:
 build: .
 ports:
 - "5000:5000"
 redis:
 image: "redis:alpine"
 volumes:
 - "redisdata:/data"

volumes:
 redisdata:
 driver: "local"

Suppose you have a small cluster of 3 nodes. (Their roles in the cluster don’t matter for example). You run your docker-compose file with your app and a Redis instance, and define a volume for it. The first time you run your docker-compose, docker creates the volume and mounts it to your service on startup.

Now you want to deploy a new version of your service. The cluster decides to schedule the Redis instance on a different node than before. What will happen to your volume?

Since the volume doesn’t exist on the node the service runs at, docker would create a new volume with the same name on the new node. The previous volume still exists with the data but, it resides on the old node. The Redis service has no access to it. Redis now actually has a new, empty volume for use. You end up in an inconsistent state. This solution doesn’t work for us.

AWS Elastic File System (EFS)

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed elastic NFS file system for use .. It is built to scale on-demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

EFS provides a NFS volume you can mount at runtime. The swarm cluster schedules to run the service on one of its nodes and mounts the NFS volume to the container.

With these configurations, if the cluster decides to move our Redis service to a different node (due to a failure, deployment, etc.), the mount (and hence the data) will move to the new node too. I won’t go through how to create an EFS volume, you can find the steps here on aws tutorial.

To achieve that, we need to install the nfs-common package on the swarm nodes. On Ubuntu, you can install it with: sudo apt-get install nfs-common

Next, we will update the volumes definition of our docker-compose, with the new driver type and the address. It looks as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# docker-compose-with-efs.yml
version: "3.7"
services:
 app:
 build: .
 ports:
 - "5000:5000"
 redis:
 image: "redis:alpine"
 volumes:
 - "redis-efs:/data"

volumes:
 redis-efs:
 driver: local
 driver_opts:
 type: nfs
 o: addr=fs-1224ea45.efs.us-east-1.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
 device: "fs-1224ea45.efs.us-east-1.amazonaws.com:/"

When we run our app now, the Redis data volume is persistent. It will move with it to whatever node Redis runs on.

This solves our original problem. We now have a way to persist data across our cluster.

Costs

This blog post uses AWS as the cloud infrastructure. When comparing EFS, the closest service AWS I could find is EBS volumes. I assume you’re familiar with it.

You should use these pricing calculators from AWS to tailor the price for your use case.

These are not apples to apples, but let’s do a quick price comparison between the two. We will compare a 100GB volume in the EU region.

EBS

Type	Charged for	Price
Storage	0.088$/GB * 730 * 100	$64.24
IOPS	3K - included	$0
Throughput	125MB/s - included	$0

You can increase the throughput or IOPS, with additional costs.

EFS

With EFS, the charges are different (shocked ah?). You pay for “Standard Storage Class” which is designed for active file system workloads.

You pay a different price for “Infrequent Access Storage Class” (IA) which is cost-optimized for files accessed less frequently. For this comparison, let’s estimate 50% of the data is frequently accessed.

Type	Charged for	Price
Standard Storage	0.33$/GB monthly * 50	$16.5
IA Storage	0.025$/GB monthly * 50	$1.25
Throughput	2.5MB/s included, 6.60$ per additional MB/s	$0

The throughput part is kinda tricky. You get 50KB/s per Stand Storage GB (50GB * 50KB = 2.5MB/s) for write operations and 150KB/s (50GB * 150KB = 7.5MB/s) for read operations.

As you see, there is a big performance impact when comparing EBS with EFS. If you need high throughput, EFS might get very expensive.

Conclusion

The problem I wrote about in this blog is a known problem when running microservices on a managed cluster using an orchestration tool. As with other problems, it has more than one solution where each has its trade-offs.

I showed that EFS is a convenient, easy solution you can use to solve the shared storage problem. Yet, it is not the right solution for everything. The performance is limited, and it can get very expensive if your app requires high throughput. In those cases, you would want to use something else.

If you can compromise on the throughput, EFS becomes an attractive option to manage your cluster shared storage.

When you define your NFS volumes inside your docker-compose file, it is part of the service definition. It takes care of the infrastructure it uses. I find this pattern a good one when dealing with microservices. If you need to move this service to another cluster, anytime in the future, the NFS mount is one thing less you need to remember to do.

Docker healthcheck experiments with Go web app

Fri, 12 Feb 2021 12:52:00 +0200

One good way to monitor your container status is to use Docker’s HEALTHCHECK feature.

As part of testing the followed actions upon an unhealthy container, I had to experiment how this works. I wrote a small Go app that replies to /health requests, and the status it responds is configurable.

The webserver listens on $PORT (defaults to 8080). To change its status, simply make an api call to one of the supported actions. You can either connect the container to execute it, or if you have mapped a port on the host, you can use it.

Supported actions are:

/sabotage will make it respond with 500.
/timeout will make it respond after 20s.
/recover will return it back to healthy state, with 200 response code.

You can find the source code here.

From the Docker docs -

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


The HEALTHCHECK instruction has two forms:

 HEALTHCHECK [OPTIONS] CMD command (check container health by running a command inside the container)
 HEALTHCHECK NONE (disable any healthcheck inherited from the base image)

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working.
This can detect cases such as a web server that is stuck in an infinite loop and unable
to handle new connections, even though the server process is still running.

When a container has a healthcheck specified, it has a health status in addition to its normal status.
This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state
it was previously in). After a certain number of consecutive failures, it becomes unhealthy.

The options that can appear before CMD are:

--interval=DURATION (default: 30s)
--timeout=DURATION (default: 30s)
--start-period=DURATION (default: 0s)
--retries=N (default: 3)

You can configure the HEALTHCHECK settings in the Dockerfile or the docker-compose.yml. In my example, I use it in the Dockerfile.

Demo

Clone the repo and build the image: docker build . -t go-healthchecker
Bring up the container, wait until it gets healthy.

To change the healthcheck response, connect to the container and update its status

How can Stackoverflow make you a better developer

Fri, 05 Feb 2021 15:30:00 +0200

“Time is the most valuable resource because you cannot get more of it.”

There isn’t a single developer who doesn’t use Stackoverflow. Most of us use it daily. How many times a day have you googled an error, and reached Stackoverflow? It is such a valuable resource that improves productivity every single day. Consider the amount of debugging time it saved you. Yet, the majority of the developers I know don’t even own an account, nevertheless trying to answer other people’s questions.

How easy and convenient it is to find a solution to the problem you faced. In this post, I’ll share with you the benefits I found of being a contributor on Stackoverflow.

Most of the experienced developers will probably stop reading right now. Individuals look to improve their coding and problem-solving skills, I hope you will own an account (and contribute) after reading this post.

If you’re not familiar with where it started, the site was introduced by Jeff Atwood on his blog codinghorror:

“Stackoverflow is sort of like the anti-experts-exchange (minus the nausea-inducing sleaze and quasi-legal search engine gaming) meets Wikipedia meets programming Reddit. It is by programmers, for programmers, with the ultimate intent of collectively increasing the sum total of good programming knowledge in the world. No matter what programming language you use, or what operating system you call home. Better programming is our goal.”

Learn something new

When I started learning Python about five years ago, I built my curriculum that was made of few books and online courses. I put the time and effort (about four months) to practice. As most of us, when I had faced challenges I couldn’t solve by myself, I used Google and found an answer on .. you guessed it right, Stackoverflow. After a couple of iterations of this scenario, I thought to myself,

“If I find answers to my questions on Stackoverflow, let’s take a peek at the most upvoted questions tagged with Python and see what it reveals”.

It was like a goldmine for me. Clear, refined questions with detailed answers and examples, for free. No subscription, no money-back guarantee, plain simple. I started checking all of them, reading thoroughly all the answers to nail the subject in question. These are practical questions with high-quality answers from experienced professionals.

Next time you plan to learn something new, give it a try. Go to Stackoverflow and look for the most upvoted questions and answers. If you haven’t ever done it before, you’re gonna be amazed.

I then challenged myself. I pulled my sleeves and decided I will try to answer questions.

Why would I spend my precious time solving other people’s problems?

That’s a damn good question. When I took the challenge, I was selfish. I wanted to gain confidence in my knowledge. I liked to solve problems using the things I have learned. I looked for new questions that I had a solution for and tried to help using the gained knowledge. I did this to improve myself as a programmer.

I feel the time I spent, was paid back double. It made me a better programmer, problem-solver, and contributed to my professional career.

“Knowledge is like money: to be of value it must circulate, and in circulating it can increase in quality and, hopefully, in value” – Louis L’Amour

Why YOU should contribute

Improve your coding and problem-solving skills: I can’t stress enough the value of spending time on this site, reading through popular questions, and try to solve them yourself. Answering other people’s questions strengthens your knowledge and confidence in the topic
Improve your debugging capabilities: It’s one of the most valuable skills of a software engineer. By helping other people, you practice debugging their code, which is harder than debugging your own
If you don’t find the question you’re looking for - ask: Let the community help, and help other people in the future who encounters the same problem as you did
Learn something new, solve a new problem, dig deeper into a topic you’re familiar with, and expand your knowledge
Use your Stackoverflow profile to promote yourself. By accurately answering questions, you demonstrate your experience in a topic

Where to start?

This blog post aimed to get you started. First and foremost, if you don’t have an account, sign up.

I recommend checking these two threads, “How do I contribute to Stackoverflow” and “How does a new user get started”. Spend the time reading to understand how this platform works and how to use it.

Either you’re learning something new, or want to improve any of your coding skills, try to participate and answer questions:

Search for the tags of interest to see only these kinds of questions (e.g, python, go, linux)
Spend ~30-60min a day for 14 days. It doesn’t need to be contiguous, you can check it while you’re waiting for the compilation or deployment to finish
Make it a habit

That’s all it takes.

Conclusion

I found that actively participating on Stackoverflow of great value, for my development skills and career. Every beginner and experienced developers alike will benefit from it if used correctly. I tried to share my own experience, what works for me, and I encourage you to share your knowledge. And have fun.

Stay humble, be kind.

nginx config template with environment vars

Fri, 29 Jan 2021 11:53:00 +0200

Until nginx v1.19, you had to either bake in the image a ready configuration file or mount it at runtime.

Often, you would need to template the nginx config file. This was made possible since v1.19 in a convenient way using environment variables.

First, you need a template to copy. Create default.conf.template file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


server {

 listen 80;
 server_name ${NGINX_PORT};


 location / {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;

 root /usr/share/nginx/html;
 index index.html index.htm;
 }
}

In the Dockerfile, we need to place the *.template file in a specific location, /etc/nginx/templates.

On startup, the nginx entrypoint script scans this directory for files with *.template suffix by default, and it runs envsubst. The envsubst parse the template using the shell interpolation and replaces shell variables with values from environment variables. It outputs to a file in /etc/nginx/conf.d/.

If you’re using $var, and there’s no such env-var, it will stay as is in the output file. In the above file, $host and $remote_addr are such examples. We want them to stay as parameters in the output file, as they are parameters used by nginx.

Let’s look at a complete example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# docker-compose.yml
version: '3.7'
services:
 nginx:
 build:
 context: nginx/
 ports:
 - "80:80"
 environment:
 - NGINX_HOST

1
2
3
4
5
6


# Dockerfile
FROM nginx:1.19.6-alpine

RUN mkdir /etc/nginx/templates
COPY default.conf.template /etc/nginx/templates
COPY index.html /usr/share/nginx/html/

Running the docker-compose with NGINX_HOST=1.1.1.1 docker-compose up -d and the generated default.conf would look:

1
2
3
4
5
6


server {

 listen 80;
 server_name "1.1.1.1";
 ..
 ..

If you’re interested, these are the internal scripts nginx uses in its image at startup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


#!/bin/sh
# docker-entrypoint.sh

set -e

if [ -z "${NGINX_ENTRYPOINT_QUIET_LOGS:-}" ]; then
 exec 3>&1
else
 exec 3>/dev/null
fi

if [ "$1" = "nginx" -o "$1" = "nginx-debug" ]; then
 if /usr/bin/find "/docker-entrypoint.d/" -mindepth 1 -maxdepth 1 -type f -print -quit 2>/dev/null | read v; then
 echo >&3 "$0: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"

 echo >&3 "$0: Looking for shell scripts in /docker-entrypoint.d/"
 find "/docker-entrypoint.d/" -follow -type f -print | sort -n | while read -r f; do
 case "$f" in
 *.sh)
 if [ -x "$f" ]; then
 echo >&3 "$0: Launching $f";
 "$f"
 else
 # warn on shell scripts without exec bit
 echo >&3 "$0: Ignoring $f, not executable";
 fi
 ;;
 *) echo >&3 "$0: Ignoring $f";;
 esac
 done

 echo >&3 "$0: Configuration complete; ready for start up"
 else
 echo >&3 "$0: No files found in /docker-entrypoint.d/, skipping configuration"
 fi
fi

exec "$@"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


#!/bin/sh
# docker-entrypoint.d/20-envsubst-on-templates.sh

set -e

ME=$(basename $0)

auto_envsubst() {
 local template_dir="${NGINX_ENVSUBST_TEMPLATE_DIR:-/etc/nginx/templates}"
 local suffix="${NGINX_ENVSUBST_TEMPLATE_SUFFIX:-.template}"
 local output_dir="${NGINX_ENVSUBST_OUTPUT_DIR:-/etc/nginx/conf.d}"

 local template defined_envs relative_path output_path subdir
 defined_envs=$(printf '${%s} ' $(env | cut -d= -f1))
 [ -d "$template_dir" ] || return 0
 if [ ! -w "$output_dir" ]; then
 echo >&3 "$ME: ERROR: $template_dir exists, but $output_dir is not writable"
 return 0
 fi
 find "$template_dir" -follow -type f -name "*$suffix" -print | while read -r template; do
 relative_path="${template#$template_dir/}"
 output_path="$output_dir/${relative_path%$suffix}"
 subdir=$(dirname "$relative_path")
 # create a subdirectory where the template file exists
 mkdir -p "$output_dir/$subdir"
 echo >&3 "$ME: Running envsubst on $template to $output_path"
 envsubst "$defined_envs" < "$template" > "$output_path"
 done
}

auto_envsubst

exit 0

Resources

https://github.com/docker-library/docs/tree/master/nginx#using-environment-variables-in-nginx-configuration-new-in-119

If you don't use a secret management tool, you're doing it wrong

Wed, 27 Jan 2021 21:58:00 +0200

Secrets management refers to the tools and methods for managing digital authentication credentials (secrets), including passwords, keys, APIs, and tokens for use in applications, services, privileged accounts, and other sensitive parts of the IT ecosystem.

While secrets management is applicable across an entire enterprise, the terms secrets and secrets management are referred to more commonly in IT about DevOps environments, tools, and processes.

“Three may keep a secret, if two of them are dead.” ― Benjamin Franklin, Poor Richard’s Almanack

Managing and sharing secrets is a complicated task. There are various environments, with many services, where each needs to authenticate itself.

If you’re working in the Operations team, you clearly faced a secret committed to a git repository at one point. Even if the repository is private, this is a big no-no. If someone accidentally changes the repository to be public, or push a file with secrets to a public repository, there’s no turning back. The secret will be out there in the wild.

Before I proceed, I want to ask you few questions:

Where does your organization keep its sensitive data? Is it in an encrypted file? A database? A shared tool such as 1Password?
Who can access it?
How easy is it to add a new secret, or update an existing one?
Do you manage your secrets or they manage you?

Hashicorp Vault

Hashicorp Vault is an open-source robust secret management. It serves as a secret repository with access control lists, auditing, and TTL access to the secrets. It also supports a variety of authentication mechanisms and storage backends.

Vault keeps your secrets encrypted on disk and on transit. It has a simple API to communicate with and great documentation. The website presents two main use-cases:

Secrets Management: centrally store, access, and distribute dynamic secrets such as tokens, passwords, certificates, encryption keys.

Data Protection: keep application data secure with centralized key management and simple APIs for data encryption.

It gives you a central place to safely store secrets. Define who can access what, with an audit on the operations. Use short-lived tokens to reduce the impact in case of secrets gets exposed. Has great APIs you can leverage to automate processes.

Vault takes their business field seriously. Its architecture is a little complex, so let’s quickly overview the components and explain them in simple terms.

Secrets Paths

Inside Vault, secrets kept on a file-system like path. Every secret is structured as a JSON object, with key-value pairs. These files are versioned and keep their history (refer to KV secrets engine)

Policies

Vault uses policies to define fine-grained access-control. A policy is made of a list of paths (with regex support). A policy is then attached to the token and determines what secrets a token can access and what actions it can perform (create, read, update, delete).

Authentication and authorization

We use the term authentication to refer to the first phase of the login process. For example, the user provides the correct username and password to authenticate. If the credentials match, the user is authenticated.

The authorization part comes next. It defines what an authenticated user has access to.

To perform operations on the Vault, a user needs to authenticate first. Vault refers to this as authentication backends. It is a list of authentication methods a user can use to authenticate.

The simplest option is a token, where you perform the login operation using a pre-defined token to gain access. This is a bad idea. There are better options.

Vault has SSO (OIDC) integration. People in the organization can access using their AzureAD or Google accounts. Once you set it up, everyone that has an account can access Vault! You don’t need to manage credentials for every person individually.

Once the user logs in, the policies applied to his account or groups in the organization are applied to his token. This defines what he can do inside Vault.

The Flow

Except for token-based authentication, all other methods work the same way. You first provide your credentials (username/password, SSO, etc.) to perform a login action. In exchange, Vault returns a temporary token for you to use in subsequent API calls. The temporary token is defined by its attributes - lifetime, is it renewable, etc.

When using the UI or the CLI, these things happen behind the scene. Yet, if you’re using the HTTP API, you’ll need to perform login action and grab the token returned by Vault.

This is a new era. Developers can now read and add secrets by themselves and share these with Operations seamlessly. This increases the velocity of both teams and improving the security as a side-effect. For example, when an employee leaves the company, his access is revoked easily. Or the usage of short lived tokens instead of static passwords.

CI/CD pipelines

The deployment platform should also access Vault. During the build and deployment processes, the platform can get the secrets using the approle authentication method. The approle method gives you the option to configure applications access type. This type should have a short time-to-live so it’s useless after the deployment process.

We can take it a step further. Now that we have a centralized secrets store with an API, we can build a Jenkins job that generates secrets and push them to the Vault. This formalizes the procedure and prevents humble human mistakes, such as weak passwords.

Increase teams productivity

As the developers’ team grows, without an appropriate tool time is going to be wasted here. Nobody likes to wait to test something just because they don’t have the password yet. How much time is spent here? This can be solved by.. using the right tool for the job.

Having an interface to read and add secrets would remove the toil of secrets maintenance.

Summary

Using a secrets management tool has numerous benefits, but I presume most of them are unseen. On the surface, it eases the process of adding or updating a secret and managing a unified access to them, by both Dev and Ops. Replacing a secret becomes an easy task for Operations.

It provides all this while keeping high-security standards and auditing. It presents an API to automate the daunting toil of managing them yourself using a less sophisticated mechanism such as files or a database.

It takes your secrets to the next level, and I haven’t touched the more advanced capabilities this tool has (dynamic secrets and AWS authentication method, for example). If you are intrigued, go check the documentation.

The new capabilities introduced to automate processes shouldn’t be considered lightly.

How do you manage your secrets?

Go's method receiver: Pointer vs Value

Sun, 17 Feb 2019 15:10:37 +0200

The topic is large, and there is plenty of information online. This blog tries to keep it short and concise, and useful for experienced programmers that are new to Go.

Coming to Go from Python introduced me to a new concept I didn’t have to put thought into. Python is a pass-by-object-reference language. You have no direct control over that. What this means is, when you pass an object (everything in Python is an object) to a function, you pass a reference to the object itself.

We can use the id() function to illustrate that. It returns the identity of an object. This identity has to be unique and constant for this object during its lifetime.

1
2
3
4
5
6
7
8
9


>>> def action_with_string(s):
... print(id(s))
...
>>> v = "sample string"
>>> id(v)
4360321520
>>> action_with_string(v)
4360321520
>>>

The object inside a function is as same as the one of the caller. Whatever you pass, the function can mutate (as long as it is mutable). It is discussed here on StackOverflow: Python functions call by reference if you are interested in reading further.

In Go, when you define a method on a struct, you have to choose the receiver type - value or a pointer receiver.

What does this means, anyway?

In simple terms, the value receiver makes a copy of the type and pass it to the function. The function stack now holds an equal object but at a different location on memory. If the method mutates the struct, it wouldn’t be reflected outside the function scope.

The pointer receiver passes the address of an object to the method. The function stack has a reference to the original object.

A simple example shows the difference. (Go Playground link)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


package main

import (
 "fmt"
)

type Bike struct {
 Model string
 Size int
}

func ValueReceiverExample(b Bike) {
 fmt.Printf("VALUE :: The address of the received bike is: %p\n", &b)
 // Address of the object is different than the ones in main.
 // Changes to the object are scoped to the method, and not reflected to the caller.
 b.Model = "BMW"
 fmt.Println("INSIDE ValueReceiverExample :: model: ", b.Model)
}

func PointerReciverExample(b *Bike) {
 fmt.Printf("POINTER :: The address of the received bike is: %p\n", b)
 // Address of the object is as same as the ones in main.
 b.Model = "BMW"
 fmt.Println("INSIDE PointerReciverExample :: model: ", b.Model)
}

func main() {
 HondaBikeValue := Bike{"Honda CBR", 650}
 SuzukiBikePointer := &Bike{"Suzuki V-Storm", 650}

 fmt.Printf("Value object address in main: %p\n", &HondaBikeValue)
 fmt.Printf("Pointer object address in main: %p\n", SuzukiBikePointer)

 ValueReceiverExample(HondaBikeValue)
 PointerReciverExample(SuzukiBikePointer)

 fmt.Println("Value's model in main: ", HondaBikeValue.Model)
 fmt.Println("Pointer's model in main: ", SuzukiBikePointer.Model)
}

OUTPUT:
Value object address in main: 0xc00000c0a0
Pointer object address in main: 0xc00000c0c0
VALUE :: The address of the received bike is: 0xc00000c0e0
INSIDE ValueReceiverExample :: model: BMW
POINTER :: The address of the received bike is: 0xc00000c0c0
INSIDE PointerReciverExample :: model: BMW
Value's model in main: Honda CBR
Pointer's model in main: BMW

So when should you use what?

Regardless of what you choose, it is best practice to keep a uniformity of the struct methods. If the struct uses both types, it is hard to track which method use which receiver type, especially if you weren’t the one who wrote the code. Consistency is an important aspect of programming.

Pointers

If you want to share a value with its methods

If the method modifies the state of the struct, you must use a pointer. Value receiver changes are local to the method scope.
If the struct is very large (optimization)

Large structs with many fields may be costly to copy every time they need to be passed around. If you are in such a case, you should consider breaking it down into smaller pieces. If that’s not possible, use a pointer. Optimization usually adds complexity. Be aware it comes with a trade-off.
One major disadvantage of pointers is they are not safe for concurrency. If you use pointers, you need to use synchronous mechanisms such as channels or use the atomic / sync builtin packages.

Values

If you don’t want to share a value, use the value receiver.
Value receivers are concurrency safe. One of Go’s advantages is concurrency, so this is a huge plus. You never know when you would need it, and if you write library code you can be sure someone at some point will use it concurrently.

Summary

That was an interesting subject for me. It is a new concept I didn’t deal with before coding in Go. After researching the subject thru blogs, documentation, videos, and Stackoverflow, I have come up with a small set of rules to remember:

Consistency counts. Use the same receiver type for all your methods. It isn’t always feasible, but have this rule in mind when you need to mix them.
Method defines the behavior of a type; if the method updates or modifies the state, use the pointer receiver.
If a method doesn’t mutate state, use-value receiver.
Functions operate on values; functions should not depend on the state of a type.

Here’s a list of great resources if you want to explore deeper:

How to run your own docker registry with password, SSL and S3 backend

Sun, 13 Jan 2019 22:43:29 +0200

I remember the day I started using Docker. The simplicity was overwhelming. I went over the official docs which are very good, and looked up some 101 tutorials to get started. I had successfully made my first image. Now that I have one, I want to deploy it over my servers. The first option was DockerHub, but a free account obligates you to share the images. If you need to keep them private, this service costs money.

Or, you can just run your own Docker Registry and have the flexability to:

Password protect it
Store the images on locally or in the cloud (S3/Azure/GoogleCloud)
Use your self-signed SSL certificates

“The Registry is a stateless, highly scalable server side application that stores and lets you distribute Docker images.”

The official Docker documentation covers the subject in-depth. You should read it to get a grasp of what is it and how to run it. In this post, I address the issues I had encountered when I deployed our registry server. I’ll explain how to implement the three items mentioned above. For the rest of this post, I assume you read the Registry Overview and Understanding Docker Registry at the official site. (~3min).

Get the registry running

The registry app itself runs in a container. It has a front-end component (which is optional) that allows you to access the registry data from a browser. Each one of the components runs inside its own container, which forms the app stack. An app stack is a group of apps (stack) that is linked together to perform certain tasks. We can use docker-compose to run the app stack, instead of running each container separatley.

Here’s the docker-compose.yml file we start with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


version: '2'
registry:
 image: registry:v2
 restart: always
 ports:
 - "5000"
 environment:
 REGISTRY_STORAGE_DELETE_ENABLED: 'true'
 REGISTRY_HTTP_ADDR: 0.0.0.0:5000
registry-frontend:
 image: konradkleine/docker-registry-frontend:v2
 restart: always
 environment:
 ENV_DOCKER_REGISTRY_HOST: 'registry'
 ENV_DOCKER_REGISTRY_PORT: 5000
 links:
 - registry
 ports:
 - "8080:80"
 expose:
 - 80

Let’s briefly break this down, as its just plain configuration (without password, S3 or SSL).

The registry and registry-frontend are the container names in my stack. You can name them any way you like. The environment variable REGISTRY_STORAGE_DELETE_ENABLED allows you delete images from the registry, and REGISTRY_HTTP_ADDR binds the listening address.

The registry-frontend container uses ENV_DOCKER_REGISTRY_HOST and ENV_DOCKER_REGISTRY_PORT as the address to which it connects. Because we use docker-compose, it is available for us by name. The links configuration links the container the another service, in our case to the registry.

Don’t you wonder what’s the difference between ports and expose settings? the difference is ports make the ports defined accessible to the host as well as other services in the compose file, whereas expose actually expose the ports only to other services without publishing them to the host machine. Only the internal port can be specified in such case.

Ports also links between an external port (defined on the host) to a container assigned port. In the case of the registry container, port 5000 is accessible from everywhere. On the other hand, registry-frontend exposes port 80 only to other services defined in the file, while accessing it from outside is bounded to port 8080.

To run the services, we use docker-compose up inside the directory where the docker-compose.yml file resides. You can validate the services are running with docker ps. If the containers are up, you can now browse http://localhost:8080 to see your registry data.

Passwords time

As part of having my private registry, I would like to add some measure of security. I want to password protect it. Once it set, the client needs to provide a password when they login the registry for the first time. This is very simple, and there is a great article covering that. Check “Private Docker Registry Part 2: let’s add basic authentication” on Medium and configure if you need it too.

Private SSL certificates

Security is always an issue. It’s something we tend to forget when we are in the middle of POC. We want to make things work first, and fast. It requires additional steps, and we tend to leave it as the last task.

My advice for you is - whenever you can use SSL, use it. It’s worth the extra time you put in it, and has no drawbacks. The docker client uses HTTPS by default, so as I see it we have 3 options here:

Make the client use HTTP instead of HTTPS
Configure the client to trust our registry with insecure-registries parameter
Add our certificate to the trusted ones, either the docker engine or the OS

I find the 3rd option as the best solution for me. I’ll use my own certificates for my server.

I assume you have generated the certificate (cert.crt, cert.pem and ca-certificate) for your server. Put the public and private keys in /var/lib/docker/certs. For the registry to be using them, we need to add the following to our configuration, under the environment of the registry:

1
2
3
4


 REGISTRY_HTTP_TLS_CERTIFICATE: /certs/registry_example.crt
 REGISTRY_HTTP_TLS_KEY: /certs/registry_example_key.pem
 volumes:
 - /var/lib/docker/certs:/certs

Ok our server side now uses our certs. Great! but we are not done yet. In order for clients to be able to connect the registry, we need their workstations to trust our CA. Otherwise, it won’t connect. If your client machine already trust your CA certificate, you’re done. Else, you need to copy your CA certificate over the machine. You can copy it to /etc/docker/certs.d and docker will trust it automatically, or for the OS to trust it you need to (On Ubuntu system, it may vary on other OS):

Copy it to /usr/local/share/ca-certificates/
sudo update-ca-certificates

Make sure to restart the docker service on the client after you do that.

Now that our registry has basic authentication and SSL encryption support, let’s continue our journey to the final configuration step. I’ll show how you connect to the registry after I explain the use of S3 as a backened.

S3 storage backend

The registry serves our images. We can pull and push images into or from it. So, these images must reside on a disk somewhere. By default, they reside on the docker host running the registry. You can explicity mount it with a different volume, but you can do something much cooler than that. I used AWS S3 storage services as my backend. What this means is, every image I upload to the registry is saved on a dedicated bucket on S3.

This has the advantage of:

I don’t need to care about backups.
My storage is unlimited.
Disaster recovery, in case the machine my registry runs at burns or something, I can deploy a new service and connect it to the S3 bucket. All data is preserved.

Re-read item 3 again. This is priceless. I can start another registry container anywhere in the world, once I configure the S3 storage in the docker-compose.yml file I gain the access to all my images.

To configure storage, we need this snippet in the environment part of the registry component:

1
2
3
4
5
6


 REGISTRY_STORAGE: s3
 REGISTRY_STORAGE_S3_ACCESSKEY: <api access key>
 REGISTRY_STORAGE_S3_SECRETKEY: <api secret>
 REGISTRY_STORAGE_S3_BUCKET: <bucket name>
 REGISTRY_STORAGE_S3_REGION: <region>
 REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: false

It’s pretty straight forward, but the last one. REGISTRY_HEALTH_STORAGEDRIVER_ENABLED is important. Before I added that I ran into problems. I don’t recall the exact error, but after googling for some time, I find out that if you run the registry with an empty bucket (mine was) this health check fails, and the service fails to start. So I had to disable it, and things worked. After you upload something, you may enable this check. I decided I don’t really care about that, so I left it disabled.

Client connection

In order to use my new registry, I need to connect the client so it can pull or push images from/to it. Remember our registry works with SSL, and the docker client too by default, so I need to trust my CA, as explained [above](# Deploy private SSL certificates).

To connect, execute docker login -u <user> <url:port> (without the https prefix).

Now, on my Ubuntu machine I encountered another error when doing this: “Error saving credentials: error storing credentials - err: exit status 1, out: Cannot autolaunch D-Bus without X11 $DISPLAY”

To solve this, I needed to delete a package from the OS: apt remove golang-docker-credential-helpers made it work. After you login to the registry, by default the credentials are kept in a file inside a hidden directory ~/.docker/config.json. Obviously this isn’t best practice, you can find better alternatives here.

Push and Pull

To pull or push images we simply refer to our registry it’s address, docker pull our-registry.com:<port>/image. You can also use the IP if you don’t own a domain.

API to check registry contents

I will use a fake domain registry.example.com:5000 for my registry server. Once I had the containers running, I could browse to https://registry.example.com and see the content of the registry, the repositories, images and tags. If you decided not to use the front-end or you need to use it within an app, you can access the registry using it’s API.

I’ll show some useful examples,

1
2
3
4
5
6
7
8
9


# list of the repositories
chen@gns:~$ curl -ksS -u admin https://registry.example.com/v2/_catalog
Enter host password for user 'admin':
{"repositories":["sample_image","nginx"]}

# list of tags for an image
chen@gns:~$ curl -ksS -u admin https://registry.example.com/v2/sample_image/tags/list
Enter host password for user 'admin':
{"name":"sample_image","tags":["0.1", "0.2", "latest"]}

Summary

I went through the process I did to deploy my registry. I know I didn’t cover it thoroughly, but that wasn’t the purpose of this post. If you need a private registry, it is very easy to set up one. There are many tutorials how to do that. I shared with you the issues I had faced when deploying a registry, and the configuration I had applied.

Applying password and SSL as security item is important, don’t forget that. If you can use a cloud provider to store your images, it can save you a lot of headache. But it’s not the end of the world if you can’t, you just need take care for the items I listed.

Here’s how the completed docker-compose.yml looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


version: '2'
registry:
 image: registry:v2
 restart: always
 ports:
 - "5000"
 environment:
 REGISTRY_STORAGE_DELETE_ENABLED: 'true'
 REGISTRY_HTTP_ADDR: 0.0.0.0:5000
 REGISTRY_HTTP_TLS_CERTIFICATE: /certs/registry_example.crt
 REGISTRY_HTTP_TLS_KEY: /certs/registry_example_key.pem
 REGISTRY_STORAGE: s3
 REGISTRY_STORAGE_S3_ACCESSKEY: <api access key>
 REGISTRY_STORAGE_S3_SECRETKEY: <api secret>
 REGISTRY_STORAGE_S3_BUCKET: <bucket name>
 REGISTRY_STORAGE_S3_REGION: <region>
 REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: false
 volumes:
 - /var/lib/docker/certs:/certs
registry-frontend:
 image: konradkleine/docker-registry-frontend:v2
 restart: always
 environment:
 ENV_DOCKER_REGISTRY_HOST: 'registry'
 ENV_DOCKER_REGISTRY_PORT: 5000
 links:
 - registry
 ports:
 - "8080:80"
 expose:
 - 80

I hope you found this one useful.

About

Mon, 01 Jan 0001 00:00:00 +0000

Who am I?

I’m Chen, a self-taught engineer with 13+ years of experience in system administration. This blog is my digital notebook, where I share real solutions to problems in projects, thoughts about cloud-native, software architecture, devops principles, reliability and software engineering.

I pen down and share my challenges because someday, you might encounter them too, and finding these solutions could be just the guidance you need.

My projects

Below are some projects I had worked on.

Search

Mon, 01 Jan 0001 00:00:00 +0000

Devopsian

Understanding Go’s Supercharged Map in v1.24

The Library Analogy

1. Tables Are Library Sections

2. Groups Are Bookshelves

3. Control Word: The Librarian’s Cheat Sheet

How It Works: Storing and Retrieving Books

Storing a Book

Retrieving a Book

Handling Collisions

Why This Design is Brilliant

1. Cache-Friendly Layout

2. Fast Probing with Metadata

Memory Layout Example

Performance Gains

Conclusion

Handling Retries When Sending Files in Go: Lessons Learned

The Problem: Retrying Sends an Empty File

Understanding io.Reader and Why This Happens

The Solution: Using a buffer (bytes.Buffer)

Exploring Alternatives to bytes.Buffer

Memory Considerations

Best Practices for Retry Logic

Key Takeaways

Wrapping Up

Why I Switched from Makefile to Taskfile

Introduction

Makefile

Why I Switched

Introducing Taskfile

Summary

The value of API-First design on side-projects

Intro

Prioritizing Your Application’s Foundation

What is Open API?

The Project Structure

Generate The APIs

Generate Server-Side Code

Interface Implementation

Generate a Typescript Client

Summary

Inside EKS Networking: Decoding the Service IP Journey

Intro

The Setup

The Problem

Exploring Kubernetes Networking Magic

Summary

How to Structure a Go Project: Start Simple, Refactor Later

Why is project structure important?

What are the common pitfalls of project structure?

How to start simple and refactor later?

The cmd package pattern

The internal package

The config package

The API package

Controllers, handlers, and storage

Approach 1: Storage and Handlers (or controllers) packages

Approach 2: Domain Entity Package

Conclusion

This is how you want to manage your Terraform modules

Intro

The problem

Towards a better future

Terralist

Summary

Go templates: customize your output using templates

The template and data

Passing data to the template

Access multiple fields of a data structure

Summary

Archives

A deep dive to Canary Deployments with Flagger, NGINX and Linkerd on Kubernetes

Intro

Why Flagger

Prerequisites

NGINX (Ingress) Canary Deployment

Deploy nginx

Linkerd (ServiceMesh) Canary Deployment with Ingress support

Deploy linkerd

Summary

Understanding `io.Reader` and Why This Happens

The Solution: Using a buffer (`bytes.Buffer`)

Exploring Alternatives to `bytes.Buffer`