DEV Community: JankariTech

Kubernetes - A Beginner's Guide to Container Orchestration

Ashim Shrestha — Wed, 01 Apr 2026 06:02:50 +0000

Introduction

It is very difficult to manage containerized applications at a large scale. When a container crashes, you need another container to spin up automatically. When there is high traffic, you need more containers to handle it. When you want to release a new version of an application, you want a way to update the running application without any downtime.

We can use Kubernetes to solve all of the above problems automatically.

What is Kubernetes?

Kubernetes, also known as K8s, is an open source container orchestration engine for automating the deployment, scaling, and management of containerized applications. The open source project is hosted by the Cloud Native Computing Foundation (CNCF). It helps developers and operators to manage containerized applications effortlessly.

Think of Kubernetes as an automated operations team for your containers.

Why Kubernetes?

Containers provide a great way for packaging applications, but in production, you need the features mentioned below:

Self-Healing: It automatically starts another container if any container crashes.
Automatic Scaling: It increases/decreases the number of containers based on traffic.
Rolling Updates: It updates containers without stopping your service or with zero downtime.
Load Balancing: It distributes traffic across healthy containers.
Storage Management: It provides persistent data that can survive even after container restarts.
Infrastructure Abstraction: It manages which container runs on which server.

You can use Kubernetes to get all the above-mentioned features automatically, allowing you to focus on developing your application instead of spending time and effort managing infrastructure.

Quick Overview of Key Concepts

Before diving into the architecture, let's understand the basic building blocks of Kubernetes:

Cluster: A group of connected machines where Kubernetes runs your applications.
Node: A machine (physical or virtual) in the cluster that runs your applications.
Pod: The smallest unit in Kubernetes. Usually a single container, but can contain multiple containers.
Deployment: Manages multiple pods and ensures they're always running and healthy.

Don't worry if these concepts seem unclear right now. We'll discuss each one in detail in the next sections.

Kubernetes Architecture

A Kubernetes cluster has two components:

Control Plane (Master Node)

The Master Node is responsible for managing the cluster and making decisions about it.

Key Components:

kube-apiserver: It acts as the API gateway, allowing communication with the cluster.
etcd: It is the database where all of the cluster configuration and state are stored.
kube-scheduler: It is responsible for deciding which node should run which pod.
kube-controller-manager: It runs controllers for watching the cluster state and making corrections whenever needed.

Worker Nodes

Worker Nodes run your containerized applications.

Key Components:

Kubelet: It ensures that desired containers are running and healthy.
Container Runtime: It pulls container images and runs containers (Docker, containerd, etc.).
kube-proxy: It is responsible for managing networking and handling the routing of services.

Core Kubernetes Concepts

Cluster

A Cluster is a group of machines (nodes) that are connected together to run containerized applications.

View cluster information:

kubectl cluster-info              # Get cluster details
kubectl get nodes                 # List all nodes
kubectl get nodes -o wide         # Get detailed node info
kubectl describe node <node-name> # Full node details
kubectl top nodes                 # View node resource usage

Node

A Node is a physical or virtual machine that provides computing resources like CPU, memory, and storage for running pods.

View nodes:

kubectl get nodes
kubectl describe node <node-name>

Pod

A Pod is the smallest deployable unit in Kubernetes. It usually contains one container, but it can contain tightly coupled containers.

Key characteristics:

Containers in a pod share the same network namespace (same IP address)
Containers in a pod can share storage volumes
Pods are ephemeral (temporary), so when they are deleted, they're gone
Pods are usually managed by Deployment

YAML to create a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - containerPort: 80

Namespace

A Namespace is a virtual cluster that helps to divide resources within a single cluster. It can separate environments like development, testing, and production.

YAML to create a Namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production

Work with namespaces:

kubectl create namespace my-app
kubectl apply -f deployment.yaml --namespace=my-app
kubectl apply -f deployment.yaml -n my-app  # shorthand

Labels and Selectors

Labels are key-value pairs that are attached to Kubernetes objects that help organize and identify them. Selectors use these labels to locate and manage groups of related objects.

Controllers use labels to know which pods to manage. Services use labels to know which pods to route traffic to.

YAML to apply labels and selectors:

# Applying labels
metadata:
  labels:
    app: nginx
    tier: frontend
    environment: production

# Selecting by label
selector:
  matchLabels:
    app: nginx

Label commands:

kubectl get pods -l app=nginx          # Get pods with label app=nginx
kubectl get pods --show-labels         # Show all labels

Deployment

A Deployment manages a set of identical pods and ensures that the desired number of replicas are always running. It provides rolling updates, rollbacks, and automatic scaling.

Key features:

It maintains the desired number of pod replicas
It automatically replaces failed pods
It enables rolling updates (update pods gradually without downtime)
It supports rollback to the previous versions

YAML to create a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: production
spec:
  replicas: 3                    # Always maintain 3 replicas
  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80

Deployment commands:

kubectl apply -f deployment.yaml           # Create deployment
kubectl get deployments                    # List deployments
kubectl get deployments -n production      # In a specific namespace
kubectl describe deployment nginx-deployment

kubectl scale deployment nginx-deployment --replicas=5    # Scale to 5 replicas
kubectl set image deployment/nginx-deployment nginx=nginx:1.22  # Update image

kubectl rollout status deployment/nginx-deployment        # Check rollout progress
kubectl rollout undo deployment/nginx-deployment          # Rollback to previous
kubectl rollout history deployment/nginx-deployment       # View update history

Service

A Service provides a stable endpoint for accessing a group of pods. Since pods are ephemeral (get created and destroyed), services provide a consistent way to reach your application.

Services give you:

Stable IP address and DNS name
Load balancing across pods
Internal or external access

YAML to create a ClusterIP Service:

ClusterIP (default) - Only accessible within the cluster:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: ClusterIP
  selector:
    app: nginx            # Route to pods with label app: nginx
  ports:
    - port: 80            # Service port (internal)
      targetPort: 80      # Pod port (where traffic goes)

YAML to create a NodePort Service:

NodePort - Accessible from outside the cluster:

apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30007     # External port (range: 30000-32767)

Service commands:

kubectl get services
kubectl get svc
kubectl describe service nginx-service

ConfigMap

A ConfigMap stores non-sensitive configuration data as key-value pairs. It decouples configuration from container images, allowing you to change configuration without rebuilding images.

YAML to create a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"
  DATABASE_HOST: "db.example.com"
  CACHE_TTL: "3600"

ConfigMap commands:

kubectl get configmaps
kubectl describe configmap app-config

Secret

A Secret stores sensitive information like passwords, API keys, and tokens. It's similar to ConfigMap, but it is intended for confidential data.

Note: Secrets are base64-encoded by default (not encrypted). For production, enable encryption for secrets.

YAML to create a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  username: dXNlcm5hbWU=     # base64-encoded "username"
  password: cGFzc3dvcmQxMjM= # base64-encoded "password123"

Base64 encoding:

echo -n "mypassword" | base64
# Output: bXlwYXNzd29yZA==

echo "bXlwYXNzd29yZA==" | base64 -d
# Output: mypassword

Secret commands:

kubectl get secrets
kubectl describe secret db-credentials

Volume

Pods and containers are ephemeral. When a container restarts, all data gets lost. Volumes provide persistent storage that survives even when container restarts.

YAML to create emptyDir storage:
emptyDir - It is a temporary storage shared between containers in a pod:

volumes:
  - name: temp-storage
    emptyDir: {}

YAML to create a PersistentVolumeClaim:

PersistentVolumeClaim (PVC) - For permanent storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce        # Only one pod can write at a time
  resources:
    requests:
      storage: 10Gi        # Request 10 GB storage

Use volume in pod:

containers:
  - name: app
    volumeMounts:
      - name: data
        mountPath: /var/data    # Where data appears in container

volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc

Ingress

An Ingress exposes services to external clients outside the cluster. It provides HTTP/HTTPS routing based on hostnames and paths.

Key characteristics:

It routes external traffic to services (not directly to pods)
It can route based on hostname (example.com, api.example.com)
It can route based on path (/api, /web, /images)
It requires an Ingress Controller to be deployed in the cluster

YAML to create an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: production
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: webapp-service
                port:
                  number: 80

Ingress commands:

kubectl get ingress
kubectl describe ingress <ingress-name>

Hands-On Demo: Deploy Your First Application

Prerequisites

This guide assumes basic familiarity with:

Containers and Docker: Understanding of what containers are and how Docker images work
Command line: Comfortable using bash or terminal commands
YAML syntax: Basic understanding of YAML file format

Local Setup: Getting Kubernetes Running

To follow along with the demo, you'll need:

1. Install kubectl

kubectl is the command-line tool to interact with Kubernetes clusters.

Installation: Follow official kubectl installation guide

Verify:

kubectl version --client

2. Install Minikube

Minikube runs a single-node Kubernetes cluster on your local machine, which is good for learning and development.

Installation: Follow Minikube Getting Started guide

Start your local cluster:

minikube start

# Verify it's running
kubectl cluster-info

Stop the cluster:

minikube stop

Deploy Your Application

Let's create a complete application with configuration and persistence.

Step 1: Create a Namespace

kubectl create namespace demo

Verify:

kubectl get namespaces

Step 2: Create ConfigMap

cat > app-config.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: demo
data:
  APP_NAME: "My Demo App"
  APP_VERSION: "1.0.0"
  LOG_LEVEL: "info"
EOF

kubectl apply -f app-config.yaml

Verify:

kubectl get configmap -n demo
kubectl describe configmap app-config -n demo

Step 3: Create Secret

cat > app-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
  namespace: demo
type: Opaque
data:
  db-password: cGFzc3dvcmQxMjM=    # base64-encoded "password123"
EOF

kubectl apply -f app-secret.yaml

Verify:

kubectl get secrets -n demo
kubectl describe secret app-secret -n demo

Step 4: Create PersistentVolumeClaim

cat > storage.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-storage
  namespace: demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

kubectl apply -f storage.yaml

Verify:

kubectl get pvc -n demo
kubectl describe pvc app-storage -n demo

Step 5: Create Deployment

cat > deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo-app

  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80

          env:
            - name: APP_NAME
              valueFrom:
                configMapKeyRef:
                  name: app-config
                  key: APP_NAME

            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secret
                  key: db-password

          volumeMounts:
            - name: app-storage
              mountPath: /data

      volumes:
        - name: app-storage
          persistentVolumeClaim:
            claimName: app-storage
EOF

kubectl apply -f deployment.yaml

Verify the deployment:

# Check deployment
kubectl get deployment -n demo

# Check replica sets
kubectl get replicaset -n demo

# Check pods
kubectl get pods -n demo

# Get detailed info
kubectl describe deployment demo-app -n demo

Step 6: Create Service

cat > service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: demo-app-service
  namespace: demo
spec:
  type: ClusterIP
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 80
EOF

kubectl apply -f service.yaml

Verify:

kubectl get service -n demo
kubectl describe service demo-app-service -n demo

Step 7: Access Your Application

Option A: Port Forward (Recommended for local testing)

kubectl port-forward svc/demo-app-service 8080:80 -n demo

Open your browser and visit: http://localhost:8080

You should see the nginx welcome page!

Option B: NodePort (Alternative access method)

Modify service.yaml to use NodePort:

cat > service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: demo-app-service
  namespace: demo
spec:
  type: NodePort
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30007
EOF

kubectl apply -f service.yaml

Then access via:

minikube ip  # Get your minikube IP
# Visit http://<minikube-ip>:30007

Step 8: Interact with Your Deployment

Scale up to 5 replicas:

kubectl scale deployment demo-app --replicas=5 -n demo

# Watch new pods being created
kubectl get pods -n demo --watch

View logs from a pod:

kubectl logs <pod-name> -n demo

# Follow logs in real-time
kubectl logs -f <pod-name> -n demo

Execute command inside a pod:

kubectl exec -it <pod-name> -n demo -- bash
# Now you're inside the container
ls /data    # View mounted volume
echo "test" > /data/test.txt
exit

Delete a pod and watch Deployment recreate it:

kubectl delete pod <pod-name> -n demo

# Watch the new pod created automatically
kubectl get pods -n demo --watch

Update the deployment image:

kubectl set image deployment/demo-app nginx=nginx:1.22 -n demo

# Check rollout status
kubectl rollout status deployment/demo-app -n demo

Step 9: Clean Up

# Delete everything in the namespace
kubectl delete namespace demo

# Verify
kubectl get namespaces

Essential kubectl Commands

Here are the most important commands for working with Kubernetes:

# Cluster Information
kubectl cluster-info
kubectl get nodes
kubectl describe node <node-name>
kubectl get events                             # View cluster events
kubectl get events -n <namespace>              # Events in namespace

# Contexts (manage multiple clusters)
kubectl config get-contexts                    # List all contexts
kubectl config current-context                 # Show current context
kubectl config use-context <context-name>     # Switch context

# Resources (Deployments, Pods, Services, etc.)
kubectl get <resource-type>                    # List resources
kubectl get <resource-type> -n <namespace>    # In specific namespace
kubectl get <resource-type> -A                 # All namespaces
kubectl describe <resource-type> <n>        # Detailed info

# Deployments
kubectl get deployments
kubectl describe deployment <n>
kubectl scale deployment <n> --replicas=<n>
kubectl set image deployment/<n> <container>=<image>
kubectl rollout status deployment/<n>
kubectl rollout history deployment/<n>
kubectl rollout undo deployment/<n>

# Pods
kubectl get pods
kubectl get pods -n <namespace>
kubectl describe pod <n>
kubectl logs <pod-name>
kubectl logs -f <pod-name>                     # Follow logs
kubectl exec -it <pod-name> -- bash            # Execute command

# Services and Ingress
kubectl get services
kubectl get svc
kubectl describe service <n>
kubectl get ingress
kubectl describe ingress <n>

# ConfigMaps and Secrets
kubectl get configmaps
kubectl get secrets
kubectl describe configmap <n>
kubectl describe secret <n>

# Create and Delete
kubectl apply -f file.yaml                     # Create or update
kubectl delete -f file.yaml                    # Delete
kubectl delete <resource-type> <n>

# Port Forwarding
kubectl port-forward pod/<pod-name> <local-port>:<pod-port>
kubectl port-forward svc/<service-name> <local>:<remote>

Best Practices

Always use Deployments - Don't use bare pods. Deployments manage scaling and updates.
Use Namespaces - Organize applications and separate environments.
Label your resources - Makes organization and management easier.
Use ConfigMap for configuration - Don't hardcode config in images.
Store secrets securely - Never commit passwords to git.
Check pod logs - First step in debugging issues.
Use meaningful names - For deployments, services, and pods.
Organize with version control - Store YAML files in git.
Test locally first - Use minikube before deploying to cloud.
Start simple - Master basics before exploring advanced features.

Playwright and GitHub Actions - Run tests in CI

Nabin Ale — Mon, 16 Mar 2026 08:49:26 +0000

This blog post explains how to run Playwright UI tests in CI (Continuous Integration) using GitHub Actions.
Before you start, you need to have a basic knowledge of GitHub Actions, Playwright, and the Playwright Trace Viewer to better understand this blog. If you are not familiar with any of these topics, below are links to our blogs that explain each one:

By the end of this blog, you will be able to set up a CI workflow that automatically runs your Playwright UI tests whenever you push code to your repository. You will also learn how to get Playwright trace reports when tests fail, making it easier to debug and fix errors.

🤔 Why run tests on CI?

Tests are run on CI to ensure the code works properly in a clean and isolated environment every time a change is made. Here are some reasons:

1. Early bug detection:
CI runs tests automatically on every push or PR, so you find problems right when they're introduced, not days or weeks later when the context is gone and fixes are harder.

2. No more "Works on my machine":
CI runs your tests in a clean environment every time. If tests pass there, it means your code works in a standard setup, not just on your own computer.

3. Protect the main branch:
Tests on CI act like a gatekeeper. If something breaks, it doesn't get merged. This keeps the main (or master) branch deployable and stable.

4. Confidence to refactor and move fast:
When tests run automatically, you can refactor aggressively. If CI is green, you didn't break existing behavior. That confidence significantly speeds things up.

5. Better collaboration:
CI gives everyone on the team the same test results. This helps avoid confusion and makes it clear whether the code is working or not.

6. Automated process:
CI systems enable developers to save time and effort on testing.

📘 About the Project

In this blog, I have taken a simple application with a frontend built using Vue.js and a backend built using json-server(a fake REST API). The GitHub repository is available at: https://github.com/nabim777/momo-restro-list.git

This is a basic application that includes login and logout functionality. E2E tests are written using Playwright to verify the login feature. You can find the test scenarios here

🛠️ Running App Locally

Before setting up CI, let's run the application and tests locally.

1. Install dependencies

npm ci

2. Start the frontend and backend servers

npm run serve              # Start frontend
npm run backend            # Start backend

🧪 Running UI Tests Locally

NOTE: Make sure both the frontend and backend are running before running the tests.

To run the UI tests locally, use the following command:

npx playwright install chromium
npm run test:e2e tests

⚙️ Setting Up CI in GitHub Actions

After verifying the app locally, the next step is to set up CI using GitHub Actions. First, create a file named ci.yml in your project using the following folder structure:

📦momo-restro-list
┗ 📂.github
┃ ┣ 📂workflows
┃ ┃ ┗ 📜ci.yml

Then, add the following code to the ci.yml file.

name: CI

on:
  push:
    branches:
      - master
  pull_request:
    branches:
      - master
  workflow_dispatch:

jobs:
  e2e-UI-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo code
        uses: actions/checkout@v6

      - name: Set up node
        uses: actions/setup-node@v6
        with:
          node-version: 24.x

      - name: Install dependencies
        run: |
          npm ci
          npx playwright install chromium

      - name: JS lint
        run: |
          npm run lint || (echo "Linting failed! Please run 'npm run lint:fix' to fix the errors." && exit 1)

      - name: Run the project
        run: |
          npm run serve &
          npm run backend &

      - name: Wait for services
        run: |
          sudo apt-get install wait-for-it -y
          wait-for-it -h localhost -p 8080 -t 10
          wait-for-it -h localhost -p 3000 -t 10

      - name: Run UI tests
        id: test-ui
        run: npm run test:e2e tests

      - name: Upload trace results
        if: ${{ failure() && steps.test-ui.conclusion == 'failure' }}
        uses: actions/upload-artifact@v6
        with:
          path: |
            trace-results/*.zip
            retention-days: 30

🔍 What does this workflow do?

This GitHub Actions file runs when you push to the master branch or create a pull request to master.

It has one job called e2e-UI-tests with these steps:

Checkout repo code - Gets the project code from GitHub.
Set up node - Installs Node.js version 24.
Install dependencies - Installs the project dependencies and Playwright browsers.
JS lint - Runs the linter to check for code quality issues. If linting fails, it will print a custom error message and exit with a failure status.
Run the project - Starts the Vue app and the backend using json-server.
Wait for services - Waits for the frontend (port 8080) and backend (port 3000) to be ready.
Run UI tests - Runs the UI tests using Playwright.
Upload trace results - If the UI tests fail, it uploads the Playwright trace results as an artifact that can be downloaded from the GitHub Actions interface. The trace files will be retained for 30 days.

📥 How to Download and View Trace Files?

If your tests fail in CI, you can download the trace files to see what went wrong:

Go to your GitHub repository
Click on the Actions tab
Select the failed workflow run
Scroll down to the Artifacts section at the bottom of the page
Click on the artifact name to download the trace files (they will be in a .zip file)
Extract the downloaded .zip file
Open the trace file using one of these methods:
- Drag and drop the trace file into trace.playwright.dev
- Or run this command locally: npx playwright show-trace path/to/trace.zip

The trace viewer will show you step-by-step what happened during the test, including screenshots, network requests, and console logs.

📝 Conclusion

Using GitHub Actions to run your Playwright tests means your app is tested automatically in a clean environment every time you make changes. It helps find bugs early, makes teamwork easier, and keeps your project stable.

CI tests turn "hope it works" into "we know it works."

Starlark-Powered Pipelines in Woodpecker CI

Prashant Gurung — Thu, 11 Sep 2025 10:49:11 +0000

Background

Continuous Integration tools are essential to build and test modern software automatically and regularly. Woodpecker CI is one such open-source tool, a lightweight CI/CD system. Woodpecker CI is an open-source continuous integration and delivery system that help developers in automating deployment, builds, and testing of code. It's an open-source fork and alternative to the popular Drone CI project with numerous design similarities and Drone pipeline compatibility. Traditionally, Woodpecker pipelines are YAML, but now that programmable pipelines are in vogue, we can finally have the liberty of defining our CI configuration through Starlark — a Python-ish config language.

In this article, I will walk you through setting up Woodpecker CI with Traefik as a reverse proxy and WCCS (Woodpecker CI Config Service) to convert Starlark configurations into YAML files so that we can write pipelines in .star files. Here is a high-level overview of what we are going through:

Spinning up a Woodpecker server with Traefik
Authenticating with GitHub
Enabling a repository
Connecting WCCS with the Woodpecker server

By the end, you’ll be able to write CI pipelines in Starlark and dynamically convert them to YAML during runtime.

Setting Up Woodpecker Server and Agent

Let's start by setting up the Woodpecker server and agent services in the docker-compose.yml file as show below:

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:v3
    environment:
      - WOODPECKER_LOG_LEVEL=debug
      - WOODPECKER_OPEN=false
      - WOODPECKER_HOST=http://<your-ci-server>
      - WOODPECKER_GITHUB=true
      - WOODPECKER_GITHUB_CLIENT=xxxxxxx   # GitHub OAuth client ID
      - WOODPECKER_GITHUB_SECRET=xxxxxxx   # GitHub OAuth secret
      - WOODPECKER_AGENT_SECRET=xxxxxxx    # shared secret between server and agents
      - WOODPECKER_ADMIN=your-github-username
      # The following variable is where we set our Woodpecker CI Config Service (Starlark Conversion Service) later
      - WOODPECKER_CONFIG_SERVICE_ENDPOINT=http://<your-config-service-server>
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    networks:
      - woodpecker-net

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:v3
    restart: always
    depends_on:
      - woodpecker-server
    volumes:
      - woodpecker-agent-config:/etc/woodpecker
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_LOG_LEVEL=debug
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
    networks:
      - woodpecker-net

volumes:
  woodpecker-server-data:
  woodpecker-agent-config:

networks:
  woodpecker-net:
    driver: bridge

Traefik Configuration for HTTPS

So far our woodpecker server runs on HTTP, so to serve it on HTTPS we are going to configure Traefik.
Traefik plays a crucial role in this architecture by handling all the networking complexities - SSL termination, routing, and load balancing.
Add the following Traefik service right above the woodpecker-server service in the same docker-compose.yml file:

services:
  traefik:
    image: traefik:v3.1
    container_name: traefik
    command:
      - "--log.level=DEBUG"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.letsencrypt.acme.email=example@example.com" #letsEncrypt contact email
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - woodpecker-net

  woodpecker-server:
    image: woodpeckerci/woodpecker-server:v3
    ...
    ...
    ...
  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:v3
    ...
    ...
    ...

This configuration:

Sets up Traefik to listen on ports 80 and 443
Configures automatic SSL certificate management via letsEncrypt
Enables Docker provider to automatically detect new services
Mounts necessary volumes for certificate storage and Docker socket access

Now that we have Traefik configured, update the Woodpecker server service by adding following labels:

  woodpecker-server:
    ...
    ...
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.woodpecker-secure.rule=Host(`your-ci-server`)"
      - "traefik.http.routers.woodpecker-secure.entrypoints=websecure"
      - "traefik.http.routers.woodpecker-secure.tls.certresolver=letsencrypt"
      - "traefik.http.routers.woodpecker-secure.tls=true"
      - "traefik.http.services.woodpecker-secure.loadbalancer.server.port=8000"
      # HTTP router + redirect to HTTPS
      - "traefik.http.routers.woodpecker-http.rule=Host(`your-ci-server`)"
      - "traefik.http.routers.woodpecker-http.entrypoints=web"
      - "traefik.http.routers.woodpecker-http.middlewares=redirect-to-https"
      # Redirect middleware
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    networks:
      - woodpecker-net

Authenticating with GitHub

Woodpecker cannot do anything by itself, it needs a forge (the service where the repositories are hosted). Woodpecker supports various forges, for this blog we will focus on GitHub.

To connect GitHub with Woodpecker:

Register a new OAuth application in your GitHub developer settings.
In your GitHub OAuth App, set the authorization callback URL to https://<your-ci-server>/authorize

Copy the generated Client ID and Secret.
Use these values in the WOODPECKER_GITHUB_CLIENT and WOODPECKER_GITHUB_SECRET env vars.

After that, you can log into the Woodpecker web UI using your GitHub ac
count.

Enabling a Repository

Before enabling a repo in the Woodpecker CI server, make sure to have a GitHub repository ready. You can use an existing repo of your own or create a new empty one. We will be making a pull request to the repo to trigger CI later.

From the Woodpecker UI:

Click on Add Repository.

Enable it. For this blog, I am using an existing repo called Drum-KIT which has already been enabled (as you can see in the last row of the image below).

Woodpecker will automatically add the necessary webhooks to the repo.

Triggering Your First CI Run

While we are trying to integrate a starlark conversion service, woodpecker CI supports the default .yml pipelines.
So. now we can trigger our first CI using a traditional YAML configuration as show below.
Create a new branch and add the following configuration in the root of your enabled github repo with name .woodpecker.yml.

steps:
  - name: hello
    image: alpine
    commands:
      - echo "Hello from CI"
      - echo "--------------------"
      - echo "This is my first CI BUILD"
      - echo "--------------------"
    when:
      event: pull_request
      branch: master  # configure it as your repo's base branch.(in my case, it's `master`)

Now that you have your .woodpecker.yaml file ready, you can commit and push it to your enabled GitHub repository and then, open a Pull Request from your branch to master/main according to you repo. GitHub's default branch is set to main.
Once the PR is created, Woodpecker will automatically fetch the pipeline configuration, and start your first CI run. You can then head over to the Woodpecker UI to watch the build logs in real time.

Now that we have already set up the woodpecker server and triggered pipelines using the traditional .yaml configuration, let's now integrate WCCS to unlock flexibility and define our pipelines in Starlark.

Setting Up WCCS-Woodpecker CI Config Service (Starlark Conversion Service)

The Woodpecker Config Conversion Service (WCCS) is a lightweight web service created and maintained by Opencloud-eu. It enables Woodpecker CI to convert pipeline definitions written in Starlark into standard YAML on the fly by receiving a signed POST request from Woodpecker.
You can easily deploy WCCS using their official Docker image available on Docker Hub: opencloudeu/wccs.
Update your docker-compose.yml to include the WCCS service:

...

  wccs:
    image: opencloudeu/wccs:latest
    container_name: wccs
    command: server
    environment:
      - WCCS_LOG_LEVEL=debug
      - WCCS_SERVER_PUBLIC_KEY=/keys/public.pem # path to public key(generated in the section below)
    volumes:
      - /opt/woodpecker/keys:/keys
      - /etc/ssl/certs:/etc/ssl/certs:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wccs.rule=Host(`your-wccs-server`)"
      - "traefik.http.routers.wccs.entrypoints=websecure"
      - "traefik.http.routers.wccs.tls.certresolver=letsencrypt"
      - "traefik.http.services.wccs.loadbalancer.server.port=8080"
    networks:
      - woodpecker-net

...

volumes:
  woodpecker-server-data:
  woodpecker-agent-config:

networks:
  woodpecker-net:
    driver: bridge

Generating the Public Key

Before running any pipeline, woodpecker server sends a POST request to an external config service (in our case WCCS) with all current config file and build information of the current repository. The external service takes the information and sends back the pipeline configurations. In our case, this is how a starlark file is being converted into yaml configurations.

And before WCCS can accept and process those requests, it needs to verify that the requests are trusted and coming from the woodpecker server.

Every request sent by Woodpecker is signed using a http-signature by a private key (ed25519) generated on the first start of the Woodpecker server. You can get the public key for the verification of the http-signature from:

https://<your-ci-server>/api/signature/public-key

Store that public key into keys/public.pem in the same directory level as your docker compose. The public.pem file is then mounted to the WCCS container as shown in the snippet above and used in env variable WCCS_SERVER_PUBLIC_KEY

Connecting WCCS with Woodpecker

To allow Woodpecker to fetch pipeline configs from WCCS, we added this to the server env:

WOODPECKER_CONFIG_SERVICE_ENDPOINT=http://wccs:8080/ciconfig

Woodpecker now sends a signed JSON payload to WCCS whenever a build is triggered. WCCS verifies the signature using the public key and responds with a YAML pipeline based on your .woodpecker.star file.

After conneting WCCS with the woodpecker server, there's one more thing that you need to do i.e. to set pipeline config path, because by default Woodpecker will take .woodpecker.yaml as it's pipeline configuration.

From the settings of your Woodpecker server, go to Repositories and click on the enabled repo's settings icon as shown below:

Then, add .woodpecker.star as pipeline config path and save it:

Add a Simple Starlark Pipeline

Here's a simple starlark pipeline configuration example that you can add in your project.
Create a new branch by checking out from your master/main branch.
Add the following configration in the .woodpecker.star file in your repo's root

def main(ctx):
    return [{
        "name": "hello",
        "steps": [
            {
                "name": "greeting",
                "image": "alpine",
                "commands": [
                    "echo Hello from CI",
                ],
                "when": {
                    "event": ["push", "pull_request"],
                    "branch": ["master"], # configure it as your repo's base branch.(in my case, it's `master`)
                },
            }
        ]
    }]

Triggering CI with starlark-powered pipeline

Now that you have your .woodpecker.star file ready, commit and push it to your GitHub repository and then, open a new Pull Request from your branch to master/main.
Once the PR is created, Woodpecker will automatically send a build request to WCCS, fetch the pipeline configuration, and start your first CI run. You can then head over to the Woodpecker UI to watch the build logs in real time.

Conclusion

This setup provides a robust, secure, and flexible CI/CD pipeline using Woodpecker CI, enhanced with WCCS for configuration management and protected by Traefik. The integration of these components creates a powerful system that can handle complex pipeline configurations while maintaining security and ease of use.

Now, we can write pipelines in Starlark and let WCCS handle the conversion on the fly. This approach brings flexibility, structure, and the power of logic-based configurations to our CI pipelines.

Advanced Docker Container Debugging - A Comprehensive Guide

Prajwol Amatya — Thu, 08 May 2025 08:36:40 +0000

Docker has revolutionized modern software development by enabling lightweight, portable, and scalable containerized applications. However, as deployments grow in complexity, so do the challenges in debugging and optimizing containers. In this blog, we'll explore powerful debugging techniques using a sample project with Node.js, Nginx, and Redis. You can get the sample project here.

Our setup consists of:

A Node.js application (port 3000)
Nginx as a reverse proxy (port 80)
Redis for caching (port 6379)

1. Container Inspection

Viewing Running Containers

First, let's check whether our containers are actually running:

docker compose ps

Example Output:

Name                       Command                         State    Ports
debug-docker_nginx_1       /docker-entrypoint.sh ngin ...  Up       0.0.0.0:80->80/tcp
debug-docker_nodejs-app_1  docker-entrypoint.sh node ...   Up       0.0.0.0:3000->3000/tcp
debug-docker_redis_1       docker-entrypoint.sh redis ...  Up       0.0.0.0:6379->6379/tcp

State: Shows if container is running/stopped
Ports: Reveals port mappings (host:container)

Inspecting Container Details

For deeper inspection of a specific container:

docker inspect debug-docker_nodejs-app_1

This returns a JSON with all container details including:

Network settings
Mounts
Environment variables
IP addresses

Filter specific information:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' debug-docker_nodejs-app_1

Let's breakdown the command:

-f or --format activates Go template formatting
{{range .NetworkSettings.Networks}} iterates through networks
{{.IPAddress}} extracts the IP for each network
{{end}} closes the loop

Example Output:

(The container's internal IP in Docker's network)

172.19.0.3

Docker uses Go Templates for the --format filtering in docker inspect. You can get more details on how to construct the filter in Go's text/template package.
Also, you can learn more on formatting output in Format command and log output section.

2. Log Analysis

Viewing Container Logs

docker compose logs

This will return the logs of containers that are part of the docker compose setup. You can also get the container specific logs using the following command.

docker compose logs nodejs-app

Example Output (when accessing the service):

debug-docker_nodejs-app_1  | Node.js server running on port 3000

This confirms your Node.js application launched successfully.

Generating Request Logs

Make a test request to generate logs:

curl -v http://localhost:3000

After the request, check the logs again to see:

debug-docker_nodejs-app_1  | Node.js server running on port 3000
debug-docker_nodejs-app_1  | GET / 200 7.001 ms - 19

To view only GET requests from the last 5 minutes

docker compose logs --since 5m nodejs-app | grep "GET"

Example Output:

GET / 200 7.001 ms - 19

3. Network Troubleshooting

Checking Container Connectivity

Test if Nginx can reach Node.js:

docker exec debug-docker_nginx_1 ping nodejs-app

docker exec: Executes a command inside a running container (debug-docker_nginx_1)
ping nodejs-app: Calls the Linux ping utility to test network reachability to the hostname nodejs-app

Example Output:

PING nodejs-app (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.060 ms

Success: <1ms response confirms network connectivity
Failure: Would show "unknown host" or timeout

Docker Compose automatically assigns hostnames to containers based on the service names defined in docker-compose.yml.

Example:

services:
  nodejs-app: # This becomes the hostname
    image: node:alpine

Run docker compose ps to see the exact service/container names:

docker compose ps --format "table {{.Name}}\t{{.Service}}"

Output:

NAME                        SERVICE
debug-docker_nginx_1        nginx
debug-docker_nodejs-app_1   nodejs-app

Alternatively, you can use service names directly from docker-compose.yml instead of full container name.

docker compose exec nginx ping nodejs-app

Example Output:

PING nodejs-app (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.042 ms

Examining Post Accessibility

Check if Node.js is listening on port 3000 inside its container:

docker exec nodejs-app netstat -tuln

docker exec: Runs a command inside a specific container (nodejs-app).
netstat -tuln: A Linux utility to list all listening network ports with the flags:
- -t: Show TCP ports
- -u: Show UDP ports
- -l: Display only listening ports (services accepting connections)
- -n: Show numeric addresses/ports

Example Output:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 :::3000                 :::*                    LISTEN

Shows Node.js listening on port 3000
No output means the service isn't running properly

4. Interactive Debugging

Executing into Containers

For Node.js application debugging:

docker exec -it nodejs-app sh

This gives you full shell access inside the container where you can run any Linux command (as long as the tool exists in the container). Following are few things that you can do.

Check running processes: ps aux

ps aux

Output:

PID   USER     TIME  COMMAND
    1 root      0:00 {MainThread} node app.js
   28 root      0:00 sh
   45 root      0:00 ps aux

Shows all running processes in the container
PID1: Your Node.js application (node app.js)
PID28: The shell session you just started
PID 45: The ps aux command itself
Confirms your application is running as the main process

Examine environment variables: printenv

printenv

Output:

NODE_VERSION=23.11.0
HOSTNAME=ada88201c429
YARN_VERSION=1.22.22
SHLVL=1
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/app

Displays all environment variables to your Node.js app

Test Redis connectivity: redis-cli -h redis ping

redis-cli -h redis ping

Output:

PONG

Tests connectivity to your Redis container
PONG response confirms network connectivity

Debugging Nginx Configuration

docker exec -it debug-docker_nginx_1 nginx -t

Example Output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

5. Health Checks and Readiness Probes

Docker health checks are automated tests that periodically verify if a container is functioning properly. Health checks transform your containers from static processes into self aware services that can catch issues like application crashes, frozen processes, dependency failures, etc.
Let's enhance our docker-compose.yml with health checks:

services:
  nodejs-app:
    # ... existing config ...
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000"]
      interval: 30s
      timeout: 10s
      retries: 3

  redis:
    # ... existing config ...
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 5s
      retries: 3

Docker runs the test command at your specified interval (e.g., every 30s)
The service is marked healthy only if the command succeeds (exit code 0)
After retries consecutive failures, it's marked unhealthy

Now check the container health:

docker ps --format "table {{.Names}}\t{{.Status}}"

Example Output:

NAMES                     STATUS
debug-docker_nginx_1      Up 5 minutes
debug-docker_nodejs-app_1 Up 5 minutes (healthy)
debug-docker_redis_1      Up 5 minutes (healthy)

6. Temporary Debug Containers

Sometimes you need additional tools that are not part of the containers you are using, for that create a temporary debug container with the needed tools in the same network:

docker run -it --rm --network debug-docker_default alpine sh

Now from this container you can:

Test DNS resolution:

nslookup nodejs-app

Example Output:

Server:     127.0.0.11
Address:    127.0.0.11:53

Non-authoritative answer:
Name:   nodejs-app
Address: 172.19.0.3

Confirms Docker's internal DNS is working
Shows the service name resolves to the correct container IP (172.19.03)

Check connectivity:

wget -qO- http://nodejs-app:3000

Example Output:

Hello from Node.js!

Shows successful TCP connection to port 3000
Returns the actual HTTP response from your Node.js app

7. Docker System Diagnostics

When facing resource issues:

docker system df

Example Output:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          17        3         2.374GB   2.295GB (96%)
Containers      3         3         2B        0B (0%)
Local Volumes   2         1         88B       88B (100%)
Build Cache     108       0         37.82MB   37.82MB

Check detailed resource usage:

docker stats

Example Output:

CONTAINER ID  NAME                      CPU %  MEM USAGE/LIMIT   MEM %  NET I/O       BLOCK I/O PIDS
5111f46d640b  debug-docker_nginx_1      0.00%  9.633MiB/31.06GiB 0.03%  41.9kB/126B   0B/4.1kB  13
e701e4d02bb0  debug-docker_nodejs-app_1 0.00%  13.31MiB/31.06GiB 0.04%  45.3kB/3.49kB 0B/0B     7
3e0399cc7510  debug-docker_redis_1      0.93%  4.691MiB/31.06GiB 0.01%  42.3kB/126B   1.43MB/0B 6

Conclusion

Effective Docker debugging requires a systematic approach. By mastering these techniques, you'll be able to diagnose and resolve even the most complex Docker issues in production environments.

PHP 8.4 - Discover the Latest and Greatest

Isha Baral — Tue, 10 Sep 2024 10:08:51 +0000

Scheduled for release on November 21, 2024, PHP 8.4 packs some exciting new features and improvements. In this blog post, we'll explore some of the most interesting additions and changes:

New array helper functions
Property hooks
'new' without parentheses
Implicitly nullable parameter declarations deprecated
New multibyte functions

1. New array helper functions

The following variants of array helper functions will be added in PHP 8.4:

array_find()
array_find_key()
array_any()
array_all()

These functions will take an array and a callback function and return the following:

functions	Return value
`array_find()`	Returns the first element that meets the callback condition; `NULL` otherwise.
`array_find_key()`	Returns the key of the first element that meets the callback condition; `NULL` otherwise.
`array_any()`	Returns `true` if at least one element matches the callback condition; `false` otherwise.
`array_all()`	Returns `true` if all elements match the callback condition; `false` otherwise.

Note: array_find() retrieves only the first matching element. For multiple matches, consider using array_filter().

Example

Given an array with key-value pairs and a callback function:

$array = ['1'=> 'red', '2'=> 'purple', '3' => 'green']

function hasLongName($value) {
  return strlen($value) > 4;
}

Here's how we can use the new functions:

array_find():

  // Find the first color with a name length greater than 4

  $result1 = array_find($array, 'hasLongName');

  var_dump($result1);  // string(5) "purple"

array_find_key():

  // Find the key of the first color with a name length greater than 4

  $result2 = array_find_key($array, 'hasLongName');

  var_dump($result2);  // string(1) "2"

array_any():

  // Check if any color name has a length greater than 4

  $result3 = array_any($array, 'hasLongName');

  var_dump($result3);  // bool(true)

array_all():

  // Check if all color names have a length greater than 4

  $result4 = array_all($array, 'hasLongName');

  var_dump($result4);  // bool(false)

2. Property hooks

PHP 8.4 introduces property hooks, offering a more elegant way to access and modify private or protected properties of a class. Previously, developers relied on getters, setters, and magic methods (__get and __set). Now, you can define get and set hooks directly on a property, reducing the boilerplate code.

Instead of ending the property with a semicolon, we can use a code block {} to include the property hook.
These hooks are optional and can be used independently. By excluding one or the other we can make the property read-only or write-only.

Example

class User
{
  public function __construct(private string $first, private string $last) {}

  public string $fullName {
    get => $this->first . " " . $this->last;

    set ($value) {
      if (!is_string($value)) {
        throw new InvalidArgumentException("Expected a string for full name,"
        . gettype($value) . " given.");
      }
      if (strlen($value) === 0) {
        throw new ValueError("Name must be non-empty");
      }
      $name = explode(' ', $value, 2);
      $this->first = $name[0];
      $this->last = $name[1] ?? '';
    }
  }
}

$user = new User('Alice', 'Hansen')
$user->fullName = 'Brian Murphy';  // the set hook is called
echo $user->fullName;  // "Brian Murphy"

If $value is an integer, the following error message is thrown:

PHP Fatal error:  Uncaught InvalidArgumentException: Expected a string for full name, integer given.

If $value is an empty string, the following error message is thrown:

PHP Fatal error:  Uncaught ValueError: Name must be non-empty

3. 'new' without parentheses

PHP 8.4 introduces a easier syntax, allowing you to chain methods on newly created objects without parenthesis. Although this is a minor adjustment, it results in cleaner and less verbose code.

(new MyClass())->getShortName();  // PHP 8.3 and older
new MyClass()->getShortName();  // PHP 8.4

Besides chaining methods on newly created objects, you can also chain properties, static methods and properties, array access, and even direct invocation of the class. For example:

new MyClass()::CONSTANT,
new MyClass()::$staticProperty,
new MyClass()::staticMethod(),
new MyClass()->property,
new MyClass()->method(),
new MyClass()(),
new MyClass(['value'])[0],

4. Implicitly nullable parameter declarations deprecated

Before PHP 8.4, if a parameter was of type X, it could accept a null value without explicitly declaring X as nullable. Starting with PHP 8.4, you can no longer declare a null parameter value without clearly stating it as nullable in the type hint; otherwise, a deprecation warning will be triggered.

function greetings(string $name = null)  // fires a deprecation warning

To avoid warnings, you must explicitly state that a parameter can be null by using a question mark (?) in the type declaration.

function greetings(?string $name)

or,

function greetings(?string $name = null)

5. New multibyte functions

A multibyte string is a sequence of characters where each character can use more than one byte of storage. This is common in languages with complex or non-Latin scripts, such as Japanese or Chinese. There are several multibyte functions in PHP such as mb_strlen(), mb_substr(), mb_strtolower(), mb_strpos(), etc. But some of the functions like trim(), ltrim(), rtrim(), ucfirst(), lcfirst() etc. lack direct multibyte equivalents.

Thanks to PHP 8.4, where new multibyte functions will be added. They include: mb_trim(), mb_ltrim(), mb_rtrim(), mb_ucfirst()and mb_lcfirst(). These functions follow the original function signatures, with an additional $encoding parameter.
Let's discuss the new mb_functions:

mb_trim():

Removes all whitespace characters from the beginning and end of a multibyte string.

Function signature:
```
  function mb_trim(string $string, string $characters = " \f\n\r\t\v\x00\u{00A0}\u{1680}\u{2000}\u{2001}\u{2002}\u{2003}\u{2004}\u{2005}\u{2006}\u{2007}\u{2008}\u{2009}\u{200A}\u{2028}\u{2029}\u{202F}\u{205F}\u{3000}\u{0085}\u{180E}", ?string $encoding = null): string {}
```
Parameters:
- $string: The string to be trimmed.
- $characters: An optional parameter that includes a list of characters to be trimmed.
- $encoding: The encoding parameter specifies the character encoding used to interpret the string, ensuring that multibyte characters are processed correctly. Common encodings include UTF-8.

mb_ltrim():

Removes all whitespace characters from the beginning of a multibyte string.

Function signature:

  function mb_ltrim(string $string, string $characters = " \f\n\r\t\v\x00\u{00A0}\u{1680}\u{2000}\u{2001}\u{2002}\u{2003}\u{2004}\u{2005}\u{2006}\u{2007}\u{2008}\u{2009}\u{200A}\u{2028}\u{2029}\u{202F}\u{205F}\u{3000}\u{0085}\u{180E}", ?string $encoding = null): string {}

mb_rtrim():

Removes all whitespace characters from the end of a multibyte string.

Function signature:

  function mb_rtrim(string $string, string $characters = " \f\n\r\t\v\x00\u{00A0}\u{1680}\u{2000}\u{2001}\u{2002}\u{2003}\u{2004}\u{2005}\u{2006}\u{2007}\u{2008}\u{2009}\u{200A}\u{2028}\u{2029}\u{202F}\u{205F}\u{3000}\u{0085}\u{180E}", ?string $encoding = null): string {}

mb_ucfirst():

Converts the first character of a given multibyte string to title case, leaving the rest of the characters unchanged.

Function signature:
```
  function mb_ucfirst(string $string, ?string $encoding = null): string {}
```
mb_lcfirst():

Similar to mb_ucfirst(), but it converts the first character of a given multibyte string to lowercase.

Function signature:
```
  function mb_lcfirst(string $string, ?string $encoding = null): string {}
```

Conclusion

I hope this blog has given you a good overview of some of the upcoming changes in PHP 8.4. The new version appears to introduce exciting updates that will enhance the developer experience. I’m eager to start using it once it’s officially released.
For more information and updates, please visit the official RFC page.

Python's shutil module for Automated Testing

Salipa Gurung — Wed, 21 Aug 2024 04:55:01 +0000

Introduction

In the dynamic domain of software development, ensuring application quality through automated testing is vital. Automated testing involves running numerous tests that require a specific setup of files and directories. Setting up and tearing down test environments efficiently is a significant part of this process which can often be a tedious and error-prone task without the right tools.

Finding the right tools to simplify tasks can make a big difference. The standard utility module known as shutil is an abbreviation for shell utilities. It provides high-level file operations for copying, moving, and removing files and directories that can simplify managing test environments. This blog explains how the shutil module can streamline file operations in automated software testing.

To ensure that each test starts with the correct environment, we need to automate the setup process, creating a consistent and isolated test environment.

Getting started with `shutil`

Shutil module is part of the Python Standard Library, so we don’t need to install anything extra. Simply import it at the beginning of your script.

# Import shell utility module
import shutil

Essential file copying methods

While testing an application, we often require a specific set of data files. Instead of manually copying files, four common methods provide an easy and efficient way to copy a file using the shutil module.

shutil.copy()
shutil.copy2()
shutil.copyfile()
shutil.copyfileobj()

1. shutil.copy() method

shutil.copy(source, destination, *, follow_symlinks=True)

This method copies the source file to the directory or file specified in the destination and returns the file's destination. It copies the file's data and the file's permission mode but metadata (file creation and modification time) of the file copied is not preserved. The source and destination should be path-like objects or strings. The destination can specify both file or directory. If the destination specifies a path to a directory, the file will be copied to the destination directory with the file name of the source path.
While working with symbolic link:

If follow_symlinks=True, the destination will be a copy of the file that the symbolic link in source points to. The default value of parameter follow_symlinks is True.
If follow_symlinks=False, destination will be created as a symbolic link.

# Import shell utility module
import shutil

#  Copy the content of source_file.txt to destination_file.txt
shutil.copy('source_file.txt', 'destination_file.txt')

2. shutil.copy2() method

shutil.copy2(source, destination, *, follow_symlinks=True)

The shutil.copy2() method provides the additional functionality of preserving all the metadata of the file it can. Other characteristics are the same as for the shutil.copy() method. The destination can specify both file or directory. If the destination is a directory, the file will be copied with the same name as the source file name.

# Import shell utility module
import shutil

#  Copy the content of source_file.txt to destination_file.txt
shutil.copy2('source_file.txt', 'destination_file.txt')

I used this method while automating test as it copies the file content along with most of its associated metadata so that we can programmatically achieve the same result as copying and pasting a file using keyboard shortcuts in a graphical file manager and it also closely replicates the behavior of a typical copy and paste operation using a mouse in graphical file manager.

3. shutil.copyfile() method

shutil.copyfile(source, destination, *, follow_symlinks=True)

Using this method, a file's content is copied from its source to its destination without metadata. The source and destination should be path-like objects or strings. Destination path can only specify a path to a file but cannot specify a path to a directory.

# Import shell utility module
import shutil

#  Copy the content of source_file.txt to destination_file.txt
shutil.copyfile('source_file.txt', 'destination_file.txt')

4. shutil.copyfileobj() method

shutil.copyfileobj(fsrc, fdst, length)

This method copies the contents of the source file object to the destination file object. The source and destination must be file-like objects. It does not preserve metadata and does not return any value. The optional length parameter specifies an integer value for buffer size.

# Import shell utility module
import shutil

# Create file objects
source_file_object = open("../hello.txt", "r")
destination_file_object = open("CopyFile/copyFalseSrcPath.txt", "w")

# Copy the content of the file object to another file object
shutil.copyfileobj(source_file_object, destination_file_object)

Preventing common mistakes:

The source and destination cannot specify the same path in shutil.copy(), shutil.copy2() and shutil.copyfile(). If the source and destination specify the same file path then the exception "SameFileError" will be raised. This error is not raised in shutil.copyfileobj() as it overwrites the destination file if it already exists.
An error "FileNotFound" will be raised if the source specifies a path to a file that is not present.

Conclusion

Each method provides different aspects of file operations, providing flexibility and power to meet various testing needs. Python's shutil module offers essential tools for efficient file operations in software testing:

shutil.copy: Great for basic file copying, including permissions.
shutil.copy2: Ideal for copying files along with most of their metadata, including timestamps.
shutil.copyfile: Best for fast, simple content-copying without metadata.
shutil.copyfileobj: Enables precise copying between file-like objects.

Some other commonly used utility functions that can be helpful while automating our tests are:

shutil.mkdirs(): Creates directories recursively. Useful for setting up directory structures needed for testing or organizing files.
shutil.copytree(): Recursively copies an entire directory tree. Ideal for duplicating complex directory structures, including all files and sub-directories, to create or backup environments for testing or deployment.
shutil.rmtree(): Deletes a directory tree recursively. Essential for cleaning up after tests or removing temporary directories and their contents completely.
shutil.move(): Moves a file or directory to another location. Efficiently relocates files or directories within the filesystem, facilitating organization or deployment process.
shutil.make_archive(): Creates an archive (such as AIP or TAR) of a directory and its contents. Useful for packaging directories for distributions or backup purposes.

More utility functions and detailed descriptions can be found in the official documentation.

The shutil module in Python serves as a versatile toolset not only in software testing but also across diverse fields of software development such as version control and deployment, data science and machine learning, IT and system administration, research and academia, and many more.

Integrating shutil into our testing processes, we can perform complex file operations with minimal code, saving our time and effort and allowing us to focus on developing high-quality software.

20 Useful and Exciting Technical Terms for Junior Programmers 🚀💻

Pradip Subedi — Wed, 21 Aug 2024 04:52:22 +0000

Many junior-level techies, especially the newcomers in coding, can admit to the fact that sometimes attending their first stand-up meeting in a professional setup can be pretty shaky simply because you have no idea what they are talking about. In this blog post, I am going to share 20 different and cool terminologies that can give you an extra edge in your knowledge and domain. With that in mind, we will delve into these key terms, allowing you to navigate the rapidly changing tech-world landscape. Let's go through these one by one:

Development Practices

1. Scrum

Scrum can be your life raft if you ever feel like any of your projects are stuck in a never-ending waterfall, with deadlines constantly whooshing away. It depicts a software development process that breaks down big tasks into manageable pieces, with regular feedback and adjustments along the way. It is all about teamwork and flexibility, allowing us to adapt to changes very quickly. You can think of it as a team sport - everyone works together in short bursts (called sprints) to deliver a specific chunk of the project. Regular follow-ups and reviews keep everyone on the same path, ensuring the final product delivers what clients truly need. Scrum is all about agility and drop-shipping value fast, making it a popular choice for software development!

2. Backlog

A backlog is a secret weapon for a company having tasks and ideas overflowing and keeping them well organized. It is mostly a prioritized to-do list for features, bug fixes and anything else that needs tackling. The most important are listed at the top, ensuring the team focuses on what matters the most. You can imagine it as a constantly evolving board of lists - new tasks get added, some get tackled and deleted, while others get rearranged based on urgency. This dynamic approach makes the development process very attentive and ensures the team delivers the most valuable features first!

3. Retrospective

A retrospective is one of the key meetings in Agile Software Development. Conducted after a project, or a sprint, it is a meeting where the team contemplates what they did right, and what they need to improve. When improvements are identified they should be documented and put in place, then revisited in future retrospectives to see if they made a difference. Generally, the participants are developers, product owners/clients and scrum master(s). Retrospectives are a priceless tool for boosting team dynamics, workflows and productivity.

4. YAGNI

YAGNI (You Aren't Gonna Need It) - It is a practice in extreme programming and other agile practices that tells not to implement any functionality until it is needed.
YAGNI tells developers to only concentrate on features that are demanded in the present task. This is to prevent solutions from being built that will never be used, and keep a codebase clean during its life. There might be some situations when you get a YAGNI comment from seniors.😸

Practical example:

Suppose you were creating a task management app. Even though you were tempted to include everything during the first iteration like email notifications, calendar integrations etc., YAGNI suggests that you should only put them into action when they are needed. The foremost priority should be core functionalities: adding, editing and deleting tasks. If users ask for those advanced features in the future, those can be added only at that time.
In this way, embracing YAGNI helps you keep your development process lean and deliver just what is needed without overengineering the list of features to be covered.

Version Control

5. Version Control System (VCS)

VCS is a platform where changes to a codebase are logged using a tool. Sometimes we may get worried that continuously rewriting the same code and one small mistake could destroy the entire project. But there's a superhero in the developer world ready to save our day: Version Control Systems (VCS)! This feature allows any number of developers to work on the same project without clashing and records changes in a history with who did what, and why. Key features of VCS include:

Version Control: Handles the history of the changes and keeps versioned.
Branching and merging: Allows developers to work separately on features or bug fixes.
Collaboration: Leads to seamless integration of modifications across multiple developers working on the same codebase.
Backup & Restore: If failures and errors occur, we can rollback to the previous version.
Solving Issues: In case of overlapping changes, we are able to resolve discrepancies.

Git, Subversion (SVN), and Mercurial are some of the popular VCS tools. For instance, Git and platforms like Github or GitLab provide an opportunity to clone repositories, create branches and merge changes which helps to coordinate to work optimally in terms of teamwork as well as project management.
Want to geek out even more? Have fun exploring, Git vs Mercurial vs SVN (Subversion)!

6. Porting Commits

Porting commits is where you move or "port" the changes from one Git branch to another. Usually, this is required if you have any commits in one branch that you want to add to another branch but not by merging the whole branch. Generally, there are two ways of porting commits which are:

Forward-Porting a commit:
Forward-porting is the practice of moving changes from an older branch to a newer branch. That way any fix or update done will be in the main development branch. This includes finding appropriate commits, cherry-picking them to the branch created for it, resolving any conflicts and running tests thoroughly on these changes to make sure nothing broke during integration. This way, a version of the codebase is made consistent and stable across different implementations.
Backporting a commit:
Backporting is when software has some patches or updates in a recent version and they need to be applied to an older version of the same software. It includes: identifying the patches to be done, adapting those patches to be compatible with older versions of the same product, dealing with any undesired side effects surfaced during the last step and finally deploying the backported patch.

7. Bump

In general, when developers increase the version number of the software they use the term Bump. Most of the time, this version number change also signals changes that have been made to the product. Think of it as updating the version of a book. A higher version number points to a newer version of the software, usually with bug fixes or new features and other improvements. Increasing the version number helps developers/users see that a change has been made and helps them ensure that they are using the most compatible and latest version.

Testing

8. Quality Assurance (QA)

Imagine a team committed to ensuring software is flawless before it reaches clients or end users. That's the world of Quality Assurance (QA)! QA is not just about testing; it is a practice to ensure that defects are identified and prevented during the development lifecycle. Consider QA engineers the gatekeepers of software brilliance. They work in tandem to developers to make sure the software possesses the highest quality standards. From reading requirements to test case designing and early identification of issues, QAs play various roles in a project. Their goal? - delivering a user-friendly and seamless working final product that is close to being perfect.

9. Unit Testing

Unit testing is a testing method in which every component of software is tested to check if it is working accurately in those smaller components. It means catching bugs and other types of errors closer to the time they are introduced which results in having more reliable code and easier maintenance.
Key Points:

Isolation: Tests are written for small units of code like functions/methods, separately from other parts of the application.
Automation: With unit testing, tests are automated and can be executed continuously as often as required to fix issues immediately.
Refactoring Safety: Verify that changes made don't break the previous functionality of the application.

10. Regression Testing

Suppose you've patched a bug or added any code/tests to your software. Great work! What if that fix/addition broke something else that previously worked fine? That's where Regression testing shines. It's a safety net that ensures recently made changes haven't caused any unintended regressions, or reintroduced old bugs.

So, nearly like checking your work before saving your changes in a document, regression testing is performing old test cases on a new release to ensure that previously implemented functionality still works as it needs to, after changes have been made. This catches any regressions right away and prevents them from reaching users.

Automation and Deployment

11. CI/CD (Continuous Integration/Continuous Deployment)

CI/CD sounds pretty complicated, but it will completely rock your world if you can manage to get it working. It automates the testing and deployment of your code using something called a pipeline. A pipeline is pretty much a conveyor belt for your code; it tests every change you make and makes it available for deployment to production. This automation helps to reduce time and minimize errors that can slow down your development process.

12. Docker

Docker helps you bundle your code and its dependencies into a package which is like a secret handshake. No matter where you deploy it, this package ensures that your code runs exactly as you designed. These packages are called Docker images. From a single image, you can create isolated environments called Docker containers. Each container acts like its mini-pc, sharing resources efficiently. This consistency makes the flow of development, testing and deployment so much easier - all a breeze!

13. Cron Jobs

Cron jobs are simply time-based tools used to automate repetitive tasks. You can also define the frequency and timing of an actionable task by using a specific syntax called the cron expression. This improves productivity and makes sure that critical procedures are carried out in a consistent way.

Some use-cases for Cron Jobs

Automatically performing a nightly backup
Automated weekly reporting
Auto-installing software

Coding Tools

14. cURL

cURL is a short term for "client URL" and is pronounced and typed as curl. It is an open-source software that is utilised in command lines or scripting to perform data transfers. cURL is built to support the transfer protocols like HTTP, HTTPS, IMAP and many others. It further gives the ability to use cookies, set proxies, provide authentication credentials and even headers while making requests. Thus it is an all-purpose tool for data transfer & network communication.

Syntax for using cURL:
curl [options] [URL]

15. Linter

The process of analyzing the source code and reporting common syntactic and semantic errors or potential bugs is called linting and is done by using a linter.
Think of a project you are working on using JavaScript. Even though you are writing the code yourself, how do you guarantee your code to be without any errors or faults and also follow coding standards? That's where a linter like ESLint is important. It automatically checks the quality of your code such as checking typos and applying coding guidelines. For example, it can warn when a variable is declared but never used or normalize quotes to either single or double.

Web Development

16. HTTP methods

Simply, HTTP methods are actions and provide a way to indicate the desired action to be performed on a given resource within a web server. They are crucial parts of the HTTP protocol that are used in communication on the web. Common HTTP methods are:

GET method: The GET method is designed to request a specific resource.
POST method: The POST method is designed to submit data to be processed by a specified resource. It is akin to completing and sending a form.
PUT method: The PUT method is designed to replace an entire resource with new content. It's pretty much similar to saving an edited document.
DELETE method: When the DELETE method is triggered, the server removes the resource identified by that request. This is like deleting a file.

Example scenario:

When you open any webpage, the browser sends a GET request to its server to get its page data. When you fill out a form it actually is sending your data over with a POST request.

17. API

An API (Application Programming Interface) is a set of specifications that allows different applications to communicate, and it provides an interface for developers to interact with other services. So we can picture it as a translator that helps two people speaking different languages understand each other.
Key Features:

Scalability: Easy to extend and adapt for interoperation with other systems and applications
Efficiency: Can be embedded in other applications with great ease, thus giving you the freedom to utilize present functionality again instead of writing code from scratch.
Security: Ensures secure ways of accessing resources and a healthy trade of data.

18. Cookies and Sessions

Cookies and sessions are tools to store user data/state in our web applications. It enables websites to remember their own users so as to create a more personalized experience.

Cookies:

Meaning: It is the tiny piece of data that is stored by a website on your browser.
Goal: For the purpose of remembering user activity, login details and preferences.
Example: Logging-in to certain websites, and checking the "remember me" box will store a cookie in your web browser.

Sessions:

Meaning: Some type of temporary storage which includes server-side and client-side components that are used to store user data during the visit from the time a visitor sits on a page until when they leave (session).
Goal: To maintain the state of the user and data during navigation across a website.
Example: When you add items to a cart on an e-commerce site, a session is created to keep the item in the cart until payment is finalized or the user leaves the site.

Authentication

19. Single Sign-On (SSO)

SSO is an authentication process which allows users to enter one set of login credentials for multiple applications. It simplifies the user experience by removing the requirement to remember unique usernames and passwords for each application. That is nice for users but it also improves the security of your application and makes controlling user access easier by having one central location where validation occurs.
Example Scenario:
Suppose, you are logging into your company's intranet. SSO allows you to log into email, project management accounts and some other tools without having to login for each.
SSO simplifies the log-in process and boosts user efficiency while increasing security as it centralizes credential management.

Documentation

20. User Documentation

User documentation is a high level guide for assisting users to understand and utilize products or services. It provides examples, FAQs, explanations, and troubleshooting advice making the service easily navigable and user-friendly.

Picture yourself learning how to use a new code library for the first time as a developer and it would be overwhelming without clear instructions. This is where User documentation saves the day! It includes step-by-step guides for how to integrate the library, answers the most common questions and provides solutions for typical problems.

Proper documentation is like having the knowledge of a colleague sitting beside you to help solve any barriers to ensure you fully utilize the features of the service/product.

Conclusion

Learning to code can be daunting, however understanding the jargon associated with it may help make the journey more interesting. With a grasp on these 20 crucial terms, you will be able to easily traverse the ever-evolving sphere of software development. Keep exploring, keep learning and most importantly; code away. Happy programming! 🚀💻

💡 Note
Image for the banner is taken from: Undraw

Integrating oCIS With Keycloak

Amrita Shrestha — Tue, 20 Aug 2024 08:35:48 +0000

In the previous blog, we explored the general functionality of Keycloak.
In this blog, I demonstrate how we can integrate oCIS(file-sync and share platform) with Keycloak using the Keycloak WebUI.
This benefits anyone who is interested in getting hands-on experience with Keycloak.

Run Services

Make sure you've installed Docker Engine and Docker Compose.

Before setting up oCIS with Keycloak, it's essential to deploy both oCIS and Keycloak services. I'll describe how to achieve it with docker compose.
oCIS and Keycloak runs behind Traefik as reverse proxy. Traefik generates self-signed certificates for oCIS and Keycloak.

You have to create a csp.yaml file with the following contents:

directives:
  child-src:
    - '''self'''
  connect-src:
    - '''self'''
    # In contrast to bash and docker the default is given after the | character
    - 'https://${KEYCLOAK_DOMAIN|host.docker.internal:8443}'
  default-src:
    - '''none'''
  font-src:
    - '''self'''
  frame-ancestors:
    - '''none'''
  frame-src:
    - '''self'''
    - 'https://embed.diagrams.net/'
  img-src:
    - '''self'''
    - 'data:'
    - 'blob:'
  manifest-src:
    - '''self'''
  media-src:
    - '''self'''
  object-src:
    - '''self'''
    - 'blob:'
  script-src:
    - '''self'''
    - '''unsafe-inline'''
  style-src:
    - '''self'''
    - '''unsafe-inline'''

You can either download the docker code from GitHub or utilize the code provided below:

Note: PROXY_CSP_CONFIG_FILE_LOCATION should contain the exact location of a csp.yaml file

services:
  traefik:
    image: traefik:2.11.0
    command:
      [
        "--log.level=ERROR",
        "--api.insecure=true",
        "--api.dashboard=true",
        "--pilot.dashboard=false",
        "--providers.docker=true",
        "--entrypoints.ocis.address=:9200",
        "--entrypoints.keycloak.address=:8443",
        "--providers.docker.exposedbydefault=false",
        "--entrypoints.websecure.http.tls.options=default",
      ]
    ports:
      - 8080:8080
      - 9200:9200
      - 8443:8443
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      traefik.enable: true
      traefik.http.routers.traefik.rule: HostRegexp(`{any:.+}`)
    restart: always

  ocis:
    image: owncloud/ocis:latest # ocis version = v5.0.5
    entrypoint:
      - /bin/sh
    command: ["-c", "ocis init || true; ocis server"]
    environment:
      # Keycloak IDP configuration
      PROXY_AUTOPROVISION_ACCOUNTS: true
      PROXY_ROLE_ASSIGNMENT_DRIVER: oidc
      OCIS_OIDC_ISSUER: https://host.docker.internal:8443/realms/oCIS # ocis is keycloak realm name
      PROXY_OIDC_REWRITE_WELLKNOWN: true
      WEB_OIDC_CLIENT_ID: web
      # general config
      OCIS_URL: https://host.docker.internal:9200
      OCIS_LOG_LEVEL: error
      OCIS_INSECURE: true
      OCIS_ADMIN_USER_ID: ""
      OCIS_EXCLUDE_RUN_SERVICES: idp
      GRAPH_ASSIGN_DEFAULT_USER_ROLE: false
      GRAPH_USERNAME_MATCH: none
      # PROXY
      PROXY_TLS: false
      PROXY_USER_OIDC_CLAIM: preferred_username
      PROXY_USER_CS3_CLAIM: username
      PROXY_CSP_CONFIG_FILE_LOCATION: /etc/ocis/csp.yaml
      KEYCLOAK_DOMAIN: ${KEYCLOAK_DOMAIN:-host.docker.internal:8443}
    volumes:
      - ./csp.yaml:/etc/ocis/csp.yaml
    extra_hosts:
      - host.docker.internal:host-gateway
    labels:
      traefik.enable: true
      traefik.http.routers.ocis.tls: true
      traefik.http.routers.ocis.rule: PathPrefix(`/`)
      traefik.http.routers.ocis.entrypoints: ocis
      traefik.http.services.ocis.loadbalancer.server.port: 9200

  postgres:
    image: postgres:alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: keycloak

  keycloak:
    image: quay.io/keycloak/keycloak:24.0.1
    command: ["start-dev", "--proxy=edge"]
    environment:
      KC_DB: postgres
      KC_DB_URL: "jdbc:postgresql://postgres:5432/keycloak"
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD: keycloak
      KC_FEATURES: impersonation
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    extra_hosts:
      - host.docker.internal:host-gateway
    labels:
      traefik.enable: true
      traefik.http.routers.keycloak.tls: true
      traefik.http.routers.keycloak.rule: PathPrefix(`/`)
      traefik.http.routers.keycloak.entrypoints: keycloak
      traefik.http.services.keycloak.loadbalancer.server.port: 8080
    depends_on:
      - postgres

volumes:
  postgres_data:

Save the above code in compose.yaml file locally and run docker command docker compose up in a terminal.

Docker compose serves three services:

oCIS on https://host.docker.internal:9200
Keycloak on https://host.docker.internal:8443
Traefik dashboard on http://host.docker.internal:8080/

Manual Steps For Setting Up oCIS With Keycloak

Go to oCIS url: https://host.docker.internal:9200/ > Note: The oCIS WebUI will not be accessible until Keycloak and oCIS are fully integrated. The discovery endpoint is unavailable, which is why the web interface is inaccessible.

Go to Keycloak url: https://host.docker.internal:8443/

Create new realm:

Realm name: oCIS > Note: the realm name should be the name provided in the oCIS environment variable: OCIS_OIDC_ISSUER: https://host.docker.internal:8443/realms/oCIS

Add a new client:

Client ID: web

Root URL: https://host.docker.internal:9200

Add realm roles:

Role name: ocisAdmin

Note: oCIS has defined role names that have been documented under Automatic Quota Assignments. So, we can't create the role with a random name.

Create a new user:

With username: admin and other information:

Create password

Role Mapping: Assign ocisAdmin role

Update roles Client scope: Client Scopes -> roles

Mappers -> realm roles -> Token Claim Name=roles > Note: oCIS default role_claim is 'roles'

Go to url https://host.docker.internal:9200/, reload if already opened Enter username: admin and password: admin to log into oCIS

Setting Up oCIS With Keycloak Using JSON Files

Keycloak provides an Importing and Exporting Realms feature. If we have JSON configuration files, then we can avoid manually setting services in Keycloak.
oCIS provides documentation to set up oCIS with Keycloak which uses JSON configuration.
If you want to automate Keycloak integration with oCIS services, then you can use json files.

In this blog post, we've talked about putting Keycloak and oCIS together. We've shown how to set them up using Docker Compose, making sure they work smoothly with Traefik.
We've also given step-by-step instructions on setting up oCIS with Keycloak manually, like making realms, adding clients and roles, and creating users.
For those who like using JSON files, Keycloak lets you do that to set things up more easily.

Introduction to Keycloak

Amrita Shrestha — Tue, 20 Aug 2024 07:05:55 +0000

Keycloak is an open-source project created by RedHat for Single Sign-On. It provides an Identity and Access Management (IAM) solution designed to secure application services.
Additionally, it enables users to authenticate through various identity providers and use fine-grained permissions for regulating access to Software as a Service (SaaS) applications.

It facilitates the creation of a user database with customizable roles and groups, offering functionalities such as user management, registration, and password policy enforcement.
This makes it a comprehensive Identity and Access Management solution for contemporary applications. The Keycloak API enables the integration of all these features into your application
without the need for additional coding.

Keycloak Features

Identity Broker:
Identity Broker acts as a middleman, linking various service providers with different identity providers. It establishes trust with external identity providers to utilize their identities for accessing internal services offered by service providers.
Imagine a company, which has two Keycloak setups: one for customers and another for internal employees. If you want your employees to access external services without creating new accounts on the customer Keycloak, you can use the customer Keycloak as Broker and the internal Keycloak as Provider in this scenario.
User Federation:
Keycloak provides the ability to integrate with existing LDAP or Active Directory servers. It also supports implementation of your own provider such as a relational database.
Standard Protocols:
Keycloak is based on three standard protocols i.e., OpenID Connect, OAuth2.0, and SAML.
Password Policies:
Keycloak contains different password policies i.e., HashAlgorithm, Hashing Iterations, Digits, Lowercase Characters, Regular Expression and so on.
Single Sign-On:
Keycloak facilitates a seamless login experience where users only need to sign in once to access multiple applications, eliminating the need for repeated logins.
This not only streamlines the user experience but also simplifies identity management for administrators.
Social Login:
Keycloak has built-in support to login via Google, GitHub, and Facebook, which helps us to use social identity providers.

Keycloak Installation

In this blog, two different ways are discussed to install Keycloak.

By Docker

Make sure Docker Engine and Docker Compose has been installed in your system.

Keycloak using docker image
Run the following docker command to serve the Keycloak image. Keycloak will be available on http://localhost:8080.
```
 docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:24.0.2 start-dev
```

Keycloak with postgres database using docker-compose

 services:
   postgres:
     image: postgres:alpine
     volumes:
       - postgres_data:/var/lib/postgresql/data
     environment:
       POSTGRES_DB: keycloak
       POSTGRES_USER: keycloak
       POSTGRES_PASSWORD: keycloak

   keycloak:
     image: quay.io/keycloak/keycloak:24.0.1
     command: ["start-dev"]
     environment:
       KC_DB: postgres
       KC_DB_URL: "jdbc:postgresql://postgres:5432/keycloak"
       KC_DB_USERNAME: keycloak
       KC_DB_PASSWORD: keycloak
       KC_FEATURES: impersonation
       KEYCLOAK_ADMIN: admin
       KEYCLOAK_ADMIN_PASSWORD: admin
     ports:
       - 8001:8080 # map keycloak port 8080 to 8001
     depends_on:
       - postgres

 volumes:
   postgres_data:

Save the above code in compose.yaml file and run docker compose up in a terminal.

Docker compose serves Keycloak on http://localhost:8001.

By Distribution File

There is another way to set up Keycloak using the Keycloak distribution file.

Download the Keycloak distribution

export KC_VERSION=24.0.4
curl -LO  https://github.com/keycloak/keycloak/releases/download/"${KC_VERSION}"/keycloak-"${KC_VERSION}".zip

Unzip the Keycloak distribution package
```
unzip keycloak-${KC_VERSION}.zip
```
Install openjdk
```
sudo apt install openjdk-17-jdk
```
Navigate to the Keycloak directory
```
cd keycloak-${KC_VERSION}
```
When we start the server for the first time, we have to set the admin user and the admin password:
```
KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=admin ./bin/kc.sh start-dev
```
Note: if port 8080 is already used, then you can map the Keycloak port using the Keycloak environment variable 'KC_HTTP_PORT=8001'
When we start again, it is not necessary to set these variables again. You can start the server with:
```
./bin/kc.sh start-dev
```

Note: start-dev runs the Keycloak application in DEV-mode. Do not use this for production.

By default, the Keycloak server is on port 8080 for HTTP and 8084 for HTTPS. They are only served from the localhost loopback address 127.0.0.1:
Keycloak serves on http://localhost:<port>.

Why Keycloak Matters

Keycloak is open source.
Keycloak supports three different authentication protocols which gives you the possibility to cover many applications with different security demands with a single tool.
Keycloak provides a web-based GUI which makes any configuration changes easy.
Keycloak has huge community support.

Keycloak Drawbacks

Despite its extensive features, Keycloak does have certain limitations. One of these is the need for a more varied implementation approach.
Additionally, the following are some of the drawbacks associated with Keycloak:

Complex Server Deployment:
The manual process involved in deploying Keycloak on a server can be complicated, potentially affecting overall productivity.
Lack of Comprehensive Support Documentation:
Despite its robust functionalities, Keycloak could benefit from more efficient and extensive support documentation. This would make it easier for users to find suitable solutions to their problems.

When It May Not Be the Best Choice

Enterprises seeking robust guarantees or support may find Keycloak lacking in this regard. As an open-source project, there are no assurances provided by its producer regarding its functionality or roadmap. Support is community-driven, typically through platforms like Stack Overflow, with no guaranteed response times.
If your application consists of a single application with just one client in the Keycloak realm, you won't benefit from Single Sign-On (SSO) capabilities.
For applications solely reliant on a pure user database, Keycloak could be overkill. A database with specific tables may offer a simpler alternative, especially if you already have one set up.

In conclusion, when considering Identity Access Management solutions, Keycloak stands out as a robust option. However, it's essential to explore alternatives such as Auth0, Zluri, Microsoft Azure Active Directory, Okta, or WSO2 Identity Server to ensure you find the best fit for your specific needs and preferences.

Understanding REST APIs - A Comprehensive Guide with Practical Example

Prajwol Amatya — Thu, 09 May 2024 06:06:52 +0000

Within the field of web development, APIs (Application Programming Interfaces) are essential for facilitating communication across various software components. Because of its simplicity, scalability, and statelessness, REST (Representational State Transfer) APIs stand out among the other types of APIs. In this blog, we will dive into basics of building a REST API.

What is a REST API?

REST is an architectural style that outlines a collection of guidelines for building web services. A REST API offers a straightforward and adaptable method to interact with web services without the need for any processing. REST is often favored for its lower bandwidth usage, simplicity, and flexibility, making it better suited for internet applications. REST is utilized to retrieve or transmit data from a web service with all interactions through a REST API occurring via HTTP requests.

Key Principles of REST

The four key principles of REST API design, derived from the broader principles of REST architecture, are:

1. Stateless: Every client request to the server must include all necessary information for comprehension and processing. The server should not retain any data about the client's latest HTTP request.

2. Client-Server Architecture: The client handles the user interface and experience, while the server manages request processing and resource management.

3. Cacheable: Clients can cache responses from the server, enhancing performance by reducing the server load and minimizing client-server interactions.

4. Uniform Interface: The API has a consistent interface, which simplifies the architecture and enhances visibility of interactions.

REST API Architecture

REST APIs utilize a client-server architecture, communicating over HTTP. The client sends requests to the server, which processes them and sends back responses. Communication is stateless, requiring each client request to contain all necessary information for processing.

Components of REST API Architecture

Resources: In REST, resources are identified by URIs, representing anything that can be named and addressed, such as documents, images, temporal services, collections of other resources, non-virtual objects, and more.
HTTP Methods: REST APIs utilize standard HTTP methods for operations on resources. Common methods include GET (to retrieve a resource), POST (to create a new resource), PUT (to update a resource), and DELETE (to remove a resource).
Representations: Resources are represented in a format easily understood by both client and server, commonly JSON (JavaScript Object Notation) or XML (eXtensible Markup Language).

Understanding Endpoints in REST API

Endpoints are the points of interaction between the client and the server in a RESTful API. Each endpoint is associated with a specific URI through which an API is accessed, and it defines a specific URI at which the resource can be accessed or manipulated. Endpoints are crucial for REST APIs as they provide a structured and standardized way for clients to interact with the API's resources.

Characteristics of Endpoints

- URI Structure: Endpoints are defined by their URI structure, which typically includes the base URI of the API and the path to the resource.
- HTTP Methods: Each endpoint is associated with one or more HTTP methods (GET, POST, PUT, DELETE, etc.), which define the type of operation that can be performed on the resource.
- Resource Representation: The data returned by an endpoint is typically in a structured format, such as JSON or XML, which represents the resource or resources associated with the endpoint.

Implementing a REST API

Now, let's start to build a REST API. Before we start, make sure you have Python installed on your system. You will need a minimum Python version of 3.6. You can download Python from python.org. We will also need Flask, which you can install using pip:

pip install Flask

We will build a REST API for managing a list of tasks. Each task will have a title and a status indicating whether it's completed or not.

Create a new Python file: app.py
Import Flask and initialize the app:

from flask import Flask, jsonify, request

app = Flask(__name__)

Define the data structure for our tasks. For simplicity, we'll use a list of dictionaries:

tasks = [
    {"id": 1, "title": "Buy groceries", "completed": False},
    {"id": 2, "title": "Clean the house", "completed": True}
]

GET Requests

A GET request is used to retrieve data from a server. We can get all the available data or specific data using a GET request.

To retrieve all tasks:

@app.route("/tasks", methods=["GET"])
def get_tasks():
    return jsonify({"tasks": tasks}), 200

The @app.route decorator provided by Flask binds a URL pattern to the function that follows it. In this case, the URL pattern in /tasks.
methods=["GET] specifies that this route should respond to the HTTP GET requests. This means that when a client sends a GET request to the /tasks URL, Flask will call the associated function.
get_tasks() function will return a JSON response containing all tasks.
jsonify({"tasks": tasks}) is a Flask function that converts the Python dictionary {"tasks": tasks} into a JSON response.
To retrieve a specific task:

@app.route("/tasks/<int:task_id>", methods=["GET"])
def get_task(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    return jsonify({"task": task}), 200

@app.route binds a URL pattern /tasks/<int:task_id> to the function that follows it.
<int:task_id> is a variable part of the URL that will be captured and passed to the function as an argument.
methods=["GET"] specifies that this route should respond to the HTTP GET requests. This means that when a client sends a GET request to the tasks/<int:task_id> URL, Flask will call the associated function.
get_task() function will take task_id as the argument that is extracted from /tasks/<int:task_id> URL. The function will then search for a task with that ID in the tasks list. If a task is found, it will return the task in JSON format. If no task is found, it will return a 404 status code with an error message.
return jsonify({"task": task}) returns a JSON response containing the task.

POST Request

A POST request is used to send data to a server to create a new resource. To create a new task, implement a route that handles POST requests:

@app.route("/tasks", methods=["POST"])
def create_task():
    new_task = request.get_json()
    tasks.append(new_task)
    return jsonify({"task": new_task}), 201

@app.route binds a URL pattern /tasks to the function that follows it.
methods=["POST"] specifies that this route should respond to the HTTP POST requests. This means when a client sends a POST request to the /tasks URL, Flask will call the associated function.
request.get_json() is a Flask function that parses the incoming request data as JSON. This function is used to extract the JSON payload sent by the client in the POST request.
create_task() function will parse the JSON payload, add the new task to the tasks list, and return a JSON response containing the newly created task. The status code 201 indicates that the task was successfully created.
tasks.append(new_task) appends the new_task dictionary to the tasks list.
jsonify({"task": new_task}) is a Flask function that converts the new_task dictionary into a JSON response. This response includes the newly created task.

PUT Request

A PUT request is used to update an existing resource on a server. To update a task, implement a route that handles PUT requests:

@app.route("/tasks/<int:task_id>", methods=["PUT"])
def update_task(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    task.update(request.get_json())
    return jsonify({"task": task}), 201

@app.route binds a URL pattern /tasks/<int:task_id> to the function that follows it.
<int:task_id> is a variable part of the URL that will be captured and passed to the function as an argument.
methods=["PUT"] specifies that this route should respond to the HTTP PUT requests. This means that when a client sends a PUT request to the /tasks/<int:task_id> URL, Flask will call the associated function.
update_task() function will take task_id as the argument that is extracted from /tasks/<int:task_id> URL. The function will then search for a task with that ID in the tasks list. If a task is found, it will update the task with the new data provided in the request and return the updated task in JSON format. If no task is found, it will return a 404 status code with an error message.
task.update(request.get_json()) updates the task with the JSON data sent by the client in the PUT request. The request.get_json() function parses the incoming request data as JSON, and the update() method merges this data into the task dictionary.
return jsonify({"task": task}) returns a JSON response containing the updated task.

DELETE Request

A DELETE request is used to remove a resource from a server. To delete a task, implement a route that handles DELETE requests:

@app.route("/tasks/<int:task_id>", methods=["DELETE"])
def delete_tasks(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    tasks.remove(task)
    return jsonify({"result": "Task deleted"}), 200

@app.route binds a URL pattern /tasks/<int:task_id> to the function that follows it.
<int:task_id> is a variable part of the URL that will be captured and passed to the function as an argument.
methods=["DELETE"] specifies that this route should respond to the HTTP DELETE requests. This means that when a client sends a DELETE request to the /tasks/<int:task_id> URL, Flask will call the associated function.
delete_task() function will take task_id as the argument that is extracted from /tasks/<int:task_id> URL. The function will then search for a task with that ID in the tasks list. If a task is found, it will remove the task from the list and return a JSON response indicating that the task was deleted. If no task is found, it will return a 404 status code with an error message.
tasks.remove(task) removes the task from the tasks list.
return jsonify({"result": "Task deleted"}) returns a JSON response indicating that the task was successfully deleted.

Running the Application

To run your application, add the following line:

if __name__ == "__main__":
    app.run(debug=True)

The entire app.py will look like this:

from flask import Flask, jsonify, request

app = Flask(__name__)

tasks = [
    {"id": 1, "title": "Buy groceries", "completed": False},
    {"id": 2, "title": "Clean the house", "completed": True}
]

@app.route("/tasks", methods=["GET"])
def get_tasks():
    return jsonify({"tasks": tasks}), 200

@app.route("/tasks/<int:task_id>", methods=["GET"])
def get_task(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    return jsonify({"task": task}), 200

@app.route("/tasks", methods=["POST"])
def create_task():
    new_task = request.get_json()
    tasks.append(new_task)
    return jsonify({"task": new_task}), 201

@app.route("/tasks/<int:task_id>", methods=["PUT"])
def update_task(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    task.update(request.get_json())
    return jsonify({"task": task}), 201

@app.route("/tasks/<int:task_id>", methods=["DELETE"])
def delete_tasks(task_id):
    task = next((item for item in tasks if item["id"] == task_id), None)
    if task is None:
        return jsonify({"error": "Task not found"}), 404
    tasks.remove(task)
    return jsonify({"result": "Task deleted"}), 200

if __name__ == "__main__":
    app.run(debug=True)

Then open your terminal and run the following command from your project directory:

python3 app.py

Your REST API is now up and running on http://127.0.0.1:5000

Testing Your API

You can test your API using tools like Postman or cURL.

1. List All Tasks

You can use the following cURL command to list all the available tasks:

curl http://127.0.0.1:5000/tasks

Expected Result:

{
  "tasks": [
    {
      "completed": false,
      "id": 1,
      "title": "Buy groceries"
    },
    {
      "completed": true,
      "id": 2,
      "title": "Clean the house"
    }
  ]
}

2. List a Specific Task

To retrieve a specific task by its ID, you can use the following cURL command, replacing <task_id> with the ID of the task you want to retrieve:

curl http://127.0.0.1:5000/tasks/<task_id>

Example:

curl http://127.0.0.1:5000/tasks/1

The curl command sends an HTTP GET request to http://127.0.0.1:5000/tasks/1, retrieving the task with ID 1 from a local server.

Expected Result:

{
  "task": {
    "completed": false,
    "id": 1,
    "title": "Buy groceries"
  }
}

3. Create a New Task

To create a new task, you can use following cURL command, replacing <task_data> with the JSON data for the new task:

curl -X POST -H "Content-Type: application/json" -d '<json_data>' http://127.0.0.1:5000/tasks

Example:

curl -X POST -H "Content-Type: application/json" -d '{"id":3,"task":"Finish the report","completed":false}' http://127.0.0.1:5000/tasks

The curl command sends an HTTP POST request with JSON data to http://127.0.0.1:5000/tasks, creating a new task.

Expected Result:

{
  "task": {
    "completed": false,
    "id": 3,
    "task": "Finish the report"
  }
}

4. Update a Task

To update an existing task, use the following cURL command, replacing <task_id> with the id of the task, and <updated_data> with the JSON data you want to update:

curl -X PUT -H "Content-Type: application/json" -d '<updated_data>' http://127.0.0.1:5000/tasks/<task_id>

Example:

curl -X PUT -H "Content-Type: application/json" -d '{"completed":true}' http://127.0.0.1:5000/tasks/3

The curl command sends an HTTP PUT request with JSON data to http://127.0.0.1:5000/tasks/3, updating the task with ID 3 to mark it as completed.

Expected Result:

{
  "task": {
    "completed": true,
    "id": 3,
    "task": "Finish the report"
  }
}

5. Delete a Task

To delete a task, use the following cURL command, replacing <task_id> with the id of the task you want to delete:

curl -X DELETE http://127.0.0.1:5000/tasks/<task_id>

Example:

curl -X DELETE http://127.0.0.1:5000/tasks/3

The curl command sends an HTTP DELETE request to http://127.0.0.1:5000/tasks/3, deleting the task with ID 3.

Expected Result:

{
  "result": "Task deleted"
}

Let's try to retrieve a task that doesn't exist in the list of tasks. For instance, we just deleted a task with ID 3. Now, let's try to retrieve the task with ID 3 and see the result.

curl http://127.0.0.1:5000/tasks/3

Expected result:

{
  "error": "Task not found"
}

We see that the JSON response returns the error Task not found, which indicates that the task doesn't exist in the list of tasks.

Common Causes of API Request Failures

Now, let's explore scenarios where API requests might fail. API requests can fail for various reasons, including sending invalid requests. This could happen if an incorrect endpoint is used, if the JSON data is invalid, or if the request is made with an unsupported HTTP method. Let's examine some instances where API requests might fail.

1. Invalid Endpoint:
With the REST API that we built, we have the following valid API endpoints:

GET /tasks: Retrieves a list of all available tasks
GET /tasks/<int:task_id>: Retrieves a specific task by its ID
POST /tasks: Creates a new task
PUT /tasks/<int:task_id>: Updates a specific task by its ID
DELETE /tasks/<int:task_id>: Deletes a specific task by its ID

If we send an API request to any endpoint that is not listed above, the request will fail with a 404 status code. For instance, let's try to get all the tasks with following cURL command:

curl http://127.0.0.1:5000/task

Here, we have used task instead of tasks in our endpoint, which makes our endpoint invalid. With this request, we get the following error with 404 status code.

Expected Result:

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

2. Invalid JSON Data:
Invalid JSON data can lead to API request failures due to syntax or structure errors. Here are some of the most common syntax and structure errors that can occur in JSON data:

Missing Quotes Around Keys: In JSON, keys must be strings, and strings are defined by enclosing them in double quotes. Omitting quotes around keys will result in invalid JSON. Invalid:

{ key: "value" }

Valid:

{ "key": "value" }

Single Quotes Instead of Double Quotes: JSON strings must be enclosed in double quotes. Using single quotes will result in invalid JSON. Invalid:

{ 'key': 'value' }

Valid:

{ "key": "value" }

Trailing Commas: JSON does not allow trailing commas after the last element in an object or array. Including a comma after the last element will make the JSON invalid. Invalid:

{
  "key1": "value1",
  "key2": "value2",
}

Valid:

{
  "key1": "value1",
  "key2": "value2"
}

Missing Comma Between Elements: In JSON objects and arrays, elements must be separated by commas. Omitting a comma between elements will make the JSON invalid. Invalid:

{
  "key1": "value1"
  "key2": "value2"
}

Valid:

{
  "key1": "value1",
  "key2": "value2"
}

Incorrect Data Types: JSON supports several data types, including strings, numbers, objects, arrays, booleans, and null. Using an incorrect data type or a value that does not match the expected type can lead to invalid JSON. Invalid:

{
  "key": "value",
  "number": "123"
}

Valid:

{
  "key": "value",
  "number": 123
}

Let's add a new task with invalid JSON data with a missing quote around one of the keys:

curl -X POST -H "Content-Type: application/json" -d '{"id":3,task:"Finish the report","completed":false}' http://127.0.0.1:5000/tasks

Here, the JSON data in the request body is invalid as the key task is not enclosed in double quotes. With this request, we get the following error with status code 400.

Expected Result:

Failed to decode JSON object: Expecting property name enclosed in double quotes: line 1 column 9 (char 8)

3. Unsupported HTTP method
When a API request is sent to an endpoint with an unsupported HTTP method, then the API request will fail with an error message Method Not Allowed. For example, trying to POST to an endpoint to get a specific task:

curl -X POST http://127.0.0.1:5000/tasks/1

Here, we have sent an API request with POST method to retrieve a specific task with ID 1. The problem with using the POST method to retrieve a specific task by its ID is that it violates the standard conventions of REST API design. In REST APIs, the POST method is intended for creating new resources, not for retrieving existing ones, due to which this API request will fail with error message Method Not Allowed.

Expected Result:

Method Not Allowed

Conclusion

In this blog, we've explored the basics of REST APIs and how to implement them using Python Flask. This simple example demonstrates the power and flexibility of REST APIs in building scalable and maintainable web applications. You can expand this basic structure to include more complex data models, authentication, and more advanced features as needed. Building a REST API is just the beginning. The real power comes from how you integrate it with other services, databases, and front-end applications. Happy coding!

Introduction to htmx

nirajacharya2 — Thu, 02 May 2024 06:37:54 +0000

htmx is a UI library written in JavaScript that extends the features of HTML. It can be extended to make AJAX requests, do CSS transitions, connect to WebSockets and handle Server Sent Events inside an HTML document with just some declarative HTML attributes.

htmx was created by Carson Gross in 2013 because he found HTML to be limited for what he was trying to achieve. htmx tries to solve the following limitations of HTML:

only <a> and <form> tags are able to make HTTP requests
only click and submit events trigger them
only GET and POST methods are available
and when using <a> and <form> the whole screen gets replaced

In this blog, we will learn how htmx enhances HTML and overcomes the limitations mentioned above.

Note:

I will be using go for the code examples given in this blog post.
If you are not familiar with go you can learn the basics from here.

Throughout this blog I will be providing small snippets of my code instead of the actual implementation (you can find the code implementation here) and I will not go into how to create a webserver to serve the endpoints. For that you might want to read this blog.

Using htmx

We can simply use htmx by adding a script tag in the header of our HTML.



<head>
  <script src="https://unpkg.com/htmx.org@1.9.5"></script>
</head>

htmx claims to be small (~14k min.gz’d), dependency-free, extendable, IE11 compatible and 67% less code when compared with react.

Now let's see how to make a simple AJAX request in htmx.

Making AJAX request (GET)

So, let's begin by creating a few endpoints for this example.



func getRoot(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "./static/index.HTML")
}

func hi(w http.ResponseWriter, r *http.Request) {
    io.WriteString(w, "<div>hi</div>")
}
func main() {
    http.HandleFunc("GET /", getRoot)
    http.HandleFunc("GET /hello", hi)
    http.ListenAndServe(":3333", nil)
}

In the above code example, we made two endpoints: / to serve the HTML file and /hello will return an HTML element. htmx gets the body of the response and renders it on the page, therefore the response of our function should be HTML code.

Now we will see how to use the hello endpoint in HTML:



<div hx-post="/hello">hello</div>

As seen above, the endpoint is working. The content of the div is replaced by the response.
Because we did not define any specific event to trigger the AJAX request, it will be triggered by clicking the hello text.

We can explicitly define it like this:



<div hx-get="/hello" hx-trigger="click">hello</div>

htmx has other predefined events like this:

hx-trigger="mouseenter" is triggered every time that the mouse enters the element
hx-trigger="mouseenter once" is triggered when the mouse enters the element for the first time and won't trigger after that
hx-trigger="click[ctrlKey]" is triggered when the element is clicked while holding the Ctrl key
hx-trigger="every 2s" is triggered every two seconds

You can find more on this subject in the htmx docs.

Currently, our example above replaces the element that makes the AJAX request, but we can specify which element is going to get replaced using the hx-target attribute:



<div id="replace" class="replace">replace me</div>
<div hx-get="/hello" hx-trigger="click" hx-target="#replace">click me to replace the element above</div>

We can also use a class name like .replace or give elements like body as the target.

In the following example we have two foo elements:



<foo id="1" class="replace">replace me 1</foo>
<foo id="2" class="replace">replace me 2</foo>
<bar hx-post="/hello" hx-trigger="click" hx-target="foo">replace foo element</bar>

The first of the foo elements will be taken as the target. So in this example the first foo element with id=1 will be replaced by the response.

Making a todo app

Now we have learned the basics, let's write a todo app.

First start with making a submit form for adding a todo task.

For that, we will only need an input box and a button in our HTML code:



<div id="app" class="container">
  <div id="todo-list"></div>
  <form
    id="myForm"
    hx-post="/save"
    hx-trigger="submit"
    hx-target="#todo-list"
  >
    <input type="text" name="task" required />
    <input type="submit" value="save" />
  </form>
</div>

Now let's have a look at what is needed on the server side:



func save(w http.ResponseWriter, r *http.Request) {
    r.ParseForm()
    insert(r.FormValue("task")) // save to database
    taskList := "<div>" + r.FormValue("task") + " </div>"
    io.WriteString(w, taskList)
}

func main() {
    ...
    http.HandleFunc("/save", save)
    ...
}

In the HTML above, the form will send a post request to the /save endpoint and htmx will replace the content in the div with the id todo-list. The response will replace everything in the target element i.e., the element with id todo-list will be replaced and looks like this:



<div id="todo-list">
  <div>item 1</div>
</div>

After adding another todo item to the list, the HTML document looks like this:



<div id="todo-list">
  <div>item 2</div>
</div>

That is not quite what we want. We want to append the new todo item after the last todo item. So we use the attribute hx-swap to let htmx know where we want the response to render. The beforeend value will append the HTML response before the end of the targeted element.

After the swap the code will look like this:



<div id="app" class="container">
  <div id="todo-list"></div>
  <form
    id="myForm"
    hx-post="/save"
    hx-trigger="submit"
    hx-target="#todo-list"
    hx-swap="beforeend"
  >
    broadcast<input type="text" name="task" required />
    <input type="submit" value="save" />
  </form>
</div>

Adding the first item (item 1):



<div id="todo-list">
  <div>item 1</div>
</div>

After adding the second item (item 2):



<div id="todo-list">
  <div>item 1</div>
  <div>item 2</div>
</div>

Adding multiple todo items is working now, but on page load (e.g. if we refresh the page) the list of tasks is not being loaded.
To make that work, we need to make a GET request.



<div id="todo-list" hx-get="/taskList" hx-trigger="load"></div>

In the above HTML, I have added hx-get with value '/tasklist' and hx-trigger with value 'load'. The 'load' value specifies when the request is sent. In this case it's when the HTML file is loaded.



func listTask(w http.ResponseWriter, r *http.Request) {
    tastList := getTODO() // gets data from the database and returns HTML
    io.WriteString(w, tastList)
}
func main() {
    ...
    http.HandleFunc("/save", listTask)
    ...
}

Now the list is shown even when the page is refreshed.

For edit let's try something different: instead of doing all our operations in a single page, we will be navigating to another page to do the edit.

First, let's make clicking on the todo item go to a different page that will contain a form for updating the todo item list.
For that, we need to get an anchor tag instead of a div with the todo text inside a div.

On the server side, I will get the task and return the proper form for updating that task.



func main() {
    ...
    http.HandleFunc("/task/", individualTask)
    ...
}

Example response from server:



  <div hx-boost="true">
      <a href="/task/1">clean room</a>
  </div>

Here the hx-boost attribute will replace the body element and add the previous page to the history entry.
That means the back button on browsers will work perfectly.

For the actual editing, it will be similar to the other examples.



<div>
  <div>clean room</div>
  <form id="myForm2" hx-put="/edit" hx-target="#app">
    <input type="text" name="task" required="" />
    <input type="hidden" name="old" value="1" />
    <input type="submit" value="edit" />
  </form>
</div>

Now let's perform deletion on the list.

First, we have to make a delete button.

For that, I will update the list template and add a delete form:



<div hx-boost="true" class="new-element">
  <a href="/task/1">clean room</a>
  <form
    hx-trigger="click"
    hx-delete="/delete"
    hx-target="closest .new-element"
    hx-swap="outerHTML swap:1s"
  >
    <input type="hidden" name="task" value="1" />
    delete
  </form>
</div>

Most of the snippets in the delete form are similar to the other examples. But in the hx-swap="outerHTML swap:1s" attribute, I added an extra value. This value will delay the swapping of the element by 1 second because I want to do a CSS animation during that time.

While htmx is doing the swapping of the element, it also adds the htmx-swapping class to the element.

Now I can add any CSS effect I want to the htmx-swapping class.



.htmx-swapping {
  opacity: 0;
  transition: opacity 1s ease-out;
}

WebSockets

Currently, if we add a todo task to our list, it won't be shown in another tab or browser opening the same page. This can be achieved using WebSockets. Fortunately htmx supports WebSockets.



<div id="ws" hx-ws="connect:/ws" hx-trigger="task"></div>

The hx-ws attribute takes the WebSockets endpoint and makes a connection to it.

After adding this line to the code, I just have to create a WebSocket that listens for input by the user and broadcasts it to all clients.



import(
    ...
    "golang.org/x/net/websocket"
    "sync"
    ...
)

func main(){
    ...
    server := NewServer()
    http.Handle("/ws", WebSockets.Handler(server.WebSocketsHandler)) // handles the saving and broadcasting of data
    ...
}

Update the form:



<form
  id="ws-form"
  hx-ws="send:submit"
  hx-target="#todo-list"
  hx-swap="beforeend"
>
  <input type="text" name="task" required />
  <input type="submit" value="save ws" />
</form>

In the hx-ws attribute of the form we are telling htmx to send the values to the WebSockets endpoint on the submit event.



func (s *Server) WebSocketsHandler(ws *WebSockets.Conn) {
    broadcast() // broadcast the response
}
func main(){
    ...
    http.Handle("/ws", WebSockets.Handler(server.WebSocketsHandler))
    ...
}

Now we can see our list getting updated everywhere.

These were some basics of htmx to get started with. Now you can build on it and make something more interesting by yourself.

And if you want to learn more about htmx, have a look at the docs here

Introduction to GitHub Actions - CI & CD

Nabin Ale — Thu, 25 Apr 2024 05:06:24 +0000

GitHub Actions is an automation and CI/CD (Continuous Integration/Continuous Delivery or Deployment) service provided by GitHub, allowing us to automate our software development processes or workflows, from building and testing code to deploying it to different environments. It's an effective tool that can enhance software quality and save our time.

Some features of GitHub Actions:

It allows you to automate various stages of SDLC (Software Development Life Cycle), such as building, testing, deploying, and releasing software.
It simplifies CI/CD integration, facilitating seamless DevOps practices.

Components of GitHub Actions:

Workflows
Events
Jobs
Steps
Actions
Runners

1. Workflows

Workflows are automated procedures that you add to your repository. They will run when triggered by an event in your repository, or they can be triggered manually, or at a defined schedule. They can be used to build, test, package, release, or deploy a project on GitHub.

2. Events

Events are specific activities such as code push, pull request creation, issue creation, comments on pull requests or issues, etc., that trigger a workflow.

3. Jobs

Jobs are sets of steps that are executed in the same runner. A workflow with multiple jobs will run in parallel by default, but it can be configured so that jobs run sequentially.

4. Steps

A step is an individual task that can run the commands in a job. These can be actions or commands.

5. Actions

Actions are the standalone commands for the GitHub Actions platform that perform complex but frequently repeated tasks. Actions help to reduce repetitive code in workflows.

6. Runner

A runner is a GitHub Actions server where our workflows run. It listens for available jobs, runs multiple jobs at a time, and reports the progress, logs, and results back to GitHub. GitHub hosted runners are based on Ubuntu, Linux, Microsoft Windows, and Mac OS.

Note:
One event can trigger many workflows, a workflow can contain many jobs, and a job can contain many steps.

Process in GitHub Actions:

Getting started:

Before we dive into an example, make sure you have a repository where you want to set up your CI/CD workflow. If you don't have one, create a new repository, for example, named action-hero.

Step 1: Clone your repository into your system

git clone <your_github_repo_url>

Step 2: Go to path action-hero and create a folder named .github

cd action-hero
mkdir .github

Step 3: Go to path .github and create a folder named workflows

cd .github
mkdir workflows

Step 4: Inside the folder workflows, create a file named hello_world.yml

cd workflows
touch hello_world.yml

Our file structure should look like this:

📦action-hero
┗ 📂.github
┃ ┣ 📂workflows
┃ ┃ ┗ 📜hello_world.yml

Step 5: Add the following code snippet to your hello_world.yml file:

name: Hello world workflow

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Print a message
      run: echo "Hello, GitHub Actions!"

In this example:

The workflow is triggered by a push event on the main branch.
The job named "build" runs on the latest version of Ubuntu.
The first step checks out the repository using the actions/checkout action.
The second step simply prints a message using the echo command.

GitHub Actions activity:

The image above is the output from our GitHub Actions workflow run. It shows that the workflow was successful and completed the Hello world workflow job, which consisted of two steps: Checkout repository and Print a message.

The step Print a message executed the command echo "Hello, GitHub Actions!", resulting in the output Hello, GitHub Actions! being displayed in the workflow log.

Conclusion

In this blog, we've explored GitHub Actions, an automation and CI/CD service provided by GitHub. We've learned about its key components and how they work together to automate software development workflows.

GitHub Actions enables us to automate tasks like building, testing, and deploying code, leading to improved software quality and time savings.

With a practical example, we've seen how easy it is to set up a basic workflow.

In summary, GitHub Actions is a powerful tool for speedy development processes, making it easier to deliver high-quality software efficiently.