What is prompt engineering?

This GitHub blog from June 2023 tells you what prompt and prompt engineering are and shares examples and best practices for communicating your desired results to the GitHub Copilot.

How to use GitHub Copilot: Prompts, tips, and use cases

Tips and Tricks for Effective Prompting

This video shows tips and tricks on how to use GitHub Copilot in Visual Studio Code with demos and examples in Python, SQLAlchemy, and Django MVC.

And If it is not enough, you can check out this video to see how GitHub Copilot helps you turning a command line app into a Flask API app.

You can now change ghost text font!

As of the latest version of Visual Studio Code, you can now change your ghost text font (but not the style 😅) by using the setting editor.inlineSuggest.fontFamily.

Get to know GitHub Copilot in VS Code

This new great video is recommended to those who is new to GitHub Copilot and want to get up and run with it in just minutes.

Copilot in GitHub Support is now available!

You can now chat with AI which trained on the official GitHub documentation and will help you find answers to your GitHub-specific questions.

Check out this GitHub blog for more information.

Copilot in GitHub Support is now available!

GitHub Copilot for Enterprise is coming

GitHub Copilot for Enterprise launches this week. The sign up for the Copilot Enterprise waitlist is now closed.

Prompt Engineering 101 Training

This introductory training course uncovers techniques to transform your coding comments into precise and actionable code.

Introduction to prompt engineering with GitHub Copilot - Training

You can subscribe to the GitHub Insider Newsletter by going to github.blog. Scroll down to the bottom, input your email address, click Subscribe. Then it will send an email which you also need to click the link within to complete the subsciption.

Microsoft Azure provides Azure Bastion service which is a jump server so you can securely access your virtual machines via its Azure Portal web interface without exposing SSH or RDP port. Here is its architecture:

This service is great except it costs $0.19 per hour. Combining with other components then this may costs you almost $150 a month. Bad news is you can’t stop it unless you delete it.

This post, I will show you how I create my own jump server on Ubuntu VM to replace Azure Bastion using Apache Guacamole which is an open source tool which provide similar functionalities (I wonder that Azure Bastion may be even built on top of it). The VM can be a small one that may costs only a few tens bucks per month.

Apache Guacamole

From its website, Apache Guacamole is a clientless remote desktop gateway that supports standard protocols like VNC, RDP, and SSH. Clientless means your clients don’t need to install anything but just use a web browser to remotely access your fleet of VMs.

The Guacamole comprises of two main components:

• Guacamole Server which provides guacd which is like a proxy server for the client to connect to the remote server.
• Guacamole Client which is a servelet container that user will log in and via web browser.

For more information about Guacamole, visit its architecture page.

As my disclaimer, installation is not simple as there are several components you need to install and configure before it is good enough to use. There may be simpler ways to deploy e.g. using Docker image or using this Helm chart but I haven’t tried them yet. Because of cost is my concern so I’d like to deploy on a small VM that may not run a Kubernetes cluster (and may be I just prefer to learn it in hard way :P).

Network Topology

You use Azure Bastion or a jump server because you want to secure your VMs behind so having the right network design is needed. Here it mine:

In my virtual network (VNet), I split into two subnets:

1. snet-gateway where I will deploy Guacamole on a Ubuntu VM which has a public IP so its web interface can be reached from the Internet.
2. snet-default where I will deploy my backend pool of VM and enable remote access via Guacamole only.

For the sake of security, you should configure Network Security Group (NSG) of your snet-gateway to limit inbound and outbound connections to/from Ubuntu VM in the same way as you do for AzureBastionSubnet. But in this example, I just associate NSG to the VM directly.

Once you setup your network then create the VMs. In this example, I create the following two VMs:

1. vm-win10 in the snet-default which is the machine I will remote access to.
2. vm-ubuntu1804 in the snet-gateway where I will install Guacamole and use it as a jump server. I choose B2s size which cost around $39 per month but, of course, you can change its size later.

Install Guacamole Server

Once your Ubuntu Server 18.04 is created then log in via SSH. You may need to expose SSH port publicly for now. You can change SSH port to something than 22 to make it more secure. See how to do it in my previous blog.

Setting Up Your Own VPN Server with just $5 a month

From its documentation, there’s no executable binary available for Guacamole server and you need to build it from source (unless you deploy from Docker image).

First you need to install build tools and all required dependencies for the build process. Missing some of them may result in missing features or build failure.

# Update & upgrade system
sudo apt-get update && sudo apt-get --yes upgrade

# Install build tools
sudo apt install --yes build-essential

# Install build dependencies
sudo apt install --yes libcairo2-dev libjpeg-turbo8-dev libpng-dev libtool-bin libossp-uuid-dev

# Install optional dependencies
sudo apt install --yes \
    libavcodec-dev libavformat-dev libavutil-dev libswscale-dev \
    freerdp2-dev \
    libpango1.0-dev \
    libssh2-1-dev \
    libtelnet-dev \
    libvncserver-dev \
    libwebsockets-dev \
    libpulse-dev \
    libssl-dev \
    libvorbis-dev \
    libwebp-dev

# Install runtime dependencies
sudo apt install --yes --no-install-recommends \
    netcat-openbsd                \
    ca-certificates               \
    ghostscript                   \
    fonts-liberation              \
    fonts-dejavu                  \
    xfonts-terminus

Next, download source codes and extract.

export GUAC_VERSION="1.3.0"
curl -fLO "https://downloads.apache.org/guacamole/${GUAC_VERSION}/source/guacamole-server-${GUAC_VERSION}.tar.gz"
tar -xzf "guacamole-server-${GUAC_VERSION}.tar.gz"

Configure the build and start the build.

cd "guacamole-server-${GUAC_VERSION}"
./configure --with-init-dir=/etc/init.d
make

Once done, install and start the service.

sudo make install
sudo ldconfig
sudo systemctl daemon-reload
sudo systemctl start guacd
sudo systemctl enable guacd

At this point, the Guacamole server (guacd) service should be up and running. Inspect by executing systemctl status guacd --no-pager.

Install Guacamole Client

Unlike the server, we don’t need to build it (but you can if you want). So we will download the WAR file and install on Tomcat server then expose it through nginx via HTTPS.

Install Tomcat

First, we need Tomcat to run the WAR file. Use this script to install it.

export TOMCAT_VERSION="8.5.65"
TOMCAT_MAJOR_VERSION=$(echo ${TOMCAT_VERSION} | awk -F . '{print $1}')
sudo apt-get install --yes default-jdk
sudo groupadd tomcat
sudo useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
sudo usermod -a -G tomcat "$USER"
curl -LO "https://downloads.apache.org/tomcat/tomcat-${TOMCAT_MAJOR_VERSION}/v${TOMCAT_VERSION}/bin/apache-tomcat-${TOMCAT_VERSION}.tar.gz"
sudo mkdir -p /opt/tomcat
sudo tar -xzf "apache-tomcat-${TOMCAT_VERSION}.tar.gz" -C /opt/tomcat --strip-components=1
sudo chgrp -R tomcat /opt/tomcat
cd /opt/tomcat
sudo chmod -R g+r conf
sudo chmod g+x conf
sudo chown -R tomcat webapps/ work/ temp/ logs/

Configure Tomcat and start the service.

JAVA_ALT_TEXT="$(update-java-alternatives -l || true)"
JAVA_HOME="$(echo "${JAVA_ALT_TEXT}" | awk '{print $3}')"
echo "[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking

Environment=JAVA_HOME=${JAVA_HOME}
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target" | sudo tee /etc/systemd/system/tomcat.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl start tomcat
sudo systemctl enable tomcat

Now the Tomcat service should be up and running. Inspect by executing systemctl status tomcat.service --no-pager.

Add Guacamole Client Servlet

Download and add the WAR file to the Tomcat.

curl -LO "https://downloads.apache.org/guacamole/${GUAC_VERSION}/binary/guacamole-${GUAC_VERSION}.war"
sudo cp "guacamole-${GUAC_VERSION}.war" "/opt/tomcat/webapps/ROOT.war"
sudo chown tomcat:tomcat "/opt/tomcat/webapps/ROOT.war"
sudo rm -rf /opt/tomcat/webapps/ROOT

You may try to access the client at http://<your-server>:8080/ by either open or forward the port and you should see the Guacamole's login screen (but you can't login now).

In case you see Tomcat instead of Guacamole, restart Tomcat using command sudo systemctl restart tomcat and try again.

Install nginx and certbot

We will use nginx as a proxy and certbot to get a certificate from Let’s Encrypt.

sudo apt install --yes nginx-core
sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot

Configure certbot with a domain and an email address and integrate with nginx.

export DOMAIN_NAME="<Your VM FQDN>"
export EMAIL="<Your Email Address>"
sudo certbot --nginx -d "${DOMAIN_NAME}" -m "${EMAIL}" --agree-tos -n

Edit the file /etc/nginx/sites-enabled/default and replace the following section:

server_name your.server.fqdn; # managed by Certbot

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

with this one:

server_name your.server.fqdn; # managed by Certbot

        location / {


                proxy_pass http://localhost:8080/;
                proxy_buffering off;
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $http_connection;
                proxy_cookie_path /guacamole/ /;
                access_log off;
        }

This will set nginx as a proxy to your servlet. You don’t need to configure to redirect HTTP to HTTPS as that is already done by certbot.

Test the configuration and restart nginx.

sudo nginx -t
sudo systemctl restart nginx

Now you should be able to access Guacamole ay https://<your-server>/

Configure User and Connections

We will create a user with two connections in the user-mapping.xml file so we can test logging on.

Create the file /etc/guacamole/user-mapping.xml an put below content. Create the directory /etc/guacamole first if it does not exist.

<user-mapping>
    <authorize username="guacadmin" password="guacadmin">
        <connection name="this-server-ssh">
             <protocol>ssh</protocol>
             <param name="hostname">localhost</param>
             <param name="port">22</param>
        </connection>
        <connection name="some-win10-rdp">
             <protocol>rdp</protocol>
             <param name="hostname">vm-win10</param>
             <param name="port">3389</param>
             <param name="username">username</param>
             <param name="password">thisisyourpassword</param>
             <param name="ignore-cert">true</param>
        </connection>
    </authorize>
</user-mapping>

The first connection is SSH to local server itself while the second one is RDP to Windows 10 VM. Don’t forget to update the username and password.

Restart Tomcat and test logging in and making the connections.

sudo systemctl restart tomcat

If everything is configured properly, you should be able to connect to your VMs via either SSH and RDP. However, you cannot change anything on the web GUI the configuration is static in the file user-mapping.xml

To make it editable via the web GUI, we need to install a database. In this example, I will use MySQL but Guacamole also supports other databases e.g. MariaDB, PostgreSQL. You can check for more information in this documentation page.

Install MySQL

Let’s install MySQL.

sudo apt install --yes mysql-server

Check MySQL service status by executing systemctl status mysql --no-pager.

After install the MySQL, we also need to install the Guacamole extension and library so it knows how to talk to the database.

# Download and install JDBC extensions
curl -fLO "https://downloads.apache.org/guacamole/${GUAC_VERSION}/binary/guacamole-auth-jdbc-${GUAC_VERSION}.tar.gz"
tar -xzf "guacamole-auth-jdbc-${GUAC_VERSION}.tar.gz"
sudo mkdir -p /etc/guacamole/extensions
sudo cp "guacamole-auth-jdbc-${GUAC_VERSION}/mysql/guacamole-auth-jdbc-mysql-${GUAC_VERSION}.jar" "/etc/guacamole/extensions/"

# Download and install MySQL Connector/J
CONNECTORJ_VERSION="8.0.24"
curl -fLO "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-${CONNECTORJ_VERSION}.tar.gz"
tar -xvf "mysql-connector-java-${CONNECTORJ_VERSION}.tar.gz"
sudo mkdir -p /etc/guacamole/lib
sudo cp "mysql-connector-java-${CONNECTORJ_VERSION}/mysql-connector-java-${CONNECTORJ_VERSION}.jar" "/etc/guacamole/lib/"

Configure Database

Next, we need to create a database and a user. In this example, I will create the database named guacamole_db and the user named guacamole_user. Please note the password must meet complexity criteria.

MYSQL_PASSWORD="<YourPasswordHere>"
sudo mysql --execute='CREATE DATABASE guacamole_db;'
sudo mysql --execute="CREATE USER 'guacamole_user'@'localhost' IDENTIFIED BY '${MYSQL_PASSWORD}';"
sudo mysql --execute="GRANT SELECT,INSERT,UPDATE,DELETE ON guacamole_db.* TO 'guacamole_user'@'localhost';"
sudo mysql --execute='FLUSH PRIVILEGES;'

Then run the provides scripts to create schema and the Guacamole default user guacadmin.

cat guacamole-auth-jdbc-1.3.0/mysql/schema/*.sql | sudo mysql guacamole_db

Lastly, you need to configure Guacamole client so it can connect to the database by creating file /etc/guacamole/guacamole.properties and put the following content.

# MySQL properties
mysql-hostname: localhost
mysql-port: 3306
mysql-database: guacamole_db
mysql-username: guacamole_user
mysql-password: <YourPasswordHere>

If you configure the user-mapping.xml file before then you may no longer need that so remove it.

sudo rm -f /etc/guacamole/user-mapping.xml

Restart Tomcat and test logging in again using default username and password i.e. guacadmin.

sudo systemctl restart tomcat

Now, you should be able to make changes in the Settings. It is recommended you create a new user and then remove the guacadmin as soon as possible.

Enable Two-Factor Authentication

If you want you can optionally install TOTP module to enable two-factor authentication to add more security to your VM pool. Just download the extension and restart Tomcat.

# Download and install TOTP extension
curl -fLO "https://downloads.apache.org/guacamole/${GUAC_VERSION}/binary/guacamole-auth-totp-${GUAC_VERSION}.tar.gz"
tar -xzf "guacamole-auth-totp-${GUAC_VERSION}.tar.gz"
sudo mkdir -p /etc/guacamole/extensions
sudo cp "guacamole-auth-totp-${GUAC_VERSION}/guacamole-auth-totp-${GUAC_VERSION}.jar" "/etc/guacamole/extensions/"

# Restart tomcat
sudo systemctl restart tomcat

Now, when you try logging in again, it will ask you to setup an authenticator and require you to input in every time you log in.

Scripts

If you don’t want to perform all above steps one by one, you may leverage bash scripts I created in this repository.

pacroy/guacamole-setup-script

Use at your own risk!

Originally published at https://dev.to on April 30, 2021.

In my previous blog about setting up Kubernetes with MicroK8s and Traefik, I used DDNS service from noip.com to create a domain name microk8s.ddns.net for testing with whoami application which was working without any problem.

But when I tried to switch to my other domain which is a DDNS service from my internet provider, the certificate cannot be issues with the error query timed out looking up CAA.

This blog is to get to know more about ACME and CAA.

Introduction

To host a website or web application securely, you may want to serve it over HTTPS (HTTP over SSL, or should be over TLS now) only and disable HTTP or, better, to always redirect it to HTTPS.

To make HTTPS works properly, you need a certificate to prove your web’s identity with your clients that you’re really from the domain you serve, not from a middleman who try to see your transferred data. Of course, the self-signed certificate won’t do the job.

To validate the certificate that it is belong to whom it claims to, the certificate must be issued by a trusted certificate authorities (CA). Every web browsers or OSs maintain the list of trusted CAs so the validation works and cross-checked.

Let’s Encrypt is a trusted CA that provide automated ways to issue certificate for your application for free using Automatic Certificate Management Environment (ACME) protocol. Learn more about how Let’s Encrypt and ACME protocol work in this documentation page.

For servers or VMs, you can use certbot to do this job. For Kubernetes clusters, you have cert-manager and Traefik that can do this job.

Traefik Error Log

Let’s get back to my situation. When I tried to setup a new IngressRoute with certificate resolver for my whoami application for a specific domain, I got these errors in the Traefik’s log:

Unable to obtain ACME certificate for domains "this.isnotwork.com": unable to generate a certificate for the domains [this.isnotwork.com]: error: one or more domains had a problem:
[this.isnotwork.com] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: query timed out looking up CAA for this.isnotwork.com

After several of the same errors, it showed this error as the last one.

Unable to obtain ACME certificate for domains "this.isnotwork.com": unable to generate a certificate for the domains [this.isnotwork.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/

*For security reason, the domain is substituted.

I searched with the error message and found this post explaining concisely about CAA error.

Let's encrypt returns "query timed out looking up CAA for " error when obtaining a new certificate

In Let’s Encrypt documentation page, it also explain about CAA.

Certificate Authority Authorization (CAA)

Certificate Authority Authorization (CAA)

In short, Let’s Encrypt (or any CA) should query for a CAA record from the DNS server to determine if it is authorized to issue a certificate for that specific domain. This is to prevent mis-issue certificates that could potentially affect every domain names.

Verify CAA

You can verify if your domain name is compatible with Let’s Encrypt ACME by using https://unboundtest.com/ which simulate like what Let’s Encrypt will perform when it’s trying to issue a certificate.

You will get i/o timeout for the domain that does not provide CAA query and hence won’t work with Let’s Encrypt.

Here is an example for one that provide CAA query and will work with Let’s Encrypt.

Solution

Contact your DNS service provider whether they can add CAA record for you or otherwise change the service provider. Most of DNS service providers should support CAA record out of the box. You may check out this list from Traefik documentation page.

Azure DNS

Here, I have tried to add a new A record under my Azure DNS zone and CAA query works fine.

Here is Unboundtest log:

Then tried to add a CAA record to explicitly authorize letsencrypt.org to issue certificate for this specific domain.

Here is Unboundtest log:

CNAME Record

The other question still persists. Can I use CNAME record for my domain name?

If your home’s public IP is dynamic so you cannot use A record as the IP keeps changing but you want to use CNAME record to map your domain name (the pretty domain that you bought) to your DDNS domain name (the uglier domain that provided by DDNS service provider e.g. noip.com).

The short answer is NO. It seems ACME protocol will resolve IP address for your domain from A (for IPv4) or AAAA (for IPv6) record only. In addition, as I tried, if you create a CNAME record, you cannot create CAA record and vice versa.

I’m still looking for solution for this scenario. One option is to use dnschallenge resolver but I’m not sure if it will work with CNAME or otherwise I need to develop a script run by a regular job to manually update DNS record.

Will share later. Cheers!

In my recent blog, I have setup a single-node Kubernetes cluster on my home lab server using k0s. It is a good tool that make it easy to setup a cluster in just one or two commands. However, as it may be relatively new, I found a problem when starting the cluster that worker node sometime isn’t up and running and you need to restart the service.

Setup Single-node Kubernetes Cluster on a Home Lab server using k0s

TL;DR

In this blog, I will try another tool called MicroK8s which is around for sometime and hopefully will be more stable than k0s to setup a single-node Kubernetes cluster on Ubuntu Linux 20.04 on my home lab box.

Prerequisites

• A public IP address from your internet provider, either dynamic or static.
• A domain name mapped to your static IP or a dynamic DNS domain name for dynamic IP which can be configured on your modem router to sync with the Dynamic DNS provider e.g. no-ip.
• You have a fresh install of Ubuntu 20.04 on your home lab server — see this blog for how-to.
• SSH has been configured for remote access on your home lab server— see my previous blogs on how to setup on home lab or Linode VM.
• kubectl, git, and helm are installed on your local machine

Setup MicroK8s

In this section, we will install MicroK8s on our Ubuntu server.

Install MicroK8s

On your server, use snap to install the MicroK8s package.

sudo snap install microk8s --classic --channel=1.21

Add yourself into the group microk8s, gain access to .kube caching directory, and refresh the session so group update takes effect.

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
su - $USER

Monitor the cluster provisioning status. This may take a few minutes until the cluster is ready.

microk8s status --wait-ready

Get nodes and services on the cluster.

microk8s kubectl get nodes
microk8s kubectl get services

Enable foundation addons dns and storage

microk8s enable dns storage

Remote Access

To enable remote access to the api-server using kubectl, edit the file /var/snap/microk8s/current/certs/csr.conf.template on your server and add domain name and/or IP address (if static) of your server under alt_names section that reachable from the internet.

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = your.dnsname.com
IP.1 = 127.0.0.1
IP.2 = 10.152.183.1
IP.3 = 192.168.1.xx
IP.4 = 123.456.789.0

Export kubeconfig to file.

microk8s config > admin.config

On your local machine, copy the kubeconfig file from the server to your machine.

scp username@your.dnsname.com:~/admin.config .

Edit the admin.config file and update clusters.cluster.server by replacing the IP and port with your domain name or public IP (if static). It is recommended to use the port other than 16443 for better security and then configure your router to forward the specified port to your server’s port 16443.

Test the connection

export KUBECONFIG=admin.config
kubectl version
kubectl get nodes

Deploy Application

In this section, we will deploy an application named whoami for testing purpose.

Install whoami

First, clone this repository and apply deployment and service on the cluster.

git clone https://github.com/pacroy/whoami.git
cd whoami
kubectl create ns whoami
kubectl apply -f whoami.yml -n whoami

Once the pod is up and running, forward port to its service.

kubectl port-forward service/whoami 8080:80 -n whoami

Go to localhost:8080 on your local machine and you should see the response like this:

Expose Service

In this section, we will expose whoami service to the outside world using metallb load balancer and traefik ingress controller.

Enable meltallb

To expose a service with LoadBalancer, we need to enable metallb which a load balancer for bare metal.

microk8s enable metallb

It will ask to input IP address range allocated for load balancers. It is good to assign s small IP address pool within your subnet that are outside your DHCP range to avoid collision.

In this case I choose 192.168.1.85–192.168.1.89.

Enabling MetalLB
Enter each IP address range delimited by comma (e.g. ‘10.64.140.43–10.64.140.49,192.168.0.105–192.168.0.111’): 192.168.1.85–192.168.1.89

Update whoami service type to LoadBalancer.

apply -f service-lb.yml -n whoami

Get service

Test the load balancer by taking note the EXTERNAL-IP and opening it in your browser (in this case it is http://192.168.1.85) and you should see the same result as before.

Restore the service type to ClusterIP as we will expose it via Ingress in the next section.

kubectl apply -f whoami.yml -n whoami

Install Traefik Ingress

On your server, enable traefik ingress controller for external access.

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
kubectl create namespace traefik
helm install traefik traefik/traefik -n traefik

Check component and you should see the traefik service is exposed via LoadBalancer on both port 80 and 443, by default.

Restore whoami service type to ClusterIP and create an ingress which route traffics at any host with URI /whoami to our service.

kubectl apply -f ingress.yml -n whoami

Configure your modem router to forward port 80 and 443 to your LoadBalancer IP address. Instruction varies by router’s brand and model and won’t be covered here.

Open your browser and go to your domain name or public IP and access URI /whoami. In my case, it is http://microk8s.ddns.net/whoami.

traefik Dashboard

You can access traefik dashboard by forwarding port 9000 from a traefik pod.

kubectl port-forward $(kubectl get pods --selector "app.kubernetes.io/name=traefik" --output=name -n traefik) -n traefik 9000:9000

Then access the traefik dashboard at http://localhost:9000/dashboard

Configure HTTPs

In this section, we will enable HTTPS by setup a mechanism to automatically generate SSL certificate for our application using ACME provider from Let’s Encrypt authority (It’s free!).

UPDATE: You may wanna check first if your domain name can work with Let’s Encrypt by follow what I did in this blog.

Introduction

Now, if you try to access https://your.dnsname.com/whoami then you will get this warning page.

This is because the traefik is using a self-signed certificate which is not trusted by the browser.

To make our whoami website trusted by the browser, we need to tell traefik to get and use a certificate that issued by a trusted certificate authority (CA) e.g. Let’s Encrypt.

Delete the ingress as we will use IngressRoute instead in the next section.

kubectl delete ingress/whoami -n whoami

IngressRoute

Instead of using standard Ingress object with lots of annotations, traefik also provides their own CRD called IngressRoute that make the configuration more readable and structured.

Let’s try IngressRoute. Edit the file ingressroute.yml and update the hostname microk8s.ddns.net with your domain name then apply.

kubectl apply -f ingressroute.yml -n whoami

Try accessing whoami again at http://your.dnsname.com/whoami and you should see the same result.

Certificate Resolver

Next, we need to upgrade the traefik to include a certificate resolver. Edit the file values.yml and update youremail@domain.com with your email address. This will be used by Let’s Encrypt to send notification email when the certificate nearly expires.

Upgrade the traefik release.

helm upgrade traefik traefik/traefik -n traefik --values values.yml

Check if the arguments are applied correctly by looking the Deployment’s manifest.

helm get manifest traefik -n traefik

Check traefik’s log and you should see no error about the certificate resolver.

kubectl logs $(kubectl get pod -n traefik -o name) -n traefik -f

IngressRoute with TLS

Now, let’s apply a new IngressRoute that using the certificate resolver. Don’t forget to update the hostname to yours!

kubectl apply ingressroute-tls.yml -n whoami

Check if the certificate is created properly.

kubectl exec -it $(kubectl get pod -n traefik -o name) -n traefik -- cat /data/letsencrypt.json

NOTE: It may take a while before the certificate shows up for the configured domain.

Test accessing your application at https://your.dnsname.com/whoami and the browser should show the page with a valid certificate without warning.

Security

This section will show you how to enhance security for our application.

TLS version 1.2

HTTPS should no longer support SSL and TLS v1.1 and TLS v1.2 as they are weak in term of security. If we scan our endpoint with SSLLabs, you will get rating at B.

We can make our traefik ingress to accept connection with minimum TLS v1.2 by creating a new TLSOption in default namespace.

kubectl apply -f tlsoption.yml

This create a default TLSoption that will apply to all traefik routers by default (if not explicitly overridden). Scan the endpoint again with SSLLabs and you will now get rating A.

Redirect to HTTPS

So far, our whoami application serves HTTP traffic at port 80 and HTTPS traffic at port 443. What if we want to redirect all HTTP traffics to HTTPS. We need to leverage RedirectScheme middleware.

Let’s create a new middleware in default namespace.

kubectl apply -f middleware-https.yml

Then replace IngressRoute with the new one with the middleware.

kubectl apply -f ingressroute-http.yml -n whoami

Test your HTTP endpoint and it should now redirect to HTTPS.

$ curl http://microk8s.ddns.net/whoami -D-
HTTP/1.1 301 Moved Permanently
Location: https://microk8s.ddns.net/whoami
Date: Mon, 12 Apr 2021 14:14:36 GMT
Content-Length: 17
Content-Type: text/plain; charset=utf-8

Moved Permanently

For more information about the traefik’s concept and configuration, please visit its documentation site.

See all available values for traefik Helm chart here.

Cheers!

I have been running a small Kubernetes (k8s) cluster on a cloud provider for some time and finding alternative to reduce my bill. I could have run my small workloads on a small server that cost only $5 a month like I did for in this blog but I prefer a k8s cluster as all my existing configurations and scripts are built for it and I can keep learning and catch-up this popular ecosystem.

Recently, I have read this blog by Luc Juggery which show how easy you can setup a k8s cluster on your home lab server with k0s. Fortunately, I have a miniPC box with CPU Atom 1.44Ghz , 4GB RAM that I don’t use much. So here come this blog.

TL;DR

This blog guides you how to setup a single-node Kubernetes cluster with k0s on a miniPC running Ubuntu 20.04 OS and install ingress-nginx and cert-manager and a sample whoami application that can be publicly accessed over the Internet.

Prerequisites

• Broadband internet with a good download/upload bandwidth — in my case it is 500/500Mbps
• A home lab PC, at least 2 CPU cores, 2GB of RAM, connected to the your home broadband network
• A public IP from your internet service provider — this may incur additional cost, check with your ISP

Install Ubuntu Server 20.04 on Home Lab server

Download Ubuntu Server 20.04 ISO file from its official site.

Make a bootable USB stick from the downloaded ISO file follow this official instruction.

Once you get the bootable USB stick, restart your PC and boot with your USB. Just follow on-screen instruction to install Ubuntu server.

Configure Ubuntu

Update System

Once the installation complete, log on and update the system.

apt-get update && apt-get -y upgrade

Configure IP Address

Configure your DHCP server in your router to issue a fixed IP address to your server. Try to assign an IP address that outside of your DHCP range to avoid collision. For example, your router may have DHCP range from 192.168.1.100 to 192.168.1.255. Then you may assign 192.168.1.10 to your server.

Configuration steps are vary by router. If not possible, you may need configure IP manually on Ubuntu. The instruction to do so is beyond this blog.

After DHCP is configured, you may restart the server and check IP address if it is configured properly.

# Reboot
sudo reboot

# Show local IP address
hostname -I | awk ‘{print $1}’

Setup SSH

Add your SSH key to the server by executing this command on another computer/laptop.

ssh-copy-id yourusername@ip_address

Edit the SSH config file

sudo nano /etc/ssh/sshd_config

Change the following lines for better security

PermitRootLogin no
PasswordAuthentication no

Restart SSH

systemctl restart sshd

Test SSH from another computer/laptop and make sure you can connect and logon.

ssh yourusername@ip_address

Configure SSH for Remote Access

If you want to SSH to your server from the internet, you need to configure port forwarding on your router. Choose the external port other than 22 for better security. This vary by router so the instruction is beyond this blog.

Once configured, you may test the connection.

ssh your_username@external_ip -p external_port

You can see your external IP address by executing this command on your server.

curl ipv4.icanhazip.com

If your public IP is static then this is fine. You can use the IP address to connect as it is fixed and will never change.

But most of ISPs won’t usually give you a static one but dynamic one i.e. your external IP keeps changing. In this case, you may need to configure Dynamic DNS (DDNS) settings on your router so you can access your server using a domain name and you don’t need to worry about the IP.

Again, the steps to configure DDNS are vary by router and will not be included in this blog.

To save some keys on SSH command, you can create the file ~/.ssh/config with the following content:

Host youralias
  User your_username
  Port external_port
  Hostname your_domain_name

Then you can easily make SSH connection in short like this

ssh youralias

Unattended Upgrade

You can also configure the system to automatically upgrade and restart by following steps I wrote in this blog. Look for Setup Unattended Upgrades title.

Install k0s

Follow its official instruction by executing this command.

curl -sSLf https://get.k0s.sh | sudo sh

Check if it is install properly

$ k0s version
v0.11.0

Setup the Cluster

Building Config

Generate default config YAML file to a folder that make it accessible by root .

sudo k0s default-config > /root/k0s.yaml

Edit the file and append the following extension. Make sure you update two things:

1. The IP address range load balancer —Use a small range of available and addressable IP addresses in your local networks that outside DHCP scope to avoid collision. In this example, I use 192.168.1.20–192.168.1.25.
2. The email address in the cluster-issuer section.

This will install ingress-nginx and cert-manager on your cluster in one shot.

Start the Cluster

Install the cluster from our configuration file and start the server.

sudo k0s install controller -c /root/k0s.yaml — enable-worker
sudo systemctl start k0scontroller.service

Wait a bit and check server status

$ sudo k0s status
Version: v0.11.0
Process ID: 1472
Parent Process ID: 1
Role: controller+worker
Init System: linux-systemd

Check server node

$sudo k0s kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
urserver1   Ready    <none>   6d    v1.20.4-k0s1

NOTE: After a few minutes, if you still don’t see the node then try to restart the service.

sudo systemctl restart k0scontroller.service

Reset the Cluster

In case your cluster go wrongly and you want to resetup the cluster then use this command to reset the cluster and restart the server before repeating the steps.

sudo k0s reset
sudo reboot

Remote Access to the Cluster

To access the cluster with kubectl, you need to copy the kubeconfig file.

sudo cp /var/lib/k0s/pki/admin.conf ~/admin.conf

And transfer to your other computer.

scp your_username@your_server:~/admin.conf ~

Edit the file and replace server field with your server IP address:port. If you decide to open this port (6443) to the internet. This could be your server’s external IP or domain name:external port.

apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ***
 server: https://your_server:port
 name: local
...

Test connection

$ export KUBECONFIG=~/admin.conf
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.4-k0s1", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-03-03T07:31:16Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

If you server’s version information then the connection is successful.

Verify Cluster

Check if all pod are up and running. You may need to wait for a while until all pods are up.

kubectl get pods --all-namespaces

Check if all Helm charts are installed as expected

helm list --all-namespaces

Check ingress external IP. It should be your load balancer IP address on your local network that you defined in k0s.yaml file when setting up the server.

kubectl get services --namespaces kube-system

Forward Ports

To make your web application accessible from the internet, you need to forward the port 80 and 443. The configuration steps should be the same as you did for SSH (port 22) and kubectl (port 6443).

Eventually, you may have these port forwarding configuration on your router (Assume 192.168.1.10 is the server’s IP and 192.168.1.20 is the LB’s IP).

Ext.Port   Int.Port   Server IP
--------   --------   ---------
12322      22         192.168.1.10
12343      6443       192.168.1.10
80         80         192.168.1.20
443        443        192.168.1.20

Test Deploying whoami

Prepare Configuration File

Create a new namespace

kubectl create namespace whoami

Create a YAML file whoami.yml with the following content

Deploy Application and Test

Apply the YAML file to the namespace

$ kubectl apply -f whoami.yml --namespace whomai
deployment.apps/whoami-deployment created
service/whoami-service created
ingress.networking.k8s.io/whoami-ingress created

Check the status

kubectl get all --namespace whoami

Try accessing your application at http://load_balancer_ip/whoami

Configure HTTPS

To have a proper SSL certificate for HTTPS connection, you need a valid domain name. Then create a DNS A record pointing to your public IP address. The instruction is vary by DNS service provider and won’t be included here.

Check if DNS record can be resolved properly

$ nslookup whoami.yourdomain.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   whoami.yourdomain.com
Address: xxx.xxx.xxx.xxx

Create a new ingress file ingress.yml with the following context:

Reapply the ingress

$ kubectl apply -f ingress.yml --namespace whoami
ingress.networking.k8s.io/whoami-ingress configured

Watch the pods. You will see the a cert-manager pod is created to auto-provision SSL certificate and then get terminated. This process usually takes less than one minute.

kubectl get pod --watch --namespace whoami

Check the certificate status and you should see True in the Ready column.

$ kubectl get certificates --namespace whoami
NAME     READY   SECRET   AGE
whoami   True    whoami   2m56s

Now, test accessing your application at http://yourdomain.com/whoami and it should redirects to HTTPS with valid SSL certificate.

Clean up

Once you finished testing, you can delete whoami.

kubectl delete all --all --namespace whoami
kubectl delete namespace/whoami

Caveats

k0s worker node is sometimes not up and running

when you start the cluster first time, sometimes the worker node is not up even after some mount of time. In this case, restarting the service is usually help.

sudo systemctl restart k0scontroller.service

Cannot use CNAME record with cert-manager

In my environment, my public IP is dynamic and mapped to a DNS using Dynamic DNS (DDNS) service. I have my own custom domain that I try to map to DDNS domain using CNAME record. But that doesn’t seem to work for cert-manager.

I end up mapping my custom domain using a A record and thinking about to setup an automated task to regularly check and update DNS A record.

But please let me know if you find a better way to deal with this.

I have recently stumbled upon a video on YouTube by Wolfgang’s Channel about how to setup your own VPN server using OpenVPN. There’s also a transcript version of the video that you can read and follow along. Or you can follow my guide below.

TL;DR

In this guide, I will show you how to setup OpenVPN server on a Linux VM hosted on Linode. The cost of the VM is $5 per month (1TB traffic inclusive) but you can get free $100 credit for 60 days from this link. Since this is your own server, you can create multiple VPN profiles for multiple devices or give them to your friend or family to use.

Find a Cloud Server

I was recently looking for a cheap virtual machines (VM) or virtual private server (VPS) solution for running my workloads. The cheapest one (1 vCPU, 0.75GB RAM) on Microsoft Azure is ~ $13/month and it only includes 5GB transfer. It could cost you ~$100/month for 1TB transfer.

I’m also looking for local providers in-country and its cheapest plan starts from ~$10/month with similar size VM (1 vCPU, 1GB RAM). But the good thing is they offer unlimited transfer. However, it may not serve the purpose of using VPN as you might not want website to know where you are from.

The cheapest one I found is Linode that offer a 1vCPU, 1GB RAM for $5/month with 1TB transfer included which should be enough for general VPN usage. It would cost $10 for each additional TB transfer.

Digital Ocean is another popular provider that offer similar price. If you see any better alternative, please also let me know. But in this guide I’ll just use Linode to host my VPN server.

Create Virtual Machine

Click Create a Linode with the following parameters:

Image: Ubuntu 20.04 LTS
Region: Choose the closest one to you
Linode Plan: Nanode 1GB ($5 monthly)
Linode Label: Name your server
Root Password: Set a secure root password (we will disable root login later)
Private ID: Select to add a private IP

Wait until the provisioning is completed when the status turns to Running.

Try to connect to the server using your favourite SSH client. In my case, I will use SSH client inside Ubuntu distro of WSL.

Configure System

Update system

Upgrade your system

apt-get update && apt-get -y upgrade

Create User

Create a new user as using root is not a good security practice.

useradd -G sudo -m yourusername -s /bin/bash

Set the password for the new user.

passwd yourusername

If you don’t have a SSH key yet, open a new terminal and create a new SSH key on your local PC.

ssh-keygen -t ed25519

You may set a passphase for better security.

Then, add the SSH key to the new user on the server.

ssh-copy-id yourusername@ip_address

Configure SSH

On your server, edit the file /etc/ssh/sshd_config

nano /etc/ssh/sshd_config

Uncomment and change the following lines:

Port 12345
PermitRootLogin no
PasswordAuthentication no

For Port, change to other number than 22 to prevent SSH port scanners.

Restart the SSH service

systemctl restart sshd

Test loggin in using SSH key in a separated terminal window but don’t close the current root terminal yet in case you cannot log on.

ssh yourusername@ip_address -p 12345

If you can log on then you can log off the root and try logging in again. You should not be able to to log on.

Disable sudo Password

If you don’t like to enter password everytime you use sudo command then edit the file /etc/sudoers by executing the following command

sudo visudo

Append the following line:

yourusername ALL=(ALL) NOPASSWD: ALL

Please note that this reduces the security.

Change Hostname

Edit the file /etc/hostname

sudo nano /etc/hostname

Change the hostname from locahost to the name you want e.g. demo-server in my case.

Edit the file /etc/hosts

sudo nano /etc/hosts

And add your new hostname like below:

127.0.0.1 localhost demo-server

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback demo-server
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Reboot the server from your Linode console.

Create SSH alias

To avoid typing long command to login to server like ssh pacroy@123.456.78.90 -p 12345. We can create an alias for it.

Create the file ~/.ssh/config on your local PC and add the following:

Host demo-server
    User yourusername
    Port 12345
    Hostname server_domain
    IdentityFile ~/.ssh/id_ed25519

Name your alias on the first line. In this case, I name it demo-server.

For Hostname, instead of using IP address which can be changed for some cloud providers, you can use server’s domain name. For Linode, you can get the server’s domain name from the console:

For IdentityFile, you need to specify if you have multiple SSH keys. If you have only one key then you can omit this line.

Test your alias with:

ssh demo-server

And you should be able to log in.

Install OpenVPN

Execute the Script

In this step, we will use the installation script from https://github.com/Nyr/openvpn-install

Execute this command to install OpenVPN:

wget https://git.io/vpn -O openvpn-install.sh && sudo bash openvpn-install.sh

It will ask a few questions which you can accept the defaults by keep hitting enter.

Welcome to this OpenVPN road warrior installer!

Which IPv4 address should be used?
 1) 139.162.54.189
 2) 192.168.169.11
IPv4 address [1]:

Which protocol should OpenVPN use?
 1) UDP (recommended)
 2) TCP
Protocol [1]:

What port should OpenVPN listen to?
Port [1194]: 443

Select a DNS server for the clients:
 1) Current system resolvers
 2) Google
 3) 1.1.1.1
 4) OpenDNS
 5) Quad9
 6) AdGuard
DNS server [1]: 3

Enter a name for the first client:
Name [client]: demo-server

OpenVPN installation is ready to begin.
Press any key to continue…

However, I choose the port to 443 and 1.1.1.1 as my DNS server as suggested in the original guide.

For the client name, I recommend to choose a name that can scale if you plan to create more clients for different devices or for multiple users. For example, demo-1, demo-2, etc.

Once ready, press anykey to start the installtion. It should take only several seconds to complete.

Copy the VPN Profile

You need this file to setup the client.

Copy the file and change owner so you can download it later via SSH.

sudo mv /root/demo-server.ovpn ~
sudo chown pacroy:pacroy ~/demo-server.ovpn

Disable the log

Edit the file /etc/openvpn/server/server.conf

sudo nano /etc/openvpn/server/server.conf

Find and change this line to verb 0

verb 0

Restart the OpenVPN service

systemctl restart openvpn-server@server.service

Setting up Your Client

Download the Profile

Use scp command to download the ovpn file to your local PC.

scp demo-server:~/demo-server.ovpn ./Downloads

Download OpenVPN client

Head to https://openvpn.net/download-open-vpn/ and download client for your device. It supports Windows, MacOS, Linux, iOS, Android.

In this case, I’ll show how to setup the client on Windows. But the method is similar on all devices. You need to use a secure way to transfer the ovpn file to your device e.g. using cloud storage, IM, etc. Avoid using email.

Drop the ovpn file.

Click Add

Try to connect

If you see the above screen then you have setup everything correctly and you’re ready to roll!

Adding More Clients

If you have multiple devices or you want to create new profile for other users. Do not use the same ovpn profile as the connection will not work well if there are multiple clients connect using the same profile. Instead, create a new client

Execute the installation script again on the server to add a new client.

sudo bash openvpn-install.sh

Follow the on-screen instruction to add a new client (or remove).

OpenVPN is already installed.

Select an option:
 1) Add a new client
 2) Revoke an existing client
 3) Remove OpenVPN
 4) Exit
Option: 1

Provide a name for the client:
Name: client-2
…

Use the same method above to download .ovpn file and send to your device to setup the client.

Setup Unattended Upgrades (Optional)

You can configure the server to automatically upgrade and reboot to apply security patches.

Install required components:

sudo apt install -y unattended-upgrades apt-listchanges

Follow on-screen instruction to install Postfix.

Enable the stable security upgrades

sudo dpkg-reconfigure -plow unattended-upgrades

Choose Yes.

Edit the config file.

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Uncomment and update the following lines as shown

Unattended-Upgrade::Remove-Unused-Kernel-Packages “true”;
Unattended-Upgrade::Remove-Unused-Dependencies “true”;
Unattended-Upgrade::Automatic-Reboot “true”;
Unattended-Upgrade::Automatic-Reboot-Time “23:00”;

Time should be in your server timezone which is UTC by default.

Test to see if it works

sudo unattended-upgrades — dry-run

Now, the system will be automatically upgraded.

We often use Docker when we talk about containers, similar to we use ‘Coke’ when we talk about carbonated soft drink. We all know that Coke is not the only brand that sells carbonated soft drink. Same to Docker, it is not the only one that does containers.

Why not Docker

Docker has been around for quite a while and it is a good tool. It is one of the very first thing I always tell junior developers to study when they start their first job from school.

Recenly, I feel annoying installing Docker on my different machines that I need to switch back and forth. Due to its big installation file so it is heavy weight. I normally use Docker to just build images and deploy workloads on my Kubernetes clusters and use it for nothing else.

I knew about podman and skopeo that may do the job I need but never have a chance to try them out until I stumble upon this blog from Martin Heinz.

You Don’t Have to Use Docker Anymore

It is pretty long one but good to read through. But if you don’t have time, here is my summary:

• Docker is a monolithic tool that tries to do everything. But the functionality can be divided into several components:

Container Engines

• Container Engines is a tool providing UI for working with images and containers (excluding running containers)
• The most prominent competitor to Docker is Podman, developed by Red Hat.
• Podman doesn’t need daemon to run and also doesn’t need root privileges which has been long-standing concern with Docker.
• Podman can also run pods which make it easier to later migrate the workloads to Kubernetes.
• Podman provides the exact same CLI commands as Docker as they are implemented using the same standard defined by the Open Container Initiatives (OCI).
• There are other Container Engines but can be considered as dead-end e.g. LXD, CRI-O, rkt (rocket)

Image Builder

• Buildah is another tool developed by Red Hat and it comes along with Podman and is called when you run podman build.
• Buildah is daemonless and rootless and produces OCI compliant images so it’s guaranteed that your images will run the same way as the ones built with Docker.
• Buildah is also able to build images from Dockerfile .
• Buildah are user specific, so you will be able to list only images you built yourself.
• buildah CLI is superset of commands included in podman build . Learn more about the differences between Podman and Buildah from this article.
• Another tools for building images are Google’s Kaniko, Docker’s buildkit, OpenShift’s Source-To-Image (S2I), Jib, and Bazel.

Container Runtime

• Container Runtime is responsible for running containers and is one part of the whole container lifecycle/stack, which you will most likely not going to mess with.
• runc is the most popular container runtime created based on OCI container runtime specification. It’s used by Docker (through containerd), Podman, and CRI-O (default for OpenShift cluster).
• Alternative to runc is crun which is a tool developed by Red Hat and fully written in C (runc is written in Go). Considering that it’s Red Hat product, we might eventually see as default for Podman or CRI-O.
• Last one to mention is containerd which is a CNCF graduating project and acts as an API facade for various container runtimes. It’s used by Docker Engine, Google Kubernetes Engine (GKE), IBM Kubernetes Service (IKS).

Image Inspection and Distribution

• Skopeo is made by Red Hat and it’s an accompanying tool for Buildah, Podman.
• Skopeo is also able to copy images using skopeo copy which allows you to mirror images between remote registries without first pulling them to local registry.
• Dive is another tool for inspecting, exploring, and analyzing images. It’s little more user friendly and provides more readable output.

Install podman, buildah, and skopeo

Now, it’s time for hands-on. Let’s install these three tools from Red Hat. Please note that here I tried on Ubuntu 18.04 so the steps may be different on other distros.

In order to map container ports to host ports, you also need slirp4netns. It must be found within PATH variable so check it with command which slirp4netns. If not found, install using the script below.

Add it to PATH variable, if you haven’t done so.

Testing podman

Run a new httpd container and forward port 8080 to host.

If you get the error ERRO[0001] unable to write pod event: "write unixgram @00018->/run/systemd/journal/socket: sendmsg: no such file or directory", you seem to run podman in WSL2. Then you need to use the flag --events-backend=file to suppress this error.

Note: Podman will search in default registries if you don’t specify full image name. The default registries are defined in /etc/containers/registries.conf. You can use command podman info to see the list of registries.

Check container status.

Try accessing the application at http://localhost:8080.

Testing buildah

Clone repository https://github.com/pacroy/flask-app and use buildah to build the image from the Dockerfile.

List local images.

Run the container

Try accessing the application at http://localhost:5000.

Testing skopeo

You can inspect properties or configuration of an image on a remote repository using skopeo.

Note: If you don’t have jq installed, you can download it from https://stedolan.github.io/jq/ and add it to PATH variable.

You can also inspect a local image.

Next, we will push the built image to Docker.io. Create a new repository on Docker Hub first if you haven’t done so. Login and push the image, just like you use Docker.

Then, we will copy the image from Docker.io to Quay.io. Create a new repository on quay.io first and then copy the image from docker.io.

Note: You may want to use access tokens (aka. robot account in quay.io) instead of password.

Now, I hope you see that we can use podman, buildah, and skopeo for working with containers and images like we use Docker. There are also some useful features that does not even exist in Docker CLI that you may love.

Next time, when you feel overwhelm with Docker installation then now you know an alternative option.

Updated Jan 2019: Azure Container Service (ACS) has been replaced by Azure Kubernetes Service (AKS). Please refer to this guide on how to easily create an AKS cluster.

Table of Contents

• 1. Create Application ID
• 2. Deploy Azure Container Service
• 3. Connect to the Master
• References

1. Create Application ID

1) Go to Azure Active Directory → App registrations and click New application registration

2) Name your app and URL then click Create

3) Note the Application ID and click Settings

4) Go to Keys. Fill in key description and expires then click Save

5) Note the secret key generated

6) Go to your Subscription → Access control (IAM) and click Add

7) Add your new app with role at least Contributor

2. Deploy Azure Container Service

1) Go to Marketplace and search for ACS. Select and click Create.

2) Name your service and resource group

3) Select Kubernetes as the orchestrator. Complete other fields and fill Application ID in Service principal client ID and Secret key in client secret.

4) Select agent size. (This size is only for agents but not the master so you may need to resize the master after deploy)

5) On the summary page, click Deploy. This may take ~15–20 minutes.

6) After the deployment is done, here are the resources you get:

3. Connect to the Master

1) Select the master VM and click Connect to see SSH command.

2) Once connected, try to see the kubectl’s version using kubectl version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

References

• Service principal for Azure Kubernetes cluster
• Create identity for Azure app in portal

Introduction

SSH is the way to connect and command your Linux machine. Doing this from Windows is even more painful as you need a client like Putty installed (Or enable built-in SSH if you’re using Windows 10).

What if you can do it right from your web browser from any OS without installing anything!?!

What is Wetty?

Wetty = Web + tty.

Terminal over HTTP and HTTPS. Wetty uses ChromeOS’ terminal emulator (hterm) which is a full fledged implementation of terminal emulation written entirely in Javascript. Also it uses websockets instead of Ajax and hence better response time.

Source: https://github.com/krishnasrinivas/wetty

Prerequisite

• Ubuntu Server with Docker installed (See this blog on how to get one on Azure)

Installation

1. Clone Wetty repository on GitHub.

$ git clone git@github.com:krishnasrinivas/wetty.git

2. Edit Dockerfile

$ vim Dockerfile

You will see the something like this:

FROM node:0.10.38
MAINTAINER Nathan LeClaire <nathan@docker.com>

ADD . /app
WORKDIR /app
RUN npm install
RUN apt-get update
RUN apt-get install -y vim
RUN useradd -d /home/term -m -s /bin/bash term
RUN echo 'term:term' | chpasswd

EXPOSE 3000

ENTRYPOINT ["node"]
CMD ["app.js", "-p", "3000"]

This tells us that:

• Wetty is a Node.js application.
• It creates a new user term which we can use to logon.
• The web interface is exposed at port 3000

Setup HTTPS — Always use HTTPS!

3. If you don’t have SSL certificates from a CA you can
create a self signed certificate using this command:

$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 30000 -nodes

You need to fill in some information. Better to put common name with your domain name you want to publish your web terminal e.g. terminal.yourdomain.com

Country Name (2 letter code) [AU]:TH
State or Province Name (full name) [Some-State]:Bangkok
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Demo Company
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:terminal.yourdomain.com
Email Address []:admin@yourdomain.com

4. Update entry point in Dockerfile to include certificates.

CMD ["app.js", "--sslkey", "key.pem", "--sslcert", "cert.pem", "-p", "3000"]

5. Copy private key id_rsa you normally use to connect to your Ubuntu server and add these lines to Dockerfile.

RUN mkdir /home/term/.ssh
COPY id_rsa /home/term/.ssh/id_rsa
RUN chmod 600 /home/term/.ssh/id_rsa && chown -Rf term /home/term/.ssh

6. Update password as you wish.

RUN echo 'term:yourpassword' | chpasswd

At the end, you Dockerfile may look like this:

FROM node:0.10.38
MAINTAINER admin@yourdomain.com

ADD . /app
WORKDIR /app
RUN npm install
RUN apt-get update
RUN apt-get install -y vim
RUN useradd -d /home/term -m -s /bin/bash term
RUN echo 'term:yourpassword' | chpasswd
RUN mkdir /home/term/.ssh
COPY id_rsa /home/term/.ssh/id_rsa
RUN chmod 600 /home/term/.ssh/id_rsa && chown -Rf term /home/term/.ssh

EXPOSE 3000

ENTRYPOINT ["node"]
CMD ["app.js", "--sslkey", "key.pem", "--sslcert", "cert.pem", "-p", "3000"]

Build Your Docker Image

$ sudo docker build -t yourusername/wetty .

Wait until it finishes, like this:

$ sudo docker build -t demo/wetty .
Sending build context to Docker daemon  1.042MB
Step 1/15 : FROM node:0.10.38
 ---> 82073591bd0c
Step 2/15 : MAINTAINER admin@yourdomain.com
 ---> Using cache
 ---> 067a8d078ccb
Step 3/15 : ADD . /app
 ---> ccd418e158ee
Step 4/15 : WORKDIR /app
Removing intermediate container 514c8c3f5511
 ---> d8611af96ced
Step 5/15 : RUN npm install
 ---> Running in e635ea2d39be
.
.
.
Step 13/15 : EXPOSE 3000
 ---> Running in f246b51e0a0f
Removing intermediate container f246b51e0a0f
 ---> 39b74e2eda59
Step 14/15 : ENTRYPOINT ["node"]
 ---> Running in 4fa38e6c2d19
Removing intermediate container 4fa38e6c2d19
 ---> ee6c549f88ca
Step 15/15 : CMD ["app.js", "--sslkey", "key.pem", "--sslcert", "cert.pem", "-p", "3000"]
 ---> Running in 26c1f3817918
Removing intermediate container 26c1f3817918
 ---> f69d767e6c92
Successfully built f69d767e6c92
Successfully tagged demo/wetty:latest

Spin Up a Container

$ sudo docker run --name termdemo -p 3000:3000 -dt demo/wetty

Test Your Wetty

Go to your domain e.g. https://terminal.yourdomain.com:3000 and log in with id term and its password.

Then, try SSH to your Ubuntu host.

Tips

To change the login timeout, edit file /etc/login.defs and look for LOGIN_TIMEOUT. (Reference)

References

krishnasrinivas/wetty

https://hub.docker.com/r/krishnasrinivas/wetty/

Related

Setting up Docker on Azure with Ubuntu server

In my last blog, I’ve explained about deployment pipeline I built for Continuous Integration and Continuous Delivery in ABAP. Another thing that our team has built was the application monitoring.

For our Java Spring Boot services, it can seamlessly integrate Prometheus module in the POM file and then your application is ready to be monitored with built-in metrics.

I managed to create Prometheus client for ABAP. I will explain how you can setup one for yours below.

What is Prometheus?

Prometheus is a time series database. It stores your data stream and it also has a web interface so you can query and visualize the data e.g. into a graph.

Imagine a temperature measured from a thermometer at regular interval. This is a time series data. In IT operations, this can be like CPU usage, memory allocated, etc. But in our case, we just want to monitor our application.

The concept I like is that the server will poll the data from applications instead of getting applications sending the data to the server. This ensures that monitoring will never break your application. If the monitoring server down, your applications are still running fine.

Visit their website to learn more.

Why do we need to monitor our application?

Well, have you ever wondered about the feature that you build, how much is it used by the users? How fast or slow is it? What if you want to experiment which button color would attract the customer better?

To answer these questions, you need some kind of monitoring, some kind of metric that you can measure.

Let’s start!

In this example, I will show you how we can monitor usage and response time of ABAP HelloWorld RESTful APIs.

Cloning repository

If you have a problem cloning the class and Shared Memory Area ZCL_SHR_PROMETHEUS_AREA then you can create it on your own as this class is automatically generated from the transaction SHMA. Make sure you configure it precisely as shown below.

Recording metrics in your application

Prepare runtime for duration metric.

Create a new instance attribute for the timer instance.

DATA runtime TYPE REF TO if_abap_runtime.

In the constructor, create the timer instance.

METHOD constructor. 
  super->constructor( ). 
  me->runtime = cl_abap_runtime=>create_hr_timer( ). 
ENDMETHOD.

Create a new method in the REST resource class (ZCL_REST_RESOURCE).

METHODS record_metric 
  IMPORTING 
    i_method TYPE string 
    i_response TYPE i.

The method accepts HTTP method and response status code which we will put in the metrics.

METHOD record_metric. 
  TRY. 
      zcl_prometheus=>set_instance_from_request( me->mo_request ).     
      zcl_prometheus=>write_multiple( VALUE #( 
        ( key = |hello_count\{method="{ i_method }",status="{ i_response }"\}| value = '1' command = zif_prometheus=>c_command-increment ) 
        ( key = |hello_duration\{method="{ i_method }",status="{ i_response }"\}| value = me->runtime->get_runtime( ) ) ) ). 
    CATCH cx_root. 
  ENDTRY.
ENDMETHOD.

set_instance* methods will set the Shared Memory Area instance that you can see in the transaction SHMM. So you can use one instance for each application and the metrics can be kept and queried separately.

set_instance_from_request will set the instance from the root endpoint by default (i.e. hello) if not explicitly specified by instance attribute or query parameter.

There are three fields you need to pass when you want to record metrics (via write* methods)

• key is the metric name and its label according to Prometheus data model and metric naming convention.
• value is the value to record
• command is optional and is ‘overwrite’ by default. Using increment will add value to the current so you don’t need to read and write on your own.

Surround the method with try…catch… so this codes will (almost) never break your application.

Then you call this method at the end of each REST handler. Don’t forget to start the timer at the beginning.

For example:

METHOD if_rest_resource~get.
  me->runtime->get_runtime( ).
  .
  .
  . 
  record_metric( i_method = mo_request->get_method( ) i_response = cl_rest_status_code=>gc_success_ok ).
ENDMETHOD.

Creating the metrics endpoint

Next, you need an endpoint for Prometheus server to call and grab the metrics data.

In REST handler class (ZCL_REST_HANDLER), insert the following routing string in the method if_rest_application~get_root_handler.

lo_router->attach( iv_template = '/hello/metrics' iv_handler_class = zcl_prometheus_rest_resource=>c_class_name ).

Testing your endpoint

Open your browser and open your /hello/metrics endpoint. You should see a blank page.

Try using your application so the metrics are recorded.

In case you’re using Postman to test the APIs you can import my Postman files from here.(Please note Postman is a commercial product but there’s a free version available)

After one GET and one POST request, refresh the metrics page and you should start seeing the data.

Now, your application is ready to be monitored

Setting up Prometheus Server

Download Prometheus from this page. Extract the package and you can run the executable without installing.

Note: If you want to install as a window service, you can use NSSM.

Open browser and go to http://localhost:9090. and you should see its web UI.

Adding a new job to monitor your application

Now, your Prometheus server does not yet recognize your application so you need to configure it first.

Edit the file prometheus.yml and add the following lines:

 - job_name: npl-test
   params:
     sap-client: ['001']
     sap-language: ['EN']
   metrics_path: /test/hello/metrics
     basic_auth:
       username: DEVELOPER
       password: yourpassword
   static_configs:
     - targets:
       - vhcalnplci.dummy.nodomain:8000

In your secured environment, you may want to use HTTPS like this:

  - job_name: npl-test
    
    scheme: https
    params:
      sap-client: ['001']
      sap-language: ['EN']
    metrics_path: '/test/hello/metrics'
    basic_auth:
      username: DEVELOPER
      password: yourpassword
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets: ['vhcalnplci.dummy.nodomain:44300']

Please note that you should add one job for each application server. Don’t use load balance URL as the metrics are bound for each application server. Go to transaction SM51 to see the list of all application servers on the system.

After saving, restart the service and access the web UI. Query for up metric and click Execute. If your job is setup properly, you should see the value 1.

You may try querying your application metric e.g. hello_count.

Setting up Grafana

Prometheus is good at collecting and querying time series data but to have a better a visualization you may need Grafana.

What is Grafana?

Grafana is a tool for data visualization & monitoring with support for Graphite, InfluxDB, Prometheus, Elasticsearch and many more databases. In short, Grafana will pull the data from Prometheus and visualize them on their dashboard web interface.

We setup a desktop PC with two monitoring screens and put it where the team can see easily.

Installation

Download from here. Extract and run it the same way you do for Prometheus.

Setting up data source

First, we need to setup Grafana to recognize Prometheus server.

Go to http://localhost:3000 and log in with default username and password (i.e. admin:admin). Select Data Source from the menu and click Add Data Source. Fill in the connection to your Prometheus server like below:

Setting up dashboard

We’re going to setup 2 graphs to monitor API usage count and response time.

Select Dashboards from the menu, click Home and click + New Dashboard.

Select Graph. Click the graph title and select Edit.

On tab General, name your graph title as you wish.

On tab Metrics, you will specify which metric data will be displayed on this graph. Fill in hello_count and put {{method}} ({{status}}) in Legend format.

On tab Axes, you can customize the graph axes.

Once done customizing, click Back to dashboard on the top-right.

Click + ADD ROW to add a new row and create a new graph.

Configure tab Metrics with hello_duration with the same Legend format.

Configure tab Axes like this:

Once done, you should see your dashboard like this.

Don’t forget to set the time range and refresh rate so your monitor screen keeps refreshing with the latest data.

Try to use your application and see how the graph reflect.

Originally published at blogs.sap.com on January 7, 2018.

Read More

Continuous Integration in ABAP using Jenkins