Since the containerization got more popular, kubernetes gained more traction than just using single VM. In previous post I explained why or when you don’t need kubernetes and when you’ll need it. At least from deployment perspective we can categorize it into 5 types (based on ownership, initial cost, and granularity of the recurring cost, the need for capacity planning):

1. On-Premise Dedicated Server, our own server, our own rack, or put it the colocation, we own the hardware, and have to replace it when it’s broken, we have to be also maintaining the network part. Usually this one is best choice for internal services (software that used only by internal staff, from the security and bandwidth especially.

2. VM, we rent the “cloud” infrastructure, this can be considered IaaS (Infrastructure as a Service), we rent a virtual machine/server or sometimes named Virtual Private/Dedicated Server, so we pay monthly when the server turned on (or based on contract). Some of notable product in this category: Google Compute Engine, Amazon EC2, Azure VM, Contabo VPS/VDS, etc. Usually this one is the best for databases (unless you are using managed database service) or other stateful applications or if the number of users are limited based on the capacity planning (not whole world will be accessing this).

3. Kubernetes, we rent or use managed kubernetes, or install kubernetes on top of our own own-premise dedicated server. Usually the company will rent 3 huge servers 64 core, 256GB RAM, with very large harddisk, and let developer to deploy containers/pod inside the kubernetes themself splitted based on their team or service’s namespace. This have constant cost (those 3 huge VMs, and the managed service’s cost), some provider also provider automatic node scale out (so the kubernetes nodes/VM (where the pods will be located) can be increased based on load). Some notable product in this category: GKE, Amazon EKS, AKS, DOKS, Jelastic Kubernetes Cluster, etc.

4. Container Engine, we use the infrastructure provider’s platform, so we only need to supply container without have to rent the VM manually, some provider will deploy the container inside a single VM, some other will deploy the container on shared dedicated server/VM. Some of them Some notable product in this category: Google AppEngine, Amazon ECS/Beanstalk/Fargate, Azure App Service, Jelastic Cloud, Heroku, etc. Usually this one is the best choice for most cases on budget-wise and for scalability side.

5. Serverless/FaaS, we only need to supply the function (mostly based on a specific template) that will run on specific event (eg. on specific time like CRON, or when load balancer receiving a request like in old CGI). Usually the function put inside a container, and used as standby instance, so the scale out only happened when it receives high load. If the function requires database as dependency, it’s recommended to use managed databases that support high number of connections/connect-disconnect or offloaded to MQ/PubSub service. Notable products in this category: Google CloudRun, AWS Lambda, Azure Functions, OpenFaaS, Netlify, Vercel, Cloudflare Workers, etc. We pay this service usually based on CPU duration, number of calls, total RAM usage, bandwidth, and any other metrics, so it would be very cheap when number of function calls are small, but can be really costly if you are writting inefficient function or have large number of calls. Usually lambda only used for handling spikes or as atomic CRON.

Because of the hype, or because it fit their use cases (bunch of teams that want to do independent service deployments), and the possibility of avoiding vendor locking, sometimes a company might decide to use kubernetes. Most of the company can survive by not following the hype by only managed database (or deploying database on VM or even using docker-compose with volume binding) + container engine (for scaling out strategy), not having to train everyone to learn Kubernetes.

But today we’re gonna try one of the fastest local kubernetes for development use-case (not for production).

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

minikube start

# use — driver=kvm2 or virtualbox if docker cannot connect internet
#sudo apt install virtualbox
#sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager
#sudo adduser `id -un` libvirt
#sudo adduser `id -un` kvm

alias kubectl=’minikube kubectl — ‘
alias k=kubectl

# will download kubectl if it’s the first time
k

# get pods from all namespace
k get po -A

# open dashboard and authenticate
minikube dashboard

# destroy minikube cluster
minikube ssh
sudo poweroff
minikube delete

create Dockerfile you want to deploy to kubernetes cluster, or if it’s just simple single binary golang project, build locally then put it to alpine docker, then push to image registry will be work just fine:

# build binary
CGO_ENABLED=0 GOOS=linux go build -o ./bla.exe

# create Dockerfile
echo ‘
FROM alpine:latest
WORKDIR /
COPY bla.exe .
CMD ./bla.exe
‘ > Dockerfile

# build docker image
VERSION=$(ruby -e ‘t = Time.now; print “v1.#{t.month+(t.year-2021)*12}%02d.#{t.hour}%02d” % [t.day, t.min]’)
COMMIT=$(git rev-parse — verify HEAD)
APPNAME=local-bla
docker image build -f ./Dockerfile . \
 — build-arg “app_name=$APPNAME” \
-t “$APPNAME:latest” \
-t “$APPNAME:$COMMIT” \
-t “$APPNAME:$VERSION”

# push image to minikube
minikube image load $APPNAME

# create deployment config
echo ‘
apiVersion: v1
kind: Pod
metadata:
name: bla-pod
spec:
containers:
 — name: bla
image: bla
imagePullPolicy: Never
env:
 — name: BLA_ENV
value: “ENV_VALUE_TO_INJECT”
# if you need access to docker-compose outside the kube cluster
# use minikube ssh, route -n, check the ip of the gateway
# and use that ip as connection string
# it should work, as long the port forwarded
restartPolicy: Never
‘ > bla-pod.yaml

# deploy
kubectl apply -f bla-pod.yaml

# check
k get pods
k logs bla-pod

# delete deployment
kubectl delete pod bla-pod

If you need NewRelic log forwarding, it’s just as easy as adding a helm chart (it would automatically attach new pod logs and send it to newrelic):

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add newrelic https://helm-charts.newrelic.com
helm search repo newrelic/
helm install newrelic-logging newrelic/newrelic-logging — set licenseKey=eu01xx2xxxxxxxxxxxxxxxRAL
kubectl get daemonset -o wide -w — namespace default newrelic-logging

The next step should be adding load balancer or ingress so that pod can receive http requests.

Originally published at http://kokizzu.blogspot.com.

Normally we use LXC/LXD, KVM, QEMU, Docker, Vagrant, VirtualBox, VMWare or any other virtualization and containerization software to spawn a VM-like instance locally. Today we’re gonna try multipass, a tool to spawn and orchestrate ubuntu VM. To install multipass, it’s as easy as running these commands:

snap install multipass
ls -al /var/snap/multipass/common/multipass_socket
snap info multipass

To spawn a VM on Ubuntu (for other OSes, see the link above), we can run:

multipass find

Image Aliases Version Description
snapcraft:core18 18.04 20201111 Snapcraft builder for Core 18
snapcraft:core20 20.04 20210921 Snapcraft builder for Core 20
snapcraft:core 16.04 20210929 Snapcraft builder for Core 16
core core16 20200818 Ubuntu Core 16
core18 20211124 Ubuntu Core 18
18.04 bionic 20220104 Ubuntu 18.04 LTS
20.04 focal,lts 20220118 Ubuntu 20.04 LTS
21.10 impish 20220118 Ubuntu 21.10
daily:22.04 devel,jammy 20220114 Ubuntu 22.04 LTS
appliance:adguard-home 20200812 Ubuntu AdGuard Home Appliance
appliance:mosquitto 20200812 Ubuntu Mosquitto Appliance
appliance:nextcloud 20200812 Ubuntu Nextcloud Appliance
appliance:openhab 20200812 Ubuntu openHAB Home Appliance
appliance:plexmediaserver 20200812 Ubuntu Plex Media Server Appliance
anbox-cloud-appliance latest Anbox Cloud Appliance
charm-dev latest A development and testing environment for charmers
minikube latest minikube is local Kubernetes

multipass launch — name groovy-lagomorph # 20.04

multipass list
Name State IPv4 Image
groovy-lagomorph Running 10.204.28.99 Ubuntu 20.04 LTS

multipass info — all
Name: groovy-lagomorph
State: Running
IPv4: 10.204.28.99
Release: Ubuntu 20.04.3 LTS
Image hash: e1264d4cca6c (Ubuntu 20.04 LTS)
Load: 0.00 0.00 0.00
Disk usage: 1.3G out of 4.7G
Memory usage: 134.2M out of 976.8M
Mounts: —

To run shell inside newly spawned VM, we can run:

multipass shell groovy-lagomorph

multipass exec groovy-lagomorph — bash

If you need to simulate ssh, according to this issue you can either:

sudo ssh -i /var/snap/multipass/common/data/multipassd/ssh-keys/id_rsa ubuntu@10.204.28.99

# or add ssh key before launch on cloud-init.yaml
ssh_authorized_keys:
 — <your_ssh_key>

# or copy ssh key manually after launch
sudo ssh-copy-id -f -o ‘IdentityFile=/var/snap/multipass/common/data/multipassd/ssh-keys/id_rsa’ -i ~/.ssh/id_rsa.pub ubuntu@10.204.28.99

If to stop/start/delete the VM:

multipass stop groovy-lagomorph
multipass start groovy-lagomorph
multipass delete groovy-lagomorph
multipass purge

What technology used by multipass? it’s QEMU, but maybe it’s different on another platform (it can run on Windows and MacOSX too).

Originally published at http://kokizzu.blogspot.com.

• CrateDB, a highly optimized for huge amount of data (they said), probably would be the best for updatable time series, also with built-in search engine, so this one is quite fit my use case probably to replace [Riot (small scale) or Manticore (large scale)] and [InfluxDB or TimescaleDB], does not support auto increment
• CockroachDB, self-healing database with PostgreSQL-compatible connector, the community edition does not support table partitioning
• MemSQL, which also can replace kv-store, there’s a limit of 128GB RAM for free version. Row-store tables can only have one PRIMARY key or one UNIQUE key or one AUTO increment column that must be a SHARD key, and it cannot be updated or altered. Column-store tables does not support UNIQUE/PRIMARY key, only SHARD KEY. The client/connector is MySQL-compatible
• MariaDB (MySQL), one of the most popular open source RDBMS, for the sake of comparison
• PostgreSQL, my favorite RDBMS, for the sake of comparison
• NuoDB on another benchmark even faster than GoogleSpanner or CockroachDB, the community edition only support 3 transaction engine (TE) and 1 storage manager (SM)
• YugaByteDB, distributed KV+SQL with Cassandra and PostgreSQL compatible protocol. Some of SQL syntax not yet supported (ALTER USER, UNIQUE on CREATE TABLE).
• ScyllaDB, a C++ version of Cassandra. All Cassandra-like databases has a lot of restrictions/annoyances by design compared to traditional RDBMS (cannot CREATE INDEX on composite PRIMARY KEY, no AUTO INCREMENT, doesn’t support UNION ALL or OR operator, must use COUNTER type if you want to UPDATE x=x+n, cannot mix COUNTER type with non-counter type on the same table, etc), does not support ORDER BY other than clustering key, does not support OFFSET on LIMIT.
• Clickhouse, claimed to be fastest and one of the most storage space efficient OLAP database, but doesn’t support UPDATE/DELETE-syntax (requires ALTER TABLE to UPDATE/DELETE), only support batch insert, does not support UNIQUE, AUTO INCREMENT. Since this is not designed to be an OLTP database, obviously this benchmark would be totally unfair for Clickhouse.

What’s the extra motivation of this post?

I almost never use distributed database, since all of my project have no more than 200 concurrent users/sec. I’ve encountered bottleneck before, and the culprit is multiple slow complex queries, that could be solved by queuing to another message queue, and process them one by one instead of bombing database’s process at the same time and hogging out the memory.

The benchmark scenario would be like this:
1. 50k inserts of single column string value, 200k inserts of 2 column unique value, 900k insert of unique
INSERT INTO users(id, uniq_str) — x50k
INSERT INTO items(fk_id, typ, amount) — x50k x4
INSERT INTO rels(fk_low, fk_high, bond) — x900k

2. while inserting at 5%+, there would be at least 100k random search queries of unique value/, and 300k random search queries, every search queries, there would be 3 random update of amount
SELECT * FROM users WHERE uniq_str = ? — x100k
SELECT * FROM items WHERE fk_id = ? AND typ IN (?) — x100k x3
UPDATE items SET amount = amount + xxx WHERE id = ? — x100k x3

3. while inserting at 5%+, there would be also at least 100k random search queries
SELECT * FROM items WHERE fk_id = ?
4. while inserting at 5%+, there also at least 200k query of relations and 50% chance to update the bond
SELECT * FROM rels WHERE fk_low = ? or fk_high = ? — x200k
UPDATE rels SET bond = bond + xxx WHERE id = ? — x200k / 2

This benchmark represent simplified real use case of the game I’m currently develop. Let’s start with PostgreSQL 10.7 (current one on Ubuntu 18.04.1 LTS), the configuration generated by pgtune website:

For slow databases, all values reduced by 20 except query-only.

[Pg] RandomSearchItems (100000, 100%) took 24.62s (246.21 µs/op)

innodb_buffer_pool_size=4G

CREATE USER ‘b1’@’localhost’ IDENTIFIED BY ‘b1’;

GRANT ALL PRIVILEGES ON b1.* TO ‘b1’@’localhost’;

sudo mysqltuner # not sure if this useful

[My] RandomSearchItems (100000, 100%) took 16.62s (166.20 µs/op)
[My] SearchRelsAddBonds (10000, 100%) took 86.32s (8631.74 µs/op)
[My] UpdateItemsAmounts (5000, 100%) took 172.35s (34470.72 µs/op)
[My] InsertUsersItems (2500, 100%) took 228.52s (91408.86 µs/op)
USERS CR : 2500 / 4994
ITEMS CRU : 17500 / 14982 + 696542 / 13485
RELS CRU : 2375 / 12871 / 6435
SLOW FACTOR : 20
CRU µs/rec : 10213.28 / 23.86 / 13097.44

Next we’ll try with MemSQL 6.7.16–55671ba478, while the insert and update performance is amazing, the query/read performance is 3–4x slower than traditional RDBMS:

$ memsql-admin start-node — all

$ go run memsql.go lib.go # 4 sec before start RU

$ go run memsql.go lib.go # SLOW FACTOR 5

[Crate] SearchRelsAddBonds (10000, 100%) took 49.11s (4911.38 µs/op)
[Crate] RandomSearchItems (100000, 100%) took 101.40s (1013.95 µs/op)
[Crate] UpdateItemsAmounts (5000, 100%) took 246.42s (49283.84 µs/op)
[Crate] InsertUsersItems (2500, 100%) took 306.12s (122449.00 µs/op)
USERS CR : 2500 / 4965
ITEMS CRU : 17500 / 14894 + 690161 / 14895
RELS CRU : 2375 / 4336 / 2168
SLOW FACTOR : 20
CRU µs/rec : 13681.45 / 146.92 / 19598.85

Next is CockroachDB 19.1, the result:

$ go run cockroach.go lib.go
[Cockroach] SearchRelsAddBonds (10000, 100%) took 59.25s (5925.42 µs/op)
[Cockroach] RandomSearchItems (100000, 100%) took 85.84s (858.45 µs/op)
[Cockroach] UpdateItemsAmounts (5000, 100%) took 261.43s (52285.65 µs/op
[Cockroach] InsertUsersItems (2500, 100%) took 424.66s (169864.55 µs/op)
USERS CR : 2500 / 4988
ITEMS CRU : 17500 / 14964 + 699331 / 14964
RELS CRU : 2375 / 5761 / 2880
SLOW FACTOR : 20
CRU µs/rec : 18979.28 / 122.75 / 19022.43

Next is NuoDB 3.4.1, the storage manager and transaction engine config and the benchmark result:

$ chown nuodb:nuodb /media/nuodb
$ nuodbmgr — broker localhost — password nuodb1pass
start process sm archive /media/nuodb host localhost database b1 initialize true
start process te host localhost database b1
 — dba-user b2 — dba-password b3
$ nuosql b1 — user b2 — password b3
$ go run nuodb.go lib.go
[Nuo] RandomSearchItems (100000, 100%) took 33.79s (337.90 µs/op)
[Nuo] SearchRelsAddBonds (10000, 100%) took 72.18s (7218.04 µs/op)
[Nuo] UpdateItemsAmounts (5000, 100%) took 117.22s (23443.65 µs/op)
[Nuo] InsertUsersItems (2500, 100%) took 144.51s (57804.21 µs/op)
USERS CR : 2500 / 4995
ITEMS CRU : 17500 / 14985 + 698313 / 14985
RELS CRU : 2375 / 15822 / 7911
SLOW FACTOR : 20
CRU µs/rec : 6458.57 / 48.39 / 8473.22

Next is YugaByte 1.2.5.0, the result:

export YB_PG_FALLBACK_SYSTEM_USER_NAME=user1
./bin/yb-ctl — data_dir=/media/yuga create
# edit yb-ctl set use_cassandra_authentication = True
./bin/yb-ctl — data_dir=/media/yuga start

./bin/cqlsh -u cassandra -p cassandra

When INSERT is not batched on Clickhouse 19.7.3.9:

$ go run clickhouse-1insertPreTransaction.go lib.go
[Click] InsertUsersItems (2500, 100%) took 110.78s (44312.56 µs/op)
[Click] RandomSearchItems (100000, 100%) took 306.10s (3060.95 µs/op)
[Click] SearchRelsAddBonds (10000, 100%) took 534.91s (53491.35 µs/op)
[Click] UpdateItemsAmounts (5000, 100%) took 710.39s (142078.55 µs/op)
USERS CR : 2500 / 4999
ITEMS CRU : 17500 / 14997 + 699615 / 15000
RELS CRU : 2375 / 18811 / 9405
SLOW FACTOR : 20
CRU µs/rec : 4951.12 / 437.52 / 52117.48

These benchmark performed using i7–4720HQ 32GB RAM with SSD disk. At first there’s a lot that I want to add to this benchmark (maybe someday) to make this huge ‘__’), such as:

• DGraph, a graph database written in Go, the backup is local (same as MemSQL, so you cannot do something like this ssh foo@bar “pg_dump | xz — -c” | pv -r -b > /tmp/backup_`date +%Y%m%d_%H%M%S`.sql.xz”)
• Cayley, a graph layer written in Go, can support many backend storage
• ArangoDB, multi-model database, with built-in Foxx Framework for creating REST APIs, has unfamiliar AQL syntax
• MongoDB, one of the most popular open source document database, for the sake of comparison, I’m not prefer this one because of the memory usage.
• InfluxDB or TimeScaleDB or SiriDB or GridDB for comparison with Clickhouse
• Redis or SSDB or LedisDB or Codis or Olric or SummitDB, obviously for the sake of comparison. Also Cete, distributed key-value but instead using memcache protocol this one uses gRPC and REST
• Tarantool, a redis competitor with ArrangoDB-like features but with Lua instead of JS, I want to see if this simpler to use but with near equal performance as Aerospike
• Aerospike, fastest distributed kv-store I ever test, just for the sake of comparison, the free version limited to 2 namespace with 4 billions object. Too bad this one cannot be started on OpenVZ-based VM.
• Couchbase, document oriented database that support SQL-like syntax (N1QL), the free for production one is few months behind the enterprise edition. Community edition cannot create index (always error 5000?).
• GridDB, in-memory database from Toshiba, benchmarked to be superior to Cassandra
• ClustrixDB (New name: MariaDB XPand), distributed columnstore version of MariaDB, community version does not support automatic failover and non-blocking backup
• Altibase, open source in-memory database promoted to be Oracle-compatible, not sure what’s the limitation of the open source version.
• RedisGraph, fastest in-memory graph database, community edition available.
• RethinkDB, document-oriented database, last ubuntu package cannot be installed, probably because the project no longer maintained
• OrientDB, multi model (document and graph database), their screenshot looks interesting, multi-model graph database, but too bad both Golang driver are unmaintained and probably unusable for latest version (3.x)
• TiDB, a work in progress approach of CockroachDB but with MySQL-compatible connector, as seen from benchmark above, there’s a lot of errors happening
• RQLite, a distributed SQLite, the go driver by default not threadsafe
• VoltDB, seems not free, since the website shows “free evaluation”
• HyperDex, have good benchmark on paper, but no longer maintained
• LMDB-memcachedb, faster version of memcachedb, a distributed kv, but no longer maintained
• FoundationDB, a multi-model database, built from kv-database with additional layers for other models, seems to have complicated APIs
• TigerGraph, fastest distributed graph database, developer edition free but may not be used for production

The chart (lower is better) shown below:

Other 2018’s benchmark here (tl;dr: CockroachDB mostly higher throughput, YugabyteDB lowest latency, TiDB lowest performance among those 3).

Originally published at http://kokizzu.blogspot.com.

# 664 MB in about 20 seconds

Or if you have time you can use aggresive GC, like this:

# 217 MB in about 5 minutes

mv ../temp/{shallow,objects} .git

# 150 MB in about 2 seconds

Next you can reclaim space from docker using this command:

For more disk usage analysis you can use baobab for linux or windirstat on windows.

Originally published at http://kokizzu.blogspot.com.

As we can see, Mum, Gencode, Colfer, Bebop, Gotiny, XDR2, MsgPack wins in terms of performace, in cost of serialization size. Let’s check the deserialization performace, the fastest are:

In this part, Bebop, XDR2, Gencode, Colfer, Mum, Gogoprotobuf, Gotiny, FlatBuffers, MsgPack wins this part. So if your bandwidth is unlimited, you can choose these format as your serialization format. You can access the reformatted spreadsheet here. Here’s combined result of serialization and deserialization and addition of serialization size and allocation needed to deserialize.

Here are the links for high performing libraries used:

• Bebop codegen from .bop (need to create schema like protobuf)
• Gencode codegen from .schema (similar to go syntax)
• XDR2 codegen version (other libs are using reflection/automatic), unmaintained
• Mum manual (must create the method to serialize and deserialize yourself), new name is enkodo
• Colfer codegen from .colf (similar to go syntax)
• Gogoprotobuf codegen from .proto
• Gotiny codegen, not recommended for production
• MsgPack codegen version (other libs are using reflection/automatic), from .go source file using go:generate, supported in lots of language
• FlatBuffers codegen, can be used in gRPC

What are the difference between codegen, automatic, manual?

• codegen means there are step to generate golang function by writing a schema definition file then run a program to convert that file to specific programming language implementation, sometimes using other format (so you cannot add custom tag to generated .go file), sometimes using golang struct with tag (like codegen version of msgpack above you need to label each property with , so you can add your own custom tag on the struct property, eg. `json:”bar,omitempty” form:”baz” bson:”blabla”` )
• manual means you must write the serialization and deserialization yourself for each property of the struct (the library only the helper), this allows highly flexible system, so for example you read 1 byte first, then if the value is 1 then you read a string, if 2 you read int32, and so on. This can also be useful to parse network packet if having the same endian.
• automatic means you don’t need to write any schema, it uses reflection so should be slower than codegen version.

Originally published at http://kokizzu.blogspot.com.

import “github.com/google/gops/agent”

If you don’t put those lines, you can still use gops limited to get list of programs running on your computer/server that made with Go with limited statistics information, using these commands:

$ go get -u -v github.com/google/gops

$ gops # show the list of running golang program

$ gops tree # show running process in tree

# PID can be replaced with GOPS host:port of that program

$ gops stack PID # get current stack trace of running PID

$ gops memstats PID # get memory statistics of running PID

$ gops gc PID # force garbage collection

$ gops pprof-cpu PID # get cpu profile graph

$ gops pprof-heap PID # get memory usage profile graph

profile saved at /tmp/heap_profile070676630

$ gops trace PID # get 5 sec execution trace

# you can install graphviz to visualize the cpu/memory profile

$ sudo apt install graphviz

# visualize the cpu/memory profile graph on the web browser

$ go tool pprof /tmp/heap_profile070676630

Next step is analyze the call graph for the memory leaks (which mostly just wrongly/forgot to defer body/sql rows or holding slice reference of huge buffer or certain framework’s cache trashing) or slow functions, whichever your mission are.

What if golang service you need to trace it inside Kubernetes pod that the GOPS address (host:port) not exposed to outside-world? Kubernetes is a popular solution for companies that manages bunch of servers/microservices or cloud like (GKE, AKS, Amazon EKS, ACK, DOKS, etc) but obviously overkill solution for small companies that doesn’t need to scale elastically (or the servers are less than 10 or not using microservice architecture).

First, you must compile gops statically so it can be run inside alpine container (which mostly what people use):

$ cd $GOPATH/go/src/github.com/google/gops

# copy gops to your kubernetes pod

$ kubectl cp ./gops $POD_NAME:/bin

$ kubectl exec -it $POD_NAME — sh

# for example you want to check heap profile for PID=1

# copy back trace file to local, then you can analyze the dump

kubectl cp $POD:/tmp/heap_profile070676630 out.dump

But if your address and port are exposed you can directly use gops from your computer to the pod or create a tunnel inside the pod if it doesn’t have public IP, for example using ngrok.

Btw if you know any companies migrating from/to certain language (especially Go), frameworks or database, you can contribute here.

Originally published at http://kokizzu.blogspot.com.

# run server using docker

docker run -it -p 4040:4040 pyroscope/pyroscope:latest server

And here’s the example on how to use the client library/agent (modifying Go’s source code, just like in DataDog or any other APM tools) and install the Pyroscope CLI to run Ruby/Python scripts:

# golang, add agent inside the source code

It would show something like this if you open the server URL (localhost:4040) in the browser, so you can check which part of the code that took most of the runtime.

Originally published at http://kokizzu.blogspot.com.

• The default kubernetes dashboard
• Octant

• IntelliJ Service Tool Window (View > Tool Window > Services, Alt+8)

• IntelliJ Cloud Code (most features focused on Google Cloud and Google Cloud projects)

• Kontenta-Lens (most beautiful one :3)

• Kui (like jupyter-notebook, pry/irb/julia, etc, have tutorial mode)

• KubeBox (terminal based GUI)

• k8dash (deployment only)
• HeadLamp (didn’t work)
• K9s (failed to go get, also require license?)

For now my recommendation would be Kontenta-Lens, you’ll get a bird-view of your cluster.

For shell autocomplete, I recommend kube-prompt (or other shell-specific autocomplete)

If you only need docker GUI, you can try Dry

If you prefer web-based GUI, you can try Portainer (it could manage much more than just kubernetes, but also local docker and docker swarm), it’s quite better than Rancher.

Originally published at http://kokizzu.blogspot.com.

Mock Approach

Given a code like this:

type Obj struct {
 *sql.DB // or Provider
}
func (o *Obj) DoMultipleQuery(in InputStruct) (out OutputStruct, err error) {
 ... = o.DoSomeQuery()
 ... = o.DoOtherQuery()
}

I’ve seen code to test with mock technique like this:

func TestObjDoMultipleQuery(t *testing.T) {
 o := Obj{mockProvider{}}
 testCases := []struct {
 name string
 mockFunc func(sqlmock.Sqlmock, *gomock.Controller) in InputStruct out OutputStruct
 shouldErr bool
 } {
 {
 name: `bast case`,
 mockFunc: func(db sqlmock.SqlMock, c *gomock.Controller) {
 db.ExpectExec(`UPDATE t1 SET bla = \?, foo = \?, yay = \? WHERE bar = \? LIMIT 1`).
 WillReturnResult(sqlmock.NewResult(1,1))
 db.ExpectQuery(`SELECT a, b, c, d, bar, bla, yay FROM t1 WHERE bar = \? AND state IN \(1,2\)`).
 WithArgs(3).
 WillReturnRows(sqlmock.NewRows([]string{"id", "channel_name", "display_name", "color", "description", "active", "updated_at"}).
 AddRow("2", "bla2", "Bla2", "#0000", "bla bla", "1", "2021-05-18T15:04:05Z").
 AddRow("3", "wkwk", "WkWk", "#0000", "wkwk", "1", "2021-05-18T15:04:05Z"))
 ...
 }, in: InputStruct{...}, out: OutputStruct{...},
 wantErr: false,
 },
 {
 ... other cases
 },
 } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T){ ... // prepare mock object o := Obj{mockProvider} out := o.DoMultipleQueryBusinessLogic(tc.in) assert.Equal(t, out, tc.out) }) }
}

This approach has pros and cons:

+ could check whether has typos (eg. add one character in the original query, this test would detect the error)

+ could check whether some queries are properly called, or not called but expected to be called

+ unit test would always faster than integration test

- testing implementation detail (easily break when the logic changed)

- cannot check whether the SQL statements are correct

- possible coupled implementation between data provider and business logic

- duplicating work between original query and the regex-version of query, which if add a column, we must change both implementation

For the last cons, we can change it to something like this:

This approach has pros and cons:

+ not deduplicating works (since it just a simplified regex of the full SQL statements

+ still can check whether queries properly called or not

+ unit test would always faster than integration test

- testing implementation detail (easily break when the logic changed)

- cannot detect typos/whether the query no longer match (eg. if we accidentally add one character on the original query that can cause sql error)

- cannot check correctness of the SQL statement

- possible coupled implementation between data provider and business logic

We could also create a helper function to replace the original query to regex version:

func SqlToRegexSql(sql string) string {
 return // replace special characters in regex (, ), ?, . with escaped version
}
db.ExpectQuery(SqlToRegexSql(ORIGINAL_QUERY)) ...

This approach has same pros and cons as previous approach.

Fake Approach

Fake testing use classical approach, instead of checking implementation detail (expected calls to dependency), we use a compatible implementation as dependency (eg. a slice/map of struct for database table/DataProvider)

Given a code like this:

It’s better to make a fake data provider like this:

So in the test, we can do something like this:

This approach have pros and cons:

+ testing behavior (this input should give this output) instead of implementation detail (not easily break/no need to modify the test when algorithm/logic changed)

+ unit test would always faster than integration test

- cannot check whether the queries are called or not called but expected to be called

- double work in Golang (since there’s no generic/template yet, go 1.18 must wait Feb 2022), must create minimal fake implementation (map/slice) that simulate basic database table logic, or if data provider not separated between tables (repository/entity pattern) must create a join logic too — better approach in this case is to always create Insert, Update, Delete, GetOne, GetBatch instead of joining.

+ should be no coupling between queries and business logic

Cannot check whether queries in data provider are correct (which should not be the problem of this unit, it should be DataProvider integration/unit test’s problem, not this unit)

Classical Approach for DataProvider

It’s better to test the queries using classical (black box) approach integration test instead of mock (white box), since mock and fake testing can only test the correctness of business logic, not logic of the data provider that mostly depend to a 2nd party (database). Fake testing also considered a classical approach, since it test input/output not implementation detail.

Using dockertest when test on local and gitlab-ci service when test on pipeline, can be something like this:

Where the prepareDb function can be something like this (taken from dockertest example):

In the pipeline the file can be something like this for PostgreSQL (use tmpfs/inmem version for database data directory to make it faster):

The dockerfile for tmpfs database if using MySQL can be something like this:

Or for MongoDB:

The benefit of this classical integration test approach:

+ high confidence that your SQL statements are correct, can detect typos (wrong column, wrong table, etc)

+ isolated test, not testing business logic but only data provider layer, also can test for schema migrations

- not a good approach for database with eventual consistency (eg. Clickhouse)

- since this is an integration test, it would be slower than mock/fake unit test (1–3s+ total delay overhead when spawning docker)

Conclusion

1. use mock for databases with eventual consistency
2. prefer fake over mock for business logic correctness because it’s better for maintainability to test behavior (this input should give this output), instead of implementation details
3. prefer classical testing over mock testing for checking data provider logic correctness

References

(aka confirmation bias :3)

https://martinfowler.com/articles/mocksArentStubs.html
https://stackoverflow.com/questions/1595166/why-is-it-so-bad-to-mock-classes
https://medium.com/javascript-scene/mocking-is-a-code-smell-944a70c90a6a
https://chemaclass.medium.com/to-mock-or-not-to-mock-af995072b22e
https://accu.org/journals/overload/23/127/balaam_2108/
https://news.ycombinator.com/item?id=7809402
https://philippe.bourgau.net/careless-mocking-considered-harmful/
https://debugged.it/blog/mockito-is-bad-for-your-code/
https://engineering.talkdesk.com/double-trouble-why-we-decided-against-mocking-498c915bbe1c
https://blog.thecodewhisperer.com/permalink/you-dont-hate-mocks-you-hate-side-effects
https://agilewarrior.wordpress.com/2015/04/18/classical-vs-mockist-testing/
https://www.slideshare.net/davidvoelkel/mockist-vs-classicists-tdd-57218553
https://www.thoughtworks.com/insights/blog/mockists-are-dead-long-live-classicists
https://stackoverflow.com/questions/184666/should-i-practice-mockist-or-classical-tdd
https://bencane.com/2020/06/15/dont-mock-a-db-use-docker-compose/
https://swizec.com/blog/what-i-learned-from-software-engineering-at-google/#stubs-and-mocks-make-bad-tests
https://www.freecodecamp.org/news/end-to-end-api-testing-with-docker/
https://medium.com/@june.pravin/mocking-is-not-practical-use-fakes-e30cc6eaaf4e
https://www.c-sharpcorner.com/article/stub-vs-fake-vs-spy-vs-mock/

Originally published at http://kokizzu.blogspot.com.

• using dynamically-typed language (JS, Python, PHP, Ruby, etc) just because it’s the most popular language — only for short/discardable project
• mocking — there’s better way
• microservice without properly splitting domain — modular monolith is better for small teams, introducing network layer just to split a problem without properly assessing surely will be a hassle in a short and long run
• overengineering — eg. adding stack that you don’t need when current stack suffice, for example, dockerizing or kubernetesizing just because everyone using it, adding ElasticSearch just because it’s search use case, but the records needs to be searched are very little and rps are very low, a more lightweight aproach more make sense: eg. TypeSense or MeiliSearch or even database’s built-in FTS are more make sense for lower rps target/simpler search feature.
• premature “clean architecture” — aka. over-layering everything that you’ll are almost never replace — dependency tracking is better
• unevaluated standard — sticking with standard just because it’s a standard, just like being brainwashed/peer-pressured by dead people’s will (tradition) without rethinking is it still make sense to be followed for this use case?
• not making SRS/Software Requirement Specification (roles/who can do what action/API) and SDS/Software Design Specification (this action/API will mutate/command or read/query which datastore or hit which 3rd party) — this helps new guy to be onboarded to the project really fast

I have one more unpopular opinion, interface (-overuse) in Golang is almost always bad for jumping around (jump to declaration-implementation) inside source code which causes everyday overhead when reading code and debugging. For example when you want to create a fake/mock/stub of certain method:

type Bla interface {
Get(string) string
Set(string)
}

struct RealBla struct {} // wraps a 3rd party/client library
func (*RealBla) Get(string) string { return `` }
func (*RealBla) Set(string) { }

struct FakeBla struct {} // our fake/stub/mock implementation
func (*FakeBla) Get(string) string { return `` }
func (*FakeBla
var b Bla = FakeBla{…}
// usually as data member of other method that depends on RealBla
b.Set(…)
x := b.Get(…)

the problem with this approach is, it’s harder to jump around between declaration and implementation (usually RealBla that we want, not FakeBla), how often we switch implementation anyway? YAGNI (vs overengineering). It’s better for our cognitive/understanding that we keep both coupled, this violates single responsibility principle from SOLID, but it’s easier to reason/understand, since the real and fake are in the same file and near each other, so we can catch bug easily without have to switch, something like this:

// declare/use 3rd party client here

UseFake bool
// create fake/in-mem here

by doing this, we could compare easily between our fake and real implementation (you could easily spot the bug, whether your fake implementation differ way too much from real implementation), and we can still jump around simply by ctrl+click the IDE on that function since there’s only 1 implementation. The only pros I could see from doing interface-based is when you are creating a 3rd party library (eg. io.Writer, io.Reader, etc) and you have more than 2 implementation ( DRY only good when its more than 2), but since you’re only making this for internal project that could be easily refactored within the project itself, it doesn’t make sense to abuse interface. See more tips from this video: Go Worst Practice.

After all being said, I won’t use this kind of thing ( UseFake property) for testing databases (2nd party), because I prefer to do integration (contract-based) testing instead of unit testing, since i’m using a fast database anyway (not a slow but popular RDBMSes).

Originally published at http://kokizzu.blogspot.com.