Nested structs for HTTP Calls

Some API's return complex structures. Here's for an example from Stocktwits API.
Instead of creating a specific type for every element in the structre, we use use a anonymous structure and define only the fields we're intrested in. In our example we'd like to see the symbols most mentioned for a given symbol (in our case AAPL). And the output:

$ go run stocktwits.go
SPY    10
TSLA    7
AMZN    5
BTC.X   5
TVIX    2

CPU Affinity in Go

Go's concurrency unit is a goroutine, the Go runtime multiplexes goroutines to operating system (OS) threads. At an upper level, the OS maps threads to CPUs (or cores). To see this goroutine/thread migration, you'll need to use some C code (note that this is Linux specific).


If you run this code and sort the output, you'll see the workers moving between cores:
$ go run affinity.go | sort -u
worker: 0, CPU: 0
worker: 0, CPU: 1
worker: 0, CPU: 2
worker: 0, CPU: 3
worker: 1, CPU: 0
worker: 1, CPU: 1
worker: 1, CPU: 2
worker: 1, CPU: 3
worker: 2, CPU: 0
worker: 2, CPU: 1
worker: 2, CPU: 2
worker: 2, CPU: 3
worker: 3, CPU: 0
worker: 3, CPU: 1
worker: 3, CPU: 2
worker: 3, CPU: 3

There is a cost for moving a thread from one core to another (see more here). In some cases this cost might be unacceptable and you'd like to pin a goroutine to a specific CPU.

Go has runtime.LockOSThread which pins the current goroutine to the current thread it's running on. For the rest of the way - pinning the thread to a CPU, you'll need to use C again.

Now if you run our code with the LOCK environment variable set, you can see each worker is pinned to a specific core.
$ LOCK=1 go run affinity.go | sort -u
worker: 0, CPU: 0
worker: 1, CPU: 1
worker: 2, CPU: 2
worker: 3, CPU: 3

direnv

I use the command line a lot. Some projects require different settings, say Python virtual environment, GOPATH for installing go packages and more.

I'm using direnv to help with settings per project in the terminal. For every project I have a .envrc file which specifies required settings, this file is automatically loaded once I change directory to the project directory or any of it's sub directories.

You'll need the following in your .zshrc

if whence direnv > /dev/null; then
    eval "$(direnv hook zsh)"
fi

Every time you create or change your .envrc, you'll need to run direnv allow to validate it and make sure it's loaded. (If you did some changes and want to check them, run "cd .")

Here are some .envrc examples for various scenarios:

Python + pipenv

source $(pipenv --venv)/bin/activate

Go

GOPATH=$(pwd | sed s"#/src/.*##")

PATH=${GOPATH}/bin:${PATH}

This assumes your project's path that looks like /path/to/project/src/github.com/project

If you're using the new go modules (in 1.11+), you probably don't need this.

Python + virtualenv

source venv/bin/activate

Python + conda

source activate env-name

Replace env-name with the name of your conda environment.

Go, protobuf & JSON

Sometimes you'd like more than one way to serve an API. In my case I'm currently working on serving both gRPC and HTTP. I'd like to have one place where objects are defined and have a nice way to serialize both from protobuf (which is the serialization gRPC uses) and JSON .

When producing Go code, protobuf adds JSON struct tags. However since JSON comes from dynamic languages, fields can have any type. In Go we can use map[string]interface{} but in protobuf this is a bit more complicated and we need to use oneof. The struct generated by oneof does not look like regular JSON and will make users of the API write complicated JSON structures.

What's nice about Go, is that we can have any type implement json.Marshaler and json.Unmarshaler. What's extra nice is that in Go, you can add these methods to the generated structs in another file (in Python, we'd have to change the generated source code since methods need to be inside the class definition).

Let's have a look at a simple Job definition


And now we can add some helper methods to aid with JSON serialization (protoc generates code to pb directory)


As a bonus, we added job.Properties that returns a "native" map[string]interface{}

Let's look at a simple example on how we can use it

And its output:
$ go run job.go
[j1]  user:"Saitama" count:1 properties: > properties: > 
[json]  {"user":"Saitama","count":1,"properties":{"retries":3,"target":"Metal Knight"}}

[j2]  user:"Saitama" count:1 properties: > properties: > 

Checking for Zero Values in Go

In Go, every type has a zero value. Which is the value a variable of this type get if it's not initialized. I had a configuration object of type map[string]interface{} and I needed to check if value exists and is not a zero value.

Here's a small piece of code that checks for zero values:

Go's append vs copy

When we'd like to concatenates slices in Go, most people usually reach out for append. Most of the time this solution is OK, however if you need to squeeze more performance - copy will be better.

EDIT: As Henrik Johansson suggested, if you pre-allocate the slice to append it'll fast as well.

Simple Object Pools

Sometimes we need object pools to limit the number of resource consumed. The most common example is database connnections.

In Go we sometime use a buffered channel as a simple object pool.

In Python, we can dome something similar with a Queue. Python's context manager makes the resource handing automatic so clients don't need to remember to return the object.


Here's the output of both programs:

$ go run pool.go
worker 7 got resource 0
worker 0 got resource 2
worker 3 got resource 1
worker 8 got resource 2
worker 1 got resource 0
worker 9 got resource 1
worker 5 got resource 1
worker 4 got resource 0
worker 2 got resource 2
worker 6 got resource 1

$ python pool.py
worker 5 got resource 1
worker 8 got resource 2
worker 1 got resource 3
worker 4 got resource 1
worker 0 got resource 2
worker 7 got resource 3
worker 6 got resource 1
worker 3 got resource 2
worker 9 got resource 3
worker 2 got resource 1

"Manual" Breakpoints in Go

When debugging, sometimes you need to set conditional breakpoints. This option is available both in gdb and delve. However sometimes when the condition is complicated, it's hard or even impossible to set it. A way around is to temporary write the condition in Go and set breakpoint "manually".

I Python we do it with pdb.set_trace(), in Go we'll need to work a little harder. The main idea is that breakpoints are special signal called SIGTRAP.

Here's the code to do this:
You'll need tell the go tool not to optimize and keep variable information:

$ go build -gcflags "-N -l" manual-bp

Then run a gdb session

$ gdb manual-bp 

(gdb) run 

 When you hit the breakpoint, you'll be in assembly code. Exit two functions to get to your code

(gdb) fin
(gdb) fin

Then you'll be in your code and can run gdb commands

(gdb) p i
$1 = 3

This scheme also works with delve

$ dlv debug manual-bp.go 
(dlv) c 

Sadly delve don't have "fin" command so you'll need to hit "n" (next) until you reach your code. 

That's it, happy debugging.

Oh - and in the very old days we did about the same trick in C code. There we manually inserted asm("int $3)" to the code. You can do with with cgo but sending a signal seems easier.

draft2gist - Publish draftin.com Documents to gist

I've started playing with draftin.com, so far very nice.

draftin.com lets you publish documents to several sites, and if the site you want to publish to is not on the list - there are WebHooks.

I've written a small AppEngine service that is a WebHook for publishing draftin.com documents to gist. Feel free to use it and let me know if you find any errors.

Below is the server code, rest of the files are here.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.