Nested structs for HTTP Calls

Some API's return complex structures. Here's for an example from Stocktwits API.
Instead of creating a specific type for every element in the structre, we use use a anonymous structure and define only the fields we're intrested in. In our example we'd like to see the symbols most mentioned for a given symbol (in our case AAPL). And the output:

$ go run stocktwits.go
SPY    10
TSLA    7
AMZN    5
BTC.X   5
TVIX    2

Go, protobuf & JSON

Sometimes you'd like more than one way to serve an API. In my case I'm currently working on serving both gRPC and HTTP. I'd like to have one place where objects are defined and have a nice way to serialize both from protobuf (which is the serialization gRPC uses) and JSON .

When producing Go code, protobuf adds JSON struct tags. However since JSON comes from dynamic languages, fields can have any type. In Go we can use map[string]interface{} but in protobuf this is a bit more complicated and we need to use oneof. The struct generated by oneof does not look like regular JSON and will make users of the API write complicated JSON structures.

What's nice about Go, is that we can have any type implement json.Marshaler and json.Unmarshaler. What's extra nice is that in Go, you can add these methods to the generated structs in another file (in Python, we'd have to change the generated source code since methods need to be inside the class definition).

Let's have a look at a simple Job definition


And now we can add some helper methods to aid with JSON serialization (protoc generates code to pb directory)


As a bonus, we added job.Properties that returns a "native" map[string]interface{}

Let's look at a simple example on how we can use it

And its output:
$ go run job.go
[j1]  user:"Saitama" count:1 properties: > properties: > 
[json]  {"user":"Saitama","count":1,"properties":{"retries":3,"target":"Metal Knight"}}

[j2]  user:"Saitama" count:1 properties: > properties: > 

Python's iter & functools.partial

Python's built-in iter function is mostly used to extract an iterator from an iterable (if you're confused by this, see Raymond's excellent answer on StackOverflow).

iter has a second form where it takes a function with not arguments and a sentinel value. This doesn't see that useful but check out the readline example in the documentation. iter will call the function repeatably until it the function return the sentinel value and then will stop.

What happens when the function  producing the values require some arguments? I this case we can use functools.partial to create a zero arguments function.

Here's an example of a simple HTTP client using raw sockets, we can directly use join on the iterator.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.