3 years of Go in Production

For the last 3 years, my microservices in production were divided into the following platforms:

• Core: Ruby and JRuby with Rails and Sinatra
• Satellites, scale: Node.js, Clojure and later: Go

A “core” toolset would live long. It would also move fast. It would depict the domain of the business and the core product solution that provides raw value.

Mostly, its performance profile doesn’t really introduce any infrastructural concerns. The satellites and “scale” toolset exhibits use cases where we bumped into scalability issues and had to tear apart a piece of the core, and rebuild it on top of a more performant stack.

It also represent a pure infrastructural concern; such as a push infrastructure, or analytics services. These things don’t change as fast as the problem domain, and they do need to be robust, fast and dependable.

I want to talk about that “scale” toolset and share a bit of my own experience. Let’s start at the end. While migrating from Node to Go, the following are the things I have noticed to be different.

Crashes

Node liked to crash when handling unexpected performance, mostly due to large use of connections and the overhead in managing them and keeping their resources in check. True, this is mostly solved by proper capacity planning, and usage of hardening patterns like circuit breaker, and throttling.

However, using these, my Node services looked like forced concoctions that crashed hard and had a horrible GC profile of minutes per collection (this is early Node days I’m talking about where you had to patch V8 to support a larger heap). It kept reminding me V8 was originally built to be run on the desktop.

Even in its early days, Go didn’t have any of that, and it was enough. And when it did crash, it recovered crazy fast, and I liked that property, and made use of it; failing processes crashed fast.

Performance

Around 2011–12, Node was at the apex of performance, and Go wasn’t. That is, until Go 1.1 mixed up that equation. I first noticed it through the excellent TechEm benchmarks:

• Round 1, March 2013 (pre Go 1.1): Go at 13k req/s, Node at 10k req/s. No big deal.
• Round 10 (latest): Go at 380k req/s Node at 225k req/s. Around 175% increase in favor of Go, and when you compare that to Node with Express, you get 145k req/s for Node, which is 260% increase in favor of Go.

Although these are a merely but a specialized variant of a micro benchmark, think about the overhead of the web framework (Express) superimposed on the host platform (Node). When Go is straddled with a typical web framework (Gin), it doesn’t react that hysterically and the reduction in performance is in the 1–3% range, Node however had a dramatic reaction.

It stands to hint which stack you’d want to pack infrastructure on. Think about it. Is this why a lot of full-on infrastructure projects ( Docker, Kubernetes, Vagrant etc.) were built on Go (hint: yes)?

Deployment

To deploy Node or Ruby, you have to deal with dependencies. Tools such as Bundler, rubygems, and npm were created to overcome dependency hell, and provided us with an ever helpful layer of abstraction, which split our products into two:

• Product (essence)
• Product (dependencies)

Essentially we could snapshot our deps, and ship our product. But notice, with Bundler and npm, we snapshot a description of our deps (unless we choose to vendor. IMHO, with npm — people usually don’t).

Every deploy might have modified the dependency tree of the product, and servers hosting these products had to reflect that. For those wanting to solve this problem, they had to ask these questions:

• Is that the responsibility of the Configuration Management infra?
• Should the deployment process or framework take care of dependencies?
• Should you bundle dependencies along with your product?
• What happens when your dependencies die? (i.e. pulled off from Rubygems)

And their answers would typically be:

• Configuration management should take care of the servers. Resources are not products.
• Yes. The deployment process should take care of deps.
• No, bundling dependencies is an anti-pattern. At worst case make our own local cache or proxy.
• When dependencies die, we can use a local cache. Or: dependencies never die.

Docker

Docker seals these questions shut — everything is snapshotted into an image, and you deploy that. This provides a layer of abstraction on top of the dependencies idea — Snapshot all the things.

But still, for what’s written here, we’re talking pre-production-docker era here (which is, only a year and a half ago).

Go

Even without Docker, Go packs a binary which is self-contained. And the answers to the above questions, are:

• Go builds its dependencies into the binary, making a self-inclusive deliverable
• Deployment framework caring about dependencies doesn’t matter anymore
• Bundling dependencies doesn’t matter anymore
• Dying dependencies doesn’t matter, dependencies live within your source tree

And even with Docker, we have no drama. A 5mb image plus your binary size, makes pulling an new image and starting up (and failing, when needed) crazy fast.

The Surprise Factor

Go makes portable Code. Java made that possible too. However, Go makes for a different kind of portability. It doesn’t compile to every platform under the sun (yet), but it does build for x86 and ARM.

Building for ARM means building for mobile, and Raspberry Pis.

My tipping point for using Go was when I looked into Python and C, for building my several ideas for home projects. I had to look at Python because it looks like that’s what the entire RPi community used, and I had to look at C because I found that a typical Python app took 27Mb of RAM blank.

Obviously for the first Raspberry Pi model I had, I didn’t have a lot of memory. So, I decided to try Go, and I cleared up a day to do that, because I guessed cross compilation and ARM were going to be a nightmare and I really wanted to use Go (better yet, I didn’t want to use C as bad).

The first 5 minutes passed and I cross-compiled and built a hello world Go binary, SCP’d it to the Pi, and it printed a ‘Hello world’ and exited. This was Go 1.0 or something of the sort.

Not making peace with how smooth everything went — I spent the following 10 minutes making sure, and double-making-sure, that I copied the correct binary, and it truly was my own Go program that was running.

I had a day to spare because everything worked perfectly, so I started working on what eventually became Ground Control.

Go is About Forgiveness

Let me tell you a story about forgiveness, and Go code.

Go is verbose. It lacks generics, it adopts code generation as an escape hatch for many things the core language lacks.

To everyone with experience — code generation is a bad smell, and this is a problem; and it should be.

However, coming from Node.js code bases, with the dreaded callback hell and a particularly low quality factor for community packages (early Node days) a Go code base looks like heaven. So we forgive.

Just when you are starting to get used to punching out very verbose Go code, you start noticing these verbosity issues you have overlooked; they bug you on a daily basis, and they are everywhere. But then, this kind of codebase would usually indicate you’re a bit more serious with Go. My guess is that you’d be at the same stage where you want to start doing concurrency work.

You discover Go’s channels, its concurrency model, and its nonblocking I/O.

Once again, you learn to appreciate it and become forgiving. By this time, the systems you’re developing are complex (in a good way), and so they want to be generic.

You want to start building infrastructure for yourself.

Generics, and lack of language abstractions start to hit you, hard. And once again, at the same time, you notice that your production environment is quiet. So quiet that it allows you to even ponder these things.

You’re noticing everything you build is very robust, performant, without a special effort on your part. Additionally, you remember that the last commits and fixes you made were relatively easy because everything was super spelled out.

You accept Go

At this point, you accept Go.

Code generation, its CSP concurrency model, its simplicity, its hyper-focused single-purpose tools such as go-vet, go-fmt, and the likes and make peace with the fact that by using Go, you’re building and getting accustomed to a colorful, vivid, tool set.

You become forgiving, because strangely, you doubted Go at every crossroad, and it didn’t let you down.