Stories by Robert Barnes on Medium

Office Setup of a Developer Advocate

Robert Barnes — Wed, 25 Nov 2020 10:02:25 GMT

I recently did a tour of my office setup on my YouTube channel, where I walked through all of my tech and desk gadgets that I use in my day to day as a Developer Advocate for HashiCorp. This blog will detail all of the components mentioned in the video with links.

https://youtu.be/Vve0KiSsjwA

Furniture

1400x600 Grey Oak Cantilever desk – https://www.officesupermarket.co.uk/collections/single-desks/products/start-600mm-deep-cantilever-desk?variant=31255928209459

Secret Labs Omega 2020 Edition gaming chair – https://secretlab.co.uk/collections/omega-series#omega_2020-ash

Screen

Samsung CHG90 49” Curved Ultra Wide monitor – https://www.amazon.co.uk/dp/B073RJQXB1/ref=cm_sw_r_cp_tai_KcWNFb9YH78MV

Screen mount – https://www.amazon.co.uk/gp/product/B084TLBBXW/ref=ppx_yo_dt_b_asin_image_o06_s00?ie=UTF8&psc=1

Compute

Apple MacBook Pro 16", Intel 9th Gen 8 core i9 processor, 16GB RAM, 512GB SSD – https://www.apple.com/uk/shop/buy-mac/macbook-pro/16-inch

iPad Pro 11" 2019 model — https://www.johnlewis.com/2018-apple-ipad-pro-11-inch-a12x-bionic-ios-wi-fi-cellular-64gb/p3834498

Apple Magic Trackpad 2 – https://store.apple.com/uk/xc/product/MRMF2Z/A

Belkin Thunderbolt 3 dock pro – https://www.amazon.co.uk/gp/product/B07Y3W21NY/ref=ppx_yo_dt_b_search_asin_image?ie=UTF8&psc=1

USB 3 Desk hub – https://www.amazon.co.uk/gp/product/B07H4MYS4C/ref=ppx_yo_dt_b_asin_title_o09_s01?ie=UTF8&psc=1

Desk grommet wireless charger – https://www.amazon.co.uk/gp/product/B07TS5QL72/ref=ppx_yo_dt_b_asin_title_o09_s02?ie=UTF8&psc=1

Mouse Mat – https://www.amazon.co.uk/gp/product/B081N7Q12S/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Headphones stand – https://www.amazon.co.uk/gp/product/B07ZYVVZKD/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

iPad stand — https://www.amazon.co.uk/gp/product/B07XT9X2N4/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Monitor light — https://www.amazon.co.uk/gp/product/B08F53WJVN/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Other

Google Nest Hub — https://store.google.com/gb/product/google_nest_hub?gclid=Cj0KCQjwufn8BRCwARIsAKzP697MyaSzTwceo7Rm6yKXHvu6U9Ek6k6Bsmqps0sjWtlgEkqwrNFpUAEaAi1JEALw_wcB&gclsrc=aw.ds&srp=/gb/product/google_home_hub

Architecting Vault — Part 4

Robert Barnes — Tue, 06 Aug 2019 21:46:36 GMT

Architecting Vault — Part 4

In this blog edition, we’ll look at what Vault’s seal status means, different ways to unseal vault servers and when we can expect to find vault in a sealed state.

Sealing is Vault protective mechanism that prevents you secrets from being accessible if the Vault server is ever compromised in any way. When you first deploy a Vault server, it is in a sealed state, which means, there aren’t many actions you can perform on the server. In fact, the only actions you can perform on Vault whilst in a sealed state are:

Check the Vault status
Initialise and unseal the Vault.

When a Vault server is first deployed, the first thing you will have to do to it is initialise it. The initialisation process generates a couple of things which are crucial to the management of the server.

Firstly, it will generate five unseal keys. As the name suggests, these keys can be used to unseal the Vault server. By default, you need any three of the five unseal keys to unseal the server.

The second thing that the initialisation process does for us is generate a root token for us to use to log in to the server and perform initial configuration tasks. The root token is similar to the root user on Linux systems in that, it has unrestricted access to everything. This token should be handled with care and should eventually be deleted once the server has been properly configured. There are mechanisms built into to Vault that allow operators to re-generate new root tokens if they have the correct permissions.

Going back to unsealing the Vault server, the unseal keys generated by the init process are equally as sensitive as the root token and too, need to be handled with care. The keys should never be stored together as that defeats the object. Instead, they should be carefully distributed out to five trusted operators to store securely.

In the event of a server restart, the Vault server will need to be unsealed, which requires three out of the 5 operators to enter their unseal keys in order to restore Vault services to normal. This is fine when the restart happens during the working day; however, it’s not so cool if the server restart occurs at 3am on a Sunday morning. This scenario would mean having to wake up engineers in the middle of the night for them to enter their unseal keys.

When Architecting your Vault deployment, there is a feature you can include in your design that solves this problem. This feature is Vault auto-unseal. This feature uses a trusted cloud Key Management System (KMS) to automatically unseal the server when this feature is configured. This is done by the KMS decrypting and encrypting the Vault server’s master key.

Vault supports auto-unsealing using KMS offerings from the four main cloud providers (Azure, AWS, GCP and Alicloud). In addition to these options, there is also an Enterprise only option of PKCS11 with HSM auto-unseal mechanism as well as the in-built Transit mechanism that you can take advantage of.

Which auto-unseal mechanism should I choose?

Much like many of the other architectural decisions that we have discussed over this series of blogs, the answer to this question will differ from one organisation to the next and will depend on one main factor, which is, what cloud strategy if any is in place?

Organisations with a cloud strategy will more than likely choose an auto-unseal mechanism with their main cloud provider; however, organisations who pay for a Vault Enterprise license may prefer to take advantage of the PKCS11 options, which after all is one of the features they are paying for. Organisations who are completely on-premise and using open source Vault are better suited to the Vault Transit mechanism. Whichever option you choose when enabling auto-unsealing, it should reflect the technology roadmap of the organisation.

Architecting Vault — Part 3

Robert Barnes — Mon, 29 Jul 2019 18:49:25 GMT

Architecting Vault — Part 3

So far in this blog series, we have covered hosting options for Vault deployments and also explored the different options available for Storage backends and some of the considerations needed when making your design decision.

This post will look at the auth methods that are available for you to enable in your Vault deployment and how to make the right decision as to which option to choose for your use case.

What is an Auth Method?

As a secrets management platform, Vault holds sensitive information to which access needs to be carefully managed and controlled. In order to understand how Vault can achieve this, there are two concepts which I will briefly discuss which are authentication and authorisation.

Authentication

Authentication is the process of assigning an identity to a user, much like what happens when you sign into most modern day applications. It will ask you for a username, which is you telling the application who you are. Next it will ask for a password which is a method of proving that you are who you say you are. Some applications may have an extra layer of security such as multi-factor authentication. Once you have signed in, the application will assign you your identity in the context of the application.

Authorisation

Authorisation is the process of managing what resources, files and data that each identity should and shouldn’t have access to. There is a dependancy on authentication to enable authorisation to work effectively. Authentication and authorisation are two halves that make up the same coin to deliver the coin’s value, access control.

With this context now set out, let’s go back and address the original question, what is an auth method? From Vault’s perspective, an auth method is a way of assigning an identity to a user accessing Vault (authentication).

Which auth method do I choose?

Vault provides several different ways to do authentication, some of which are more suitable to some organisations than others. Below is a list of some of the commonly used auth methods available to us to use in our Vault deployment design.

GitHub
Okta
LDAP
Username and password
AWS
Azure
GCP

Which auth method you choose will depend on what you use to manage identities for other platforms and applications. For example, I think it’s fair to say that majority of enterprise size organisations would use something like Active Directory to manage user accounts in the organisation. In this example, a lot of time and effort would have already been placed into the design of the directory structure, organisational units and groups. You can all but guarantee that the directory contains user accounts for the entire organisation and security groups have been set up according to the business structure and business logic. In this case, it make the most sense to use the LDAP auth method as most of the structural work is already done for us.

I want to point out that Vault allows you to enable multiple auth methods in your deployment so that gives you flexibility when choosing which auth methods to enable.

To demonstrate this, I’ll use a different example use case which is a small tech startup with 1–20 users. These organisations typically would not have an Active Directory in place as it’s more than likely not cost effective for the number of users they have. They do use GitHub to store and manage their source code, which means most of the organisation’s developers would have GitHub accounts which are linked to the organisation’s GitHub account. The organisation may also have a Marketing Director and a Finance & HR Director who both don’t have GitHub accounts for understandable reasons. In this case, it makes the most sense to enable both the GitHub auth method and the username & password auth method.

Application identities

For the most part, the methods listed above are geared towards assigning users an identity for vault, but what about assigning identities to applications? Vault offers some auth methods which are better suited this, as listed below:

AppRole
JWT / OIDC
TLS
Kubernetes

Depending on the nature of your application, these options should provide effective ways of it authenticating. If your application is made up of micro-services which are orchestrated using Kubernetes, then this will also be a sensible choice of auth method.

Approle is in my opinion, the most flexible of the above listed auth methods for applications as it allows for multiple differing workflows, which makes it suitable for almost all applications and services, no matter how different they are. You should bare in mind that this level of flexibility can easily inject added complexities into your design if you don’t design the workflows with the wholistic view in mind.

Summary

When choosing which auth methods to enable, you need to understand some of the following things in order to produce a suitable design:

Who will need to log in to Vault and why?
How is the organisation currently managing identity and user accounts?
What platforms are existing services and applications being hosted on?
How are existing applications currently handling authentication?
What are the future plans on the roadmap for existing applications and services as well as new ones?

The answers to these questions will guide you into choosing the right combination of auth methods. Auth methods can be enabled and disabled at any time so you can evolve your Vault platform as your roadmap moves along. Another thing to consider is if the organisation has or is moving towards a Single Sign On (SSO). If there is already SSO in place or plans to implement it, your choice of auth method should reflect this.

It’s also worth noting that you may need to re-architect parts of your existing applications to work with some of these auth methods so you will also need to ascertain how much technical debt it will incur and if this debt is acceptable.

Architecting Vault — Part 2

Robert Barnes — Sun, 21 Jul 2019 21:58:00 GMT

Architecting Vault — Part 2

In the previous blog post, i discussed hosting options for Hashicorp Vault and things to consider when making decisions about the platform to deploy your production Vault Cluster on to. This post will focus on the next architectural decision that you need to make when designing your vault cluster.

Which backend do i want to use for my Vault Cluster?

Firstly, let’s define what a backend is in the context of Vault and what capabilities a backend can enable for our cluster. Backend in the world of Vault is actually a storage backend which holds all of Vault’s information. Vault supports many different storage backends, 19 to be precise, each of which have their pro’s and cons. Let’s look at some of the main backend options in more detail.

Consul

Consul is another offering from Hashicorp which provides a Key/Value store amongst other services which are beyond the scope of this article. As this backend is also made by Hashicorp, it functions seamlessly as a backend for vault and provides High Availability clustering possibilities. As this offering is not specific to any one particular cloud vendor, it makes it a good option for organisations who want to avoid vendor lock-in as part of their cloud strategy.

Azure

Organisations that are heavily invested in Azure cloud may wish to use azure storage containers as a backend for their vault instances. Azure storage blobs provide a cheap storage option which is useful for organisations with budget constraints. Whilst this may be an attractive option for these organisations, there are some limitations that need to be taken into consideration.

Firstly, this backend does not have High Availability support, which I imagine would be unacceptable for many organisations; however, this will depend on how you plan on using Vault. If uptime is not a concern for your organisation then this may not be a deal breaker for you.

The next thing to note about this back end option is the storage limitation it has for Vault’s information. As of the time of writing this, Azure storage containers can only support a maximum of 4MB of data per blob.

It’s also worth pointing out that this backend is supported by the Hashicorp Azure community. In general, the community support is really good; however, some organisations may deem that to be an unacceptable risk for their Secrets Management Platform.

Similarly to the Azure storage backend, organisations that are heavily invested in AWS may also find this to be an attractive option, especially as S3 provides cheap storage. S3 also lacks High Availability support and is also community supported.

DynamoDB

Organisations using AWS also have the option of using DynamoDB as a storage backend for Vault. DynamoDB is a NoSQL document database which makes it suitable as a Key/Value store for Vault.

Unlike the S3 option, DynamoDB does offer support for High Availability clustering but this is a more expensive option than S3 storage.

Etcd

This is a simple Key/Value store which is also used in Kubernetes to store cluster state data. Etcd provides High Availability capabilities to Vault. This backend is also community supported and is very easy to setup.

Filesystem

This backend uses the directory structure of the local file system to store data for vault. For obvious reasons, this backend does not provide High Availability functionality. Filesystem is an official backend for Vault and as such, is supported by Hashicorp. In general, this option is not the most suitable of options for production Vault deployments that require zero downtime.

Access to the file system where the Vault data persists requires close attention to avoid unauthorised access to the information held there.

Google Cloud Storage

GCS is one of the storage offerings from GCP. Just like Azure blob storage and AWS S3 storage buckets, this offers cheap and flexible storage. Organisations that are heavily invested in GCP will find this to be an attractive option. This backend can also be configured for High Availability which is a plus for these organisations. This storage backend is also community supported.

In-Memory

In memory is a storage backend that persists the Vault data entirely in the memory of the local machine. Not only is this not a good option for production, it’s not an option at all. In-memory is useful for spinning up local test instances, which is what the Vault server uses when started up in dev mode using the -dev flag. When the platform hosting Vault is restarted, all the data that was previously persisted in memory is lost.

There are many other backend options which include various SQL offerings, both cloud native and traditional SQL databases like MySQL, PostgreSQL and MSSQL. Some of these options do offer High Availability options. Further reading of the official Vault documentation is recommended.

When considering which backend to use for Vault, there are some questions that need answering to find the most suitable solution.

Is the organisation heavily invested in a particular Cloud Vendor?

Some organisations, like Microsoft Gold Partners, are likely already invested heavily in a particular cloud vendor, in this case, Azure. This is more than an investment, this is a technological partnership between the organisation and Microsoft.

Similarly, many organisations are also partnered with Google, Amazon and Alicloud. The existence of such partnerships strongly suggest that the chances of these organisations moving to a different cloud vendor are minimal to non-existent in the long term technology roadmap. The point here is that as an architect, we do not need to take vendor lock-in concerns into consideration when making this design decision for the organisation.

These organisations will likely already have the skill set required to set up and manage the cloud vendor specific storage backend, which lessens the support overhead of the backend. This could be an important factor in whatever decision is made.

How will the organisation use the Vault deployment?

Organisations have different use cases for deploying Vault, for example, some will use it to serve secrets to applications and services which need to be accessible at all times. Other organisations may be using it as a storage tool for secrets. There are of course many other use cases but for the purposes of this blog, the two mentioned will suffice.

The answer to this question will more than likely dictate the answer to the next two questions.

How important is High Availability to the organisation?

In the first example above, those that wish to use it to serve applications and services with secrets will more than likely value High Availability over most things. In this example, applications and services cannot perform their functions if Vault is unavailable for a period of time. In this scenario, you almost certainly choose an option which provides High Availability like Consul or maybe Google Cloud Storage if you’re heavily invested in Google Cloud Platform.

Backup and restore process

To some organisations, the data being stored in vault is of the upmost importance and as such will value the ease of backup and restore process above most other factors. Does the organisation already possess the skillsets to easily implement backup and restore processes for a particular storage backend option? This is something that will also need to be taken into consideration.

Summary

As you can see from the above discussion, choosing the correct storage backend for each individual use-case requires a lot of thought and consideration. In order to make the right decision, you will need to understand the organisation as it is today, where the organisation is heading and the relevant skill sets they already possess. You will also need to understand the areas where they will need to up-skill, how much support overhead each option will cost them and if this cost is acceptable.

Understanding the future ambitions of the organisation will aid in producing a future proofed design with the appropriate backend. The above is by no means an exhaustive list of questions; however, it provides a good base to start the right conversations to extract the information needed to produce the best possible design for the organisation.

Architecting Vault — Part 1

Robert Barnes — Sun, 14 Jul 2019 19:12:35 GMT

Architecting Vault — Part 1

In the modern world, applications and the infrastructure they run on are moving to a multi-cloud, multi-platform and multi-service approach. This means that applications are being separated into service components that make up an application stack and run on a number of different platforms to deliver the business value, for example, some services may run in a container which is orchestrated by Kubernetes or Nomad, some may be run on Virtual Machines and the others run as Serverless functions using services like Lambda or FunctionApp. These services, whichever platform they are being run on, can be run either on-premise and/or in one or more of the biggest cloud providers (GCP, AWS, Azure, Alicloud) to create a multi-cloud hosted application.

The shift to this approach requires a change in mindset on many different fronts, especially security and secrets management as the networks that these applications use, move from a trusted network approach to a trustless network approach. In a trusted Network, credentials and secrets which allow application services to talk to one another are typically stored in the application code or in a configuration map as there is 100% control over the network and and it’s perimeter, meaning all services and users inside the perimeter can be trusted. There are flaws in this approach but it was and still is a common practice for many organisations. Moving to a cloud approach means loosing some of that control over the network perimeters as the cloud providers are partially responsible for network security under the Shared Responsibility Model.

This breeds the introduction a different paradigm in Trustless Network approach as we can no longer have the same perimeters in place and can no longer trust the perimeters that are not completely under our control. In this new approach, secrets can no longer be stored in the same way using config maps or hardcoded in the application code, which presents a new challenge for application developers and DevOps engineers who are developing, deploying and running these applications and infrastructure at scale.

HashiCorp Vault is a secure secrets management platform which solves this problem, along with other problems we face in modern day application engineering including:

Encryption as a service
Key management

When architecting your vault deployment, there are some fundamental questions you need to answer before designing your cluster. This series of Architecting Vault blogs will address these questions in more detail and explore ways to find the most suitable solution for individual use-cases.

What platform do I host my vault cluster on?

The first decision we need to make is what platform to use to host a Production Vault cluster. The most appropriate decision will differ from one organisation to the next and will depend on their current cloud strategy (if they have one), their industry compliance requirements, and the business value of the secrets that will be stored in the Vault cluster.

In addressing this question, let’s look at what options there are to host Vault and common hosting platforms in use today, from on-premise data centres to cloud providers.

Containers

Deploying Vault into a container is a useful option for local development as you can spin up local instances quickly and reliably; however, the very nature of how containers work make it the most risky platform to deploy a Production Vault cluster on. Containers are virtual workloads which share the kernel, libraries and binaries of the underlying host operating system, which is what makes them so lightweight and an attractive option to run micro-service architecture applications at scale.

The things that make Containers such an attractive option for many services are also the very things that make Containers the least secure option for running Vault clusters. The shared kernel, binaries and libraries presents a larger attack surface for the Vault cluster and exposes it to any vulnerabilities in the container run time which can be exploited by malicious attackers.

This increased attack surface is an acceptable risk to some organisations as the secrets stored in the Vault clusters are not deemed to be business critical and so this may still be a viable deployment option for many organisations. If using a container orchestration platform like Kubernetes, you will need to be aware that the more tenants (services) you have running in your Kubernetes cluster, the more risks there are to access your organisations secrets held in Vault clusters hosted on the same Kubernetes clusters as other tenants.

I reached out to HashiCorp for their advice on running Vault in Kubernetes and Co-founder & CTO Mitchell Hashimoto responded with the following.

I attended the HashiConf EU conference in Amsterdam last week and took the opportunity to speak to many of the HashiCorp Vault Software Engineers about this dilemma of running Vault on a Kubernetes cluster. The underlying message I got is that HashiCorp do not recommend running Vault in Production inside containers; however, they do recognise that some organisations will accept the increased risk and will deploy into Kubernetes, which is the reason they are currently working on Helm Charts to help these organisations deploy Vault into Kubernetes clusters quickly and reliably.

Virtual Machines

Running Vault Clusters on virtual machines is the most secure way to host Vault in Cloud environments. The underlying technology that enables virtual machines to work is different to the underlying technology that enables containerisation and has a smaller attack surface. Virtual machines are run on Hypervisors, which is a technology that allows physical resources such as Storage disks, CPU cores, Memory and Network Interface Cards (NICs) to be accessed and consumed by virtual machines. Hypervisors typically sit on top of an operating system, much like KVM sits on top of Linux Operating systems and Hyper-V sits on top of Windows operating systems. These Operating Systems are installed on bare metal servers.

The reason that this a more secure way to run Vault in production than containers is the fact that each virtual machine has its own operating system installed on it, which each have their own kernel, libraries and binaries. The image below depicts the difference between container technologies and Hypervisor technologies

Bare Metal

For organisations that still have on-premise data centres hosting physical servers, running your production vault clusters on bare metal may be an option for them. The bare metal option is the most secure way of running Vault clusters in production as the hardware doesn’t share itself with any other tenants. It provides you with complete control of the operating system and the underlying hardware.

This option does come with the most upfront cost to implement as you will need to procure dedicated server hardware for this deployment and will require physical resources to rack & stack the required hardware in the data centre and configure the networking accordingly. This option is most commonly used by financial organisations who’s secrets are business critical and need the minimum attack surface exposure for compliance reasons as their systems hold financial data that underpins their very existence and in some cases the national / international economy.

Making a decision

When making a decision as to how to host and run your Vault production clusters, I want to emphasise the fact that architectural design in general is a business driven exercise and should always exist to serve the purpose of achieving the business goals. There are some questions to which you need to seek answers for to help make the right decisions

What is the cloud strategy of the organisation — If your organisation has a cloud strategy, you should understand what the long term goals are for this strategy, for example, if currently there is a mix of on-premise and cloud work loads, is the aim to migrate everything to the cloud and retire the data centre? In this example, bare metal is probably not the most appropriate option to align to your business goals.
What is the value of the secrets to the organisation — The answer to this question is key and can often dictate whether your Vault cluster should be hosted and run on premise or in the cloud. The example mentioned earlier with secrets unlocking access to financial data would more often than not, influence the decision to use Bare metal installations.
Budget constraints — If cost of running and time to value is of the upmost importance then you may consider using VMs to get you up and running quickly but if in the cloud, these VMs will increase your cloud costs and still impact any budget constraints. If you are already running Kubernetes clusters in the cloud, running Vault clusters in containers is probably the most cost effective solution; however, it comes with the security trade-off mentioned earlier. Bare metal installations will take the longest amount of time to deploy as you will need to undertake a procurement exercise and hardware installation tasks before you can start configuring your Vault instances and this can take many weeks or in some cases months.

Somewhere between the answers to these three questions will be the answer to your main question which is, what platform do I host my vault cluster on?

To conclude this chapter, I will leave you with a quote from the Vault Production Hardening guide for you to ponder.

“Similarly, running on bare metal should be preferred to a VM, and a VM preferred to a container. This reduces the surface area introduced by additional layers of abstraction and other tenants of the hardware.”

Stories by Robert Barnes on Medium

Office Setup of a Developer Advocate

Furniture

Screen

Compute

Audio

Visual

Peripherals

Other

Architecting Vault — Part 4

Architecting Vault — Part 4

Architecting Vault — Part 3

Architecting Vault — Part 3

Architecting Vault — Part 2

Architecting Vault — Part 2

Architecting Vault — Part 1

Architecting Vault — Part 1