..see what I did there?
Hash Brown Inc. is a Japanese multinational widget manufacturer headquartered in Tokyo, Japan. They’ve heard all about this new DevOps movement and are relying on it to modernise, and give the company an edge to outshine their competitors, after a year that has had investors calling for drastic action.
The technology devision, responsible for handling 100% of internal software development projects, from idea to production to maintenance- is in a bad way. And it’s costing Hash Brown billions.
The developers have set up their flavoured water and fake plant refuge, barricading themselves from the infrastructure guys- whose desk setup has been dubbed “the slums“. Both teams refuse to communicate, throwing their projects at each other over the virtual “wall” via email.
Nothing is built using a CI. Nothing is tested. Silent prayers are thrown around whenever something new is deployed, and upgrading is unheard of.
A developer manually packages the application, puts it in a ZIP file and sends it via email to the one infrastructure guy they can stand to talk to.
The company has been using Linode for about a year. Nothing is documented, nobody knows what servers run critical applications, there is no staging environment, and everyone has administrator privileges.
Everything is stored in a single cluster of PostgreSQL database servers on Linode. Everything. That includes server secrets, customer information, product, stock, Human Resources, payroll, and Legal’s own internal apps.
Legend had it, if you so much as logged in or touched an instance’s networking rules- not only would the Networking Team have a field day, but the entire application would most likely break, unable to communicate with some component.
Deployments are all or nothing. Everything would go down for a 24 hour “maintenance window“, which certainly isn’t unheard of- but was always a nightmare. That window was almost always exceeded, causing vendors and customers to go ballistic. Thankfully, they haven’t dropped the production database yet.
The application that we, the newly incepted Rangers team are going to fix first, is known internally as Seattle. It consists of a simple AJAX-powered frontend, and a magic, black-box binary backend for an API. We are going to demonstrate how easy it is to get started with the “DevOps Mentality” by implementing the Modern Application Delivery Lifecycle.
For information on the “Modern Application Delivery Lifecycle“, please read this post on Hashicorp’s website.
Provision
Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the “works on my machine” excuse a relic of the past.
– https://www.vagrantup.com/intro/index.html
# seattle-vagrant/Vagrantfile
Vagrant.configure(“2”) do |config|
config.vm.box = “generic/alpine310”
config.vm.box_check_update = true
config.vm.network “forwarded_port”, guest: 80, host: 8080
config.vm.provider “virtualbox” do |vb|
vb.memory = “1024”
end
config.vm.synced_folder “seattle-ui/”, “/var/www/html/ui/”
config.vm.synced_folder “seattle-api/”, “/seattle/api/”
config.vm.provision “shell”, inline: <<-SHELL
apt-get update
apt-get install -y apache2
wget -O /tmp/seattle.zip https://ci.private.hashbrown.co.jp/
unzip /tmp/seattle.zip -d /var/www/html/ -x api/
unzip /tmp/seattle.zip -d /seattle/ -x ui/
SHELL
end
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.
– https://www.packer.io/intro/
# seattle-packer/linode.json
{
“type”: “linode”,
“linode_token”: “xxxaaaxxxaaaxxx”,
“image”: “linode/debian10”,
“region”: “ap-northeast”,
“instance_type”: “g6-nanode-1”,
“instance_label”: “seattle-staging-{{timestamp}}”,
“image_label”: “seattle-image-staging-{{timestamp}}”,
“image_description”: “Seattle Staging Environment Image”,
“ssh_username”: “root”
}
# seattle-packer/docker-ui.json
{
“type”: “docker”,
“image”: “httpd”,
“commit”: true,
“changes”: [
“USER www-data”,
“WORKDIR /var/www”,
“ENV HOSTNAME staging.seattle.private.hashbrown.co.jp”,
“COPY ./seattle-ui/ /usr/local/apache2/htdocs/”
“EXPOSE 80”,
“LABEL version=1.0”,
]
}
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
– https://www.terraform.io/intro/index.html
# seattle-terraform/terraform.tf
resource “linode_instance” “web” {
label = “seattle-staging-${timestamp()}”
image = “hashbrowninc/seattle-image-staging-2019-12-31T09:30:10Z”
region = “ap-northeast”
type = “g6-nanode-1”
authorized_keys = [“ssh-rsa AAAA…Gw== [email protected]”]
group = “seattle-staging”
tags = [ “seattle” ]
private_ip = true
}
Secure
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
– https://www.vaultproject.io/docs/what-is-vault/index.html
The team sets up Vault to use for internal secrets using Linode’s excellent guide. https://www.linode.com/docs/applications/configuration-management/use-hashicorp-vault-for-secret-management/
Connect
Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. Each of these features can be used individually as needed, or they can be used together to build a full service mesh.
– https://www.consul.io/intro/index.html
# ./consul.d/seattle-ui.json
{
“service”: {
“name”: “seattle-ui”,
“tags”: [“seattle”],
“port”: 80,
“check”: {
“args”: [“curl”, “localhost”],
“interval”: “10s”
}
}
}
# ./consul.d/seattle-api.json
{
“service”: {
“name”: “seattle-api”,
“port”: 8181,
“connect”: { “sidecar_service”: {} }
}
}
$ dig @127.0.0.1 -p 8600 seattle-ui.service.consul
; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 8600 seattle-ui.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28340
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;seattle-ui.service.consul. IN A
;; ANSWER SECTION:
seattle-ui.service.consul. 0 IN A 10.122.122.13
;; ADDITIONAL SECTION:
seattle-ui.service.consul. 0 IN TXT “consul-network-segment=”
;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Tue Jul 16 14:26:53 PDT 2019
;; MSG SIZE rcvd: 99
Run
Nomad is a flexible workload orchestrator that enables an organization to easily deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run a diverse workload of Docker, non-containerized, microservice, and batch applications.
– https://www.nomadproject.io/intro/index.html
# seattle-nomad/seattle-ui.hcl
job “seattle-ui” {
region = “jp”
datacenters = [“ap-northeast”]
type = “service”
update {
stagger = “30s”
max_parallel = 2
}
group “seattle-webservers” {
count = 3
task “frontend” {
driver = “docker”
config {
image = “hashbrowninc/seattle-ui”
}
service {
port = “http”
check {
type = “http”
path = “/”
interval = “10s”
timeout = “2s”
}
}
env {
“DB_HOST” = “db.seattle.private.hashbrown.co.jp”
“DB_USER” = “web”
“DB_PASS” = “AAAABBBBCCCCDDDD0000”
}
resources {
cpu = 1000 # MHz
memory = 512 # MB
network {
mbits = 1000
port “http” {}
}
}
}
}
}
And with that, developers have a build box that mimics the build environment thanks to Vagrant. Multiple packaging methods that can be tailored to any environment are available with Packer. Infrastructure is written in Terraform code to keep environments alike. Our applications can connect to Vault to store secrets, connect to each other via Consul and have their deployments managed via Nomad.
Perhaps Hash Brown Inc. will have a good year…?