Infinite Databases with Nomad and Traefik

I'm no SQL wizard but I enjoy working with databases a lot — I myself am a PostgreSQL and DynamoDB fan. I've always viewed databases as tablestakes for any application since every good one has some sort of basic data persistence requirement.

I have also admired newer fully managed database-as-a-services (DaaS) like Neon and PlanetScale, as well as platform-as-a-services, with database additions, like Railway, Heroku, and most recently Vercel. However, I haven't really used these since the need to has never came up, and I've always been conformtable with cloud-provider options like AWS RDS.

My work to-date has really only necessitated a production, staging, and sandbox database at most, and provisioning these — it has always taken place in AWS via Terraform — has always been viewed as a non-trivial, heavy-handed action (this excludes click-ops). And maybe it's just who I am, but I have always been curious about how to spin up multiple databases on the fly, like Heroku Postgres, but on cloud infrastructure that I owned.

I finally did it.

The goal

My goal was to create some sort of system for rapidly provisioning databases on demand. This "system" could be anything from a few Makefile commands that wrap other tools, to a fully baked provisioning engine... obviously though, I leaned towards simplicity and less work.

The loose constraints I imposed on myself were that the databases had to run on infrastructure I owned, they must be accessible from the public internet via unique URL, and they had to be created quickly. For reference, simply creating AWS RDS instances takes 5-10 minutes, which is unacceptably slow.

The approach

Going into this, I had already ruminated on potential solutions for some time. I knew Nomad and its Docker driver would make it a breeze to run PostgreSQL containers, but I had never understood how to handle the networking or routing, or in other words, "connect the pieces".

Luckily, and I don't remember exactly how, I discovered Traefik which would take care of all of my routing needs. Routing was probably the most complex and nebulous piece to me personally.

So, Nomad, Docker, and Traefik were going to be the three core components of my approach.

Some other tools and technical callouts:

• I used a single AWS EC2 t2.large instance (that's ~$67/month for you pricing nerds out there) for running everthing: the Nomad server, Docker, and Traefik plus PostgreSQL containers.
  • I initially ran a t2.micro but I think it hit memory limits after the first PostgreSQL container.
• I used Packer to create an Amazon machine image (AMI) — what the EC2 instance boots up with — with Nomad and Docker pre-installed.
• AWS Route53 for DNS and subdomain routing.

The easy parts

What went well or felt like a breeze

• Automatic service discovery and routing with Traefik — this part is pure magic
• Running a Nomad job — nomad run ./traefik.nomad.hcl
  • Running a PostgreSQL Nomad job is extremely fast
• Learning Traefik through their Docker Compose example

The hard parts

What was difficult or felt like slog

• Writing a Nomad job spec. While this is just HCL, the Nomad-isms are unique enough that this is basically learning a new configuration format.
• Trafik's tag syntax in Nomad job specs
• Navigating Nomad documentation
• Navigating the Nomad UI — I don't understand 95% of the information displayed.
• No dedicated guide; Figuring out things by trial and error
• Too many ways to do certain things, like passing a static configuration to Traefik...
  • You can mount a real host system file
  • You can use Nomad's template stanza
  • You can pass container command arguments

UI and architecture

The intended user interface is a PostgreSQL connection string. This is not novel or different than that of any other database provider. (Though, where the good ones shine is in their splendid end-to-end experience — beautiful web interfaces, documentation, CLI tools, SDKs, APIs, etc. — everything else that accompanies that string.)

For this effort, I as an end user would be perfectly happy with a usable connection string.

Through a little bit of backwards working I laid out the architecture that supports the proposed user experience.

Oh, and did I say that this is not intended to be used in production workloads? ... This is not intended to be used in production workloads.

Real world use case

This effort is really just an alternate timeline of a project that I shipped: migrating my team off of Heroku.

This write up is effectively what likely would've happened if I had proceeded down path #1 from above.

Play by play

The play-by-play of how I got this working. Pardon me if the language starts to get inconsistent. Writing is not my forté and I always find myself unintentionally alternating between a reflective post and something like a how-to guide.

Prepare AMI

This part is optional as you can ssh into an EC2 instance and install everything manually, but I strived to codify as many moving parts as possible into some semblance of a system, as previously mentioned.

I used Packer to build an Amazon machine image (AMI) with Nomad and Docker pre-installed and ready to run on start up.

• Given the following file, run: packer build ./packer/nomad.pkr.hcl.

  ./packer/nomad.pkr.hcl

  packer {
    required_plugins {
      amazon = {
        version = ">= 1.1.1"
        source  = "github.com/hashicorp/amazon"
      }
    }
  }
  
  # https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs#tag-example
  source "amazon-ebs" "basic" {
    ami_name      = "nomad-x86_64"
    source_ami    = "ami-03a6eaae9938c858c" # Amazon Linux 2023 AMI 2023.2.20230920.1 x86_64 HVM kernel-6.1
    instance_type = "t2.small"
    region        = "us-east-1"
    ssh_username  = "ec2-user"
  }
  
  
  build {
    name = "packer-build"
    sources = [
      "source.amazon-ebs.basic"
    ]
  
    # install junk
    provisioner "shell" {
      inline = [
        "sudo yum -y install zsh",
        "sudo yum -y install git",
      ]
    }
  
    # multiline script in packer - https://stackoverflow.com/a/68216479
    # sudo cat not working - https://stackoverflow.com/a/18836896
    provisioner "shell" {
      inline = [
  "sudo mkdir -p /etc/nomad.d/",
  <<EOT
  sudo bash -c 'cat <<EOF > /etc/nomad.d/conf.hcl
  bind_addr = "0.0.0.0"
  
  acl {
    enabled = true
  }
  
  client {
    enabled = true
  }
  
  server {
    enabled          = true
    bootstrap_expect = 1
  }
  EOF'
  EOT
      ]
    }
  
    # Install nomad & run it
    # Nomad adds a systemd file post-installation.
    provisioner "shell" {
      inline = [
        "sudo yum -y install yum-utils",
        "sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo",
        "sudo yum -y install nomad",
        "sudo systemctl enable nomad.service",
        "sudo systemctl start nomad.service"
      ]
    }
  
    # Install docker & run it; This will be a driver that nomad will use
    # https://www.cyberciti.biz/faq/how-to-install-docker-on-amazon-linux-2/
    provisioner "shell" {
      inline = [
        "sudo yum update -y",
        "sudo yum -y install docker",
        "sudo usermod -a -G docker ec2-user",
        "id ec2-user",
        "newgrp docker",
        "sudo systemctl enable docker.service",
        "sudo systemctl start docker.service",
      ]
    }
  }
  

Run EC2

Here is a single file Terraform configuration that launches an EC2 instance with the AMI created by Packer. This requires that AWS credentials are set in your environment, which can be quickly done with aws configure or verified with cat ~/.aws/credentials.

• Given the following file, run the following commands:
  • terraform init - basically npm install for Terraform
  • terraform plan - view the proposed changes
    • it should say "Plan: 2 to add, 0 to change, 0 to destroy."
  • terraform apply
  • Grab the outputs to either ssh into the EC2 instance or set NOMAD_ADDR and interact with Nomad from your local machine.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.19.0"
    }
  }
}

#############
# Variables #
#############
variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "key_name" {
  description = "Key name for SSH access - hardcoded for demonstration purposes"
  type        = string
  default     = "test"
}

variable "vpc_id" {
  description = "The VPC ID — hardcoded for demonstration purposes"
  type        = string
  default     = "vpc-139b3769" # default VPC
}

variable "ami" {
  description = "The AMI output from packer — hardcoded for demonstration purposes"
  type        = string
  default     = "ami-05dc771b6a201cf29"
}

#############
# Providers #
#############
provider "aws" {
  region = var.region
}

##########
# Locals #
##########
locals {
  tags = {
    repo = "traefik-test"
  }
}

################
# Data sources #
################
data "aws_vpc" "default" {
  id = var.vpc_id
}

#############
# Resources #
#############

# Consider more restrictive security group rules
resource "aws_security_group" "allow_all" {
  description = "Allow all public traffic"
  vpc_id      = data.aws_vpc.default.id

  ingress {
    description      = "Allow all inbound public traffic"
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    description      = "Allow all outbound public traffic"
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  tags = local.tags
}


resource "aws_instance" "nomad-leader" {
  ami                    = var.ami
  instance_type          = "t2.large"
  key_name               = var.key_name
  vpc_security_group_ids = [aws_security_group.allow_all.id]

  tags = local.tags
}

###########
# Outputs #
###########

output "nomad_addr" {
  value       = "export NOMAD_ADDR = http://${aws_instance.nomad-leader.public_ip}:4646"
  description = "convenience command to set NOMAD_ADDR"
}

output "ssh_string" {
  value = "ec2-user@${aws_instance.nomad-leader.public_dns}"
}

This option is great as you can conveniently destroy the EC2 instance and security group with terraform destroy when you're done — No click-ops required!

Verify Nomad

From your local machine, without having to ssh into the EC2 instance, you can set NOMAD_ADDR and use the nomad CLI to verify that Nomad is ready:

export NOMAD_ADDR="http://$EC2_IP_ADDRESS:4646"

nomad agent-info

View the Nomad UI

View the Nomad UI @ http://$EC2_IP_ADDRESS:4646/ui/jobs

Run Traefik

Traefik is a reverse proxy that will serve as a front door on the EC2 instance. It has a full integration with Nomad, and through a relatively simple — but hard to know if you're right — tag syntax, Traefik can automatically discover and handle routing to all your services allocated by Nomad. My non-service-networking mind cannot comprehend this magic.

• Given the following Nomad job spc, run: nomad run ./nomad/traefik.nomad.hcl

  traefik.nomad.hcl

  # nomad run ./nomad/traefik.nomad.hcl
  job "traefik" {
    datacenters = ["dc1"]
    type        = "service"
  
    group "traefik" {
      count = 1
  
      network {
        mode = "host"
  
        port "http" {
          # to     = 8080 # container port the app runs on
          static = 80
        }
  
        port "https" {
          static = 443
        }
  
        port "traefik" {
          static = 8080
          to     = 8080
        }
  
        port "db" {
          static = 5432
          to     = 5432
        }
      }
  
  
      service {
        name     = "traefik-http"
        provider = "nomad"
        port     = "traefik"
      }
  
      task "server" {
        driver = "docker"
        config {
          # network_mode = "bridge"
          image = "traefik:v3.0"
          ports = [
            "http",
            "https",
            "traefik",
            "db"
          ]
          volumes = [
            "local/traefik.toml:/etc/traefik/traefik.toml",
          ]
        }
  
        # https://doc.traefik.io/traefik/getting-started/configuration-overview/#configuration-file
        template {
          data = <<EOH
  [entryPoints]
    [entryPoints.http]
      address = ":80"
    [entryPoints.https]
      address = ":443"
    [entryPoints.traefik]
      address = ":8080"
    [entryPoints.db]
      address = ":5432"
  
  [api]
    dashboard = true
    insecure = true
    debug = true
  
  [providers.nomad]
    refreshInterval = "5s"
    [providers.nomad.endpoint]
      address = "http://{{ env "attr.unique.network.ip-address" }}:4646"
      token   = "$NOMAD_TOKEN"
  
  [log]
    level = "DEBUG"
  EOH
  
          destination = "local/traefik.toml"
        }
      }
    }
  }
  

Run a few databases

This is where the fun starts.

I use a single Nomad job spec, and modify it slightly before running new Postgres instances. I modify a few key values like the job ID ($POSTGRES_JOB_ID) and hostname ($HOSTNAME) Traefik rule.

Nomad job IDs, like postgres-1 need to be unique, and I haven't yet figured out how to run multiple jobs from one Nomad file, hence pre-run, manual modifications.

• Given the following Nomad job spec, I modify the highlighted values, then run nomad run ./nomad/postgres.nomad.hcl for each new database. In theory, this could be repeated for an infinite number of databases... I think.

  ./nomad/postgres.nomad.hcl

  # nomad run ./nomad/postgres.nomad.hcl
  job "$POSTGRES_JOB_ID" {
    datacenters = ["dc1"]
  
    type = "service"
  
    group "db" {
      count = 1
  
      network {
        mode = "host"
  
        port "db" {
          to = 5432
        }
      }
  
      service {
        name     = "postgres"
        port     = "db"
        provider = "nomad"
  
        tags = [
          "traefik.enable=true",
          "traefik.tcp.routers.route_db.rule=HostSNI(`*`)",
          "traefik.tcp.routers.route_db.entryPoints=db",
          "traefik.http.routers.postgres.rule=Host(`$HOSTNAME`)",
          "traefik.http.services.db.loadbalancer.server.scheme=postgres",
        ]
      }
  
      task "server" {
        env {
          POSTGRES_DB       = "$POSTGRES_DB"
          POSTGRES_USER     = "$POSTGRES_USER"
          POSTGRES_PASSWORD = "$POSTGRES_PASSWORD"
        }
  
        driver = "docker"
  
        config {
          # image = "postgres:15-bookworm"
          image = "timescale/timescaledb:latest-pg13"
          ports = ["db"]
        }
      }
    }
  }
  

Create A records

The last piece here is to ensure that each database's unique hostname actually exists.

I already had a domain name (thekevinwang.com) and Route53 hosted zone that I could add additional subdomains to, so I may gloss over some details.

For my dashboards, I routed nomad.thekevinwang.com to the EC2 instance, and relied on ports :4646 and :8080 to view the Nomad UI and Traefik respectively.

For each on-demand database, I created an A record that pointed to the EC2 instance, and made the subdomain match the hostname value in the traefik.http.routers.postgres.rule=Host(`$HOSTNAME`) tag.

• Given the following JSON blob, I run:

  $ ~/

  aws route53 change-resource-record-sets \
    --hosted-zone-id $HOSTED_ZONE_ID \
    --change-batch file://./route53.json
  

• JSON blob. For demonstration purposes, there are only two database hostnames in here, but you could add more, do some templating, or even use multiple files.

  ./route53.json

  {
    "Comment": "optional comment about the changes in this change batch request",
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "nomad.thekevinwang.com",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [{ "Value": "$EC2_IP_ADDRESS" }]
        }
      },
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "postgres-1.thekevinwang.com",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [{ "Value": "$EC2_IP_ADDRESS" }]
        }
      },
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "postgres-2.thekevinwang.com",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [{ "Value": "$EC2_IP_ADDRESS" }]
        }
      }
    ]
  }
  

The test

If everything is setup correctly — networking, databases, nomad jobspec tags, Route53 A Records — you should be able to run psql postgres://$POSTGRES_USER:$POSTGRES_PASSWORD@$HOSTNAME -c "SELECT \* from NOW()" and get a response from each of the unique databases.

For me, it worked! See this Twitter/X thread from 09/22/2023 for reference.

Conclusion

Finally being able to implement this felt pretty epic. I know this isn't a viable approach for any production workloads but it feels like a pretty monumental breakthrough and huge addition to my generalist skillset as an engineer.

And, not that it was my intention, this feels like a significant personal inflection point towards platform engineering.