Infrastructure as Code To Save Time

Let's face it. Writing code takes a ton of time. If you're a software engineer like me, then you probably find yourself constantly switching contexts on the regular — personal projects, to work and then back to personal projects. One of the biggest time sucks that I've experienced is simply resuming personal projects after 8+ hours of work .

I get paid for my time to write code at work, and as long as I ship features or complete work by the end of my sprint, there's no problems or issues with time there (optimizations can definitely still be made though). On the other side of things, I don't get paid (yet) to work on my personal projects, so I really need to optimize at every corner, otherwise I'm pretty much wasting my life doing things like remembering where I left off, or figuring out what state I left my cloud provider infrastructure in the day or even week before.

Basic Needs

Pretty much every app needs some bit of infrastructure. This could be a custom domain, an AWS S3 Bucket to host static assets, or a GCP Compute Engine to run your webserver. A cloud provider's console is an intimidating place to be in. How many things do you have to click just to create a bucket, or provision a database? What specs do you provide? What do you name it? What other cloud resource do you want to connect? What specs do you want for those? Figuring this out takes a ton of time, not to mention even learning about the different available features.

This also gets messy really quick. How do you keep track of which of your 10 projects each piece of infrastucture belongs to? Environment variables? Oof. This is where the infrastructure as code paradigm comes in to save the day. Github and IDEs — essentially where your lines of code are — are most developers' happy places, so having infrastructure live there makes sense.

Cloudformation is how AWS orchestrates multiple bits of infrastructure from one source of truth. But no one likes its bizarre syntax. Terraform and AWS Cloud Development Kit are two tools that help you maintain your infrastructure as code, without having to deal with Cloudformation directly. I know large organizations like AirBnb use Terraform but I've personally become familiar and taken a liking to the aws-cdk since there's no 3rd party to deal with, just you and AWS. You install a CLI, write some TypeScript some and provision all your infrastructure from your terminal with one command — cdk deploy (or if you're fancy, deploy it from a Github action. I have yet to figure this part out).

Code Example

Here's a diagram of some arbitrary infrastructure that you might have.

• API Gateway → SNS Service Integration
• Subscribed SQS Queue → Lambda function → Dynamo DB Table
• Dynamo Stream → 2nd Lambda function → 2nd Dynamo DB Table

There's 8 unique pieces of infrastructure here. In 2021, no one has time to go into 8 different AWS console pages, and create each of these manually. This is also extremely fragile. If one resource is removed, the blast radius could be huge, and you have no audit trail of what happened and when. If your infrastructure changes were version controlled, none of this would be an issue.

The code might look something like this...

export class ArbitraryAppStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const topic = new sns.Topic(this, "ArbitraryAppTopic", {
      fifo: true,
      topicName: "ArbitraryAppTopicFIFO",
    });

    const deadLetterQueue = new sqs.Queue(
      this,
      "ArbitraryAppAppDeadLetterQueue",
      {
        retentionPeriod: cdk.Duration.days(14),
        deliveryDelay: cdk.Duration.millis(0),
        fifo: true,
        // contentBasedDeduplication: true,
      },
    );

    const queue = new sqs.Queue(this, "ArbitraryAppAppQueue", {
      visibilityTimeout: cdk.Duration.seconds(30),
      fifo: true,
      // contentBasedDeduplication: true,
      deadLetterQueue: {
        maxReceiveCount: 1,
        queue: deadLetterQueue,
      },
    });

    const sub = new SqsSubscription(queue);
    topic.addSubscription(sub);

    const table = new dynamodb.Table(this, "ArbitraryAppTable", {
      partitionKey: { name: "PK", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "SK", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
      replicationRegions: ["us-east-1"],
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });
    const replicaTable = new dynamodb.Table(this, "ArbitraryAppReplicaTable", {
      partitionKey: { name: "PK", type: dynamodb.AttributeType.STRING },
      sortKey: { name: "SK", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    const sqsToDynamoHandler = new lambda.NodejsFunction(
      this,
      "ArbitraryAppHandler",
      {
        entry: "resources/lambda/sqsToDynamo/index.ts",
        // depsLockFilePath is implied
        // depsLockFilePath: "resources/lambda/sqsToDynamo/yarn.lock",
        handler: "handler",
        timeout: cdk.Duration.seconds(10),
        environment: {
          TABLE_NAME: table.tableName,
          QUEUE_URL: queue.queueUrl,
        },
        bundling: {
          minify: true,
          environment: {
            NODE_ENV: "production",
          },
        },
      },
    );

    const dynamoToReplicaHandler = new lambda.NodejsFunction(
      this,
      "ArbitraryAppReplicaHandler",
      {
        entry: "resources/lambda/dynamoToReplica/index.ts",
        handler: "handler",
        timeout: cdk.Duration.seconds(10),
        environment: {
          REPLICA_TABLE_NAME: replicaTable.tableName,
        },
        bundling: {
          minify: true,
          environment: {
            NODE_ENV: "production",
          },
        },
      },
    );

    sqsToDynamoHandler.addEventSource(new SqsEventSource(queue));
    dynamoToReplicaHandler.addEventSource(
      new DynamoEventSource(table, {
        startingPosition: StartingPosition.LATEST,
      }),
    );

    table.grantWriteData(sqsToDynamoHandler);
    table.grantStreamRead(dynamoToReplicaHandler);
    replicaTable.grantWriteData(dynamoToReplicaHandler);

    queue.grantConsumeMessages(sqsToDynamoHandler);
  }
}

Though this example is incomplete, it gives you an idea of the code necessary — about 100 lines for this case — to provision multiple pieces of cloud infrastructure. There's no need to touch AWS's bizarre CloudFormation YAML, or even Hashicorp's proprietary HCL.

Anecdotal Time Savings

I created jaus.io (Just Another URL Shortener) in 3 days. 20% of the project is a NextJS statically generated page, and the other 80% is AWS resources,

• S3 Bucket
• Cloudformation Distribution
• API Gateway
• Lambda funciton
• Dynamo DB Table
• Route 53 Hosted Zone
• Route 53 Registered Domain
• ACM TLS/SSL Certificate
• A-Record

I attribute the speed and development ease to being able to manage infrastructure from code, and rarely having to log in to the AWS dashboard to do any manual work, although I did spend about a day just getting up to speed with particular AWS resources, as well as the CDK itself.

Clone and Destroy

If I wanted spin up another app with similar infrastructure, all the infrastructure is available for some good ole copy paste. Rename a few things, and the infrastructure will be ready in a few minutes.

If I needed to clean up and cloud resources, I could tear everything down with a single command — cdk destroy.