Weekend

Over the past weekend, I built a fun little feature that I’m calling Regeneration History. You can see it on the right side column (desktop width only).

I also watched a lot of Big Timber in the background, and worked on some CloudWatch dashboards for Lambda function metrics.

See: /wiki/cloud/aws/cloudwatch

Regeneration History

I self-host & deploy my site on AWS so I can view some additional bits of info that you forfeit when using a fully managed platform-as-a-service like Netlify or Vercel.

I use serverless-next.js to build my site. This package implements Next.js’ ISR by storing statically generated pages in an S3 bucket. Anytime a page is re-generated, the S3 object is modified, and if you have versioning enabled, the object will get a new version rather than being overwritten.

For example, the S3 object for /wiki/cloud/aws/s3 will be an .html file, located in the bucket at this particular path:

S3 Object Versioning

Regular traffic to a page, which implies regular static regeneration, will result in an object version dataset like so this...

Usefulness

An audit log or append-only-like log such is certainly useful in other contexts, but I don’t think there is much value in this particular case of a website’s static files.

For fun though, in case you wanted to build some sort a time-travel feature (thinking about something similar to the React.js tic-tac-toe time-travel feature), you could accomplish that this mechanism... but again, not particularly useful.

Work week

Tuesday, Aug 2, 2022

Terraform Core Versioned Docs

Today, I, along with my colleague Bryce, delivered versioned docs for the Core docs on terraform.io. This includes:

• https://www.terraform.io/intro
• https://www.terraform.io/cli
• https://www.terraform.io/language
• https://www.terraform.io/internals

While the trivial on the surface — you get a nifty little dropdown — the underlying architecture is massively more complex. More on that in a dedicated post maybe...

Wednesday, Aug 3, 2022

Automated GitHub Releases

• https://github.com/hashicorp/terraform-docs-agents/pull/12

Mermaid Rabbit Hole & GitHub GraphQL API Rate Limit

After work, I went into one of the worst rabbit holes in recent times.

The Problem: I had an initial implementation of mermaid diagrams being rendered in MDX. However, these diagrams would render at build time only, which takes place on my machine. Subsequent builds that would take place on AWS Lambda in a regeneration lambda, during ISR would fail with an difficult to diagnose error like:

_projectRoot is undefined. Unable to create a BrowserFetcher.

This lead me to use https://github.com/alixaxel/chrome-aws-lambda.

However this ended up resulting in file trace issues that were caused by some incompatibility with the target: "serverless" config that serverless-next.js sets.

...The ugly part...

Throughout this process, I had built my site several times. My site makes calls to the GitHub GraphQL API for data to statically render pages — mostly the pages on this wiki portion of the site. I hadn’t yet optimized these calls, but I quickly realized that I was exceeding the 5000 req/hr rate limit.

I continuously monitored my usage at https://docs.github.com/en/graphql/overview/explorer and continued debugging two issues simultaneously — mermaid/puppeteer/chromium, and excessive API calls to the GitHub API.

I needed to replace my troublesome code, which was already deployed on AWS, with a new deployment but the static page generation during the build process was blocked my GitHub API rate limit.

Both the deployed code and build attempts were simultaneously exacerbating the rate limit.

Oh lord...

GitHub resets your rate limit every hour, so I stayed up throughout 3 of these hour windows... from roughly 1:30AM to 4:30AM. In the first two windows, I had failed to find/solve the culprit, and rate limit was reached within minutes. I eventualy optimized and deleted some repeat API calls, and delete other unecessary ones.

Here's one for fetching all the files in a tree shape. I deleted this.

fragment treeEntryFields on TreeEntry {
  oid
  name
  type
}

query GetFileTree($owner: String!, $name: String!, $expression: String!) {
  rateLimit {
    cost
    remaining
    used
  }
  repository(owner: $owner, name: $name) {
    name
    object(expression: $expression) {
      ... on Tree {
        entries {
          ...treeEntryFields
          name
          object {
            ... on Tree {
              entries {
                ...treeEntryFields
                name
                object {
                  ... on Tree {
                    entries {
                      ...treeEntryFields
                      name
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Logical Nullish Assignment (??=)

One takeaway and learning was this logical nullish assigment operator.

In this bare bones example, fetch only gets called once, despite the wrapping callApi function being called three times.

let cache: Record<string, any> = {};

async function callApi(key) {
  cache[key] ??= await fetch(key);
}

await callApi('foo');
await callApi('foo');
await callApi('foo');

I eventually optimized a few repeat API calls, resolved my rate limit nightmare, never ended up getting Mermaid diagrams to render on Lambda, and eventually went to bed.

Awful life decision of staying up.

Todo

Thursday, Aug 4, 2022

I basically felt like garbage all day, and was hit by back to back Vercel gotchas at work.

Next.js && Nested Middleware incompatibility

# Error: Builder returned invalid routes: ["Route at index 25 has invalid `src` regular expression \"^(/_next/data/W1kPu1ISmrdZOGFG4eV3j)?((?:^/docs(?:(/.*)?)(\\.json)?$)$\"."]

From JJ Kasper:

There should be a build error if nested middleware is used past 12.1.7-canary.9

Nix Shell + Mermaid CLI

mmdc is the CLI command for the Mermaid CLI.

nix-shell -p nodePackages.mermaid-cli

# [nix-shell:~/repos/public-docs]

cat <<EOF | mmdc -o test.svg
graph LR
    A-->B
    B-->C
    C-->D
    C-->F
EOF