Posts

Dynamic Matrices in GitHub Actions

...from JSON payloads that you send! — This was a recent rabbit hole 🐰🕳 that took me 2 days to figure out. I couldn't find a quick and clear answer on Google so I figured I'd write about it.

Kevin WangSeptember 19, 2021

I recently needed to understand GitHub actions' matrix strategy for work.

...A matrix allows you to create multiple jobs by performing variable substitution in a single job definition.

For example, you can use a matrix to create jobs for more than one supported version of a programming language, operating system, or tool...

Unfortunately, the docs didn't cover my specific use case and the examples had hardcoded matrix values:

os: [macos-latest, windows-latest, ubuntu-18.04]
node: [8, 10, 12, 14]

I wanted to figure out how to create dynamic matrices... and more specifically, from JSON payloads that I'd be sending via repository_dispatch events.

So, how do you send dynamic matrix values from repository dispatch payloads?

Trial and error

My dev process was pretty clunky and was not particularly pleasant either. At the time of development, I hadn't yet tried nekos/act, which could've helped speed up parts of my process.

Set up a repo with a workflow

I first created a repo — well, shoved actions into an existing repo — and created a workflow that listened for repository_dispatch events, and specific event_types.

name: Dispatch Listener

on:
  repository_dispatch:
    types:
      - "test run"
      - "foobar"
      - "look-wildcard-here:*"
# more she-yaml-gans to come...

`POST /repos/{owner}/{repo}/dispatches`

The next thing I did was call the GitHub REST API to trigger the repository's workflow manually. I did so via Postman, but in actual implementation, I'd probably be calling it via another workflow with something like HTTP Request Action or from server-side code using Octokit and authenticating as a GitHub application.

See "Create a rpository dispatch event" for endpoint authentication and details

curl --location --request POST 'https://api.github.com/repos/{{owner}}/{{repo}}/dispatches' \
--header 'Authorization: Bearer {{PERSONAL_ACCESS_TOKEN}}' \
--header 'Content-Type: application/json' \
--data-raw '{
  "event_type": "test run",
  "client_payload": {
    "services": [
        "cat",
        "dog",
        "bird",
        "hamster"
    ],
    "versions": [2,4,6,8]
  }
}'

I debugged the live workflow runs via the GitHub UI's "Actions" tab, which was not the worst given the simplicity of my workflow, but it was certainly not fun. The lack of IDE linting and type-system made it feel like I was developing in the dark.

Success

Skipping over a boring streak of unsuccessful workflows, here's the yaml for a workflow that finally worked.

This workflow file allows you to

Post variable JSON arrays in a client_payload
Pass them from 1 job run to a following job
Trigger a matrix

name: Dispatch Listener

on:
  repository_dispatch:
    types:
      - "test run"
      - "test::*"

jobs:
  # ------------------------
  say_hello:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.generate-matrix.outputs.services }}
      versions: ${{ steps.generate-matrix.outputs.versions }}
    steps:
      - name: Generate Matrix
        id: generate-matrix
        run: |
          SERVICES='${{ toJSON(github.event.client_payload.services) }}'
          echo ::set-output name=services::${SERVICES}
          VERSIONS='${{ toJSON(github.event.client_payload.versions) }}'
          echo ::set-output name=versions::${VERSIONS}
  # ------------------------
  my_echo:
    runs-on: ubuntu-latest
    needs:
      - say_hello
    steps:
      - name: Echo previous outputs
        run: echo "${{ toJSON(needs.say_hello.outputs) }}"
  # ------------------------
  my_matrix:
    runs-on: ubuntu-latest
    needs:
      - my_echo
      - say_hello
    strategy:
      matrix:
        service: ${{ fromJSON(needs.say_hello.outputs.services) }}
        version: ${{ fromJSON(needs.say_hello.outputs.versions) }}
    steps:
      - name: Matrix => (${{ matrix.service }} ${{ matrix.version}})
        run: |
          echo ${{ matrix.service }}
          echo ${{ matrix.version }}

Notes

You could pass the client_payload keys directly to a matrix value

strategy:
  matrix:
    service: ${{ github.event.client_payload.services }}
    version: ${{ github.event.client_payload.versions }}

Or you can set them as outputs from prior jobs, leading up to a job with a matrix.

- name: Generate Matrix
  id: generate-matrix
  run: |
    SERVICES='${{ toJSON(github.event.client_payload.services) }}'
    echo ::set-output name=services::${SERVICES}
    VERSIONS='${{ toJSON(github.event.client_payload.versions) }}'
    echo ::set-output name=versions::${VERSIONS}

The key here was the single quotes '' wrapping the toJSON function. Without it, the outputs were set to either Array or Object — the types of the underlying JSON — or [ or { — the first line of pretty-printed JSON.

`nektos/act`

After finding my solution, I went back to try act, a tool for running GitHub actions/workflows locally. Given the same JSON payload I was sending via Postman,

{
  "event_type": "test::something",
  "client_payload": {
    "services": ["Terraform", "Waypoint", "Consul", "Vault"],
    "versions": [0, 1, 2, 3]
  }
}

I run act repository_dispatch -e event.json and can see the various steps and jobs of my workflow being executed.

At the time of writing this post, act doesn't handle matrices yet.

Other

Some complex workflows that people dropped in some threads that I encountered.