How to Build & Publish a Terraform Provider

I’m going to go out on a limb and say many of us have had a similar experience with Terraform provider development:

“I tried briefly once before and totally gave up”

The barrier to initial entry is quite high but it doesn’t need to be. Hopefully my steps can save others some time.

Backstory

Yesterday, October 4, 2023, was my last day at HashiCorp after a great two year run on the awesome Web Presence team. Prior to my departure, I shipped “Developer AI” as a closed beta. This is an LLM-powered Q&A interface with retrieval augmented generation (RAG).

The backend API that powers this widget has two datastores. The first is a PostgreSQL database for storing messages in a tree-like structure. (This is an arbitrary implementation detail, based on what I could glean from the experience on chat.openai.com). The second datastore is a Pinecone vector database, populated with OpenAI GPT3.5 vector embeddings, created from HashiCorp product docs, like Terraform and Consul.¹

In preparing the project for effectively maintenance mode after my departure, I wanted to make sure that all infrastructure — AWS, Azure OpenAI, and Pinecone — was captured in infrastructure as code (IAC) with Terraform.

The AWS parts constituted 95% of the infrastructure, and was easy to capture as it was already familiar. However, Pinecone was a new service, and no “Official” or “Partner” Terraform provider existed, so by pragmatic choice, that IAC effort was tabled.

On my last 24 hours, I started cranking out my first provider... and published it to the registry just today! I’d say it took me a collective 12-18 hours of development to get a minimum viable product.

Anyways, on to the business...

Critical

The process of building a Terraform provider is not too bad! But the end to end experience is strewn with numerous paper cuts and the information is presented in a fragmented manner, making for a subpar development experience.

Here is a rough collection of what I felt were critical pieces worth documenting, in a unified and ordered view. This is biased towards the publishing process, rather than the development process, because:

”If it’s not deployed, it doesn’t exist.”

Don’t you agree?

Feedback loop

A quick feedback loop is critical to any sane development process. Setting this up for terraform provider development enables running terraform plan, apply, destroy locally, to verify each code change.

From here on, the writing will take a very much “do this, then do that” tone, so bear with me!

Update ~/.terraformrc with the following. The key in the dev_overrides stanza is mostly arbitrary, but will be referenced in Terraform HCL code.²

provider_installation {
  dev_overrides {
    "thekevinwang.com/terraform-providers/pinecone" = "Users/kevin/go/bin"
  }

  direct {}
}

And the usage in Terraform code that you want to test locally looks like:

terraform {
  required_providers {
    pinecone = {
      source = "thekevinwang.com/terraform-providers/pinecone"
    }
  }
}

For me, testing each source code change involved two terminal windows:

1. Reinstall the provider binary
   $ ~/terraform-provider-pinecone

   go install .
   
2. Run a terraform command of choice
   $ ~/terraform-provider-pinecone/examples/basic

   terraform plan
   

Documentation generation

Documentation is technically optional and doesn’t prevent a provider from making it to the registry, but I consider it a critical requirement for any trusted provider.

The designated tool for this is tfplugindocs. ³

How do you use it?

Add /tools/tools.go to the provider project.

//go:build tools

package tools

import (
	// Documentation generation
	_ "github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs"
)

And run the following commands to verify installation, and generate docs:

export GOBIN=$PWD/bin
export PATH=$GOBIN:$PATH
go install github.com/hashicorp/terraform-plugin-docs/cmd/tfplugindocs
which tfplugindocs
tfplugindocs

Running tfplugindocs will generate a tree of docs based on the go code written. Take note of fields like (schema.Schema).MarkdownDescription.

For example:

.
├── docs
│   ├── index.md
│   ├── resources
│   │   └── index.md
│   └── data-sources
│       └── collection.md

These docs will ultimately be rendered on the registry in a matching tree format.

Releasing

Here’s my distilled overview of the release process, which was not necessarily a code-heavy part, but involved many moving parts.

Valid repository name

The GitHub repository name must adhere to terraform-provider-{lowercase} in order for Terraform Registry to accept it. ⁴

Add a manifest

Add a terraform-registry-manifest.json file to the project root. This is a special file that the registry looks for in each release.

{
  "version": 1,
  "metadata": {
    "protocol_versions": ["6.0"]
  }
}

I used the Terraform Plugin Framework so protocol_versions is set to 6.0. ⁵

Registry authorization

These are a series of mostly one-time, web-based steps to authorize the registry to detect your provider and read its releases.

Sign in to the Terraform Registry and authorize the GitHub app.

Get a GPG key from the following command and add it to the Signing Keys view.

gpg --armor --export "[[Key_ID_or_email]]"

# -----BEGIN PGP PUBLIC KEY BLOCK-----
# abcdedfg...
# -----END PGP PUBLIC KEY BLOCK-----

GitHub release

At this point, Terraform Registry should be able to auto detect the provider, based on the name, and read its GitHub releases for expected artifacts.

(No releases should exist yet though.)

Of the few options listed on the publishing docs, I chose using goreleaser locally since I had used it before, and wanted to avoid the additional complexity of say, working with a remote CI environment.

Add .goreleaser.yml

Copy this file from terraform-provider-scaffolding-framework

# Visit https://goreleaser.com for documentation on how to customize this
# behavior.
before:
  hooks:
    # this is just an example and not a requirement for provider building/publishing
    - go mod tidy
builds:
  - env:
      # goreleaser does not work with CGO, it could also complicate
      # usage by users in CI/CD systems like Terraform Cloud where
      # they are unable to install libraries.
      - CGO_ENABLED=0
    mod_timestamp: "{{ .CommitTimestamp }}"
    flags:
      - -trimpath
    ldflags:
      - "-s -w -X main.version={{.Version}} -X main.commit={{.Commit}}"
    goos:
      - freebsd
      - windows
      - linux
      - darwin
    goarch:
      - amd64
      - "386"
      - arm
      - arm64
    ignore:
      - goos: darwin
        goarch: "386"
    binary: "{{ .ProjectName }}_v{{ .Version }}"
archives:
  - format: zip
    name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
checksum:
  extra_files:
    - glob: "terraform-registry-manifest.json"
      name_template: "{{ .ProjectName }}_{{ .Version }}_manifest.json"
  name_template: "{{ .ProjectName }}_{{ .Version }}_SHA256SUMS"
  algorithm: sha256
signs:
  - artifacts: checksum
    args:
      # if you are using this in a GitHub action or some other automated pipeline, you
      # need to pass the batch flag to indicate its not interactive.
      - "--batch"
      - "--local-user"
      - "{{ .Env.GPG_FINGERPRINT }}" # set this environment variable for your signing key
      - "--output"
      - "${signature}"
      - "--detach-sign"
      - "${artifact}"
release:
  extra_files:
    - glob: "terraform-registry-manifest.json"
      name_template: "{{ .ProjectName }}_{{ .Version }}_manifest.json"
  # If you want to manually examine the release before its live, uncomment this line:
  # draft: true
changelog:
  skip: true

Then create and push a tag, and run goreleaser.

git tag [[v0.1.1]]
git push origin [[v0.1.1]]
GITHUB_TOKEN=$(gh auth token) goreleaser release --clean

• Debugging

  If there's a error about GPG_FINGERPRINT not being set, run the following command to retrieve and set it.

  $ ~/

  gpg --list-keys
  
  # /Users/kevin/.gnupg/pubring.kbx
  # -------------------------------
  # pub   rsa4096 2021-12-15 [SC]
  #       [[AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJ]]
  # uid           [ultimate] Kevin Wang <email>
  # sub   rsa4096 2021-12-15 [E]
  
  export GPG_FINGERPRINT=[[AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJ]]
  

  I’m not a security expert, so do not take these GPG-related steps as scripture. Refer to Signing Key steps if anything seems off!

goreleaser should create a new GitHub release with various artifacts included.

The Terraform Registry should detect this new release and create a new version — like magic.

That’s it!

This marks the end of the release process of all the critical steps that I remember.

Secondary

Ok, broad strokes aside, here are some other, more lower level issues that involved non-trivial effort. Not critical but still worth sharing for future reference.

Scaffolding

There are two GitHub repositories that serve as standards for copy-paste — every developer’s absolute best friend.

1. Terraform Provider Scaffolding Framework

2. Terraform Provider HashiCups PF — This repository is nearly identical to the one above, but is intended to accompany a HashiCorp tutorial. I personally did not find this tutorial worth going through. The source code is an OK reference though, and just a slightly more realistic representation of a provider than the base scaffolding repository. Ignore the docker-compose parts and use a real external service instead!

Lifecycle of a resource

Reasoning about the lifecycle of a resource for the first time is pretty daunting since there are so many permutations of commands and state.

And because it’s not super clear what the implicated code methods being called are, development can feel quite opaque or very much like trial-and-error.

There are a few very helpful pages that are either not on https://developer.hashicorp.com or are simply hidden behind the complex navigation.

1. Terraform plugin framework's internal RPCs
2. Terraform core resource change diagram

I painstakingly ran multiple permutations of plan, apply, import, and destroy with a state and no state, and picking out log statements I added, in order to get a firsthand look at the exact call order of various resource methods.

This is what I gathered:

And my approach behind this was:

1. Add log statements to the top of each method. For example,

   ~/terraform-provider-pinecone/internal/resources/index.go

   // Delete resource information.
   func (r *indexResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
   	tflog.Info(ctx, "indexResource.Delete", map[string]any{"req": req, "resp": resp})
   
   

2. Run terraform with JSON logging. ⁶

   $ ~/terraform-provider-pinecone/examples/basic

   TF_LOG=json TF_LOG_PROVIDER=info TF_LOG_PATH=logs.jsonl terraform destroy
   

3. Meticulously comb through the logs with the help of a regular expression like, "indexResource.[a-z]+".+tf_rpc.

   $ ~/terraform-provider-pinecone/examples/basic

   {"@level":"trace","@message":"GRPCProvider.v6: PlanResourceChange",
   "@timestamp":"2023-10-06T18:36:20.926340-04:00"}
   {"@caller":"/Users/kevin/repos/terraform-provider-pinecone/internal/resources/index.go:108",
   "@level":"info","@message":"indexResource.Configure","@module":"pinecone","@timestamp":
   "2023-10-06T18:36:20.926821-04:00","req":{"ProviderData":...
   

4. Diagram my findings.

Tactics

For a sane provider development experience, pick a service with fully spec’d API docs, like Pinecone. Provider development is essentially a ton of CRUD operations, so having a clear understanding of the target API is very important.

Start with a Data Source. Data Sources are read-only and significantly easier to get to a working state than with a Resource.

The rest is Go

After all this, the rest of the provider development gauntlet is just writing Go code. Being able to connect the dots between a Terraform HCL configuration and the provider code is plenty enough to achieve a minimal, working provider. I will acknowledge that this is not actually easy by any means, but it is not worth expanding on here.

A simple heuristic for starting to understand how a Terraform provider’s code works is to view the source code of a simple provider...

Oh hey, I have one of those...terraform-provider-pinecone 🥳.