Adventures in Terraform: How and why we built our Terraform Provider

Hey there - my name is Gagan Trivedi and I am an engineer at Flagsmith. A few months back I had absolutely no experience building Terraform Providers. Flagsmith needed a Terraform provider, so we built one! I wanted to document my journey to help other people looking to build providers for their products.

Context

At Flagsmith, we offer an open-source feature flagging tool. Our goal is to help teams ship faster and continuously improve their products. Over the last few years, we have seen a growing number of teams that have adopted Terraform to automate parts of their CI/CD processes. This product has been an absolute game changer in moving more and more of the build process into code, ensuring consistent approaches and best practices to building software. 

Understanding this approach, we decided to build and launch our own Terraform Provider a few months back. This provider is great for CI/CD automation and lets dev teams control and manage feature flags/toggles as part of their Terraform workflows.

A big part of being an Open Source company is really leaning into transparency. This article will look at how / why we made the decision to build the Terraform provider and how to get started if you are looking to build your own. If you are interested in these topics, we wrote about our scariest release ever and our infrastructure costs of running SaaS at scale (billions of requests/month).

Why we decided to make this integration

Whenever we talk to our potential clients we always make notes about their requests. Since the beginning of 2022, we’ve noticed that more and more teams are adopting Infrastructure as Code (IaC) best practices. The platform that consistently came up was Terraform.

To make a decision about whether we actually need to invest in this and if so, how we should build this integration we started the customer discovery calls.

The feedback was extremely helpful for our initial design and allowed us to scope an MVP plus a near-term roadmap for improvements to the provider over future releases. Once we were aligned on what we wanted to build, it was time to start writing code!

Building the Terraform Provider

Step 1: Let’s create a basic terraform configuration first to give us some idea of what we need to build, and we can build it backwards from there.

In our case, for the version v0.1.0, it was just the ability to update a feature state

and our configuration looked something like this:

terraform {
  required_providers {
    flagsmith = {
      source = "github.com/Flagsmith/flagsmith"
    }
  }
}

provider "flagsmith" {
  master_api_key = "<master_api_key>"
  base_api_url = "<http://localhost:8000/api/v1>"
}

resource "flagsmith_flag" "feature_1_dev" {
  enabled         = false
  environment     = 2
  feature         = 14
  environment_key = "<environment_key>"
  feature_name    = "some_feature"
  feature_state_value = {
    type         = "unicode"
    string_value = "some_flag_value"
  }
}

With what we need to build answered, the next step is it to figure out the how are we going to build that for that, we decided to use https://github.com/hashicorp/terraform-provider-scaffolding-framework because we believe it has a clear and concise interface. If you want, you can take a look at https://www.terraform.io/plugin/which-sdk and decide which one suits your requirements better.

With the most important building block out of the way, it was time to build the Golang library that will sit between our plugin and Flagsmith server, in other words, our terraform plugin is going to need a way to interact with our API and our Golang library is going to do just that.

Since we only need to GET and PUT (update) the feature states (what about create and delete, you might ask, since we need to implement them as part of the provider interface? Well more on that later), our Golang library is only going to have those two public methods (apart from the one used for initializing the client)

Now, let’s get back to our plugin. We start by replacing the `internal/provider` directory with `flagsmith`, i.e. the name of our provider.

Next, let’s create some files:

[gagan@gpad flagsmith]$ ls  
models.go  models_test.go  provider.go  provider_test.go  resource_flag.go  resource_flag_test.go

We decided to create a file called models.go to hold our resource data struct instead of it being part of resource_flag.go which makes the abstraction a bit clearer, and we also decided to implement some helper methods on that resource data to make it interoperable with the data structures used by our golang library.

Because we have implemented To/From methods to convert data structure to and from our golang library, our plugin code looks much simple:

Now to answer your question; what about creating and deleting

Now create is a bit tricky since we do read and an update in place of create. In Flagsmith whenever we create a feature, the feature states for the all the environments are automatically created; hence we can't ‘create’ a feature state but in order to implement the create functionality we first read the feature state and if there are any changes between what we have read vs what is specified in terraform we perform an update.?

Let’s talk about delete first: Well for delete we do nothing, we just remove the resource from the state. The reason for this is that in Flagsmith, whenever we create a feature, the feature states for all the environments are automatically created, i.e: they live and die with the environment, hence it does not make sense to delete an environment feature state. The logic is pretty similar to why we do read and update instead of create.

In Flagsmith whenever we create a feature, the feature states for the all the environments are automatically created, i.e: they live and die (gets deleted) with the environment, hence it does not make sense to delete an environment feature state

func (r flagResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
	var data FlagResourceData

	diags := req.Config.Get(ctx, &data)
	resp.Diagnostics.Append(diags...)

	if resp.Diagnostics.HasError() {
		return
	}
	// First we read the resource by calling the `Read` method 
  // Manually 
	readResponse := resource.ReadResponse{State: resp.State}
	r.Read(ctx, resource.ReadRequest{
		State: tfsdk.State{
			Raw:    req.Plan.Raw,
			Schema: req.Plan.Schema,
		},
		ProviderMeta: req.ProviderMeta,
	}, &readResponse)

	if readResponse.Diagnostics.HasError() {
		resp.Diagnostics.Append(readResponse.Diagnostics...)
		tflog.Error(ctx, "Create: Error reading resource state")
		return
	}

	// Next, we call the `Update` method manually 
	updateResponse := resource.UpdateResponse{State: resp.State}
	r.Update(ctx, resource.UpdateRequest{
		Config:       req.Config,
		Plan:         req.Plan,
		State:        readResponse.State,
		ProviderMeta: req.ProviderMeta,
	}, &updateResponse)

	if updateResponse.Diagnostics.HasError() {
		resp.Diagnostics.Append(updateResponse.Diagnostics...)
		tflog.Error(ctx, "Create: Error updating resource state")
		return
	}
  // finally we set the state from update response
	resp.State = updateResponse.State
	resp.Diagnostics.Append(updateResponse.Diagnostics...)
}

Check it out!

Now you can use this provider https://registry.terraform.io/providers/Flagsmith/flagsmith/latest

and control feature flagging from your Terraform environment. Stay tuned for updates! We are doing a lot of work in this area to make our customer’s lives easier and move more of their infrastructure into code. 

Thanks,

Gagan