variable-reuse-main-hero-img

Disclaimer:The thoughts expressed in this case study are my own and do not necessarily reflect those of HashiCorp and their business. The designs, content and any other assets belong to the company in mention and should not be copied or re-used in any way. Some information and screenshots have been redacted for privacy and security purposes.

  • RoleSr. Product Designer

  • DurationQ3 2021

  • TeamTerraform @ HashiCorp

Creating and managing variables can be an overwhelming task, as it can create vulnerabilities within your system

At a high level, the essence of the problem is users are required to do repetitive tasks and build non-trivial custom tooling when managing workspace variables. Which are a key piece when using Terraform Cloud. The solution was to allow users to create a variable one time and apply it to all or selected workspace. This project was in partnership with the engineering, product & design (EPD) teams. It took place over a quarter while employed at HashiCorp and was launched into public beta in November 2021.

Managing the same variable across multiple workspaces within an organization
Managing the same variable across multiple workspaces within an organization

The Challenge

Managing Variables is Difficult

​​Workspace variables let you customize configurations, modify Terraform’s behavior, and store information like provider credentials. Terraform Cloud applies workspace variables to all runs within that workspace. Basically if you want to build infrastructure, use a webhook, api, etc. you must define those variables. Users have to define and rotate variables one-by-one for each workspace in the UI, for context some customers have thousands of workspaces. Some users have built custom pipelines to read and map variables from a large JSON file.

Managing one variable set that gets applied to multiple workspaces
Managing one variable set that gets applied to multiple workspaces

The Solution

Variable Reusability

We created a solution that allows users to create a collection of variables once, known as a variable set. Then apply that variable set to one, several or all of an organization’s workspaces. This enables users to manage, rotate or even revoke a variable once at any time.

Our guiding principles

  • Credential management and rotation: Define once and rotate all credentials at the same time. Encourage adopting best practices.
  • Ease of administration: Provide a variable set for your team to use for a workspace, a module or a set of credentials.
  • Better developer ergonomics: Instead of looking for what variables to apply to your workspace or module, use pre-defined sets and optionally overwrite them.
  • Integrate well with existing pipelines: Enable users to manage variable sets through the API.
Managing variables today and managing variable sets tomorrow

The power of an EPD team

Roles & Responsibilities

I was the design lead for this project and partnered my product management counterpart to conduct research and co-author the product requirements.

This document was informed by research with internal and external customers during the initial discovery phase. Which also included an analysis of tools customers use that might imply a pre-existing mental model or pattern they would expect for a solution like this.

Triad: Engineering, Product and Design collaborating from the start
Triad: Engineering, Product and Design collaborating from the start

We did a few rounds of short brainstorming sessions to discuss potential solutions. This led to me hosting a design jam workshop with the team to explore potential solutions. I was also responsible for defining the critical user journey (CUJ) and hi-fi prototype that I used to run iterative usability testing around a potential solution.

The following goes more in-depth of the process and solution.

Variable reuse critical user journey

User Research

Data Informed Decisions

During our initial user research with users, we began reviewing the data to have a better understanding of what percentage of users would benefit from this type of feature. Essentially asking the question:

How many organizations currently have duplicate variables across all their workspaces?

We found that across all 4 tiers 78% – 97% had 0 – 20 variables per workspace.

It should be noted that not all of these variables would be categorized as reusable. Only variables with matching keys and values would be considered reusable. For privacy and security reasons we do not have access to the value, just the key. Which means there is a gap in the full picture of the data we found.

Identified key personas
Identified key personas

The outcome of this research was a problem summary document and the final product requirement document, which outlined the requirements and success metrics. This helped the entire EPD team gain a shared vision and understanding of the problem space. Which led to facilitated mini-brainstorming sessions and a design jam workshop exploring potential solutions.

Design Jam Workshop With Cross Functional Teams

Testing with customers

Iterative Usability Testing

Coming out of the early research and workshops I spent two weeks designing a hi-fi prototype that we could take to new customers and ones we’ve already spoken with. Walking them through our current thinking and proposed solution, essentially saying “this is what we heard you say”.

Detect and parse variables from state or terraform config file
Detect and parse variables from state or terraform config file

An early iteration led us down the path of exploring how we could offer recommendations or insights into organizations that already have a sprawl of variables being applied to their workspaces. The thinking was reducing the cognitive load and lowering the bar to adopt this feature. Ultimately the technical constraints we’d be met with and the low priority our customers put on a feature like this, we moved it to abandoned ideas. There was also a trust factor involved.

Early exploration of variable set insights and recommendations
Early exploration of variable set insights and recommendations

This phase allowed us to test and iterate on the direction we took for both the UI and API. As we exploited this iterative solutions with customers, one of the key theme we found was:

Users really need a much more simplified workflow for creating these new variable sets within the UI.

A four step process of creating, applying and confirming variable sets.
A four step process of creating, applying and confirming variable sets.
Add variables 1 by 1, bulk copy & paste, upload/drag & drop
Add variables 1 by 1, bulk copy & paste, upload/drag & drop
Apply variable sets by tag selection

Key Themes

Permissions and Overwrites

Two of the key technical aspects we had to really focus on were the permission model and variable precedence.

Permission model

Variables are a key piece for Terraform Cloud so we needed to give all users visibility into variable sets at the organization level. This deviates from our existing permission model that only users with management permissions can view organization wide settings. After talking with our customers and understanding their problem more, we knew that was a non-negotiable requirement.

Ensuring security to these sensitive objects

Variable precedence

Variables are defined in a config file, API or the UI. When they are defined in a config file we don’t always have insight into those variables until a run has kicked off. This introduced a new layer of variables that can be applied to a workspace outside of the existing mechanism. Which means we now had to solve the variable priority problem. If variables are being applied to a workspace from several sources and there are conflicts, which one gets used… That is where scope and date applied come into play.

Understanding what variables are getting used

Variable precedence hierarchy
Variable precedence hierarchy
Users are informed when variables are overwritten and what variable is doing the overwriting.
Users are informed when variables are overwritten and what variable is doing the overwriting.
Conflicting global variable sets
Conflicting global variable sets
Meeting Users Where They Are - Sample payload for creating a variable set via API
Meeting Users Where They Are - Sample payload for creating a variable set via API

Getting it shipped

Pragmatic Designs

A project goal was to get this feature out into beta within 6 weeks of implementation. This required a pragmatic design approach to evaluate what components are available that we could reuse, even if it wasn’t an optimal user experience. Working closely with my engineering counterparts, we were able to identify similar patterns to reuse that would save on engineering effort. This approach was acceptable because we wanted to get this feature in front of users quickly and learn from their feedback. Our intention was to iterate and improve on this feature, which included this part of the user experience.

I also jumped in on some front end development work to help the team focus on the more complex priorities, as well as assisting in writing the docs and providing support for our learn platform.

Displaying Workspace Specific Variables and Variable Sets Applied to a workspace
Displaying Workspace Specific Variables and Variable Sets Applied to a workspace
Improving the ability to consume variable sets applied to a workspace.
Improving the ability to consume variable sets applied to a workspace.
Variable Set Creation and Management
Variable Set Creation and Management

Metrics and a release plan

What Defines Success

A feature isn’t ready for external beta testing until unless we have the metrics defined, the product tool’d, and the api and documentation are ready.

Success metrics are used to attempt to measure what matters in a product, but it’s also important to understand that what we measure is a proxy of what matters. What are we trying to achieve with building this feature?

  • Reduce manual toil on rotating credentials on the UI.
  • We want to provide a better mechanism to make variables more manageable in the context of environment/component creation and updates.
  • We want to improve the UX of variables by providing a path to better visibility and audibility.
Early feedback from public beta
Early feedback from public beta

We attempt to measure the outcome in the following ways.

Quantitative

We expected to see a big drop in the amount of users manually entering variables one-by-one within the UI.

  • Number of organizations start using variable sets.
  • Track how an organization created a variable set and what type of variables they used.
  • Number of variable sets that are created by users, including using the API, Terraform provider and UI.
  • Average number of workspaces connected to a variable set.
  • Percentage of new workspaces within an organization that are sharing variables.
  • Percentage of new variables created that are part of a variable set.

Qualitative

Verify customers who previously experienced the variable reusability pain points no longer have these issues.

Early adoption data points from public beta
Early adoption data points from public beta

Acting on Feedback

Early Indicators

Once we announced our public beta we started to get an influx of praises, feature requests and a few U and technical bugs we were able to act on fairly quickly.

As we continue through our public beta phase we will continue to refine the feature and prioritize high impact follow up work based on customer feedback and ongoing followup customer interview. We are seeing a few key themes.

Create and apply variable sets through conditional statements of tags, teams and prefix.
Create and apply variable sets through conditional statements of tags, teams and prefix.