Disclaimer:The thoughts expressed in this case study are my own and do not necessarily reflect those of HashiCorp and their business. The designs, content and any other assets belong to the company in mention and should not be copied or re-used in any way. Some information and screenshots have been redacted for privacy and security purposes.
At a high level, the essence of the problem is users are required to do repetitive tasks and build non-trivial custom tooling when managing workspace variables. Which are a key piece when using Terraform Cloud. The solution was to allow users to create a variable one time and apply it to all or selected workspace. This project was in partnership with the engineering, product & design (EPD) teams. It took place over a span of 5 months while employed at HashiCorp and was launched into public beta in November 2021.
The Challenge
Managing Variables is Difficult
Workspace variables let you customize configurations, modify Terraform’s behavior, and store information like provider credentials. Terraform Cloud applies workspace variables to all runs within that workspace. Basically if you want to build infrastructure, use a webhook, api, etc. you must define those variables. Users have to define and rotate variables one-by-one for each workspace in the UI, for context some customers have thousands of workspaces. Some users have built custom pipelines to read and map variables from a large JSON file.
The Solution
Variable Reusability
We created a solution that allows users to create a collection of variables once, known as a variable set. Then apply that variable set to one, several or all of an organization’s workspaces. This enables users to manage, rotate or even revoke a variable once at any time.
Our guiding principles
- Credential management and rotation: Define once and rotate all credentials at the same time. Encourage adopting best practices.
- Ease of administration: Provide a variable set for your team to use for a workspace, a module or a set of credentials.
- Better developer ergonomics: Instead of looking for what variables to apply to your workspace or module, use pre-defined sets and optionally overwrite them.
- Integrate well with existing pipelines: Enable users to manage variable sets through the API.
The power of an EPD team
Roles & Responsibilities
I was the design lead for this project and partnered my product management counterpart to conduct research and co-author the product requirements.
This document was informed by research with internal and external customers during the initial discovery phase. Which also included an analysis of tools customers use that might imply a pre-existing mental model or pattern they would expect for a solution like this.
We did a few rounds of short brainstorming sessions to discuss potential solutions. This led to me hosting a design jam workshop with the team to explore potential solutions. I was also responsible for defining the critical user journey (CUJ) and hi-fi prototype that I used to run iterative usability testing around a potential solution.
The following goes more in-depth of the process and solution.
User Research
Data Informed Decisions
During our initial user research with users, we began reviewing the data to have a better understanding of what percentage of users would benefit from this type of feature. Essentially asking the question:
How many organizations currently have duplicate variables across all their workspaces?
We found that across all 4 tiers 78% – 97% had 0 – 20 variables per workspace.
It should be noted that not all of these variables would be categorized as reusable. Only variables with matching keys and values would be considered reusable. For privacy and security reasons we do not have access to the value, just the key. Which means there is a gap in the full picture of the data we found.
The outcome of this research was a problem summary document and the final product requirement document, which outlined the requirements and success metrics. This helped the entire EPD team gain a shared vision and understanding of the problem space. Which led to facilitated mini-brainstorming sessions and a design jam workshop exploring potential solutions.
Testing with customers
Iterative Usability Testing
Coming out of the early research and workshops I spent two weeks designing a hi-fi prototype that we could take to new customers and ones we’ve already spoken with. Walking them through our current thinking and proposed solution, essentially saying “this is what we heard you say”.
An early iteration led us down the path of exploring how we could offer recommendations or insights into organizations that already have a sprawl of variables being applied to their workspaces. The thinking was reducing the cognitive load and lowering the bar to adopt this feature. Ultimately the technical constraints we’d be met with and the low priority our customers put on a feature like this, we moved it to abandoned ideas. There was also a trust factor involved.
This phase allowed us to test and iterate on the direction we took for both the UI and API. As we exploited this iterative solutions with customers, one of the key theme we found was:
Users really need a much more simplified workflow for creating these new variable sets within the UI.
Key Themes
Permissions and Overwrites
Two of the key technical aspects we had to really focus on were the permission model and variable precedence.
Permission model
Variables are a key piece for Terraform Cloud so we needed to give all users visibility into variable sets at the organization level. This deviates from our existing permission model that only users with management permissions can view organization wide settings. After talking with our customers and understanding their problem more, we knew that was a non-negotiable requirement.
Ensuring security to these sensitive objects
Variable precedence
Variables are defined in a config file, API or the UI. When they are defined in a config file we don’t always have insight into those variables until a run has kicked off. This introduced a new layer of variables that can be applied to a workspace outside of the existing mechanism. Which means we now had to solve the variable priority problem. If variables are being applied to a workspace from several sources and there are conflicts, which one gets used… That is where scope and date applied come into play.
Understanding what variables are getting used
Getting it shipped
Pragmatic Designs
A project goal was to get this feature out into beta within 6 weeks of implementation. This required a pragmatic design approach to evaluate what components are available that we could reuse, even if it wasn’t an optimal user experience. Working closely with my engineering counterparts, we were able to identify similar patterns to reuse that would save on engineering effort. This approach was acceptable because we wanted to get this feature in front of users quickly and learn from their feedback. Our intention was to iterate and improve on this feature, which included this part of the user experience.
I also jumped in on some front end development work to help the team focus on the more complex priorities, as well as assisting in writing the docs and providing support for our learn platform.
Metrics and a release plan
What Defines Success
A feature isn’t ready for external beta testing until unless we have the metrics defined, the product tool’d, and the api and documentation are ready.
Success metrics are used to attempt to measure what matters in a product, but it’s also important to understand that what we measure is a proxy of what matters. What are we trying to achieve with building this feature?
- Reduce manual toil on rotating credentials on the UI.
- We want to provide a better mechanism to make variables more manageable in the context of environment/component creation and updates.
- We want to improve the UX of variables by providing a path to better visibility and audibility.
We attempt to measure the outcome in the following ways.
Quantitative
We expected to see a big drop in the amount of users manually entering variables one-by-one within the UI.
- Number of organizations start using variable sets.
- Track how an organization created a variable set and what type of variables they used.
- Number of variable sets that are created by users, including using the API, Terraform provider and UI.
- Average number of workspaces connected to a variable set.
- Percentage of new workspaces within an organization that are sharing variables.
- Percentage of new variables created that are part of a variable set.
Qualitative
Verify customers who previously experienced the variable reusability pain points no longer have these issues.
Acting on Feedback
Early Indicators
Once we announced our public beta we started to get an influx of praises, feature requests and a few U and technical bugs we were able to act on fairly quickly.
Highlighting one specific UX issue was the order in which we placed the variable category selector. This is a new data point users are not used to so they glossed right over it, causing unknown issues when performing actions within Terraform Cloud. To address this issue we added a badge and restructured the form to read from top to bottom instead of left to right. This does break our current form patterns, but the value add warranted the deviation.
As we continue through our public beta phase we will continue to refine the feature and prioritize high impact follow up work based on customer feedback and ongoing followup customer interview. We are seeing a few key themes.
Team Members
This work was done while employed by HashiCorp, working in on an EPD (engineering, product and design) team.
