Onboarding an Enterprise onto a New Stack
Tech Stack
Early on as a consultant at Insight, my team and I were tasked with onboarding an existing enterprise onto a number of platforms in a thoughtful and systematic way. The new stack included Terraform Cloud, Google Cloud Platform, Terraform as an IaC tool, and GitHub as the primary git version control system. None of these tools were being used at scale within the organization individually, and there wasn't widespread experience or exposure to any of them within the teams we supported, so the task was to learn these platforms, build a robust governance and observability framework that enabled and didn't block, figure out the best way for them to interact with each other, understand how to implement them at scale, and train and onboard teams across the stack.
Naturally, the intention of this stack was to enable teams to move from click-ops to a more robust, trackable solution for their infrastructure management. Github repos would be the source of truth for their infra code, Terraform Cloud would host their Terraform state file, and Terraform HCL would be their sole IaC solution.
Anyone whose worked on a project like this, though, understands all too well that a stack of expensive SaaS products doesn't a solution make. Our team was tasked with driving ease of use and adoption among app teams across the organization, and effort that took a remarkable amount of thought and solutioning. Turning a disjointed grouping of products into something that would accelerate and strengthen the infra deployment process across 300+ applications was a true learning experience.
Starting Principles
Firstly, we had to figure out how to govern each tool individually with regards to access, and also how each tool would reliably interact with others programmatically. Before diving into either, we set a few standards we knew our solutions needed to follow, including the standard that everything had to be git-backed, automated, repeatable, and programmatic. In practice, that meant when a user on an app team needed something, there was already a clear path they could follow and a clear pattern they fell into.
Finding the Lines So We Could Color Inside
Before we could design any governance model, we had to understand the organizational units of each platform and how configurations mapped to them. Teams, repositories, and orgs in GitHub; projects, teams, workspaces, and variable sets in Terraform Cloud; projects, folders, workload identity federation, service agents, and shared networking services in GCP.
We then had to assemble a catalog of the constraints each platform imposed. That process started with naming conventions: which characters were allowed (including what a name could start and end with), whether names were case sensitive and where, whether they needed to be globally unique or only unique within an org, and whether they could be changed after creation. GCP project IDs, for example, are limited to 6 to 30 characters, must be lowercase, must be globally unique, and are permanent once assigned. Terraform Cloud workspaces and GitHub repositories, on the other hand, allow longer names, only need to be unique within their org, and can be renamed, because each one is assigned an immutable internal ID at the time of creation. Workload Identity Federation added another wrinkle. The google.subject attribute has a hard limit of 127 bytes, and since Terraform Cloud builds the token's subject from a combination of the organization, project, workspace, and run phase, long workspace names would push it past that limit. That effectively capped how long Terraform Cloud workspace names could be.
Because a single identifier often travels across all three platforms (a GCP project backed by a Terraform Cloud workspace, whose code lives in a GitHub repository), the most restrictive rule on any one platform would effectively became a limitation for all of them. And where that wasn't practical, we had to document how the names related to each other.
Since Terraform would be our source of truth, we codified these naming limitations as regex patterns. I've shared some of them below.
We also documented structural limits, such as how many projects a single folder can hold, how deeply folders can be nested, and how many workspaces or teams an organization can support. Those ceilings would dictate how far any hierarchy we designed could scale before it'd have to be reorganized. One handy piece of information that isn't widely shared is that there's essentially no limit to how many workspaces a single Terraform Cloud org can host. At least, I haven't found the operational ceiling yet. The same can't be said for how many resources a single workspace's state file can manage. Once you get above 2,500 resources, things start to get cumbersome, and past 5,000, you're bound to see plans and applies degrade with wait times moving towards 10-15 minutes.
Finally, we mapped the metadata options available on each platform and where their limits sat. GCP offers both labels and tags, and they behave very differently. Labels are simple key-value pairs attached to individual resources and aren't inherited, while tags are centrally defined, inherited down the resource hierarchy, and can be referenced by IAM and organization policies. Terraform Cloud supports workspace tags for filtering and grouping, and GitHub offers repository topics and organization-level custom properties. Each metadata type came with its own caps on the number of keys, the length of keys and values, and the allowed character sets. Understanding these differences told us which platform could carry which piece of governance information, such as ownership, cost center, environment, or data classification, and where we'd need to enforce consistency ourselves because no single platform could do it for us. Again, where possible, we codified this into our Terraform. Those limits directly reshaped the design. We initially planned to nest folders under a single top-level folder per line of business in GCP, but the 300 folder-in-folder limit ruled that out. Rather than build a deeper LOB-based hierarchy to work around it, we went the other direction -- each app code got its own folder under a top-level folder, and we used round robin to distribute app folders across several top-level folders instead of grouping them by LOB.
Platform Limits at a Glance
| GitHub | Google Cloud Platform | Terraform Cloud | |
|---|---|---|---|
| Org unit hierarchy | Organization > Team > Repository (flat) | Organization > Folder (up to 10 levels deep) > Project | Organization > Project > Workspace |
| Naming characters | ASCII letters, digits, . - _ | Project ID: lowercase letters, digits, hyphens; must start with a letter, no trailing hyphen | Workspace: letters, digits, hyphens, underscores |
| Naming length | Repository: up to 100 characters | Project ID: 6 to 30 characters | Workspace: 90 characters or fewer on its own, but when used with GCP Workload Identity Federation, the org, project, and workspace names combined can't exceed 78 bytes |
| Naming regex | ^[A-Za-z0-9._-]{1,100}$ | ^[a-z][a-z0-9-]{4,28}[a-z0-9]$ | Workspace: ^[A-Za-z0-9_-]{1,90}$. Full WIF subject: ^(?=.{1,127}$)organization:[^:]+:project:[^:]+:workspace:[A-Za-z0-9_-]+:run_phase:(plan|apply)$ |
| Case sensitivity | Case is preserved in the name, but uniqueness and URLs are case-insensitive | Lowercase only | Case is preserved in the name |
| Uniqueness scope | Unique within the org or user account | Project ID: globally unique across all of GCP | Unique within the org |
| Renamable | Yes, with automatic redirects from the old name | Project ID: no. Display name: yes | Yes |
| Immutable internal ID | Numeric repository ID | Project number | Workspace ID (ws- prefix) |
| Structural limits | None documented at this level | A folder can't hold more than 300 direct child folders; no per-folder project limit, but projects count against org and billing quotas | No practical workspace limit observed per org; state becomes cumbersome above about 2,500 resources per workspace |
| Labels / tags | Topics (up to 20 per repo, lowercase, up to 50 characters each) and org-level custom properties for key-value metadata | Labels: up to 64 per resource, keys and values up to 63 characters, not inherited. Tags: centrally defined, inherited, usable in IAM and org policy conditions | Tags up to 255 characters; letters, digits, colons, hyphens, underscores |
| Identity / federation constraints | OIDC token claims include repo and org names | google.subject mapping capped at 127 bytes; WIF pool and provider IDs 4 to 32 characters | Dynamic credentials build the token subject from org, project, workspace, and run phase, so long names can exceed GCP's 127-byte subject limit |
For any resource type that goes into a pending-deletion or soft-delete state after it's removed, I tend to prefer to append a random ID to the end of the name. This will come up more often than you'd expect. GCP projects sit in a pending-deletion state for 30 days, and their IDs can't be reused, even after they've been fully purged. Azure subscriptions, Key Vaults, and several other resource types behave in similar ways. If part of your provisioning process fails partway through, often because proper idempotency hasn't been achieved just yet, a straightforward retry could get blocked by a resource still sitting in that to-be-deleted state under the name needed. Appending a random suffix means a failed or retried creation won't collide with whatever the last attempt left behind.
There's several other reasons to consider random suffixes, in moderation. Some resources, like GCP project IDs and Cloud Storage buckets, have to be globally unique, so a name that seems safe inside your org might already be taken elsewhere across GCP. The catch is that those characters aren't free. Every character in a random suffix cuts into your naming budget. You'll find that length budgets get tight quickly.
Personally, I've always been partial to random_pet. Instead of a hex string, it generates readable names like quick-otter, and they're much easier to spot in a console, a log, etc. Again, just keep an eye on length, since two or three words plus separators can eat into a tight limit fast.
Importing resources that use random IDs into Terraform used to be a little tricky, but there's a way around it. You can import a random_id resource using its b64_url value, with an optional prefix, which lets you bring an existing name under Terraform's management without triggering a diff.
Access and Governance, Platform by Platform
With the platform shapes and constraints understood, the next round of research focused on how teams and roles mapped to each org unit on each platform, and how we'd manage privileged access when it was needed.
The approach centered on Entra ID. Each app code got a base group for everyday access, plus several privileged groups aligned to each environment. Those privileged groups plugged into Privileged Identity Management, so teams could get time-boxed, approved access to higher permissions instead of standing access.
Automating the creation of all these groups turned out to be a body of work in itself. That wasn't because it was technically hard, but because Entra ID was a platform our team didn't own, and the number of groups needed to scale up very quickly to support this model. We had to work closely with the team that did own it to define a process and pattern for creating and maintaining groups at that scale. After multiple iterations, we landed on an API that gave us a limited ability to create the groups we needed, within a very well-defined structure. That let our automation move at the pace the model required without giving us broad control over a directory we didn't own.
Group lifecycle is a harder problem, and it'll likely always be a sensitive one for any enterprise. Removing a group's access, or deleting the group entirely, could cut off real users from the resources they depend on. Every enterprise will approach this differently, but at a minimum, there need to be multiple overlapping validation steps so nothing gets removed without notice.
Once the groups existed in Entra ID, they had to reach each platform. In GCP, groups synced from Entra ID into Google Cloud, where they could be bound to IAM roles on folders and projects. That sync ran on a schedule and typically took 30 minutes to an hour to land. In Terraform Cloud, we used SAML SSO with team mapping, so a user's Entra group membership determined which Terraform Cloud teams they landed in. We also managed some teams directly in Terraform. GitHub used Entra team sync, which kept GitHub team membership aligned with the corresponding Entra groups.
The privileged groups were specific to both environment and access level. I won't go further into that design here for security reasons, and the right environment breakdown will vary from one enterprise to the next anyway.
PIM itself worked well and without much issue. Activations were typically auto-approved for requests under a certain duration, so teams could get elevated access quickly without waiting on someone to sign off, while longer requests still needed approval. The main quirk was that access didn't always show up right away. After activating, you'd often have to refresh the console several times before the new permissions took effect.
In Terraform Cloud, we didn't want to give teams direct access to state, since state files can contain secrets in plain text. We addressed this with custom team permissions. Teams could trigger speculative plans from their local CLI, but they couldn't run applies, and they had no direct access to state. Any code that'd actually be applied had to go through a defined PR approval process in GitHub, and it was applied to the team's workspace at merge time. That meant the apply was gated by the PR process, not by anything a person could trigger by hand in Terraform Cloud.
GitHub didn't follow the same tiered, privileged-access pattern, since its permission model is flatter. Instead, teams were code owners of their own repositories, and every change needed two approvals from within their own team before it could merge.
How GitHub and Terraform Cloud Talk to Each Other
One of the earliest and hardest decisions was whether to run on Terraform Cloud's native VCS-connected workflow, which links individual GitHub repos and branches directly to a workspace, or build a custom process on Terraform Cloud's CLI-driven workflow, triggered from GitHub Actions.
| VCS-connected workflow | Custom CLI-driven workflow | |
|---|---|---|
| Time to stand up | Fast, ready on day one | Slower, required building and maintaining our own tooling |
| Vendor support | Supported directly by HashiCorp | Owned entirely by our team |
| Plan visibility on PRs | Built in from day one | Had to be built ourselves |
| GitHub-side validation of plan and apply | Limited | Whatever we chose to build |
| Room for future growth | Limited | Open |
| Authentication to Terraform Cloud | Handled by the VCS OAuth connection; no secrets for us to manage | Required a secret management solution for team tokens |
| Ongoing maintenance burden | Low | High: reusable GitHub Actions workflows, per-workspace tokens |
We decided to build and maintain our own reusable Terraform workflow rather than use the native VCS connection. Each Terraform Cloud workspace got its own unique, scoped team token, and workspaces were driven through a versioned Terraform Cloud action in GitHub Actions. That put the burden of secret management and reusable workflow maintenance on our team, but in exchange we kept the scalability and flexibility to grow the process later.
A developer opens a PR, which triggers our reusable GitHub Actions workflow. That workflow pulls a team token from the secret store and calls the Terraform Cloud workspace to run a plan, which is posted back to the PR for review. Once the PR is approved and merged, the same workflow triggers an apply against the workspace, which lands the changes in GCP.
Each workspace's team token was scoped to that workspace alone, so a leaked token couldn't be used to apply against another team's infrastructure.
Authenticating Terraform Cloud into GCP
We started with individual, unique service account keys per workspace, but quickly determined that would be a wildly problematic approach at scale -- a static key to create, rotate, and secure for every workspace, and a growing blast radius if any one of them leaked.
We moved to workload identity federation instead, built as environment-specific pools in GCP. That let a Terraform Cloud workspace authenticate into GCP automatically, with no static credential for us to manage or rotate.
This paired naturally with the per-workspace team token model on the Terraform Cloud side. The team token identified and scoped the workspace within Terraform Cloud, and workload identity federation handled the trust from that workspace into GCP, so neither side of the authentication chain depended on a long-lived secret we had to rotate by hand.
Was the Flexibility Worth It?
Looking back, I'd call it close to even rather than a clear win. We spent a lot of time in the first few months testing and iterating on the reusable Terraform Cloud workflow before it was solid, and the PR-visible plan output that the native VCS connection gave us on day one took us a few years to reproduce at a comparable level of readability. The flexibility we bet on paid off eventually, but the upfront cost was real and lasted longer than we expected.
Management Layers: Who Owns What
One layer of complexity a platform team might struggle with early on is separating management responsibilities for org units across platforms. Every org unit on every platform has a decision buried in it: which team's the owner, which team's a contributor, who can delete it, and who can request that it be deleted.
Hold too much control centrally, and app teams feel like they don't really own their space. Hold too little, and the platform team loses the ability to intervene, audit, or reclaim anything when something goes wrong.
The platform team needs to keep ultimate ownership of every org unit, across every platform, without that ever getting in the way of app teams feeling like their space is really theirs.
Core, Factory, and Leased
One pattern that works across every platform splits ownership into three tiers:
- Core: resources the platform team directly owns, manages, and uses itself. This includes the org-level governance the whole enterprise depends on, such as org policies, custom roles, workload identity federation pools, and shared networking (often in partnership with a networking team), along with the projects that host the platform team's own tooling.
- Factory: the layer that creates everything else. This covers the app-level folders, the automated creation of new projects, and the foundational resources and configuration inside them.
- Leased: the projects handed off to app teams, who get a clear path to view, deploy into, and use them through IaC and their own GitHub repository.
In GCP:
| Tier | What it covers |
|---|---|
| Core | Shared network services, org policies, org custom roles, workload identity federation pools, top-level folders |
| Factory | App-level folders, projects, and high-level project configuration, including which services are turned on by default |
| Leased | App-team projects, created by the factory and handed off with a clear IaC and GitHub-based path to view, deploy into, and use |
In GitHub:
| Tier | What it covers |
|---|---|
| Core | GitHub teams, GHAS policies, reusable workflows |
| Factory | Branch protections, team access to individual repositories |
| Leased | Repository codebases and individual branches |
In Terraform Cloud:
| Tier | What it covers |
|---|---|
| Core | Projects, team tokens, variable sets, Terraform agents and runners, org tokens, Sentinel policies |
| Factory | Workspace creation and configuration, workspace-specific team tokens, team access to workspaces, state files |
| Leased | The workspaces themselves, which teams deploy into through the GitHub PR process |
Even on a leased project, the platform team keeps the ability to update the factory-created resources inside it. It owns those resources, so it can add or remove features across them through the factory itself, which should be built to roll changes out across every factory-created resource at once instead of one at a time. That's the ownership the platform team holds onto on a leased project. It isn't day-to-day control, but the standing ability to evolve the foundation underneath a team without needing their permission first.