Simpler.Grants.gov Public Wiki
Grants.govSimpler.Grants.govGitHubDiscourse
  • πŸ‘‹Welcome
  • GET INVOLVED
    • Why open source?
    • How to contribute code
    • How to file issues
      • Report a bug
      • Request a feature
      • Report a security vulnerability
    • Community guidelines
      • Code of Conduct
      • Reporting and removing content
      • Incident response protocol
    • Community events
      • Fall 2024 Coding Challenge
        • Event Submissions & Winners
      • Spring 2025 Collaborative Coding Challenge
        • Event Submissions & Winners
    • Communication channels
  • Product
    • Roadmap
    • Deliverables
      • 🏁Static site soft launch
      • 🏁Static site public launch
      • 🏁GET Opportunities
      • 🏁Open source onboarding
      • 🏁Co-Design Group planning
    • Decisions
      • ADR Template
      • ADRs
        • Dedicated Forum for Simpler.Grants.gov Community
        • Recording Architecture Decisions
        • Task Runner for the CI / CD Pipeline
        • API Language
        • Use Figma for design prototyping
        • ADR: Chat
        • DB Choices
        • API Framework and Libraries
        • Back-end Code Quality Tools
        • Front-end Language
        • Communications Tooling: Wiki Platform
        • Use Mural for design diagrams and whiteboarding
        • Ticket Tracking
        • Front-end Framework
        • Front-end Code Quality Tools
        • Front-end Testing & Coverage
        • Backend API Type
        • Front-end Testing & Coverage
        • Deployment Strategy
        • Use U.S. Web Design System for components and utility classes
        • FE server rendering
        • Use NPM over Yarn Architectural Decision Records
        • U.S. Web Design System in React
        • Communications Tooling: Video Conferencing
        • Back-end Production Server
        • Communications Tooling: Analytics Platform
        • Commit and Branch Conventions and Release Workflow
        • Cloud Platform to Host the Project
        • Infrastructure as Code Tool
        • Data Replication Strategy & Tool
        • HHS Communications Site
        • Communications Tooling: Email Marketing
        • Communications Tooling: Listserv
        • Use Ethnio for design research
        • Uptime Monitoring
        • Database Migrations
        • 30k ft deliverable reporting strategy
        • Public measurement dashboard architecture
        • Method and technology for "Contact Us" CTA
        • E2E / Integration Testing Framework
        • Logging and Monitoring Platform
        • Dashboard Data Storage
        • Dashboard Data Tool
        • Search Engine
        • Document Storage
        • Document Sharing
        • Internal Wiki ADR
        • Shared Team Calendar Platform
        • Cross-Program Team Health Survey Tool
        • Adding Slack Users to SimplerGrants Slack Workspace
        • Repo organization
        • Internal knowledge management
        • Migrate Existing API Consumers
      • Infra
        • Use markdown architectural decision records
        • CI/CD interface
        • Use custom implementation of GitHub OIDC
        • Manage ECR in prod account module
        • Separate terraform backend configs into separate config files
        • Database module design
        • Provision database users with serverless function
        • Database migration architecture
        • Consolidate infra config from tfvars files into config module
        • Environment use cases
        • Production networking long term state
    • Analytics
      • Open source community metrics
      • API metrics
  • DESIGN & RESEARCH
    • Brand guidelines
      • Logo
      • Colors
      • Grid and composition
      • Typography
      • Iconography
      • Photos and illustrations
    • Content guidelines
      • Voice and tone
      • Editoral style
      • Terminology usage
    • User research
      • Grants.gov archetypes
  • REFERENCES
    • Glossary
  • How to edit the wiki
Powered by GitBook
On this page
  • Context
  • Notes
  • Benefits of separate tfvars and tfbackend files
  • Cons of separate tfvars and tfbackend files

Was this helpful?

Edit on GitHub
  1. Product
  2. Decisions
  3. Infra

Separate terraform backend configs into separate config files

  • Status: accepted

  • Deciders: @lorenyu @shawnvanderjagt @kyeah @bneutra

  • Date: 2023-05-09

Context

Up until now, most projects adopted an infrastructure module architecture that is structured as follows: Each application environment (prod, staging, etc) is a separate root module that calls a template module. The template module defines all the application infra resources needed for an environment. Things that could be different per environment (e.g. desired ECS task count) are template variables, and each environment can have local vars (or somewhat equivalently, a tfvars file) that customizes those variables. Importantly, each environment has it’s own backend tfstate file, and the backend config is stored in the environment module’s main.tf.

An alternative approach exists to managing the backend configs. Rather than saving the backend config directly in main.tf, main.tf could contain a partial configuration, and the rest of the backend config would be passed in during terraform init with a command like terraform init --backend-config=prod.s3.tfbackend. There would no longer be a need for separate root modules for each environment. What was previously the template module would instead act as the root module, and engineers would work with different environments solely through separate tfbackend files and tfvar files. Doing this would greatly simplify the module architecture at the cost of some complexity when executing terraform commands due to the extra command line parameters. To manage the extra complexity of running terraform commands, a wrapper script (such as with Makefile commands) can be introduced.

The approach can be further extended to per-environment variable configurations via an analogous approach with variable definitions files which can be passed in with the -var-file command line option to terraform commands.

Notes

For creating accounts, can't use the .tfbackend backend config file approach because the main.tf file can only have one backend configuration, so if we have the backend configuration as a partial configuration of backend "s3" {}, then we can't use that same module to configure a new account, since the process for configuring a new account requires setting the backend configuration to backend "local" {}. We could have a separate duplicate module that's has backend set to local. or we could also temporarily update the backend from "s3" to "local", but both of those approaches seem confusing.

Another alternative is to go back to the old way of bootstrapping an account i.e. to do it via a script that creates an S3 bucket via AWS CLI. The bootstrap script would only do the minimal configuration for the S3 bucket, and let terraform handle the remainder of the configuration, such as creating the dynamodb tables. At this point, there is no risk of not having state locking in place since the account infrastructure has not yet been checked into the repository. This might be the cleanest way to have accounts follow the same pattern of using tfbackend config files.

Benefits of separate tfvars and tfbackend files

  • Reduce risk of differences between environments – When different environments have their own root modules, development teams have historically sometimes added one-off resources to specific environments without adding those resources to the template module and without realizing that they're violating an important goal of having multiple environments – that environments are isolated from each other but function identically. This creates differences between environments that are more than just configuration differences. By forcing the differences to be limited to the .tfvars (-var) file, it limits how badly someone can get an environment out of skew.

  • DRY backend configuration – With only a single module, there is less duplication of infrastructure code in the main.tf file. In particular, provider configurations, shared partial backend configuration, and certain other top level local variables and data resources no longer need to be duplicated across environments, and provider versions can also be forced to be consistent.

  • Make receiving updates from template-infra more robust – Previously, in order for a project to receive updates from the template-infra repo, the project would copy over template files but then revert files that the project has changed. Currently, the many main.tf root module files in the template are expected to be changed by the project since they define project specific backend configurations. With the separation of config files, projects are no longer expected to change the main.tf files, so the main.tf files in infra/app/build-repository/, infra/project-config/, infra/app/app-config/, etc. can be safely copied over from template-infra without needing to be reverted.

  • Reduce the cost of introducing additional infrastructure layers – In the future we may want too add new infrastructure layers that are created and updated independently of the application layer. Examples include a network layer or a database layer. We may want to keep them separate so that changes to the application infrastructure are isolated from changes to the database infrastructure, which should occur much less frequently. Previously, to add a new layer such as the database layer, we would need two additional folders: a db-env-template module and a db-envs folder with separate root modules for each environment. This mirrors the same structure that we have for the application. With separate backend config and tfvar files we would only need a single db module with separate .tfbackend and .tfvars files for each environment.

Cons of separate tfvars and tfbackend files

  • Extra layer of abstraction – The modules themselves aren't as simple to understand since the configuration is spread out across multiple files, the main.tf file and the corresponding .tfvars and .tfbackend file, rather than all in one main.tf file.

  • Requires additional parameters when running terraform – Due to configuration being separated into .tfvars and .tfbackend files, terraform commands now require a -var-file and -backend-config command line options. The added complexity may require a wrapper script, introducing yet another layer of abstraction.

PreviousManage ECR in prod account moduleNextDatabase module design

Last updated 1 year ago

Was this helpful?