How It Works

This guide explains TerraCi's internal architecture and data flow.

Overview

TerraCi processes your Terraform project in four stages:

Stage 1: Module Discovery

TerraCi scans your directory structure to find Terraform modules.

How It Works

1. Walk the directory tree starting from project root
2. Look for directories at the configured depth containing .tf files
3. Parse the path into named segments based on the configured pattern

The pattern is configurable (e.g., {service}/{environment}/{region}/{module}), and segment names determine the keys stored in the module's components map.

Example

platform/stage/eu-central-1/vpc/main.tf
   │       │         │       │
   │       │         │       └── segment "module": vpc
   │       │         └── segment "region": eu-central-1
   │       └── segment "environment": stage
   └── segment "service": platform

Module ID

Each module's ID is its relative path: platform/stage/eu-central-1/vpc

This ID is used for:

• Dependency matching
• Job naming
• State file path resolution

Stage 2: HCL Parsing

TerraCi parses each module's .tf files to extract dependencies.

What Gets Parsed

1. terraform_remote_state blocks - Primary dependency source
2. locals blocks - Variable resolution for dynamic paths

Remote State Example

data "terraform_remote_state" "vpc" {
  backend = "s3"
  config = {
    bucket = "my-state"
    key    = "platform/stage/eu-central-1/vpc/terraform.tfstate"
  }
}

TerraCi extracts:

• Backend type: s3
• State path: platform/stage/eu-central-1/vpc/terraform.tfstate
• Resolved module: platform/stage/eu-central-1/vpc

Path Resolution

TerraCi resolves variables in state paths:

locals {
  env = "stage"
}

data "terraform_remote_state" "vpc" {
  config = {
    key = "platform/${local.env}/eu-central-1/vpc/terraform.tfstate"
  }
}

Becomes: platform/stage/eu-central-1/vpc/terraform.tfstate

for_each Handling

When for_each is present, TerraCi expands to multiple dependencies:

data "terraform_remote_state" "services" {
  for_each = toset(["auth", "api", "web"])
  config = {
    key = "platform/stage/eu-central-1/${each.key}/terraform.tfstate"
  }
}

Creates dependencies on: auth, api, web modules.

Stage 3: Graph Building

TerraCi builds a Directed Acyclic Graph (DAG) of module dependencies.

Algorithm

1. Create a node for each discovered module
2. Add edges from each module to its dependencies
3. Detect cycles (error if found)
4. Topologically sort using Kahn's algorithm

Topological Sort

Kahn's algorithm ensures modules are ordered so dependencies come first:

Output order:

• Level 0: vpc
• Level 1: eks, rds (parallel)
• Level 2: app

Execution Levels

Modules are grouped into levels for parallel execution:

Level	Modules	Can Run In Parallel
0	vpc	Yes (no deps)
1	eks, rds	Yes (same deps)
2	app	After level 1

Cycle Detection

TerraCi detects circular dependencies:

Error message:

Error: circular dependency detected
  vpc -> eks -> app -> vpc

Stage 4: Pipeline Generation

TerraCi generates CI pipeline configuration from the sorted module graph. The provider is selected via TERRACI_PROVIDER, auto-detected from the environment (GITLAB_CI env var selects GitLab, GITHUB_ACTIONS selects GitHub Actions), or inferred from a single active provider.

Job Generation

For each module, TerraCi generates:

• Runs terraform plan -out=plan.tfplan
• Saves plan as artifact

2. Apply job
   • Depends on plan job (needs)
   • Runs terraform apply plan.tfplan
   • Can be made manual or conditional with provider overwrites

DAG Stage Mapping

Job dependencies are topologically layered. GitLab renders those layers as stages:

stages:
  - deploy-0   # first DAG layer
  - deploy-1   # second DAG layer
  - deploy-2   # third DAG layer
  - deploy-3   # fourth DAG layer

Dependency Chain

plan-vpc:
  stage: deploy-0

apply-vpc:
  stage: deploy-1
  needs: [plan-vpc]

plan-eks:
  stage: deploy-2
  needs: [apply-vpc]  # Waits for vpc to be applied

apply-eks:
  stage: deploy-3
  needs: [plan-eks]

Data Flow Diagram

Each stage:

Step	Function	What it does
1	workflow.PlanProject()	Scan filesystem, apply filters, parse HCL, build dependency graph, resolve optional targets
2	runflow.Prepare(...) contributions + Terraform runtime	Gather plugin-contributed DAG jobs and normalize Terraform/OpenTofu runtime; invalid contributions fail before IR build
3	pipeline.BuildProjectIR(req)	Construct provider-agnostic immutable job DAG (*pipeline.IR) — single execution input
4	provider.NewGenerator(ir) + Generate()	Bind IR to provider; transform IR into GitLab CI YAML or GitHub Actions workflow

The IR is the single source for both pipeline generation and terraci local-exec: providers don't reach for the dependency graph or contribution list separately — the IR already encodes them and is consumed through getters.

Key Types

Module

Represents a discovered Terraform module. Instead of hardcoded fields, the module uses a components map keyed by segment names from the configured pattern:

type Module struct {
    components map[string]string // {"service": "platform", "environment": "stage", ...}
    segments   []string          // ordered segment names from pattern

    Path         string  // /abs/path/to/vpc
    RelativePath string  // platform/stage/eu-central-1/vpc
    Parent       *Module
    Children     []*Module
}

func (m *Module) Get(name string) string  // m.Get("service") → "platform"
func (m *Module) ID() string              // returns RelativePath

This design allows fully configurable patterns. For example, with pattern {team}/{env}/{module}, you would use m.Get("team") and m.Get("env") instead of fixed field names.

RemoteStateRef

Represents a terraform_remote_state dependency:

type RemoteStateRef struct {
    Name         string            // "vpc"
    Backend      string            // "s3"
    Config       map[string]string // bucket, key, region
    WorkspaceDir string            // resolved module path
}

DependencyGraph

Manages module relationships:

type DependencyGraph struct {
    nodes map[string]*Module
    edges map[string][]string  // from -> [to, to, ...]
}

func (g *DependencyGraph) AddEdge(from, to *Module)
func (g *DependencyGraph) TopologicalSort() ([]*Module, error)
func (g *DependencyGraph) ExecutionLevels() [][]*Module
func (g *DependencyGraph) DetectCycles() [][]string

Performance

TerraCi is designed for speed:

Project Size	Modules	Parse Time	Generate Time
Small	10	~100ms	~50ms
Medium	50	~300ms	~100ms
Large	200	~1s	~300ms

Tips for large projects:

• Use exclude patterns to skip irrelevant directories
• Use --changed-only for incremental pipelines
• Enable caching in generated pipelines

See Also

• Project Structure - Directory layout requirements
• Dependencies - Dependency detection details
• Pipeline Generation - Generated output format