External Validation Webhooks

Overview

External validation webhooks allow you to run IaC validation on your own infrastructure (GitHub Actions, GitLab CI, Jenkins, etc.) instead of using Cloudgeni’s internal runners.

Choose Integration Mode

Three integration modes:

GitHub Actions: Cloudgeni triggers your workflow_dispatch directly. Provide workflow filename. No incoming webhook to handle.
GitLab CI: Cloudgeni triggers your GitLab pipeline with variables. Add a job to .gitlab-ci.yml that runs when CLOUDGENI_UOW_ID is set.
Custom Webhook: Provide your webhook endpoint URL. Cloudgeni sends signed POST requests. You verify signatures, start asynchronous validation and return 202 Accepted.

Step 1: Configure in Cloudgeni

Go to: Settings → IaC Repositories → Select integration → Select repository → Configure Webhook
Choose mode:
- GitHub Actions: Enter workflow filename (e.g., terraform-validation.yml)
- GitLab Pipeline: No additional config needed - uses .gitlab-ci.yml
- Custom Webhook: Enter webhook endpoint URL
Copy CLOUDGENI_WEBHOOK_SECRET
Store securely (GitHub Secrets, GitLab CI/CD Variables, environment variables, etc.)

Step 2: Implementation

GitHub Actions
GitLab CI
Custom Webhook

Cloudgeni triggers your workflow via workflow_dispatch with uowId and callbackUrl inputs. The uowId is the unique identifier for this validation “unit of work” (UOW).

Workflow Requirements

Run IaC steps: init, validate, and plan (or equivalent for your IaC tool, we currently support Terraform and Bicep)
Required: The plan step (or what-if for Bicep) is mandatory when executionFailure=false.
Capture outputs: stdout for init/validate, file for plan JSON
Sign callback payload with CLOUDGENI_WEBHOOK_SECRET
POST results to callbackUrl

Example Workflow

This workflow is a template. You must customize the authentication, environment matrix, and Terraform commands (e.g. init args, variable files) to match your specific infrastructure setup.

View complete workflow YAML

# .github/workflows/terraform-validation.yml
name: Cloudgeni External Validation

on:
  workflow_dispatch:
    inputs:
      uowId:
        description: 'UOW execution ID'
        required: true
        type: string
      callbackUrl:
        description: 'Cloudgeni callback URL'
        required: true
        type: string
      branch:
        description: 'Branch to validate'
        required: false
        default: 'main'
        type: string

jobs:
  # ============================================================
  # Job 1: Run Terraform validation for each environment
  # ============================================================
  validate-environment:
    runs-on: ubuntu-latest
    continue-on-error: true  # Don't fail the workflow if validation fails
    strategy:
      fail-fast: false
      matrix:
        # CUSTOMIZE: Replace with your environment names
        env: [staging, prod]

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.branch }}

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: latest
          terraform_wrapper: false

      # CUSTOMIZE: Replace with your cloud provider authentication
      # Example for GCP:
      - name: Setup Cloud Credentials
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}

      - name: Run Terraform (init/validate/plan)
        id: terraform
        shell: bash
        working-directory: environments/${{ matrix.env }}  # CUSTOMIZE: Your path
        continue-on-error: true
        run: |
          set -o pipefail
          
          # Run each command and capture exit codes separately
          terraform init -no-color 2>&1 | tee "$RUNNER_TEMP/init.txt"
          echo "${PIPESTATUS[0]}" > "$RUNNER_TEMP/init.exit"
          
          terraform validate -no-color 2>&1 | tee "$RUNNER_TEMP/validate.txt"
          echo "${PIPESTATUS[0]}" > "$RUNNER_TEMP/validate.exit"
          
          terraform plan -out=tfplan.binary -no-color 2>&1 | tee "$RUNNER_TEMP/plan.txt"
          echo "${PIPESTATUS[0]}" > "$RUNNER_TEMP/plan.exit"
          
          # Generate plan JSON if plan succeeded
          if [ "$(cat $RUNNER_TEMP/plan.exit)" = "0" ]; then
            terraform show -json tfplan.binary > "$RUNNER_TEMP/plan.json" 2>&1 || echo '{}' > "$RUNNER_TEMP/plan.json"
          else
            echo '{}' > "$RUNNER_TEMP/plan.json"
          fi
          
          echo "${{ matrix.env }}" > "$RUNNER_TEMP/env.name"

      - uses: actions/upload-artifact@v4
        with:
          name: results-${{ matrix.env }}
          path: |
            ${{ runner.temp }}/env.name
            ${{ runner.temp }}/init.txt
            ${{ runner.temp }}/init.exit
            ${{ runner.temp }}/validate.txt
            ${{ runner.temp }}/validate.exit
            ${{ runner.temp }}/plan.txt
            ${{ runner.temp }}/plan.exit
            ${{ runner.temp }}/plan.json

  # ============================================================
  # Job 2: Aggregate results and send callback to Cloudgeni
  # ============================================================
  send-callback:
    runs-on: ubuntu-latest
    needs: [validate-environment]
    if: always()  # Always run, even if validation job failed
    steps:
      - uses: actions/download-artifact@v4
        continue-on-error: true
        with:
          path: ${{ runner.temp }}/results

      - name: Send callback to Cloudgeni
        env:
          CALLBACK_URL: ${{ inputs.callbackUrl }}
          WEBHOOK_SECRET: ${{ secrets.CLOUDGENI_WEBHOOK_SECRET }}
        shell: bash
        run: |
          python3 - <<'PYTHON_CALLBACK'
          import os
          import sys
          import json
          import hmac
          import hashlib
          import requests
          
          def send_callback(payload, callback_url, webhook_secret):
              """Send signed callback to Cloudgeni."""
              payload_json = json.dumps(payload, separators=(',', ':'))
              signature = hmac.new(
                  webhook_secret.encode('utf-8'),
                  payload_json.encode('utf-8'),
                  hashlib.sha256
              ).hexdigest()
              
              headers = {
                  'Content-Type': 'application/json',
                  'X-Cloudgeni-Signature-256': f'sha256={signature}'
              }
              
              response = requests.post(callback_url, data=payload_json, headers=headers, timeout=30)
              print(f'Callback status: {response.status_code}')
              return response
          
          def read_file(path, default=''):
              """Safely read a file, returning default if it fails."""
              try:
                  with open(path, 'r', encoding='utf-8') as f:
                      return f.read()
              except Exception:
                  return default
          
          # Get environment variables
          callback_url = os.environ.get('CALLBACK_URL', '')
          webhook_secret = os.environ.get('WEBHOOK_SECRET', '')
          results_dir = os.path.join(os.environ.get('RUNNER_TEMP', '/tmp'), 'results')
          
          # Check required variables
          if not callback_url or not webhook_secret:
              print('ERROR: Missing CALLBACK_URL or WEBHOOK_SECRET')
              sys.exit(1)
          
          try:
              environments = []
              
              # Process each environment's results
              if os.path.isdir(results_dir):
                  for env_dir in sorted(os.listdir(results_dir)):
                      env_path = os.path.join(results_dir, env_dir)
                      if not os.path.isdir(env_path):
                          continue
                      
                      # Read results for this environment
                      env_name = read_file(os.path.join(env_path, 'env.name')).strip() or env_dir.replace('results-', '')
                      
                      init_out = read_file(os.path.join(env_path, 'init.txt'))
                      init_exit = read_file(os.path.join(env_path, 'init.exit'), '1').strip()
                      init_ok = init_exit == '0'
                      
                      val_out = read_file(os.path.join(env_path, 'validate.txt'))
                      val_exit = read_file(os.path.join(env_path, 'validate.exit'), '1').strip()
                      val_ok = val_exit == '0'
                      
                      plan_txt = read_file(os.path.join(env_path, 'plan.txt'))
                      plan_exit = read_file(os.path.join(env_path, 'plan.exit'), '1').strip()
                      plan_ok = plan_exit == '0'
                      
                      # Process plan JSON
                      plan_json = '{}'
                      if plan_ok:
                          raw_json = read_file(os.path.join(env_path, 'plan.json'), '{}')
                          try:
                              parsed = json.loads(raw_json)
                              plan_json = json.dumps(parsed, separators=(',', ':'))
                          except json.JSONDecodeError:
                              plan_ok = False
                              plan_txt = plan_txt or 'Invalid plan JSON generated'
                      
                      # Build steps array
                      steps = [
                          {
                              'step': 'init',
                              'success': init_ok,
                              'error': None if init_ok else (init_out or 'init failed'),
                              'data': init_out if init_ok else None
                          },
                          {
                              'step': 'validate',
                              'success': val_ok,
                              'error': None if val_ok else (val_out or 'validate failed'),
                              'data': val_out if val_ok else None
                          },
                          {
                              'step': 'plan',
                              'success': plan_ok,
                              'error': None if plan_ok else (plan_txt or 'plan failed'),
                              'data': plan_json if plan_ok else None
                          }
                      ]
                      
                      environments.append({
                          'environmentName': env_name,
                          'iacType': 'TERRAFORM',
                          'path': f'environments/{env_name}',
                          'steps': steps
                      })
              
              # Build payload
              if environments:
                  payload = {
                      'executionFailure': False,
                      'executionFailureMessage': None,
                      'environments': environments
                  }
              else:
                  payload = {
                      'executionFailure': True,
                      'executionFailureMessage': 'No environment results found',
                      'environments': None
                  }
              
              # Send callback
              response = send_callback(payload, callback_url, webhook_secret)
              response.raise_for_status()
              print('Callback sent successfully!')
              
          except Exception as e:
              # Always try to send a failure callback
              print(f'ERROR: {e}')
              try:
                  failure_payload = {
                      'executionFailure': True,
                      'executionFailureMessage': f'Workflow error: {str(e)}',
                      'environments': None
                  }
                  send_callback(failure_payload, callback_url, webhook_secret)
                  print('Sent failure callback')
              except Exception as e2:
                  print(f'Failed to send failure callback: {e2}')
              sys.exit(1)
          PYTHON_CALLBACK

Cloudgeni triggers your GitLab CI pipeline with variables. Add a job to your .gitlab-ci.yml that only runs when CLOUDGENI_UOW_ID is set.

Pipeline Variables

Cloudgeni sets these CI/CD variables when triggering your pipeline:

CLOUDGENI_UOW_ID - Unique identifier for this validation
CLOUDGENI_CALLBACK_URL - URL to POST results back to
CLOUDGENI_BRANCH - Branch being validated

Required GitLab CI/CD Variable

You must add this variable in GitLab (Settings → CI/CD → Variables):

CLOUDGENI_WEBHOOK_SECRET - Copy from Cloudgeni webhook configuration

Terraform
Terragrunt

Example `.gitlab-ci.yml`

This is a template. Customize authentication, environment matrix, and Terraform commands for your setup.

View complete GitLab CI YAML

# .gitlab-ci.yml

# Your regular CI jobs (run on push/merge)
test:
  script:
    - bun run test
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

# Cloudgeni validation job - ONLY runs when triggered by Cloudgeni
cloudgeni-validation:
  image:
    name: hashicorp/terraform:latest
    entrypoint: [""]  # Override to allow shell scripts
  rules:
    - if: $CLOUDGENI_UOW_ID  # Only run when this variable exists
  variables:
    # Add your cloud credentials as CI/CD variables in GitLab
    # AWS_ACCESS_KEY_ID: from CI/CD settings
    # AWS_SECRET_ACCESS_KEY: from CI/CD settings
  before_script:
    - apk add --no-cache python3 py3-requests
  script:
    - |
      # IMPORTANT: Do NOT use 'set -e' - we must always send callback to Cloudgeni
      set -o pipefail  # Capture correct exit codes through pipes
      
      mkdir -p /tmp/results
      
      # Initialize exit code files with failure state (in case commands don't run)
      echo "1" > /tmp/results/init.exit
      echo "1" > /tmp/results/validate.exit
      echo "1" > /tmp/results/plan.exit
      echo "" > /tmp/results/init.txt
      echo "" > /tmp/results/validate.txt
      echo "" > /tmp/results/plan.txt
      echo "{}" > /tmp/results/plan.json
      
      # Run Terraform commands - capture output and exit codes
      # Adjust path to your environment
      cd environments/staging || { echo "Directory not found" > /tmp/results/init.txt; }
      
      # Init
      terraform init -no-color 2>&1 | tee /tmp/results/init.txt; echo "${PIPESTATUS[0]}" > /tmp/results/init.exit
      
      # Only validate if init succeeded
      if [ "$(cat /tmp/results/init.exit)" = "0" ]; then
        terraform validate -no-color 2>&1 | tee /tmp/results/validate.txt; echo "${PIPESTATUS[0]}" > /tmp/results/validate.exit
      fi
      
      # Only plan if validate succeeded
      if [ "$(cat /tmp/results/validate.exit)" = "0" ]; then
        terraform plan -out=tfplan.binary -no-color 2>&1 | tee /tmp/results/plan.txt; echo "${PIPESTATUS[0]}" > /tmp/results/plan.exit
        
        # Generate plan JSON if plan succeeded
        if [ "$(cat /tmp/results/plan.exit)" = "0" ]; then
          terraform show -json tfplan.binary > /tmp/results/plan.json 2>&1 || echo "{}" > /tmp/results/plan.json
        fi
      fi
      
      # ALWAYS send callback to Cloudgeni - wrapped in Python for reliability
      python3 - <<'PYTHON_CALLBACK'
      import os
      import sys
      import json
      import hmac
      import hashlib
      import requests
      
      def read_file(path, default=''):
          try:
              with open(path, 'r') as f:
                  return f.read()
          except Exception:
              return default
      
      def send_callback(payload, callback_url, webhook_secret):
          """Send signed callback to Cloudgeni."""
          payload_json = json.dumps(payload, separators=(',', ':'))
          signature = hmac.new(
              webhook_secret.encode('utf-8'),
              payload_json.encode('utf-8'),
              hashlib.sha256
          ).hexdigest()
          
          headers = {
              'Content-Type': 'application/json',
              'X-Cloudgeni-Signature-256': f'sha256={signature}'
          }
          
          response = requests.post(callback_url, data=payload_json, headers=headers, timeout=30)
          print(f'Callback status: {response.status_code}')
          return response
      
      # Get environment variables with safe defaults
      callback_url = os.environ.get('CLOUDGENI_CALLBACK_URL', '')
      webhook_secret = os.environ.get('CLOUDGENI_WEBHOOK_SECRET', '')
      
      # Check required variables
      if not callback_url or not webhook_secret:
          print('ERROR: Missing CLOUDGENI_CALLBACK_URL or CLOUDGENI_WEBHOOK_SECRET')
          print('Cannot send callback without these variables.')
          sys.exit(1)
      
      try:
          # Read results
          init_out = read_file('/tmp/results/init.txt')
          init_exit = read_file('/tmp/results/init.exit', '1').strip()
          
          validate_out = read_file('/tmp/results/validate.txt')
          validate_exit = read_file('/tmp/results/validate.exit', '1').strip()
          
          plan_txt = read_file('/tmp/results/plan.txt')
          plan_exit = read_file('/tmp/results/plan.exit', '1').strip()
          plan_json = read_file('/tmp/results/plan.json', '{}')
          
          # Determine success based on exit codes
          init_success = init_exit == '0'
          validate_success = validate_exit == '0'
          plan_success = plan_exit == '0'
          
          # Validate plan JSON
          try:
              json.loads(plan_json)
          except json.JSONDecodeError:
              plan_json = '{}'
              plan_success = False
          
          # Build steps array
          steps = [
              {
                  'step': 'init',
                  'success': init_success,
                  'error': None if init_success else (init_out or 'Init failed'),
                  'data': init_out if init_success else None
              },
              {
                  'step': 'validate',
                  'success': validate_success,
                  'error': None if validate_success else (validate_out or 'Validate failed'),
                  'data': validate_out if validate_success else None
              },
              {
                  'step': 'plan',
                  'success': plan_success,
                  'error': None if plan_success else (plan_txt or 'Plan failed'),
                  'data': plan_json if plan_success else None
              }
          ]
          
          payload = {
              'executionFailure': False,
              'executionFailureMessage': None,
              'environments': [{
                  'environmentName': 'staging',  # Adjust to your environment name
                  'iacType': 'TERRAFORM',
                  'path': 'environments/staging',  # Adjust to your terraform directory
                  'steps': steps
              }]
          }
          
          response = send_callback(payload, callback_url, webhook_secret)
          response.raise_for_status()
          print('Callback sent successfully!')
          
      except Exception as e:
          # If anything fails, try to send an execution failure callback
          print(f'ERROR during validation: {e}')
          try:
              failure_payload = {
                  'executionFailure': True,
                  'executionFailureMessage': f'Pipeline error: {str(e)}',
                  'environments': None
              }
              send_callback(failure_payload, callback_url, webhook_secret)
              print('Sent execution failure callback')
          except Exception as e2:
              print(f'Failed to send failure callback: {e2}')
          sys.exit(1)
      PYTHON_CALLBACK

Example `.gitlab-ci.yml` for Terragrunt

This example uses terragrunt run --all to plan all units in a single command. Each unit (e.g., dev/network, prod/db) becomes a separate environment in the Cloudgeni callback.

What you MUST customize:

TERRAGRUNT_VERSION and TERRAFORM_VERSION
ENVIRONMENTS_PATH - Path to your Terragrunt environments directory
Cloud authentication in before_script

What you CAN customize:

Add extra Terragrunt/Terraform flags as needed for your project
Add environment variables, pre/post steps, etc.

What MUST stay the same (required for Cloudgeni):

--log-format json - Enables per-unit error parsing
--no-color - Clean output without ANSI codes
--json-out-dir "$JSON_OUT_DIR" - Writes plan JSON per unit
--queue-ignore-errors - Continues on failures so all units are reported
set +e at the start - Ensures callback always runs
The Python callback script (parses logs and sends results)

View complete Terragrunt GitLab CI YAML

# .gitlab-ci.yml

# Cloudgeni validation job - ONLY runs when triggered by Cloudgeni
cloudgeni-validation:
  image: ubuntu:22.04
  rules:
    - if: $CLOUDGENI_UOW_ID  # Only run when this variable exists
  variables:
    # ============================================================
    # CUSTOMIZE: Versions and paths for your project
    # ============================================================
    TERRAGRUNT_VERSION: "0.77.22"
    TERRAFORM_VERSION: "1.13.3"
    ENVIRONMENTS_PATH: "terragrunt/environments"  # Path to your environments
    
    # ============================================================
    # CUSTOMIZE: Cloud provider credentials (add in GitLab CI/CD Variables)
    # Example for Azure:
    # ARM_CLIENT_ID: from CI/CD settings
    # ARM_CLIENT_SECRET: from CI/CD settings
    # ARM_TENANT_ID: from CI/CD settings
    # ARM_SUBSCRIPTION_ID: from CI/CD settings
    # ============================================================
  
  before_script:
    # Base dependencies
    - apt-get update
    - apt-get install -y --no-install-recommends ca-certificates curl unzip python3 python3-requests bash
    
    # ============================================================
    # CUSTOMIZE: Install your cloud CLI if needed
    # Example for Azure:
    # - curl -sL https://aka.ms/InstallAzureCLIDeb | bash
    # - az login --service-principal -u "$ARM_CLIENT_ID" -p "$ARM_CLIENT_SECRET" --tenant "$ARM_TENANT_ID"
    # ============================================================
    
    # Install Terraform
    - curl -Lo terraform.zip "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
    - unzip terraform.zip -d /usr/local/bin && rm terraform.zip
    - terraform version
    
    # Install Terragrunt
    - curl -Lo /usr/local/bin/terragrunt "https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64"
    - chmod +x /usr/local/bin/terragrunt
    - terragrunt --version
  
  script:
    - |
      #!/bin/bash
      
      # IMPORTANT: Disable automatic exit on error - we must always send callback
      set +e
      set -o pipefail
      
      ENV_ROOT="$CI_PROJECT_DIR/$ENVIRONMENTS_PATH"
      JSON_OUT_DIR="/tmp/cloudgeni-plans"
      PLAN_LOG="/tmp/cloudgeni-plan.jsonl"
      TG_EXIT_FILE="/tmp/cloudgeni-terragrunt.exit"
      
      rm -rf "$JSON_OUT_DIR"
      rm -f "$PLAN_LOG" "$TG_EXIT_FILE"
      mkdir -p "$JSON_OUT_DIR"
      
      echo "=== Running terragrunt plan (run --all) ==="
      cd "$ENV_ROOT" || echo "WARN: failed to cd to $ENV_ROOT"
      
      # Run terragrunt with JSON logging and JSON plan output
      # 
      # REQUIRED FLAGS (do not remove):
      #   --log-format json    : Structured logs for parsing per-unit errors
      #   --no-color           : Clean output without ANSI codes
      #   --json-out-dir       : Write tfplan.json for each successful unit
      #   --queue-ignore-errors: Continue planning other units even if some fail
      #
      # You can add additional flags as needed for your project.
      #
      terragrunt run --all \
        --log-format json \
        --no-color \
        --json-out-dir "$JSON_OUT_DIR" \
        --queue-ignore-errors \
        plan \
        -- -lock=false \
        2>&1 | tee "$PLAN_LOG"
      echo "${PIPESTATUS[0]}" > "$TG_EXIT_FILE"
      
      echo "=== Terragrunt finished with exit code $(cat $TG_EXIT_FILE) ==="
      
      # ALWAYS send callback to Cloudgeni
      python3 - <<'PYTHON_CALLBACK'
      import os
      import sys
      import re
      import json
      import hmac
      import hashlib
      import requests
      from collections import defaultdict
      
      def read_file(path, default=''):
          try:
              with open(path, 'r', encoding='utf-8', errors='replace') as f:
                  return f.read()
          except Exception:
              return default
      
      def strip_ansi(text):
          """Remove ANSI escape codes from text."""
          return re.sub(r'\x1b\[[0-9;]*m', '', text)
      
      def send_callback(payload, callback_url, webhook_secret):
          """Send signed callback to Cloudgeni."""
          payload_json = json.dumps(payload, separators=(',', ':'))
          signature = hmac.new(
              webhook_secret.encode('utf-8'),
              payload_json.encode('utf-8'),
              hashlib.sha256
          ).hexdigest()
          headers = {
              'Content-Type': 'application/json',
              'X-Cloudgeni-Signature-256': f'sha256={signature}'
          }
          response = requests.post(callback_url, data=payload_json, headers=headers, timeout=30)
          print(f'Callback status: {response.status_code}')
          return response
      
      def parse_plan_log_jsonl(log_path, env_root_abs):
          """Parse JSONL log and extract per-unit errors."""
          unit_errors = defaultdict(list)
          if not os.path.isfile(log_path):
              return unit_errors
          
          for line in read_file(log_path).splitlines():
              line = line.strip()
              if not line:
                  continue
              try:
                  entry = json.loads(line)
              except json.JSONDecodeError:
                  continue
              
              working_dir = entry.get('working-dir', '')
              if not working_dir:
                  continue
              
              # Compute unit path relative to env root
              try:
                  unit = os.path.relpath(os.path.normpath(working_dir), env_root_abs)
              except ValueError:
                  unit = os.path.basename(working_dir)
              unit = unit.replace(os.sep, '/')
              
              # Skip invalid paths
              if unit in ('.', ''):
                  continue
              if unit.startswith('../') or '/.terragrunt-cache/' in f'/{unit}/' or unit.endswith('/.terragrunt-cache'):
                  continue
              
              level = entry.get('level', '')
              if level in ('error', 'stderr'):
                  msg = strip_ansi(entry.get('msg', ''))
                  if msg:
                      unit_errors[unit].append(msg)
          
          return unit_errors
      
      def discover_units_from_json_out(json_out_dir):
          """Find all units that produced tfplan.json (successful plans)."""
          units = set()
          if not os.path.isdir(json_out_dir):
              return units
          for root, dirs, files in os.walk(json_out_dir):
              dirs[:] = [d for d in dirs if d != '.terragrunt-cache']
              if 'tfplan.json' in files:
                  rel = os.path.relpath(root, json_out_dir).replace(os.sep, '/')
                  if rel and rel != '.' and not rel.startswith('..'):
                      units.add(rel)
          return units
      
      def discover_units_from_repo(env_root):
          """Find all terragrunt.hcl files to discover all units."""
          units = set()
          if not os.path.isdir(env_root):
              return units
          for root, dirs, files in os.walk(env_root):
              dirs[:] = [d for d in dirs if d != '.terragrunt-cache']
              if 'terragrunt.hcl' in files:
                  rel = os.path.relpath(root, env_root).replace(os.sep, '/')
                  if rel and rel != '.' and not rel.startswith('..'):
                      units.add(rel)
          return units
      
      # ============================================================
      # Main callback logic
      # ============================================================
      callback_url = os.environ.get('CLOUDGENI_CALLBACK_URL', '')
      webhook_secret = os.environ.get('CLOUDGENI_WEBHOOK_SECRET', '')
      json_out_dir = '/tmp/cloudgeni-plans'
      plan_log = '/tmp/cloudgeni-plan.jsonl'
      env_root = os.path.join(os.environ.get('CI_PROJECT_DIR', '.'), os.environ.get('ENVIRONMENTS_PATH', '.'))
      env_root_abs = os.path.abspath(env_root)
      
      if not callback_url or not webhook_secret:
          print('ERROR: Missing CLOUDGENI_CALLBACK_URL or CLOUDGENI_WEBHOOK_SECRET')
          sys.exit(1)
      
      try:
          # Parse logs for per-unit errors
          unit_errors = parse_plan_log_jsonl(plan_log, env_root_abs)
          
          # Discover all units from multiple sources
          units_from_json = discover_units_from_json_out(json_out_dir)
          units_from_repo = discover_units_from_repo(env_root_abs)
          units_from_logs = set(unit_errors.keys())
          
          all_units = units_from_json | units_from_repo | units_from_logs
          # Filter out any remaining invalid paths
          all_units = {u for u in all_units if u and u != '.' and not u.startswith('../') and '.terragrunt-cache' not in u}
          
          environments = []
          for unit in sorted(all_units):
              plan_json_path = os.path.join(json_out_dir, unit, 'tfplan.json')
              
              if os.path.isfile(plan_json_path):
                  # Unit succeeded - include plan JSON
                  plan_data = read_file(plan_json_path)
                  try:
                      json.loads(plan_data)  # Validate JSON
                      success = True
                      error = None
                  except json.JSONDecodeError:
                      success = False
                      error = 'Invalid plan JSON'
                      plan_data = None
              else:
                  # Unit failed - collect errors from logs
                  success = False
                  errors = unit_errors.get(unit, ['Plan failed (no output)'])
                  error = '\n'.join(errors[:50])  # Limit error size
                  plan_data = None
              
              environments.append({
                  'environmentName': unit,  # e.g., "dev/network", "prod/db"
                  'iacType': 'TERRAFORM',
                  'path': unit,  # Same as environmentName for Terragrunt units
                  'steps': [{
                      'step': 'plan',
                      'success': success,
                      'error': error,
                      'data': plan_data
                  }]
              })
          
          if environments:
              payload = {
                  'executionFailure': False,
                  'executionFailureMessage': None,
                  'environments': environments
              }
          else:
              payload = {
                  'executionFailure': True,
                  'executionFailureMessage': 'No Terragrunt units found',
                  'environments': None
              }
          
          response = send_callback(payload, callback_url, webhook_secret)
          response.raise_for_status()
          print(f'Callback sent successfully! ({len(environments)} environments)')
          
      except Exception as e:
          print(f'ERROR: {e}')
          try:
              failure_payload = {
                  'executionFailure': True,
                  'executionFailureMessage': f'Pipeline error: {str(e)}',
                  'environments': None
              }
              send_callback(failure_payload, callback_url, webhook_secret)
              print('Sent execution failure callback')
          except Exception as e2:
              print(f'Failed to send failure callback: {e2}')
          sys.exit(1)
      PYTHON_CALLBACK

How It Works

Single terragrunt run --all plan: Plans all units in one command
--json-out-dir: Writes tfplan.json to <dir>/<unit>/tfplan.json for successful plans
--log-format json: Structured JSONL logs with per-unit working-dir for error attribution
--queue-ignore-errors: Continues planning other units even if some fail
Python script:
- Discovers all units from repo (terragrunt.hcl files), JSON output, and logs
- Filters out internal paths (.terragrunt-cache, ., ..)
- For each unit: includes plan JSON if successful, or aggregated errors if failed
- Sends signed callback with all environments

Example Callback Payload

{
  "executionFailure": false,
  "executionFailureMessage": null,
  "environments": [
    {
      "environmentName": "dev/db",
      "iacType": "TERRAFORM",
      "path": "dev/db",
      "steps": [{ "step": "plan", "success": true, "error": null, "data": "{...plan json...}" }]
    },
    {
      "environmentName": "dev/network",
      "iacType": "TERRAFORM",
      "path": "dev/network",
      "steps": [{ "step": "plan", "success": false, "error": "Error: Failed to get existing workspaces...", "data": null }]
    }
  ]
}

Key Points

Use rules: - if: $CLOUDGENI_UOW_ID to ensure the job only runs when triggered by Cloudgeni
Store CLOUDGENI_WEBHOOK_SECRET in GitLab CI/CD Variables (Settings → CI/CD → Variables)
The pipeline is triggered via GitLab API with the variables set automatically
Send results back to CLOUDGENI_CALLBACK_URL with HMAC signature
Fallback error handling: If anything fails, sends executionFailure: true to Cloudgeni

Cloudgeni sends signed POST requests to your webhook endpoint. Verify signature, return 202 Accepted, then process asynchronously.

Webhook Request

Method: POSTHeaders:

Content-Type: application/json
X-Cloudgeni-Signature-256: sha256=<hex>

Payload:

{
  "uowId": "550e8400-e29b-41d4-a716-446655440000", # The `uowId` is the unique identifier for this validation "unit of work" (UOW).
  "callbackUrl": "https://ai.cloudgeni.ai/api/v1/callback/uow/550e8400-e29b-41d4-a716-446655440000",
  "data": {
    "repoName": "my-org/my-terraform-repo",
    "branch": "cloudgeni-remediation-abc123",
    "gitCommitSha": "1234567890"
  }
}

Expected Response

Return 202 Accepted immediately. Process validation asynchronously.

Cloudgeni retries if not 202 within 5 seconds (exponential backoff, up to 3 attempts).

Signature Verification

Verify HMAC-SHA256 on raw request body using CLOUDGENI_WEBHOOK_SECRET.

Code examples (Python, Node, Go)

import hmac
import hashlib
from fastapi import Request, HTTPException, status

async def verify_cloudgeni_signature(request: Request, webhook_secret: str):
    signature = request.headers.get("X-Cloudgeni-Signature-256")
    if not signature or not signature.startswith("sha256="):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    raw_body = await request.body()
    received_sig = signature[7:]
    
    expected_sig = hmac.new(
        webhook_secret.encode('utf-8'),
        raw_body,
        hashlib.sha256
    ).hexdigest()
    
    if not hmac.compare_digest(expected_sig, received_sig):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    return True

@app.post("/webhooks/cloudgeni/validation")
async def handle_webhook(request: Request):
    await verify_cloudgeni_signature(request, os.environ["CLOUDGENI_WEBHOOK_SECRET"])
    payload = await request.json()
    
    # Trigger validation asynchronously
    trigger_validation_workflow(
        payload["uowId"],
        payload["callbackUrl"],
        payload["data"]["repoName"],
        payload["data"]["branch"],
        payload["data"]["gitCommitSha"]
    )
    
    return Response(status_code=202)

Step 3: Send Callback Response

Terraform
Bicep

After validation completes, send signed results to Cloudgeni.Endpoint: POST /api/v1/callback/uow/{uow_id} (use callbackUrl from webhook)Headers:

Content-Type: application/json
X-Cloudgeni-Signature-256: sha256=<hex>

Payload (Terraform):

{
  "executionFailure": false,
  "executionFailureMessage": null,
  "environments": [
    {
      "environmentName": "main",
      "iacType": "TERRAFORM",
      "path": "infra/production",
      "steps": [
        {
          "step": "init",
          "success": true,
          "error": null,
          "data": "Terraform initialized successfully"
        },
        {
          "step": "validate",
          "success": true,
          "error": null,
          "data": "Success! The configuration is valid."
        },
        {
          "step": "plan",
          "success": true,
          "error": null,
          "data": "{\"format_version\":\"1.2\",\"terraform_version\":\"1.5.0\",\"resource_changes\":[]}"
        }
        // Add any other steps you want to report. If any of these steps fail, the Cloudgeni agent will use that information to iterate on the IaC code during a remediation.
      ]
    }
  ]
}

Example command to generate plan JSON:

terraform plan -out=tfplan.binary && terraform show -json tfplan.binary > plan.json

Each environment MUST include a step called “plan”. This step is mandatory and must contain:

Step name: “plan”
Success status: true/false
JSON data: The data field must be a JSON string (not an object) containing the Terraform plan output in JSON format

Example: { "step": "plan", "success": true, "error": null, "data": "{...plan json...}" }

executionFailure: Set to true only for complete execution failures (cannot run any validation).
When executionFailure=true: executionFailureMessage is required, environments must be null or empty.
When executionFailure=false: environments is required, executionFailureMessage must be null.
steps[].data for plan must be a JSON string, not an object.
environmentName should match your environment (e.g., “main”, “environments/production”).
iacType must be “TERRAFORM”.
path: Directory path relative to repo root where terraform files are located (e.g., “infra/production”). Required for accurate resource matching when multiple execution scopes exist.

After validation completes, send signed results to Cloudgeni.Endpoint: POST /api/v1/callback/uow/{uow_id} (use callbackUrl from webhook)Headers:

Content-Type: application/json
X-Cloudgeni-Signature-256: sha256=<hex>

Payload (Bicep):

{
  "executionFailure": false,
  "executionFailureMessage": null,
  "environments": [
    {
      "environmentName": "main",
      "iacType": "BICEP",
      "path": "infra/production",
      "steps": [
        {
          "step": "what-if",
          "success": true,
          "error": null,
          "data": "{\"changes\":[],\"scope\":\"subscription\"}"
        }
        // Add any other steps you want to report. If any of these steps fail, the Cloudgeni agent will use that information to iterate on the IaC code during a remediation.
      ]
    }
  ]
}

Example command to generate what-if JSON (example at subscription scope):

az deployment sub what-if -f main.bicep -p bicepparam.json -l westus2 -o json > what-if.json

Each environment MUST include a step called “what-if”. This step is mandatory and must contain:

Step name: “what-if”
Success status: true/false
JSON data: The data field must be a JSON string (not an object) containing the Azure what-if output in JSON format

Example: { "step": "what-if", "success": true, "error": null, "data": "{...what-if json...}" }

executionFailure: Set to true only for complete execution failures (cannot run any validation).
When executionFailure=true: executionFailureMessage is required, environments must be null or empty.
When executionFailure=false: environments is required, executionFailureMessage must be null.
steps[].data for what-if must be a JSON string, not an object.
environmentName should match your environment (e.g., “main”, “environments/production”).
iacType must be “BICEP”.
path: Directory path relative to repo root where bicep files are located (e.g., “infra/production”). Required for accurate resource matching when multiple execution scopes exist.

Full schema: https://ai.cloudgeni.ai/docs → “External Validation Callback”

Signing the Callback

import hmac
import hashlib
import json
import requests

def sign_callback(payload_dict, secret):
    payload_json = json.dumps(payload_dict, separators=(',', ':'))
    signature_hex = hmac.new(
        secret.encode('utf-8'),
        payload_json.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    return f"sha256={signature_hex}", payload_json

def send_callback(callback_url, payload, secret):
    signature, payload_json = sign_callback(payload, secret)
    response = requests.post(
        callback_url,
        data=payload_json,
        headers={
            "Content-Type": "application/json",
            "X-Cloudgeni-Signature-256": signature
        },
        timeout=30
    )
    response.raise_for_status()
    return response

Security Best Practices

1. Protect Your Webhook Secret

Never commit secrets to version control
Use environment variables or secret managers
Rotate secrets periodically through the cloudgeni console

2. Always Verify Signatures

Reject requests without signatures
Verify before parsing JSON
Use timing-safe comparison functions
Log failed verification attempts

3. Use HTTPS

Always use HTTPS for webhook endpoints
Validate TLS certificates
Consider IP allowlisting if possible

4. Implement Timeouts

Set reasonable timeouts for callbacks (30-60 seconds)
Handle network failures gracefully
Implement retry logic with exponential backoff

5. Validate Payload Schema

Verify all required fields are present
Validate data types and formats
Sanitize inputs before processing

Getting Started

Setup & Configuration

Features

Support

Overview

Choose Integration Mode

Step 1: Configure in Cloudgeni

Step 2: Implementation

Workflow Requirements

Example Workflow

Pipeline Variables

Required GitLab CI/CD Variable

Example `.gitlab-ci.yml`

Example `.gitlab-ci.yml` for Terragrunt

How It Works

Example Callback Payload

Key Points

Webhook Request

Expected Response

Signature Verification

Step 3: Send Callback Response

Signing the Callback

Security Best Practices

Getting Started

Setup & Configuration

Features

Support

​Overview

​Choose Integration Mode

​Step 1: Configure in Cloudgeni

​Step 2: Implementation

​Workflow Requirements

​Example Workflow

​Pipeline Variables

​Required GitLab CI/CD Variable

​Example .gitlab-ci.yml

​Example .gitlab-ci.yml for Terragrunt

​How It Works

​Example Callback Payload

​Key Points

​Webhook Request

​Expected Response

​Signature Verification

​Step 3: Send Callback Response

​Signing the Callback

​Security Best Practices

Overview

Choose Integration Mode

Step 1: Configure in Cloudgeni

Step 2: Implementation

Workflow Requirements

Example Workflow

Pipeline Variables

Required GitLab CI/CD Variable

Example `.gitlab-ci.yml`

Example `.gitlab-ci.yml` for Terragrunt

How It Works

Example Callback Payload

Key Points

Webhook Request

Expected Response

Signature Verification

Step 3: Send Callback Response

Signing the Callback

Security Best Practices