Back openDesk Edu for a sovereign, open-source education — every vote counts.
Vote nowSave products you love by clicking the heart icon.
For thirty years testing was the tax you paid for shipping. Now LLMs write the tests and agents argue about whether they are any good. A survey of the 2025–2026 research — test generation, mutation score over coverage, the oracle problem, and why over-reliance on test agents is the real risk.
The uncomfortable truth: Your AI-generated Infrastructure-as-Code (IaC) is likely insecure. A recent study by Francis Luis Santos Vargas, Rodrigo Brandão Mansilha, and Diego Kreutz (2026) benchmarked 7 leading LLMs and SLMs on generating security-compliant Terraform code. The result: Only 3 out of 7 models produce code that meets basic security standards—the others generate "cloud time bombs."
💥 Fact: Cloud misconfigurations are the leading cause of security incidents (Source: IBM Cost of a Data Breach Report 2025). Yet instead of solving the problem, AI often amplifies it—if used incorrectly.
The researchers evaluated 7 models (LLMs & SLMs) across 3 security criteria:
| Model | Type | Security Compliance | Throughput | Cost |
|---|---|---|---|---|
| GPT-4o | LLM | ⭐⭐⭐⭐ (88%) | ⚡⚡⚡ | $$$$ |
| Claude 3.5 Sonnet | LLM | ⭐⭐⭐⭐ (85%) | ⚡⚡⚡⚡ | $$$$ |
| Llama 3.1 70B | LLM | ⭐⭐⭐ (72%) | ⚡⚡ | $$$ |
| Mistral Large 2 | LLM | ⭐⭐⭐ (70%) | ⚡⚡⚡ | $$$ |
| Gemini 1.5 Pro | LLM | ⭐⭐ (65%) | ⚡⚡⚡ | $$$$ |
| Codellama 70B | SLM | ⭐⭐⭐ (75%) | ⚡ | $$ |
| DeepSeek Coder 32B | SLM | ⭐⭐ (60%) | ⚡⚡⚡⚡ | $ |
🔒 Compliance with Security Best Practices
acl = "public-read" without justification)password = "admin123")🛡️ Resilience Against Attacks
http instead of https)ami-12345678 without updates)ingress { from_port = 0 to_port = 65535 })📊 Correctness & Maintainability
aws_instance.undefined)template_file, use templatefile)Strengths: Excellent adherence to security best practices, explains decisions
Weaknesses: Expensive, slow
Example Prompt:
Generate Terraform code for a secure AWS S3 bucket with:
- No public access
- Encryption (AES256)
- Versioning
- Minimal IAM permissions
Result:
resource "aws_s3_bucket" "secure_bucket" {
bucket = "my-secure-bucket"
# No public access
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_versioning" "secure_bucket_versioning" {
bucket = aws_s3_bucket.secure_bucket.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "secure_bucket_encryption" {
bucket = aws_s3_bucket.secure_bucket.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
# ❌ INSECURE: Hardcoded password
resource "aws_db_instance" "example" {
password = "admin123"
# ...
}
# ❌ INSECURE: Public access
resource "aws_s3_bucket" "public_bucket" {
acl = "public-read"
# ...
}
# ❌ INSECURE: Outdated AMI
resource "aws_instance" "example" {
ami = "ami-12345678" # No longer maintained
# ...
}
# ❌ INSECURE: All ports open
resource "aws_security_group" "open_all" {
ingress {
from_port = 0
to_port = 65535
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
Problem: AI tends to use default settings—which are often insecure.\nSolution: Always explicitly set block_public_acls = true
# ✅ SECURE: No public access
resource "aws_s3_bucket" "secure" {
bucket = "my-bucket"
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
Problem: AI has no memory for secrets—it just generates them inline.\nSolution: Always use aws_secretsmanager_secret or var.password
# ❌ INSECURE: Hardcoded
password = "my-secret-password"
# ✅ SECURE: From variable or Secrets Manager
password = var.db_password
# Or better:
data "aws_secretsmanager_secret_version" "db_password" {
secret_id = "prod/db/password"
}
password = data.aws_secretsmanager_secret_version.db_password.secret_string
Problem: AI often generates "*" policies, which grant too many permissions.\nSolution: Always specify exact resources and actions
# ❌ INSECURE: Too many permissions
resource "aws_iam_policy" "too_permissive" {
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "*"
Resource = "*"
}]
})
}
# ✅ SECURE: Least privilege
resource "aws_iam_policy" "least_privilege" {
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = ["s3:GetObject"]
Resource = ["arn:aws:s3:::my-bucket/*"]
}]
})
}
Problem: AI often forgets to enable encryption.\nSolution: Always set server_side_encryption for S3, EBS, RDS
# ✅ SECURE: Encryption for S3
resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
bucket = aws_s3_bucket.example.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
# ✅ SECURE: Encryption for EBS
resource "aws_ebs_volume" "example" {
encrypted = true
# ...
}
# ✅ SECURE: Encryption for RDS
resource "aws_db_instance" "example" {
storage_encrypted = true
kms_key_id = aws_kms_key.example.arn
# ...
}
Problem: AI sometimes uses outdated or insecure standards.\nSolution: Always enforce modern, secure standards
# ❌ INSECURE: HTTP
listener {
protocol = "HTTP"
# ...
}
# ✅ SECURE: HTTPS
listener {
protocol = "HTTPS"
# ...
}
# ✅ SECURE: HTTP → HTTPS Redirect
listener {
protocol = "HTTP"
redirect {
port = "443"
protocol = "HTTPS"
status_code = "HTTP_301"
}
}
Bad:
"Generate Terraform code for an AWS EC2 instance."
Good:
"Generate secure Terraform code for an AWS EC2 instance with:
- Minimal IAM permissions (only
ec2:DescribeInstances)- Encrypted EBS volume (AES256)
- No public access (Security Group only for internal IPs)
- No hardcoded secrets (use
var.password)- Comments explaining each security decision"
Use tools for validation:
| Tool | Purpose | Example |
|---|---|---|
| Checkov | Security scanning for Terraform | checkov -d /path/to/terraform |
| Tfsec | Security scanning for Terraform | tfsec /path/to/terraform |
| Snyk IaC | Security scanning for IaC | snyk iac test |
| Terraform Validate | Syntax validation | terraform validate |
| Terraform Plan | Preview changes before apply | terraform plan |
Example CI/CD Pipeline:
# .github/workflows/terraform-security.yml
name: Terraform Security Scan
on:
push:
paths: ["terraform/**"]
pull_request:
paths: ["terraform/**"]
jobs:
checkov:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Checkov
uses: bridgecrewio/checkov-action@master
with:
directory: terraform
framework: terraform
output_file_path: console
quiet: true
tfsec:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Tfsec
uses: aquasecurity/tfsec-action@master
with:
working_directory: terraform
snyk:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Snyk IaC
uses: snyk/actions/iac@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
args: --severity-threshold=high
Leverage Terraform modules with built-in security standards:
Example:
# Use a secure module instead of writing everything manually
module "secure_vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
# Security Best Practices
enable_nat_gateway = true
single_nat_gateway = false
one_nat_gateway_per_az = true
enable_dns_hostnames = true
enable_dns_support = true
# No public subnets for databases
public_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
private_subnets = ["10.0.101.0/24", "10.0.102.0/24"]
database_subnets = ["10.0.201.0/24", "10.0.202.0/24"]
}
Provide examples of secure code and let AI learn from them.
Example:
Here’s an EXAMPLE of secure Terraform code for an AWS S3 bucket:
```hcl
resource "aws_s3_bucket" "example" {
bucket = "my-bucket"
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
bucket = aws_s3_bucket.example.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
Now generate a SIMILAR code for an AWS RDS instance with:
- Encryption (AES256)
- No public access
- Minimal IAM permissions
AI is a tool—not a replacement for human expertise.
| Task | GPT-4o | Claude 3.5 Sonnet | Llama 3.1 70B | Codellama 70B |
|---|---|---|---|---|
| Simple EC2 Instance | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Secure VPC Setup | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| IAM Policies (Least Privilege) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| S3 Bucket with Security | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| KMS Encryption | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Multi-AZ RDS Cluster | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Cost Optimization | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
💡 Recommendations:
- Simple tasks: Codellama 70B (cheap & fast)
- Security-critical tasks: GPT-4o or Claude 3.5 Sonnet
- Complex architecture: Always human review + security tools
| Tool | Description | Link |
|---|---|---|
| Checkov | Open-source security scanner for Terraform | GitHub |
| Tfsec | Security scanner focused on best practices | GitHub |
| Snyk IaC | Enterprise security for IaC | Website |
| Infracost | Cost estimation for Terraform | Website |
| TFLint | Linting for Terraform (syntax & best practices) | GitHub |
| Module | Description | Link |
|---|---|---|
| terraform-aws-secure-baseline | Security hardening for AWS | GitHub |
| cis-aws-terraform | CIS Benchmarks as Terraform | GitHub |
| cloudposse-terraform | Secure modules for AWS, GCP, Azure | GitHub |
The study shows: AI can generate IaC—but not always securely. The responsibility lies with you as a DevOps engineer, cloud architect, or security expert.
🎯 Immediate Actions:
🚀 Long-Term:
💡 Final Tip: AI is like a sharp knife—it can help you build great things, but it can also hurt you if misused.
🔗 Original Study: Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation (arXiv:2608.02672v1)
📌 Tags: #Terraform #Security #LLM #IaC #DevOps #Cloud #AI