Inline vs. discrete rules for AWS Security Groups in Terraform

There are two ways to configure AWS Security Groups in Terraform. You may define rules inline with a aws_security_group resource or you may define additional discrete aws_security_group_rule resources.

My first instinct was to define a “base” Security Group using inline rules and then extend on it using external rules. Bad idea. More on that later.

For the two valid options though, there are important implications and I found these were not clear at the time of writing (circa Terraform v0.9.11). After a little research and experimentation I have a much clearer understanding and hope to save you all the bother.

This article focuses on managing AWS Security Groups in Terraform but you will find that all of the principles explored here apply equally to Network ACLs and Route Tables - both of which allow inline or external rule management.

Two approaches

Here’s how an inline Security Group definition looks:

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Above there are two rules, an ingress and egress rule defined inside or inline with the aws_security_group resource block.

Here’s how the same idea can be expressed using external rules via the aws_security_group_rule resource:

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"
}

resource "aws_security_group_rule" "ingress" {
  type        = "ingress"
  from_port   = 0
  to_port     = 0
  protocol    = -1
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

resource "aws_security_group_rule" "egress" {
  type        = "egress"
  from_port   = 0
  to_port     = 0
  protocol    = -1
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

The Security Group and each of its rules are defined as discrete resources, intimately linked together in loving union by the security_group_id attribute.

A reasonable person might posit that the outcome of both configurations would be the same, but they are different in subtle ways - ways that might hurt a bit if not clearly understood.

Option 1: External rules

I had hoped that external rules would function similar to Puppet’s concat module - gathering partial resources defined anywhere in the graph and then enforcing the sum state. The reality however, which does make sense, is that the desired state is managed non-destructively.

You can test this, by manually adding a rule to the Security Group created by the Terraform code above (the snippet with only external rules). If you run terraform apply, it will ignore this manually created rule.

What this means, is that you can add and enforce rules on a Security Group that was created elsewhere. Just be cautious of conflicts with existing rules, precedence and collisions in your rule numbers.

Pros

Cons

Option 2: Inline rules

When rules are defined inline, a Security Group is managed destructively. That is, any rule not defined inline, including rules defined elsewhere in Terraform and rules added manually or via other tools, will be unapologetically destroyed whenever Terraform next runs.

Pros

Cons

Why not both?

My naive first approach was to blend both approaches. I hoped to create a configurable aws_security_group “module” that contained some mandatory rules, like allowing ingress SSH, monitoring, etc. This module could then be extended with additional rules using aws_security_group_rules resources. A kinda pseudo-OOP-abstract-class approach.

Fortunately, the Terraform documentation contained a well-lit warning sign:

At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

What happens when you combine both methods? Here’s a cool high-school science lab experiment for you!

The following Terraform code defines both inline rules, and an external ingress_http rule.

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group_rule" "ingress_http" {
  type        = "ingress"
  from_port   = 80
  to_port     = 80
  protocol    = 6
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

Instructions

  1. Apply this code with terraform apply - it should create the ingress_http rule

  2. Apply the same code again with terraform apply - it should remove the newly created ingress_http rule

  3. Go to step 1

Terraform will create and then destroy the external rule on each alternating invocation - like that scene with Dormammu in Marvel’s Dr Strange.

Bug? No. It actually kinda makes sense. Jake Champlin from HashiCorp explains it on a related GitHub issue.

Summary

I’ve come to prefer using inline rules where possible. It means our Security Groups match the code and phantom ‘allow all’ rules can’t be introduced that would break our security model and the integrity of our tests.

It does mean we have to duplicate some rules in a few Security Groups, and keep these definitions in sync, but I share the opinion that a little copying is better than a little dependency.

For more information about AWS Security Groups in Terraform, please see:

Comments