Customizations for Control Tower (CfCT)

Overview

The Customizations for AWS Control Tower solution combines AWS Control Tower and other highly-available, trusted AWS services to help customers more quickly set up a secure, multi-account AWS environment using AWS best practices. Before deploying this solution, customers need to have an AWS Control Tower landing zone deployed in their account. This solution enables customers to easily add customizations to their AWS Control Tower landing zone using an AWS CloudFormation template and service control policies (SCPs). You can deploy the custom template and policies to individual accounts and organizational units (OUs) within your organization. This solution integrates with AWS Control Tower lifecycle events to ensure that resource deployments stay in sync with the customer’s landing zone. For example, when a new account is created using the AWS Control Tower account factory, the solution ensures that all resources attached to the account’s OUs will be automatically deployed. Click here for more information about the solution.

Architecture

architecture overview

What to expect in this lab:

  • Set up the Customizations for Control Tower (CfCT) Solution
  • Deploy an IAM Role in AWS Control Tower Account (Simple Lab)
  • Setup Central Networking using the Serverless Transit Network Orchestrator (STNO) Solution (Advanced Lab)
  • Clean up the TGW lab

Set up the Customizations for Control Tower (CfCT) Solution

  • Deploy Solution

  • After solution is successfully deployed, update KMS Key policy to allow access to S3 bucket (S3 bucket is protected with a KMS key)

    • Navigate to the AWS Key Management Service console.
    • In Customer Managed Keys, select CustomControlTowerKMSKey.
    • Select the Key policy tab. Then, select Edit.
    • In the Edit key policy page, find the Allow Use of the key section in the code, and add one of the following permissions:
      • To add an administration role:
        • arn:aws:iam::<account-ID>:role/<administrator-role>
      • To add a user::
        • arn:aws:iam::<account-ID>:user/<username>
    • Select Save Changes.
  • Download Configuration package from S3 bucket

    • Navigate to the Amazon S3 console, find the S3 bucket (custom-control-tower-configuration-acccountid-region) containing the configuration zip file (_custom-control-tower-configuration.zip), and select download.
    • Unzip the package: it contains an example configuration package.
    • For details about configuration package and how to set it up, see CfCT Developer Guide.

Deploy an IAM Role in AWS Control Tower Account (Simple Lab)

  • Goal: deploy an IAM role in members accounts under Core OU that allows lambda to assume role to make EC2 DescribeRegions API call
  • Set up IAM role configuration package - zip file
    • Download package: custom-control-tower-configuration.zip
      • The package contains manifest.yaml and describe-regions-iam-role.template
        • It will deploy an IAM role in members accounts under Core OU that allows lambda to assume role to make EC2 DescribeRegions API call
  • Upload custom_control_tower_configuration.zip to s3 bucket (custom-control-tower-configuration-acccountid-region).
  • Code Pipeline is triggered. Wait for pipeline execution.
  • After pipeline execution is successfully completed, switch to AWS Management Console for the Core member account to validate that the IAM role is created is the specified member account and region.

Setup Central Networking using Serverless Transit Network Orchestrator (STNO) (Advanced Lab)

STNO setup - Pre-requisite

  • Enable AWS RAM for AWS Organizations Accounts. (Reference here)
    • Sign in to the AWS Organizations console.
    • In the navigation pane, select Settings, scroll down to AWS RAM, and select Enable access.

stno ram setup

  • Enable the sharing option in the AWS RAM console.

stno ram settings

Deployment Steps

  • Create parameter file for hub and spoke.

    • Create a file under custom-control-tower-configuration/parameters/aws-transit-network-orchestrator-hub.json with following input - make the changes to the fields marked like

      [
      {
      "ParameterKey": "Principals",
      "ParameterValue": "arn:aws:organizations::<MASTER_ACCOUNT_ID>:organization/<ORG_ID o-XXXXXXX>"
      },
      {
      "ParameterKey": "PrincipalType",
      "ParameterValue": "AWS Organization ARN"
      },
      {
      "ParameterKey": "ApprovalNotification",
      "ParameterValue": "<Yes/No>"
      },
      {
      "ParameterKey": "ApprovalNotificationEmail",
      "ParameterValue": "<EMAIL_ADDRESS>"
      },
      {
      "ParameterKey": "ConsoleLoginInformationEmail",
      "ParameterValue": "<EMAIL_ADDRESS>"
      }
      ]
      
  • Create a file under custom-control-tower-configuration/parameters/aws-transit-network-orchestrator-spoke.json with following input - again change any to match your environment

    [
    {
    "ParameterKey": "HubAccount",
    "ParameterValue": "<ACCOUNT_NUMBER>"
    }
    ]
    
  • Create a file under custom-control-tower-configuration/parameters/aws-stno-vpc.json with following input - change the regions if desired to match your Control Tower region

    [
    {
    "ParameterKey": "AvailabilityZones",
    "ParameterValue": "us-east-1a,us-east-1b"
    },
    {
    "ParameterKey": "NumberOfAZs",
    "ParameterValue": "2"
    }
    ]
    
  • Updating manifest file with S3 link as a source template

    - name: stno-hub
    template_file: s3://solutions-reference/serverless-transit-network-orchestrator/v1.0.0/aws-transit-network-orchestrator-hub.template
    parameter_file: parameters/aws-transit-network-orchestrator-hub.json
    deploy_method: stack_set
    deploy_to_ou: []
    deploy_to_account:
      - Audit
    regions:
      - us-east-1
          
    - name: stno-vpc
    template_file: s3://solutions-features-reference/aws-serverless-transit-network-orchestrator/v1.0.0beta/aws-stno-vpc.template.yaml
    parameter_file: parameters/aws-stno-vpc.json
    deploy_method: stack_set
    deploy_to_ou: []
    deploy_to_account:
      - Log Archive
    regions:
      - us-east-1
    
    - name: stno-spoke
    template_file: s3://solutions-reference/serverless-transit-network-orchestrator/v1.0.0/aws-transit-network-orchestrator-spoke.template
    parameter_file: parameters/aws-transit-network-orchestrator-spoke.json
    deploy_method: stack_set
    deploy_to_ou:
      - Core
    deploy_to_account: []
    regions:
      - us-east-1
    
  • Upload custom_control_tower_configuration.zip to s3 bucket (custom-control-tower-configuration-acccountid-region).

  • Code Pipeline is triggered. Wait for pipeline execution.

  • After pipeline execution is successfully completed

  • Note: It takes about 30 minutes for the deployment to finish - Most of the time is waiting for CloudFront Distribution Resource creation. You will receive two emails with credentials for the AWS Transit Network Management Console.

Create Transit Gateway Attachment, Association, Propagation and Default Route to TGW

PreCheck

  1. Perform the following verifications after deployment but before running any tests. You will check the same items after running tests, and see the difference before and after tests.
  2. Use SSO Console to login to the Audit (hub account).
    1. Go to the AWS Transit Gateway Attachment Console in hub account, verify that there is no transit gateway attachment for the vpc created by the VPC template.
    2. Go to the AWS Transit Gateway Route Tables Console in hub account, verify that there are four route tables: On-premises, Flat, Infrastructure and Isolated. Go to Associations, *Propagations *and *Routes *tab for each route table, verify that there are no entries under each tab.
  3. Use SSO Console to login to the Log-Archive (spoke account) where we have created the VPC, Subnets and Route Tables.
    1. Go to *Subnets *Console (inside VPC) → select an STNO subnet → Route Table tab, verify that there is one internet gateway (destination 0.0.0.0/0) for all public subnets, and multiple nat gateways (destination 0.0.0.0/0) for all private subnets, one nat gateway per private subnet in the VPC. No existing default route to existing destination is overwritten by STNO routes.

Tagging the resources in the spoke account

Create TGW VPC Attachment

  1. Verify that you are logged with the Log-Archive (spoke account)
  2. Add a tag to subnet 1 in spoke account: Select an STNO subnet (for example: stno-PublicSubnet1) → Tags tab → Add/Edit Tags → add the tag below:
    1. Key: Attach-to-tgw
    2. Value: yes add edit tags
    3. Verify
      1. [Optional] Verifythat the STNO state machine in hub account (Audit) is invoked and a subnet-tagged event is created:
        1. Go to AWS Step Functions Console → go to State machines → select TransitNetworkOrchestratorStateMachine → go to Executions tab, see the latest subnet-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
      2. Go to the AWS Transit Gateway Attachment Console in hub account (Audit) → See a new AWS Transit Gateway Attachment is created → select the AWS Transit Gateway Attachment, got to Details tab → verify that the Subnet ID is listed here. tgw attachment
      3. Go to AWS Subnets Console in spoke account (Log-Archive) → select the subnet being tagged → select Tags tab → Verify that key STNOStatus-Subnet has proper timestamp and information about adding the subnet to the transit gateway in Value column. stno subnet

Update TGW-VPC Attachment - add Subnet

  1. Add a tag to subnet 2 in spoke account: Select another STNO subnet (for example: stno-PublicSubnet2) → Tags tab → Add/Edit Tags → add the tag below. Note this will invoke state machine and create a subnet-tagged event .
    1. Key: Attach-to-tgw
    2. Value: yes
    3. Verify
      1. [Optional] Verify that the STNO state machine in hub account (Audit) is invoked and a subnet-tagged event is created.
        1. Go to AWS Step Functions Console → go to State machines → select TransitNetworkOrchestratorStateMachine → go to Executions tab, see the latest subnet-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
      2. Go to the AWS Transit Gateway Attachment Console in hub account (Audit) → select the attachment → go to Details tab → verify that the Subnet ID is listed here.
      3. Go to *Subnets *Console (inside VPC) in spoke account (Log-Archive) → select the subnet being tagged → select Tags tab → Verify that key STNOStatus-Subnet has proper timestamp and information about adding the subnet to the transit gateway in Value column.

Add TGW Route Table Association and Enable Propagation

  1. Add tags to VPC in spoke account (Log-Archive) : Select the stno-VPC → Tags tab → Add/Edit Tags → add tags:
    1. tag 1:
      1. Key: Associate-with
      2. Value: Flat
    2. tag 2:
      1. Key: Propagate-to
      2. Value: Flat, On-premises, Infrastructure vpc tags
    3. Verify:
      1. [Optional] Verify that the STNO state machine is invoked and a vpc-tagged event is created in hub account (Audit) .
        1. Go to AWS Step Functions Console → go to State machines → select TransitNetworkOrchestratorStateMachine → go to *Executions *tab, see the latest vpc-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
      2. Go to the AWS Transit Gateway Route Tables Console → select route tables one by one → For Flat, go to *Associations *and *Routes *tab, verify that there is an entry for the transit gateway attachment for the VPC under each tab -> For Flat, On-premises and Infrastructure, go to *Propagations *tab, verify there is an entry for the transit gateway attachment for the VPC under *Propagations *tab for each route table. tgw route propagations
      3. Go to AWS VPC Console in spoke account (Log-Archive) → select the stno-VPC → go to Tags tab, verify that these tags are present and have proper timestamp and information in Value column: STNOStatus-VPCAssociation, STNOStatus-VPCAttachment, STNOStatus-VPCPropagation. tag status

Remove subnet(s) from the TGW-VPC Attachment

  1. Remove the tag Attach-to-tgw:yes from one of the STNO subnets that were added to the STNO subnets in spoke account (Log-Archive) in the steps above.
  2. Verify:
    1. Verify that the STNO state machine is invoked and a subnet-tagged event is created.
      1. (Optional) Go to* AWS Step Functions* Console in hub account (Audit) → go to State machines → select TransitNetworkOrchestratorStateMachine → go to *Executions *tab, see the latest subnet-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
    2. Go to *Subnets *Console (inside VPC) in spoke account (Log-Archive) → select the subnet modified → select *Tags *tab → Verify that key STNOStatus-Subnet has proper timestamp and information about removing the subnet from the transit gateway attachment in Value column.
    3. Go to the AWS Transit Gateway Attachment Console in hub account (Audit) → select the attachment for the VPC → go toDetails tab → verify that the Subnet ID is removed from the ID list.
    4. Remove the tag Attach-to-tgw:yes from all STNO subnets.
    5. When ALL the STNO tags are removed from subnets, verify that the Transit Gateway Attachment is deleted.

Update TGW RT Association/Propagation

  1. Change VPC tag in spoke account (Log-Archive): Select the STNO VPC → Tags tab → Add/Edit Tags → update tag:
    1. tag 1:
      1. Key: Associate-with
      2. Value: Infrastructure
    2. tag 2:
      1. Key: Propagate-to
      2. Value: Flat, On-premises, Isolated
    3. Verify:
      1. (Optional) Verify that the STNO state machine is invoked and a vpc-tagged event is created in hub account (Audit).
        1. Go to AWS Step Functions Console in hub account → go to State machines → select TransitNetworkOrchestratorStateMachine → go to *Executions *tab, see the latest vpc-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
      2. Select the STNO VPC in spoke account (Log-Archive) → Tags tab, verify that STNOStatus-VPCPropagation tag has been updated with latest timestamp and information about updating VPC propagation in Value column.
      3. Go to the AWS Transit Gateway Route Tables Console in the hub account (Audit) → select route tables one by one → For Infrastructure, go to Associations and Routes tab, verify that there is an entry for the transit gateway attachment for the VPC under each tab -> For Flat, On-premises and Isolated, go to Propagations tab, verify there is an entry for the transit gateway attachment for the VPC under Propagations tab for each route table.

Remove THE REMAINING subnets from the TGW-VPC Attachment

  1. Remove the tag Attach-to-tgw:yes from the remaining STNO subnets in spoke account (Log-Archive) in the steps above.
  2. Verify:
    1. Verify that the STNO state machine is invoked and a subnet-tagged event is created.
      1. (Optional) Go to* AWS Step Functions* Console in hub account (Audit) → go to State machines → select TransitNetworkOrchestratorStateMachine → go to *Executions *tab, see the latest subnet-tagged event and verify that the execution of the event is completed successfully. In case of failure, click failed step in red in Visual Workflow → go to Step details → document step name, input and exceptions.
    2. Go to *Subnets *Console (inside VPC) in spoke account (Log-Archive) → select the subnet modified → select *Tags *tab → Verify that key STNOStatus-Subnet has proper timestamp and information about removing the subnet from the transit gateway attachment in Value column.
    3. Go to the AWS Transit Gateway Attachment Console in hub account (Audit) → select the attachment for the VPC → go toDetails tab → verify that the Subnet ID is removed from the ID list.
    4. When ALL the STNO tags are removed from subnets, verify that the Transit Gateway Attachment is deleted (together with the associations and propagations).

Cleaning up the TGW lab

I. From the Master account delete the TGW Attachment Spoke StackSet instances within the StackSet

  1. Log in to your AWS Control Tower Master account with the *AWSAdministratorAccess *Role
  2. Make sure you are in the region where CT was deployed in.
  3. Access https://console.aws.amazon.com/cloudformation/stacksets/ to jump to *StackSets *console.
  4. Click on the [STNO-Spoke-] and expand *Actions *button.
  5. Select Manage stacks in StackSets
  6. Select Delete Stacks and click on Next
  7. Select Delete stacks from account. Enter the Log-Archive account number.
  8. Scroll down to Specify regions, and select all the regions under Available regions and click Add->
  9. Click *Next *to continue, and click on Delete stacks
  10. Once all the Stacks are deleted. Click on Delete StackSet button on the top right.

II. From the Master account delete the Transit Gateway Hub StackSet instances with in the StackSet

  1. Log in to your AWS Control Tower Master account with the *AWSAdministratorAccess *Role
  2. Make sure you are in the region where you deployed the StackSet.
  3. Access https://console.aws.amazon.com/cloudformation/stacksets/ to jump to StackSets console.
  4. Click on the [STNO-Hub-] and expand *Actions *button.
  5. Select Manage stacks in StackSets
  6. Select Delete Stacks and click on Next
  7. Select Delete stacks from account. Enter the Audit account number.
  8. Scroll down to Specify regions, and select all the regions under Available regions and click Add->
  9. Click *Next *to continue, and click on Delete stacks
  10. Once all the Stacks are deleted. Click on Delete StackSet button on the top right.

III. From the Master account delete the Transit Gateway VPC StackSet instances with in the StackSet

  1. Log in to your AWS Control Tower Master account with the *AWSAdministratorAccess *Role
  2. Make sure you are in the region where you deployed the StackSet.
  3. Access https://console.aws.amazon.com/cloudformation/stacksets/ to jump to StackSets console.
  4. Click on the [STNO-VPC-] and expand *Actions *button.
  5. Select Manage stacks in StackSets
  6. Select Delete Stacks and click on Next
  7. Select Delete stacks from account. Enter the Audit account number.
  8. Scroll down to Specify regions, and select all the regions under Available regions and click Add->
  9. Click *Next *to continue, and click on Delete stacks
  10. Once all the Stacks are deleted. Click on Delete StackSet button on the top right.

Clean up the Lab

  • Sign in to the AWS Management CloudFormation Console
  • Delete the Customizations for Control Tower stack

References: