Centralized Logging with ES

Overview

In this lab we will walk through how to deploy an Elasticsearch cluster and configure the search index through

We will be effectively deploying the landing zone v2.3.1 centralized logging solution manually using StackSets. The AWS Centralized Logging solution used in landing zone is effectively the same version available publicly here.

Prerequisites

  • This lab requires an account with Administrator privileges and Control Tower.
  • Download the landing zone centralized logging add-on zip file located at Centralized Logging Content in your laptop.

Architecture

architecture overview

Steps

  1. Unzip the landing zone centralized logging add-on zip file you downloaded as part of pre-requisites in to a local directory on Linux or OSX OS.

    unzip ~/Downloads/aws-centralized-logging-solution.zip -d ~/Downloads/aws-centralized-logging-solution

    Windows Users : Use appropriate method to extract the content to a local directory.

  2. Login using the SSO account-user created in the lab Deployment.

Create the CloudFormation StackSet for the primary stack

There are two methods to deploy the Centralized Logging.

  1. Via Stack Set pushing 1 account to another (Master Account to Audit Account)
  2. Via Stack 1 account (Audit account only)

Via Stack Set pushing 1 account to another (Master Account to Audit Account)

We are going to use the aws-landing-zone-centralized-logging-primary.template to deploy a single stack instance in the cross account audit account.

  1. Login to the master account
  2. Right click and Open the AWS CloudFormation Console in a new tab
  3. Verify your region
  4. Choose Create StackSet in StackSets.
  5. Choose Upload a template file. Note: This the template you downloaded as part of pre-requisites
    • located at ~/Downloads/aws-centralized-logging-solution/templates/core_accounts
  6. Select the aws-landing-zone-centralized-logging-primary.template
  7. Give the StackSet a good name, alias@AWSLZCLPrimary.
  8. Parameters:

  9. Use + addressing to enter your personal e-mail address, eg alias+ctclv2@amazon.com for both the Elasticsearch Domain Admin email address and the Cognito Admin email address.

    Name Value
    StackSet description default
    OrgID organization ID (o-zzzzzzz)
    DemoVPC default
    ClusterSize small
    DemoSubnet default
    DomainAdminEmail ALIAS+ctlab24@amazon.com
    CognitoAdminEmail ALIAS+ctlab24@amazon.com
    DemoTemplate No
    DemoVPC default
    DOMAINNAME Initialscentralizedlogging
    — —-
    IAM admin role ARN AWSControlTowerStackSetRole
    IAM execution AWSControlTowerExecution
    Account number cross account audit account
    Specify regions region with CT installed
    Deployment options 1 , 1

  10. Choose I acknowledge that AWS CloudFormation might create IAM resources with custom names. Choose I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND

  11. Click Next

  12. Select “Deploy stacks in accounts” and enter the account number for the cross account audit account (AWS Organizations console)

  13. Select the primary region where you have Control Tower deployed

  14. Click Next

  15. For IAM Admin Role ARN, select service-role AWSControlTowerStackSetRole from drop down and AWSControlTowerExecution for IAM Execution Role Name

  16. Click Next

  17. Acknowledge both IAM boxes

  18. Click Submit

  19. Wait for the stack instance to deploy (~15min). If you want to watch the progress, you can use SSO and use the administration console in the cross account audit account to watch the stack complete.

  20. Navigate to the cross account audit account CloudFormation console and copy all of the outputs to a temporary notebook

  21. Watch out the inbox of email address provided earlier for a temporary password and SNS Subscription Notification. We will use the password in next section and go ahead and subscribe for SNS notification.

Create the StackSet for the centralized logging spoke stacks

We are going to use the aws-landing-zone-centralized-logging-spoke.template to create the StackSet for deploying the spokes to the accounts in our organization.

Most customers should consider deploying the spoke stack in all of their accounts, since they are already keeping the logs from those accounts in the archive log bucket. However, for this lab, we are going to deploy to one account in the interest of time.

  1. Login to the master account
  2. Right Click and Open the AWS CloudFormation Console in a new tab
  3. Verify your region
  4. Choose Create StackSet in StackSets
  5. Choose Upload a template to Amazon S3. Note: This the template you downloaded as part of pre-requisites
    • located at ~/Downloads/aws-centralized-logging-solution/templates/aws_baseline
  6. Select the aws-landing-zone-centralized-logging-spoke.template
  7. Give the StackSet a good name, alias@AWSLZCLSpoke.
  8. Parameters:
  9. Use the outputs that you saved from the primary stack
    • Elasticsearch Endpoint
    • Master Account Role
    • Cluster Size
  10. Open a new console tab and get the name of the CloudWatch – Logs – CloudWatchLogs LogGroup – most likely it will be aws-controltower/CloudTrailLogs.
  11. Change the parameter for CloudTrailCloudWatchLogsGroupName to the name you found above, e.g. aws-controltower/CloudTrailLogs

  12. Use the primary region you used for Control Tower for the CloudTrailRegion

    Name Value
    Elasticsearch Endpoint (ESDomain) DomainEndpoint output value from the previous step
    Master Account Role MasterRole output value from the previous step
    Cluster Size small
    Sample Logs No
    VPC CIDR for Sample Sources default
    Subnet for Sample Web Server default
    CloudTrailCloudWatchLogsGroupName aws-controltower/CloudTrailLogs
    CloudTrailRegion specify your region (e.g. us-east-1)
    IAM Admin Role ARN AWSControlTowerStackSetRole
    IAM Execution Role Name AWSControlTowerExecution

  13. Click Next

  14. Select the audit account

  15. Select the primary region you just entered as a parameter and click Add

  16. Click Next

  17. Acknowledge both IAM boxes

  18. Click Create

  19. Now wait (~1min) until at least one of the stack instances is complete before moving to the next step

Login to the Kibana dashboard .

AWS

  1. Using the outputs we save from the primary stack the KibanaLoginURL.
  2. You should have an e-mail in your inbox with the username and password. alt text

You have completed the lab. For more on how to use Kibana see Kibana How To.

Cleanup the lab

  1. Login to the master account.
  2. Right Click and Open the AWS CloudFormation Console in a new tab and check your region.
  3. Delete the spoke StackSet.
  4. Delete the primary StackSet.
  5. Delete the ElasticSearch domain manually in the audit account.