Create Static Resources

Create the Organization data collector using a CloudFormation Template Console

This section is optional and automates the creation of the AWS organizations data collection using a CloudFormation template. You will require permissions to modify CloudFormation templates, create an IAM role, S3 Bucket, Lambda and create a Glue Crawler. If you do not have the required permissions skip over this section to continue using the standard setup.

• Click Launch CloudFormation Template if you are deploying to your linked account (recommended)

• Click Launch CloudFormation Template if you wish to deploy straight into your management account

2. Input the stack name as Organization-data-collector-stack. Then filled in the Parameters. Click Next.

• DatabaseName - Athena Database name where you table will be created
• DestinationBucket - The name you would like of the bucket that will be created in to hold org data, you will need to use a with cost at the start, (we have used cost-aws-lab-organisation-bucket)
• ManagementAccountId - Your Management Account Id where your Optimization is held
• Tags - List of tags from your Organization you would like to include separated by a comma.

3. Scroll down and click Next

4. Scroll down and tick the box acknowledging that this will create and IAM Role. Click Create stack

5. Wait for the CloudFormation to deploy, this can be seen when it has CREATE_COMPLETE under the stack name.

6. Repeat the above steps in your Management account using the Management.yaml template to deploy an IAM Role to allow the lambda to pull data into the Cost Optimization account. If you cannot deploy a CloudFormation into your management account please see the Create IAM Role and Policies in Management account Step further down this page to create manually.

7. Now go back to your linked account and find your deployed CloudFormation template. Select your stack and click on Resources and find the lambda function LambdaOrgData and click on the link to take you to the lambda.

Test Lambda Function

Now that you have deployed the CloudFormation, you must test your Lambda function to get your first set of data in Amazon S3.

1. To test your lambda function click Test

2. Enter an Event name of Test, click Create:

3. Click Test

4. The function will run, it will take a minute or two given the size of the Organizations files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in Organizations file size over time:

5. Go to your S3 bucket and into the organisation-data folder and you should see a file of non-zero size:

6. Go to the Glue Service page:

7. Now that you have deployed your CloudFomation, jump to step 11 on Create Glue Crawler on Utilize Organization Data Source page to run your Glue crawler which will create your Athena table.

Create the Organization data collector using Terraform

There is an AWS Github Repo which has a module to deploy all the resources needed in this lab. Please deploy using the instructions in the github repo then return to the step below.

Test Lambda Function

Now you have deployed the Terraform then you can test your lambda to get your first set of data in Amazon S3.

1. Go to the Lambda service page :

2. Search for your new function called Lambda_Org_Data and click on it. To test your lambda function click Test

3. Enter an Event name of Test, click Create:

4. Click Test

5. The function will run, it will take a minute or two given the size of the Organizations files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in Organizations file size over time:

6. Go to your S3 bucket and into the organisation-data folder and you should see a file of non-zero size is in it:

7. Go to the Glue Service page:

8. Now you have deployed your Terraform jump to step 11 on Create Glue Crawler on Utilize Organization Data Source page to run your crawler to create your athena table.

Create the Organization data collector manually

Create Amazon S3 Bucket and Folders

We’ll create an S3 bucket to store the organizations data to be combined with your cost and usage report. This will hold your organisation data so we can connect it to Athena.

1. Login via SSO to your Cost Optimization account, go into the S3 console:

2. Click Create bucket and create a bucket. You will need to use a unique bucket name with cost at the start, (we have used cost-aws-lab-organisation-bucket). Make a note of this as we will be using it later.

Create IAM Role and Policies

We’ll create an IAM role and policy for the AWS Lambda function to access the organizations data & write it to S3. This role will be used to get the list of accounts in the Organization and the meta data attached to them such as name and email. This is then placed in our S3 bucket.

1. Go to the IAM Console

2. Select Policies and Create policy

3. On the JSON tab the following policy and replace (bucket name) with your bucket name from before and replace (account id) with your Management Account id which manages your Organization. Enter the following policy, click Review policy:

   {
       "Version":"2012-10-17",
       "Statement":[
           {
               "Sid":"S3Org",
               "Effect":"Allow",
               "Action":[
                   "s3:PutObject",
                   "s3:DeleteObjectVersion",
                   "s3:DeleteObject"
               ],
               "Resource":"arn:aws:s3:::(bucket name)/*"
           },
           {
               "Sid":"OrgData",
               "Effect":"Allow",
               "Action":[
                   "organizations:ListAccountsForParent",
                   "organizations:ListRoots",
                   "organizations:ListCreateAccountStatus",
                   "organizations:ListAccounts",
                   "organizations:ListTagsForResource",
                   "organizations:DescribeOrganization",
                   "organizations:DescribeOrganizationalUnit",
                   "organizations:DescribeAccount",
                   "organizations:ListParents",
                   "organizations:ListOrganizationalUnitsForParent",
                   "organizations:ListChildren"
               ],
               "Resource":"*"
           },
           {
               "Sid":"Logs",
               "Effect": "Allow",
               "Action": [
                   "logs:CreateLogGroup",
                   "logs:CreateLogStream",
                   "logs:PutLogEvents",
                   "logs:DescribeLogStreams"
               ],
               "Resource": "arn:aws:logs:*:*:*"
           },
           {
               "Sid": "assume",
               "Effect": "Allow",
               "Action": "sts:AssumeRole",
               "Resource": "arn:aws:iam::(account id):role/OrganizationLambdaAccessRole"
           }
       ]
   }
   

4. Fill in the following

• Policy Name LambdaOrgPolicy
• Description Access to S3 for Lambda function to collect organization data
• Click Create policy

5. Select Roles, click Create role

6. Select Lambda, click Next: Permissions:

7. Type lambda into the search and select the LambdaOrgPolicy policy, click Next: Tags

8. Click Next: Review

9. Role name LambdaOrgRole, click Create role:

Create IAM Role and Policies in Management account

As we need to pull the data from the Management account we need to allow our role to do this.

1. Log into your Management account

2. Go to the IAM Console

3. Select Policies and Create policy. Copy steps 2 - 4 from above to create the below policy called ListOrganizations.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrgData",
            "Effect": "Allow",
            "Action": [
                "organizations:ListAccountsForParent",
                "organizations:ListRoots",
                "organizations:ListCreateAccountStatus",
                "organizations:ListAccounts",
                "organizations:ListTagsForResource",
                "organizations:DescribeOrganization",
                "organizations:DescribeOrganizationalUnit",
                "organizations:DescribeAccount",
                "organizations:ListParents",
                "organizations:ListOrganizationalUnitsForParent",
                "organizations:ListChildren"
            ],
            "Resource": "*"
        }
    ]
}

4. Select Roles, click Create role

5. Choose Another AWS account, and enter your sub account id which is where we started the lab, Next: Permissions

6. Search for Organizations and select the ListOrganizations policy you just made. Click Next: Tags then click Next: Review

7. Role name OrganizationLambdaAccessRole, click Create role:

8. Search for your new role in the roles page and click on the role name. Click on Trusted relationships tab then Edit trusted relationship

9. On the JSON tab the replace the current json with the following policy and replace (sub account id) with your sub account id from before, click Update Trust policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::(sub account id):role/LambdaOrgRole"
            },
            "Action": "sts:AssumeRole"
            }
        ]
    }
   

Bonus Check AWS Organizations Tags

In the next step we will be setting up a lambda to pull the data from your AWS Organization. If you wish to pull your tags from this data too then follow these steps to see your tags.

If you can use the AWS CLI then you can run the below command in your terminal where you have access to your management account to see an individual accounts tags:

aws organizations list-tags-for-resource --resource-id (account id)

Bonus Check AWS Organizations Tags from Console

1. From your Amazon console on the top right of your screen click on the drop down and chose My Organization.

2. Open AWS accounts and you will see your Organization and your Organization structure. This holds your Organizational Units and your Accounts. Click on an account name that you wish to see the tags for.

3. At the bottom of the page, you will see the Tags section. Make a note the Tags Keys in the left-hand box which you wish to collect to use with the CUR in the next step. Note, these are case sensitive so we recommend copy and pasting them.