Lab complete!
Now that you have completed this lab, make sure to update your Well-Architected review if you have implemented these changes in your workload.
Click here to access the Well-Architected Tool
To deploy the resource, you need for this lab you have three options. You can deploy using a AWS CloudFormation template or a Terraform module, either allows you to complete the lab in less than half the time as the standard setup. Or, you can deploy manually if you do not have access to deploy using CloudFormation or would like hands one experience going through the steps.
This section is optional and automates the creation of the AWS organizations data collection using a CloudFormation template. You will require permissions to modify CloudFormation templates, create an IAM role, S3 Bucket, Lambda and create a Glue Crawler. If you do not have the required permissions skip over this section to continue using the standard setup.
Click Launch CloudFormation Template if you are deploying to your linked account (recommended)
Click Launch CloudFormation Template if you wish to deploy straight into your management account
Scroll down and click Next
Scroll down and tick the box acknowledging that this will create and IAM Role. Click Create stack
Wait for the CloudFormation to deploy, this can be seen when it has CREATE_COMPLETE under the stack name.
Repeat the above steps in your Management account using the Management.yaml template to deploy an IAM Role to allow the lambda to pull data into the Cost Optimization account. If you cannot deploy a CloudFormation into your management account please see the Create IAM Role and Policies in Management account Step further down this page to create manually.
Now go back to your linked account and find your deployed CloudFormation template. Select your stack and click on Resources and find the lambda function LambdaOrgData and click on the link to take you to the lambda.
Now that you have deployed the CloudFormation, you must test your Lambda function to get your first set of data in Amazon S3.
To test your lambda function click Test
Enter an Event name of Test, click Create:
Click Test
The function will run, it will take a minute or two given the size of the Organizations files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in Organizations file size over time:
Go to the Glue Service page:
Now that you have deployed your CloudFomation, jump to step 11 on Create Glue Crawler on Utilize Organization Data Source page to run your Glue crawler which will create your Athena table.
There is an AWS Github Repo which has a module to deploy all the resources needed in this lab. Please deploy using the instructions in the github repo then return to the step below.
Now you have deployed the Terraform then you can test your lambda to get your first set of data in Amazon S3.
Search for your new function called Lambda_Org_Data and click on it. To test your lambda function click Test
Enter an Event name of Test, click Create:
Click Test
The function will run, it will take a minute or two given the size of the Organizations files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in Organizations file size over time:
Go to the Glue Service page:
Now you have deployed your Terraform jump to step 11 on Create Glue Crawler on Utilize Organization Data Source page to run your crawler to create your athena table.
We’ll create an S3 bucket to store the organizations data to be combined with your cost and usage report. This will hold your organisation data so we can connect it to Athena.
We’ll create an IAM role and policy for the AWS Lambda function to access the organizations data & write it to S3. This role will be used to get the list of accounts in the Organization and the meta data attached to them such as name and email. This is then placed in our S3 bucket.
Go to the IAM Console
Select Policies and Create policy
On the JSON tab the following policy and replace (bucket name) with your bucket name from before and replace (account id) with your Management Account id which manages your Organization. Enter the following policy, click Review policy:
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"S3Org",
"Effect":"Allow",
"Action":[
"s3:PutObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject"
],
"Resource":"arn:aws:s3:::(bucket name)/*"
},
{
"Sid":"OrgData",
"Effect":"Allow",
"Action":[
"organizations:ListAccountsForParent",
"organizations:ListRoots",
"organizations:ListCreateAccountStatus",
"organizations:ListAccounts",
"organizations:ListTagsForResource",
"organizations:DescribeOrganization",
"organizations:DescribeOrganizationalUnit",
"organizations:DescribeAccount",
"organizations:ListParents",
"organizations:ListOrganizationalUnitsForParent",
"organizations:ListChildren"
],
"Resource":"*"
},
{
"Sid":"Logs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Sid": "assume",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::(account id):role/OrganizationLambdaAccessRole"
}
]
}
Fill in the following
Click Next: Review
Role name LambdaOrgRole, click Create role:
As we need to pull the data from the Management account we need to allow our role to do this.
Log into your Management account
Go to the IAM Console
Select Policies and Create policy. Copy steps 2 - 4 from above to create the below policy called ListOrganizations.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OrgData",
"Effect": "Allow",
"Action": [
"organizations:ListAccountsForParent",
"organizations:ListRoots",
"organizations:ListCreateAccountStatus",
"organizations:ListAccounts",
"organizations:ListTagsForResource",
"organizations:DescribeOrganization",
"organizations:DescribeOrganizationalUnit",
"organizations:DescribeAccount",
"organizations:ListParents",
"organizations:ListOrganizationalUnitsForParent",
"organizations:ListChildren"
],
"Resource": "*"
}
]
}
Search for Organizations and select the ListOrganizations policy you just made. Click Next: Tags then click Next: Review
Role name OrganizationLambdaAccessRole, click Create role:
Search for your new role in the roles page and click on the role name. Click on Trusted relationships tab then Edit trusted relationship
On the JSON tab the replace the current json with the following policy and replace (sub account id) with your sub account id from before, click Update Trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::(sub account id):role/LambdaOrgRole"
},
"Action": "sts:AssumeRole"
}
]
}
Now you have completed this section there is a bonus part where you can check the Tags on your AWS Accounts. These can be used in the lambda if you wish.
In the next step we will be setting up a lambda to pull the data from your AWS Organization. If you wish to pull your tags from this data too then follow these steps to see your tags.
If you can use the AWS CLI then you can run the below command in your terminal where you have access to your management account to see an individual accounts tags:
aws organizations list-tags-for-resource --resource-id (account id)
Now that you have completed this lab, make sure to update your Well-Architected review if you have implemented these changes in your workload.
Click here to access the Well-Architected Tool