Data Collection Modules and Testing Deployment

Data Collection Modules

These modules templates are managed in AWS owned buckets. If you do not wish to have updates on them from AWS then please save a copy to a bucket in your account and use instead.

Below are the modules we have available in this lab. You can read more about them by expanding the sections. You have selected your chosen modules in the Deploy Main Resources section so no action is needed.

Cost Explorer Rightsizing Recommendations
Inventory Collector
Trusted Advisor
Compute Optimizer Collector
ECS Chargeback Data
RDS Utilization Data
AWS Organization Data Export
AWS Budgets Export

Testing your deployment

Once you have deployed your modules you will be able to test your Lambda function to get your first set of data in Amazon S3.

  1. Depending on the module which you would like to test the following Lambda functions should be triggered:
  • Inventory Collector module -> Accounts-Collector-Function-OptimizationDataCollectionStack Lambda function
  • Trusted Advisor module -> Accounts-Collector-Function-OptimizationDataCollectionStack Lambda function
  • ECS Chargeback Data module -> Accounts-Collector-Function-OptimizationDataCollectionStack Lambda function
  • RDS Utilization Data module module -> Accounts-Collector-Function-OptimizationDataCollectionStack Lambda function
  • AWS Budgets Export module module -> Accounts-Collector-Function-OptimizationDataCollectionStack Lambda function
  • Cost Explorer Rightsizing Recommendations module -> Rightsize-Data-Lambda-Function-OptimizationDataCollectionStack Lambda function
  • Compute Optimizer Collector module -> ComputeOptimizer-Trigger-Export Lambda function
  • AWS Organization Data Export module -> Lambda_Organization_Data_OptimizationDataCollectionStack Lambda function
  1. To test your Lambda function open respective Lambda in AWS Console and click Test Images/lambda_test_cf.png

  2. Enter an Event name of Test, click Create:


  1. Click Test

  2. The function will run, it will take a minute or two given the size of the Organizations files and processing required, then return success. Click Details and view the output.

For Compute Optimizer Collector module processing can take up to 30 mins (15 mins for Compute Optimizer to produce exports requested by lambda, and then another 15 mins for the replication from region buckets to the main bucket)

  1. Go to the Athena service page


  1. You will be able to see your data in the optimization_data Database


  1. If your module has a saved query you will be able to see it in the Saved queries section. Images/Saved_queries.png Otherwise you can query each table directly by clicking on Preview Table button Images/athena_query_table.png

If you have just deployed all resources into your Management Account please see below.

Deployed just into Management Account

If you would like to make your own modules then go to the next section to learn more on how they are made!

Now you have your data in AWS Athena you can use this to identify optimization opportunities using Athena Queries or visualizing data in Amazon QuickSight.