Magpie Cookbook #3: AWS Asset Reports Made Easy
Overview
Welcome to part 3 of the Magpie Cookbook series (Part 1, Part 2), where we demonstrate practical uses for Magpie in your environment. The goal of these cookbooks is to provide easy to follow guides for extracting useful information, actions, or automations using Magpie.
Today’s cookbook is generating asset lists out of AWS in CSV format. The AWS CLI does allow for limited asset lists, but we’re going to utilize the flexibility of Magpie + jq to go beyond what the AWS CLI offers.
The AWS CLI lacks the ability to generate the necessary multi-region and multi-service lists for critical visibility: audit responses, inventory management or specific investigations. Today, we’ll do both with Magpie.
Prerequisites
For this cookbook you’ll need the following on your local system:
To see examples on using credentials stored in ~/.aws/credentials instead, see Cookbook #1
Running the Commands
The first command will create a multi-region single-service (EC2) report showing all running EC2 instances for an account. The fields we put into the CSV are arbitrary, we’ve chosen the following: arn, resource name, region, and instance type.
To generate this CSV we’ll make use of jq’s @csv annotation.
docker run -a stdout -a stderr --env MAGPIE_CONFIG_="{'/plugins/magpie.aws.discovery/config/services': ['ec2']}" \
-e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \ quay.io/openraven/magpie:latest 2> magpie.log \
| jq --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -r 'select ( .resourceType == "AWS::EC2::Instance") | select(.configuration.state.name == "running") | [.arn, .resourceName, .awsRegion, .configuration.instanceType]' \
| jq -r -s '["arn", "resourceId", "region", "instanceType"], .[]| @csv'
The next command does a multi-region, multi-service discovery. In this example we’ve chosen Route53, RDS, and VPC:
docker run -a stdout -a stderr --env \
MAGPIE_CONFIG="{'/plugins/magpie.aws.discovery/config/services': ['rds', 'vpc', 'route53']}" \
-e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
quay.io/openraven/magpie:latest 2> magpie.log \
| jq --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -r 'select (.resourceType == "AWS::EC2::VPC" or .resourceType == "AWS::Route53::HostedZone" or .resourceType == "AWS::RDS::DBInstance") | [.resourceType, .arn, .resourceName, .awsRegion]' \
| jq -r -s '["resource type", "arn", "resourceId", "region"], .[]| @csv'
Commands Breakdown
We’ve incrementally built upon the scan command first introduced in Cookbook #1. Let’s review some of the aspects here.
The first few lines of the first command:
docker run -a stdout -a stderr \
--env MAGPIE_CONFIG_="{'/plugins/magpie.aws.discovery/config/services': ['ec2']}" \
-e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \|
quay.io/openraven/magpie:latest 2> magpie.log \
Executes Docker, attaching both stdin and stdout to the container, allowing us to redirect errors, logging, and JSON output to files or pipe it to another command. We also limit our Magpie scan to only the EC2 scanner. We’ll narrow this filter even further later in the command.
We attach the required AWS_* environmental variables to the container. The AWS SDK utilized by Magpie will pick these up as one of their default authentication sources. We then specify the Magpie Docker image to use and redirect errors and logging to magpie.log.
Next we harness the power of jq to filter, extract, and transform the JSON results:
| jq --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -r 'select ( .resourceType == "AWS::EC2::Instance") | select(.configuration.state.name == "running") | [.arn, .resourceName, .awsRegion, .configuration.instanceType]' \
The first line allows consumption of the output in stream form, meaning we don’t need to wait for Magpie completion before running it through the filter chain. Next, we filter out only EC2 instance resources that are in the running state. Then it is transformed into an array consisting of ARN, resource name, region, and EC2 instance type.
The final line prepends header titles to the output and then uses jq’s @csv annotation to output it as a CSV.
| jq -r -s '["arn", "resourceId", "region", "instanceType"], .[]| @csv'
The second command operates in a similar fashion except that we specify multiple services to discover, we don’t filter based on resource type, and we choose slightly different output fields.
Results
The output is a CSV consisting of running EC2 instances across all regions and is trivial to import into a spreadsheet (the output has been sanitized for security and privacy):
The second command results in a single CSV for all Route 53 domains, VPCs, and RDS databases across all regions:
Magpie provides easy integration with a variety of toolings that fit in your automation pipelines. So far we’ve demonstrated JSON, Nmap integration, and now CSV asset reports. See the Magpie Github Repository for source, builds, and documentation.
