Magpie Cookbook #2: Discover and Portscan Your Public EC2 Instances
Overview
Welcome to part 2 of the Magpie Cookbook series, where we'll be demonstrating practical uses for Magpie in your environment. The goal of these cookbooks is to provide easy to follow guides for extracting useful information, actions, or automations using Magpie.
Today's cookbook is discovering and port scanning public EC2 instances within your AWS account. It's worth noting that pen testing on your AWS resources are explicitly permitted for certain services.
AWS leaves much to be desired for enumerating active services and their accessibility. We've built this cookbook to help you easily discover all public EC2 instances along with their listening services.
Prerequisites
For this cookbook you'll need the following on your local system:
All examples in this post assume you have AWS credentials set via environmental variables. To use your ~/.aws/credentials folder please see the examples from Cookbook #1.
Running the Command
Let's start off by generating a JSON report of all instances with public IPs:
docker run -a stdout -a stderr --env MAGPIE_CONFIG_="{'/plugins/magpie.aws.discovery/config/services': ['ec2']}" \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN quay.io/openraven/magpie:latest 2> magpie.log \
| jq --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -c '.instances[]? | select (.publicIpAddress != null) | {"instanceId" : .instanceId, "availabilityZone" : .placement.availabilityZone, "publicIp" : .publicIpAddress'}
This uses the same technique from Cookbook #1 to filter Magpie's JSON output and translate them into report objects. The output will stream until you have the full report:
{"instanceId":"i-cafebeef0f2907d21","availabilityZone":"us-east-1a","publicIp":"xx.yyy.24.173"}
{"instanceId":"i-cafebeefdaba07b1a","availabilityZone":"us-west-2b","publicIp":"xx.yyy.97.113"}
{"instanceId":"i-cafebeef9b4ef1041e","availabilityZone":"us-west-2a","publicIp":"xx.yyy.161.126"}
{"instanceId":"i-cafebeefe58e39242f","availabilityZone":"us-west-2d","publicIp":"xx.yyy.149.158"}
{"instanceId":"i-cafebeef70f2907d21","availabilityZone":"us-east-1a","publicIp":"xx.yyy.24.173"}
{"instanceId":"i-cafebeefdaba07b1a","availabilityZone":"us-west-2b","publicIp":"xx.yyy.97.113"}
{"instanceId":"i-cafebeef9b4ef1041e","availabilityZone":"us-west-2a","publicIp":"xx.yyy.161.126"}
{"instanceId":"i-cafebeefe58e39242f","availabilityZone":"us-west-2d","publicIp":"xx.yyy.149.158"}
We can take this one step further and transform the output into something that can be parsed by nmap. This allows us to run a portscan on each public instance:
docker run -a stdout -a stderr --env MAGPIE_CONFIG_="{'/plugins/magpie.aws.discovery/config/services': ['ec2']}" \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN quay.io/openraven/magpie:latest 2> magpie.log \
| jq --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -c '.instances[]? | select (.publicIpAddress != null) | .publicIpAddress' \
| tr -d '"' | nmap -Pn -iL=- -oN -
This will stream directly to nmap and we'll see the following output:
# Nmap 7.80 scan initiated Tue Apr 13 17:00:38 2021 as: nmap -Pn -iL=- -oN -
Nmap scan report for ec2-xx-yyy-24-173.compute-1.amazonaws.com (xx.yyy.24.173)
Host is up (0.022s latency).
Not shown: 999 filtered ports
PORT STATE SERVICE
22/tcp open ssh
Nmap scan report for ec2-xx-yyy-97-113.us-west-2.compute.amazonaws.com (xx.yyy.97.113)
Host is up (0.095s latency).
Not shown: 999 filtered ports
PORT STATE SERVICE
22/tcp open ssh
3306/tcp open mysql
...[output snipped]...
Nmap scan report for ec2-xx-yyy-149-158.us-west-2.compute.amazonaws.com (xx.yyy.149.158)
Host is up (0.096s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
22/tcp open ssh
5432/tcp open postgresql
Command Breakdown
The scan command starts off in the same manner as our previous cookbook, so we'll focus on parsing the nmap command here.
| jq -c '.instances[]? | select (.publicIpAddress != null) | .publicIpAddress' \
We start off by looking at any response objects that contain an 'instances' array. The question mark after the brackets allows us to gracefully ignore objects that don't contain this field. Next we filter out any fields that have a null 'publicIpAddress' field and then pipe that raw value as output (note: this is not a JSON object like last week).
The final line of the command is:
| tr -d '"' | nmap -Pn -iL=- -oN -
The first piped command runs tr and filters out the double quotes wrapping the IP addresses. We then run nmap with the following parameters:
-Pn (don't attempt to ping the target first)
-iL=- (read from stdin, with IP addresses separated by a newline, space, or comma)
-oN - (print a compacted output, which is easier to read when multiple hosts are scanned).
By default nmap will perform a TCP SYN scan against a default set of 1000 commonly used ports. Modify the command as needed to perform full, stealth, or UDP scans.
Results
If your account has EC2 instances with public IP addresses, you'll see the output of an nmap scan, one per address. If you have no public instances, congratulations! You'll instead see:
# Nmap 7.80 scan initiated Wed Apr 14 13:46:54 2021 as: nmap -Pn -iL=- -oN -
WARNING: No targets were specified, so 0 hosts scanned.WARNING: No targets were specified, so 0 hosts scanned.
# Nmap done at Wed Apr 14 13:46:54 2021 -- 0 IP addresses (0 hosts up) scanned in 0.02 seconds
If you'd like to know more about Magpie check out the Github repository. To get regular development updates and chat with the core Magpie developers and the community join us on Slack.
