(In case you’re wondering, yes! This is a hat tip to the lovely book by Al Sweigart, Automate The Boring Stuff With Python)
This blog post is a tribute to my fellow pentesters out there who, like me, love the creative and highly analytical aspects of pentesting but are bored with the highly necessary but highly repetitive tasks that you have to run at the beginning of an engagement. Activities like recon and scanning are crucial to identifying the surface of attack; but the setup of these activities seldom differs from one assessment to another, apart from the domains and IP ranges of course. So these are excellent candidates for automation - which consists in writing a set of scripts to set up and run the scans.
Base tools
For my automation, I’ll be leveraging two tools:
- Terraform to spin up the base environment (https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
- Ansible to configure and run the environment
I use a pipenv virtual environment to set up ansible, which you’ll see in the sample repository.
In part one of this series, I’m going to provide an example to spin up a container and run recon.
Next part, I will provide an example for running a port scan with a VM.
For this kind of work, I’ve used the two interchangeably for the most part; I figured that it would be nice to show both, for completeness.
Sample code
All code related to this post is published here: https://github.com/Inf0Junki3/automate-pentest-example.
Spin up a recon with a container
For this example, I’m going to spin up a container on my local machine with Terraform. I could use ansible for this - but I prefer to be consistent. Terraform = spinning up / down. Ansible = configuring / running. So I have two folders in my project:
project
|
+- terraform
| |
| +- main.tf
|
+- ansible
|
+- ansible.cfg
|
+- config.yml
|
+- site.yml
I’m using the excellent Spiderfoot project for my recon. Since we’re using the container here, a little bit of prep is in order. From your box, you clone and build the container image:
git clone https://github.com/smicallef/spiderfoot
cd spiderfoot
docker build -t spiderfoot .
At this point, you have a working spiderfoot container. You could upload the image to your registry to be able to re-use it, or you could just keep it local. The next part is to write a terraform script that will spin up an instance of this container. You want to be able to support several containers for several concurrently running assessments, so the best is to use a variable to name your container, like so:
variable "assessment_name" {
type = string
}
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
version = "3.0.1"
}
}
}
# The local docker provider
provider "docker" {
host = "unix:///var/run/docker.sock"
}
# Pulls the image
resource "docker_image" "spiderfoot" {
name = "spiderfoot"
keep_locally = true
}
# Create a container
resource "docker_container" "recon_container" {
image = docker_image.spiderfoot.image_id
name = var.assessment_name
ports {
internal = 5001
external = 5001
ip = "127.0.0.1"
}
}
Now that you have the main.tf file written to your terraform directory, you can navigate to that directory and run it:
cd terraform
terraform init
terraform apply -state=acme.state
var.assessment_name
Enter a value: acme
[...]
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
docker_image.spiderfoot: Creating...
docker_image.spiderfoot: Creation complete after 0s [id=sha256:4cf1dd2d0651cb3125a879abae35c48d23bda1a8df0f6a5001b314f536d22c24spiderfoot]
docker_container.recon_container: Creating...
docker_container.recon_container: Creation complete after 1s [id=80b466b47c2fbd35a0c973446398e8c90e64b99e14c494ab9b453ee5a2499373]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Congratulations! You now have an instance of the spiderfoot container running. Let’s use it to perform our recon.
Setting yourself up for automated recon
Rather than supply arguments via command line, I prefer to use a variable file. So for this
example, I’m going to create a config.yml
file in the root of my ansible directory, which
will contain the following simple settings:
scan_type: passive
scan_timeout_seconds: 7200
assessment_name: acme
domains:
- idontexist.home.arpa
ip_ranges:
- 192.168.1.1/32
Then, I execute the commands in my container with ansible. Because recon can run for a long time, I make the scan actions concurrent and set a timeout of two hours:
---
- name: Run recon from a container
hosts: localhost
connection: local
tasks:
- name: Grab variables from the targets file
ansible.builtin.include_vars:
file: config.yml
- name: Run Recon
community.docker.docker_container_exec:
container: "{{ assessment_name }}"
command: ./sf.py -s "{{ item }}" -u {{ scan_type }}
loop: "{{ domains | union(ip_ranges) }}"
async: "{{ scan_timeout_seconds }}"
poll: 0 # Concurrent.
To run this, I navigate to my ansible folder:
cd ../ansible
pipenv shell
# This command need only be run once:
pipenv install
ansible-playbook site.yml
The long game: why automate?
You might point out that this approach is arguably much longer to set up than just typing the relatively simple commands straight into the command line, and rightfully so. My objective here, however, is consistency:
- I want to make my scans repeatable
- I want to be able to make these activities automatically run, as part of a setup workflow for example.
- I want to be able to be able to consult logs detailing when scan activities occurred.
In the third part of this series, I’ll explain how to integrate these scripts into a system so that they can start bringing more value.