(In case you’re wondering, yes! This is a hat tip to the lovely book by Al Sweigart, Automate The Boring Stuff With Python)

This blog post is a tribute to my fellow pentesters out there who, like me, love the creative and highly analytical aspects of pentesting but are bored with the highly necessary but highly repetitive tasks that you have to run at the beginning of an engagement. Activities like recon and scanning are crucial to identifying the surface of attack; but the setup of these activities seldom differs from one assessment to another, apart from the domains and IP ranges of course. So these are excellent candidates for automation - which consists in writing a set of scripts to set up and run the scans.

Base tools

For my automation, I’ll be leveraging two tools:

I use a pipenv virtual environment to set up ansible, which you’ll see in the sample repository.

In part one of this series, I’m going to provide an example to spin up a container and run recon.

Next part, I will provide an example for running a port scan with a VM.

For this kind of work, I’ve used the two interchangeably for the most part; I figured that it would be nice to show both, for completeness.

Sample code

All code related to this post is published here: https://github.com/Inf0Junki3/automate-pentest-example.

Spin up a recon with a container

For this example, I’m going to spin up a container on my local machine with Terraform. I could use ansible for this - but I prefer to be consistent. Terraform = spinning up / down. Ansible = configuring / running. So I have two folders in my project:

project
|
+- terraform
|  |
|  +- main.tf
|
+- ansible
   |
   +- ansible.cfg
   |
   +- config.yml
   |
   +- site.yml

I’m using the excellent Spiderfoot project for my recon. Since we’re using the container here, a little bit of prep is in order. From your box, you clone and build the container image:

git clone https://github.com/smicallef/spiderfoot
cd spiderfoot
docker build -t spiderfoot .

At this point, you have a working spiderfoot container. You could upload the image to your registry to be able to re-use it, or you could just keep it local. The next part is to write a terraform script that will spin up an instance of this container. You want to be able to support several containers for several concurrently running assessments, so the best is to use a variable to name your container, like so:

variable "assessment_name" {
  type = string
}

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "3.0.1"
    }
  }
}

# The local docker provider
provider "docker" {
  host = "unix:///var/run/docker.sock"
}

# Pulls the image
resource "docker_image" "spiderfoot" {
  name = "spiderfoot"
  keep_locally = true
}

# Create a container
resource "docker_container" "recon_container" {
  image = docker_image.spiderfoot.image_id
  name  = var.assessment_name
  ports {
    internal = 5001
    external = 5001
    ip = "127.0.0.1"
  }
}

Now that you have the main.tf file written to your terraform directory, you can navigate to that directory and run it:

cd terraform
terraform init
terraform apply -state=acme.state
var.assessment_name
  Enter a value: acme
[...]
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

docker_image.spiderfoot: Creating...
docker_image.spiderfoot: Creation complete after 0s [id=sha256:4cf1dd2d0651cb3125a879abae35c48d23bda1a8df0f6a5001b314f536d22c24spiderfoot]
docker_container.recon_container: Creating...
docker_container.recon_container: Creation complete after 1s [id=80b466b47c2fbd35a0c973446398e8c90e64b99e14c494ab9b453ee5a2499373]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Congratulations! You now have an instance of the spiderfoot container running. Let’s use it to perform our recon.

Setting yourself up for automated recon

Rather than supply arguments via command line, I prefer to use a variable file. So for this example, I’m going to create a config.yml file in the root of my ansible directory, which will contain the following simple settings:

scan_type: passive
scan_timeout_seconds: 7200
assessment_name: acme
domains:
  - idontexist.home.arpa
ip_ranges:
  - 192.168.1.1/32

Then, I execute the commands in my container with ansible. Because recon can run for a long time, I make the scan actions concurrent and set a timeout of two hours:

---
- name: Run recon from a container
  hosts: localhost
  connection: local
  tasks:
  - name: Grab variables from the targets file
    ansible.builtin.include_vars:
      file: config.yml
  - name: Run Recon
    community.docker.docker_container_exec:
      container: "{{ assessment_name }}"
      command: ./sf.py -s "{{ item }}" -u {{ scan_type }}
    loop: "{{ domains | union(ip_ranges) }}"
    async: "{{ scan_timeout_seconds }}"
    poll: 0 # Concurrent.

To run this, I navigate to my ansible folder:

cd ../ansible
pipenv shell
# This command need only be run once:
pipenv install
ansible-playbook site.yml

The long game: why automate?

You might point out that this approach is arguably much longer to set up than just typing the relatively simple commands straight into the command line, and rightfully so. My objective here, however, is consistency:

  • I want to make my scans repeatable
  • I want to be able to make these activities automatically run, as part of a setup workflow for example.
  • I want to be able to be able to consult logs detailing when scan activities occurred.

In the third part of this series, I’ll explain how to integrate these scripts into a system so that they can start bringing more value.