I came across WebDriver (and Selenium) years ago when I was working as a web developer and wanted to unit test my code. It’s a utility that is especially useful for testing out how a browser will react to your web application; however, it can be a very practical tool to have in your bag of tricks during a pentest.
One should note that as far as security tools are concerned, we pentesters are spoilt for choice. I automate most of my pentesting activities with tools such as OWASP ZAP or Burp Suite. In fact, David Fletcher of Black Hills wrote a great article a few years ago about leveraging macros with intruder to automate multi-form fuzzing (https://www.blackhillsinfosec.com/using-simple-burp-macros-to-automate-testing/), which is tremendously useful. However, there are times when automating the browser is faster and simpler - at least for me. Here are a couple of use cases that I’ve encountered, along with some sample source code.
Getting started: setting up your environment
The minimum setup you’ll need will include python and the selenium package. I would also recommend having packages like BeautifulSoup and requests. Finally, you’ll need a driver for the browser of your choice. You can find these here: https://www.seleniumhq.org/download/. The driver (i.e. geckodriver) will need to be deployed to a directory in your PATH.
One thing I would highly recommend is to use jupyter notebook. I’ve found it invaluable in documenting my pentests, it is a simple setup and it produces nicely-formatted text that is easy to copy-paste into reports. Not 100% necessary of course, but since it is easy to set up, I find it’s better to have it and not need it than need it and not have it.
Case 1: automated website screenshots
Consider the following scenario: you’re conducting a pentest of a client’s external perimeter. You have a limited amount of time to short-list the hosts that you’re going to examine in-depth, and you want to be able to identify different types of services so that you can get a representative sample of what’s running on your client’s public-facing infrastructure.
With a large scope of IP addresses, you’re going to find that some IP’s redirect to the same host; alternatively, your client may have several branches and use the same software for each branch, such as VPN’s, file sharing services, etc. One method I use to quickly comb through lots of hosts is by using website screenshots: even with hundreds of IP addresses, I can pick out interesting subsets of hosts by perusing through their screenshots in a folder.
I’ll often start my pentests with a vulnerability scan. The example I’ve put together uses a Nessus report to enumerate sites to screenshot, but one could easily imagine adapting this code to something open source like Nmap or OpenVAS. In fact, it may be smarter to pull the host information from a tool that can parse several scanners’ input such as dradis.
Without further ado, here is some code:
from bs4 import BeautifulSoup
# Step one: open the nessus report so that it can be manipulated with BeautifulSoup.
current_scan_file = "my_scan.nessus" # my_scan is the name of your nessus file
with open(current_scan_file, "r") as nessus_file:
nessus_contents = nessus_file.read()
soup = BeautifulSoup(nessus_contents, "xml")
# Step two: identify hosts of interest.
syn_items = soup.find_all(pluginID="11219")
syn_blacklist = []
hosts = []
for syn_item in syn_items:
cur_ip = syn_item.parent.find("tag", {"name": "host-ip"}).text.strip()
port = syn_item.get("port")
if port not in syn_blacklist:
type_of_service = syn_item.get("svc_name").strip()
if type_of_service in ("www", "http", "http?", "https", "https?", "ii-admin", "ii-admin?"):
hosts.append("{}:{}".format(cur_ip, port))
# Step three: grab screenshots of each host.
from selenium import webdriver
driver = webdriver.Firefox()
for host in hosts:
try:
if ":80" in host or ":81" in host: # This is simplistic and would need to be tweaked
proto = "http://"
else:
proto = "https://"
driver.get("{}{}".format(proto, host))
print(driver.title)
driver.save_screenshot("screenshots/{}.png".format(host))
except Exception:
print("Exception occurred for {}".format(host))
pass
driver.close()
The script is largely self-explanatory - it opens a nessus file, looks for entries associated to the “SYN scan” plugin and for each entry, checks if the service reported is a web service. If it is a web service, the script appends the endpoint to the list of endpoints to check.
Then, the script creates a webdriver.Firefox instance. This instance is what pilots the geckodriver driver. It then pilots the browser. Let’s focus on three important lines in the script:
driver.get("{}{}".format(proto, host))
This causes the browser to navigate to the endpoint. I put this in a try block because sometimes the web application will respond to a SYN but not actually allow a connection, perhaps due to application-level filtering or authentication via client certificates.
print(driver.title)
If your target range is large, it is useful to know how far you’ve gotten. Printing out the page’s title is a simple way of keeping track how many requests have been made and how many are left.
More importantly still: WebDriver allows you to interact with the page, either by reading information from objects in the page or by clicking or entering text in fields of the web page. We’ll see another example of this in the next use case.
driver.save_screenshot("screenshots/{}.png".format(host))
Extracting a screenshot of the page is merely a question of calling the save_screenshot method of the driver object, specifying a location for the resulting PNG file. Once the screenshots have been saved, use your favorite file manager to peruse through the screenshot thumbnails in order to identify hosts of interest.
Case 2: fuzzing complicated forms
Burp macros are great for automating relatively uncomplicated multi-form fuzzing; however, their downside is that they can be a real pain to set up, especially if the application makes complicated calls to many different forms. One case that I recently had to deal with was finding cross-site scripting issues in a SAML implementation: multiple endpoints are called, with random field values, redirecting the browser from one site to another. It could probably be automated with Burp macros – but previous experience with macros suggested that setting this up with Burp would take a whole lot more time and patience that I had at my disposal. I therefore chose to automate my fuzzing with WebDriver. This greatly simplified the process.
from selenium import webdriver
from time import sleep
driver = webdriver.Firefox()
# This is our target URL. The curly braces is where we will inject our fuzz values:
target_url = "https://mysite.local/authenticate.php?username=placeholder{}"
# Note the presence of "placeholder". We'll use this later on to look for results.
# Start with a couple of basic values
fuzz_values = ["<img>", "<img onerror='alert(1)'"]
# Let's try every possible character to see which if it triggers errors, or if the output is encoded.
for i in range(256):
fuzz_values.append("%{}aaa".format(i)) # We add "aaa" at the end to see if anything gets truncated.
# Let's try the big list of naughty strings (https://github.com/minimaxir/big-list-of-naughty-strings):
with open("blns.txt") as blns:
cur_line = blns.readline()
while cur_line:
if cur_line[0] != "#":
fuzz_values.append(cur_line.strip())
cur_line = blns.readline()
# Write results.
with open("fuzz_results.txt", "wb") as fuzz_results:
for fuzz_value in fuzz_values:
fuzz_results.write(b"Fuzz value: {}".format(fuzz_value)) # We track what was tested in the results file.
driver.get(target_url.format(fuzz_value))
sleep(1) # Pause while waiting for the form submission's effect. Adjust this time to your convenience.
page_source = driver.page_source
# If the page is empty, we assume there is an error.
if len(page_source.split("\n")) == 0:
fuzz_results.write(u"\tResult: Error.\n")
continue
# Now we go through the page's source code, looking for our "placeholder".
for line in page_source.split("\n"):
if 'placeholder' in line:
m = re.search('(?<=placeholder).*', line)
if m is not None:
result = m.group(0).encode("utf-8")
fuzz_results.write(b"\tResult: {}\n".format(result))
else:
fuzz_results.write(u"Result: Error.\n")
fuzz_results.flush()
driver.close()
This script is a bit rougher, and would undoubtedly need tweaking on a per-application basis. Here are a few highlights:
driver.get(target_url.format(fuzz_value))
sleep(1)
Here, we request the page that we wish to fuzz. However: once the browser makes the request, it gets redirected to several endpoints automatically. For the sake of simplicity, we pause the execution of the script here with sleep enough time to let it redirect to the final endpoint. One could refine this by instructing the script to wait until driver.current_url matches a certain keyword. This may help if the service does not respond consistently.
page_source = driver.page_source
There are several ways of retrieving values from a web page with WebDriver. I had started by using driver.find_element_by_id and writing out the resulting object’s text property (see https://selenium-python.readthedocs.io/locating-elements.html for more info). However, the resulting text was interpreted by the browser - in other words, any html-entities encoded text did not show up as such. If you’re looking at output encoding, the best thing to do is to use page_source.
The bigger picture
Selenium’s WebDriver is a great way of quickly stimulating and extracting information from web applications, and can be a great, simple alternative to Burp macros. I’ve gone over two use cases for which I’ve used this tool, but it can do so much more as evidenced by the documentation. Remember that you can also combine this tool with pentesting proxies such as Burp for even more flexibility!
Happy hunting.