If you are a red teamer or a penetration tester you got to love tools which automate your discovery process and make your life easier. One powerful tool you should definetly try out to perform automated OSINT for your security assessments is SpiderFoot.
What is SpiderFoot
SpiderFoot is an automated open source intelligence (OSINT) tool which integrates with more than 100 data sources to help you gather and analyze data about your target. Some integrations include internet scanners (e.g. shodan, censys), breach databases and email contact databases.
SpiderFoot is an open source tool which you can download and install for free from its github page. It also provides a paid service called SpiderFoot HX which is cloud based and has more capabilities like multiple targets per scan, TOR integration, multi-user collaboration and more.
Why Use It?
Employing OSINT as part of your security assessments is crucial in identifying areas of exposure for your target. For example, you may discover exposed AWS S3 buckets, critical information in Git repositories and leaked credentials.
Understanding what a person or an organization exposes to the outside world is significant, both for the defender and the attacker.
How to Install SpiderFoot
To start leveraging the spiderfoot’s automated OSINT capabilities for your security assessments or penetration tests, simply download the build from github, install the python requirements and run it. Note that you will need at least python 3.7 to be installed on your machine.
wget https://github.com/smicallef/spiderfoot/archive/v3.5.tar.gz
tar zxvf v3.5.tar.gz
cd spiderfoot-3.5
pip3 install -r requirements.txt
python3 ./sf.py -l 127.0.0.1:5001Get Your API Keys
To get the most out of this tool, it is highly recommended that you register for the services which SpiderFoot is using and get an API key. You will get a lot more information from your scans if you do.
It will take you some time to complete the registrations and configure and acquire the API keys, but it will be worth it.
Once you get your API keys, run spiderfoot and go to its web gui, navigate to Settings and you will see the list of services spiderfoot intergrates with. The ones with the lock icon indicate that you need to have an API key.
Once you have all your keys setup, i recommend that you export your API keys and save them in a secure location. This way you can quickly import them if you need them for another installation.
Run Your Automated OSINT Investigation
First enter the scan target, depending on the information you have on your target. It may be a domain name of a company, if you are doing OSINT as part of a penetration test exercise, an IP address, email address of a person/employee, username or even a real human name.
You may now run your automated OSINT investigation using one of many options.
By Use Case
By Required Data
By Module
The scan time will depend on your selection.
Examining the Scan Results
You have waited patiently for the scan the complete. Now you can go to your your “Scans” and click on the one you want to view the results for.
The summary of the results will look something like this:
Clicking on the “Browse” button will show you a list of the type of scan performed on the target and the data elements found.
If you are doing OSINT for a company, you will want to check fields like: the IP address, Email addresses, Nameservers, username, SPF and TXT records.
Doing a search on a person using a username you may know the person is using or his/her full name is trickier. You will need to go through the results manually and verify their correctness. Many people may have the same name and may use the same username.
Now, go out there and do some recon sherlock.
Dimitris is an Information Technology and Cybersecurity professional with more than 20 years of experience in designing, building and maintaining efficient and secure IT infrastructures.
Among others, he is a certified: CISSP, CISA, CISM, ITIL, COBIT and PRINCE2, but his wide set of knowledge and technical management capabilities go beyond these certifications. He likes acquiring new skills on penetration testing, cloud technologies, virtualization, network security, IoT and many more.
