OSINT: Tools & Techniques for Open Source Intel

OSINT (Open Source Intelligence) is the foundation of every professional penetration test: Before an attacker actively penetrates a system, they often spend hours or days gathering publicly available information. The goal: to obtain a complete picture of the attack surface without sending a single packet to the target network. What attackers see, defenders must also know.

OSINT Framework and Phases

Passive OSINT (without direct interaction with the target):

DNS information from public sources
WHOIS data
Certificate Transparency logs
Google Dorking
Shodan/Censys (cached scans)
Social media and company websites
Job postings (technology stack identifiable!)
Pastebin / Dark Web leaks
Code repositories (GitHub, GitLab)

Semi-Passive OSINT (not directly identifiable):

DNS resolution of public records (A, MX, SPF, DMARC)
Certificate Transparency log queries
Wayback Machine (archive.org)
Web crawling with passive fingerprinting

Active reconnaissance (direct, identifiable):

Port scanning (nmap)
Banner grabbing
Web technology fingerprinting
WAF detection
Subdomain brute force (active, sending DNS requests)

OSINT Framework Categories: osintframework.com provides a categorized overview of all tools. Categories include: Username, Email, Domain, IP/Network, Social Media, Dark Web, Documents, Images, Phone, Business.

DNS Enumeration

# DNS Enumeration - Discovering subdomains and infrastructure

# Passive Subdomain Enumeration (without sending DNS queries to the target)

# subfinder (Project Discovery):
subfinder -d example.com -o subdomains.txt
subfinder -d example.com -all -recursive -o subdomains.txt
# Sources: Certificate Transparency, VirusTotal, Shodan, etc.

# amass (OWASP):
amass enum -passive -d example.com -o amass-passive.txt
amass enum -active -d example.com -o amass-active.txt
amass db -names -d example.com  # From amass's own database

# Certificate Transparency Logs:
  # crt.sh - Query all certificates for a domain:
  curl -s "https://crt.sh/?q=%.example.com&output;=json" | \
    jq -r &#x27;.[].name_value&#x27; | \
    sort -u | grep -v "*"

  # crt.sh also for wildcard subdomains:
  curl -s "https://crt.sh/?q=%.%.example.com&output;=json" | \
    jq -r &#x27;.[].name_value&#x27; | sort -u

# DNS Brute Force (active):
dnsx -l subdomains.txt -a -resp -o resolved.txt
# Tool: puredns (with wordlist)
puredns bruteforce wordlist.txt example.com -r resolvers.txt

# Analyze DNS record types:
  # MX records (email infrastructure):
  dig MX example.com
  # → Google Workspace? Microsoft 365? Own mail server?

  # SPF, DMARC, DKIM:
  dig TXT example.com | grep spf
  dig TXT _dmarc.example.com
  # → Shows email security configuration (or lack thereof!)

  # NS records (nameservers):
  dig NS example.com
  # → Cloudflare? AWS Route 53? Own NS?

  # Attempt an AXFR zone transfer (often not allowed, but try anyway):
  dig @ns1.example.com example.com AXFR
  # If successful: ALL DNS records for the domain!

Google Dorking

# Google Dorking - Sensitive information via search engines

# Basic operators:
#   site:example.com           - Only this domain
#   filetype:pdf               - Only PDFs
#   inurl:/admin               - URL contains /admin
#   intitle:"Index of /"       - Directory listings
#   intext:"confidential"      - Text in the document
#   -site:www.example.com      - Exclude this subdomain

# Practical dorking combinations:

# Find subdomains:
  site:*.example.com -site:www.example.com

  # Login pages:
  site:example.com inurl:login OR inurl:signin OR inurl:auth

  # Configuration files:
  site:example.com filetype:env OR filetype:config OR filetype:cfg

  # Error messages with stack traces:
  site:example.com "stack trace" OR "exception" OR "debug"

  # Open redirects:
  site:example.com inurl:redirect= OR inurl:url= OR inurl:return=

  # Passwords (often in old files):
  site:example.com filetype:txt password OR username

  # Backup files:
  site:example.com filetype:bak OR filetype:backup OR filetype:sql

  # Exposed .git:
  site:example.com inurl:/.git/config

  # phpinfo():
  site:example.com inurl:phpinfo.php

  # Jenkins/CI-CD:
  site:example.com inurl:jenkins OR inurl:gitlab OR inurl:bitbucket

# Google Dorks Automation:
# ghdb-scraper (Google Hacking Database):
python3 ghdb.py -q "site:example.com" -d "vulnerability"
# GoogD0rker:
python3 googd0rker.py -q "site:example.com" -t web

Shodan and Censys

# Shodan - Search engine for connected devices

# Basic Shodan searches:
# Direct IP address:
host: 203.0.113.1

# By ASN:
org:"Example GmbH"
asn:"AS12345"

# By technology:
product:"Microsoft IIS"
product:"Apache httpd" version:"2.4.51"

# By port/service:
port:3389 org:"Example GmbH"    # RDP to org
port:22 country:DE               # SSH in Germany

# Certificate information:
ssl.cert.subject.cn:"*.example.com"
ssl.cert.expired:true org:"Example GmbH"  # Expired certificates!

# Shodan CLI:
# Installation:
pip install shodan
shodan init YOUR_API_KEY

# Domain info:
shodan domain example.com

# IP info:
shodan host 203.0.113.1

# Download search:
shodan search --fields ip_str,port,org "org:'Example GmbH'" \
  --limit 1000 > shodan_results.txt

# Censys (Alternative, more SSL/TLS focus):
# search.censys.io or censys.io/api

# Python API:
from censys.search import CensysHosts
h = CensysHosts()
results = h.search("ip_addresses.reverse_dns.reverse_dns:'example.com'")

# All IPs of an organization:
results = h.search("autonomous_system.organization_id:'NNNNNN'")

# Certificates:
from censys.search import CensysCertificates
c = CensysCertificates()
certs = c.search("parsed.subject.common_name:'*.example.com'")

# What you'll find:
# - Forgotten test servers (example-test.example.com:8080)
# - Outdated SSL certificates (known CVEs)
# - Exposed admin panels (Grafana, Jenkins, Kibana)
# - Default credentials on network devices
# - Exposed databases (MongoDB, Elasticsearch, Redis)

GitHub and Code Repositories

# GitHub OSINT - Finding source code leaks

# GitHub search:
# Search for organization:
org:example-gmbh

# Sensitive files:
org:example-gmbh filename:.env
org:example-gmbh filename:config.json password
org:example-gmbh filename:docker-compose.yml

# API keys in the code:
org:example-gmbh "AWS_SECRET_ACCESS_KEY"
org:example-gmbh "AKIA"               # AWS Key Prefix
org:example-gmbh "-----BEGIN RSA PRIVATE KEY-----"
org:example-gmbh "ghp_"               # GitHub Personal Access Token

# Automated GitHub search:

# trufflehog (Secrets scanner):
trufflehog github --org=example-gmbh
trufflehog git https://github.com/example-gmbh/repo

# gitleaks:
gitleaks detect --source . --report-path leaks.json
gitleaks detect --source https://github.com/example-gmbh/repo

# gitrob:
gitrob analyze --github-access-token TOKEN \
  --organization example-gmbh

# Analyze commits - Search git log (locally):
git log --all -p | grep -i "password\|secret\|key\|token"

# git-secrets (prevents secrets from being committed):
git secrets --install
git secrets --register-aws

# Common Leaks:
# - AWS Access Keys (AKIA... prefix)
# - Private SSH/TLS keys
# - Database passwords in .env files
# - API keys (Stripe, Twilio, SendGrid, etc.)
# - JWT secrets (HMAC key in plain text)
# - Hardcoded production credentials in tests

LinkedIn:

List of employees at the target organization
Technology stack from job postings ("Seeking Python developers with Django and AWS experience" reveals the technology stack)
Derive email format (vorname.nachname@example.com?)
Organizational chart / decision-making structures

LinkedIn OSINT Tools: linkedin2username (employee lists to generate email list), ScrapedIn / ProspectIn (automated LinkedIn scraping)

Email Enumeration:

hunter.io API - returns email format and known email addresses:

curl "https://api.hunter.io/v2/domain-search?domain=example.com&api_key=KEY"

emailhippo / verifalia - check email validity without sending.

WHOIS / Domain Registration:

whois example.com
# Registrant (often anonymized), registrar, creation date
# For older registrations: real contact information!

# Reverse WHOIS (same owner, other domains):
# viewdns.info/reversewhois/?q=admin@example.com

# Domain history: domaintools.com / whoisology.com

Wayback Machine - Old versions of the website: web.archive.org/web/*/example.com

Older versions often contain:

Old employee pages with real email addresses
Previous technology versions (older CMS, etc.)
Deleted sensitive pages (internal, documentation)

Analyze job postings - Job postings reveal the technology stack before the first attack:

"Experience with Fortinet FortiGate preferred" - Firewall vendor known!
"AWS Certified Solutions Architect" - AWS as cloud provider
"SIEM experience with Splunk" - SIEM system known
"Knowledge of SAP ERP" - Business software

OSINT Frameworks and Automation

Maltego:

Graphical OSINT visualization and correlation
Transforms: automated data queries (Shodan, Censys, etc.)
Community Edition: free (limited transforms)
Ideal for: visualizing relationships, identifying attack paths

SpiderFoot (Open Source):

# Installation:
pip install spiderfoot
python3 sf.py -l 127.0.0.1:5001

# CLI scan (automatically: subdomains, IPs, emails, social profiles, leaks):
python3 sfcli.py -s example.com -t DOMAIN -m all -o json

Over 200 modules integrated: Shodan, Censys, VirusTotal, HaveIBeenPwned, Dehashed, Certificate Transparency, Google, Bing, DuckDuckGo, GitHub, GitLab, LinkedIn, Twitter (limited).

Recon-ng (modular recon framework):

recon-ng
# [recon-ng]> marketplace install all
# [recon-ng]> workspaces create example-gmbh
# [recon-ng]> db insert domains
# Domain: example.com
# [recon-ng]> modules load recon/domains-hosts/hackertarget
# [recon-ng]> run
# Subdomains found

theHarvester - Quick email/subdomain enumeration:

theHarvester -d example.com -l 500 -b google,bing,linkedin,shodan

OSINT Report Template:

Target and Scope
Passive Findings (without target system contact)
Subdomain List (+ active IPs)
Email addresses (for phishing simulation scope)
Technology stack analysis
Exposed services (Shodan)
Credential leaks (HIBP, DeHashed)
Recommendations: What should be addressed immediately?