appsecco / practical-recon-levelup0x02 Sign up

Code Issues 0 Pull requests 0 Projects 0 Security Pulse

Dismiss
Join GitHub today
GitHub is home to over 36 million developers working together to host
and review code, manage projects, and build software together.

Sign up

Tree: 200c43b58e practical-recon-levelup0x02 / practical_recon.md Find file Copy path

0xbharath Updated twitter handle 200c43b on Jan 24

1 contributor

782 lines (463 sloc) 21.7 KB Raw Blame History

Practical recon techniques for bug hunters & pen testers

 #### Bharath Kumar

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 LevelUp 0x02 | May 26th 2018

About me
Bharath Kumar
Live from Bangalore, India
Security Engineer @Appsecco
Offensive Security Certified Professional(OSCP)

Demo environment

Feel free to run the DNS & DNSSEC attacks mentioned in this talk against the following nameservers and
domain names

**Nameservers**
ns1.insecuredns.com
ns2.insecuredns.com

**Domains** - **totallylegit.in** - **insecuredns.com**

What is this talk about?

This talk is about practical recon techniques that are useful for bug bounty hunters and penetration testers
The objective of this talk is to cover exhaustive number of practical recon techniques, tools of trade and
tips/tricks

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Note: By practical I mean that these techniques covered can actually be used during a security assessment. The
talk will be crisp and concise. We demonstrate quick and effective ways to apply a technique in such a way that the
audience can use them in their assessments right away

WHAT IS RECONNAISSANCE?

Reconnaissance is the act of gathering preliminary data or intelligence on your target. The data is
gathered in order to better plan for your attack. Reconnaissance can be performed actively or passively.

What do we look for during recon?

1. Info to increase attack surface(domains, net blocks)

2. Credentials(email, passwords, API keys)
3. Sensitive information
4. Infrastructure details

Enumerating domains

The objective is to find/correlate all domain names owned by a single entity of our interest.

Types of domain correlation

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://0xpatrik.com/asset-discovery/

What is sub-domain enumeration?

Sub-domain enumeration is the process of finding subdomains for one or more domain(s).

Using popular search engines

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Search engines like Google and Bing supports various advanced search operators to refine search queries.
site: is helpful in doing vertical domain correlation(sub-domains)

ip: is helpful in doing horizontal domain correlation

Using 3rd party information aggregators

VirusTotal runs its own passive DNS replication service, built by storing DNS resolutions performed when
visiting URLs submitted by users.

https://www.virustotal.com/#/home/search

A script that uses VirusTotal to find sub-domains

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Script - https://git.io/vhqBF

Quick tip

I like using shell functions to quickly perform some recon tasks

find-subdomains-vt()
{ curl -s https://www.virustotal.com/ui/domains/$1/subdomains\?limit\=$2 | jq .data[].id; }

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Using 3rd party information aggregators
viewdns.info is a handy service for all the DNS and WHOIS related recon

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Certificate Transparency
Under CT, a Certificate Authority(CA) will have to publish all SSL/TLS certificates they issue in a public log
Anyone can look through the CT logs and find certificates issued for a domain
Details of known CT log files - https://www.certificate-transparency.org/known-logs

https://blog.appsecco.com/certificate-transparency-part-2-the-bright-side-c0b99ebf31a8

Certificate Transparency - side effect

CT logs by design contain all the certificates issued by a participating CA for any given domain
By looking through the logs, an attacker can gather a lot of information about an organization’s
infrastructure i.e. internal domains, email addresses in a completely passive manner

https://blog.appsecco.com/certificate-transparency-part-3-the-dark-side-9d401809b025

Searching through CT logs

There are various search engines that collect the CT logs and let’s anyone search through them
i. https://crt.sh/
ii. https://censys.io/
iii. https://developers.facebook.com/tools/ct/
iv. https://google.com/transparencyreport/https/ct/

A script that searches SSL/TLS certificates issued for a domain using crt.sh

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Script - https://git.io/vhqRd

Keeping track of an organisation's sub-domains

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://developers.facebook.com/tools/ct/

Downside of CT for recon

CT logs are append-only. There is no way to delete an existing entry
The domain names found in the CT logs may not exist anymore and thus they can’t be resolved to an IP
address

https://blog.appsecco.com/a-penetration-testers-guide-to-sub-domain-enumeration-7d842d5570f6

CT logs + massdns
You can use tools like massdns along with CT logs script to quickly identify resolvable domain names.

python3 ct.py example.com | ./bin/massdns -r resolvers.txt -t A -a -o -w results.txt -

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Using certspotter

find-cert()
{ curl -s https://certspotter.com/api/v0/certs?domain=$1 | jq -c '.[].dns_names' | grep -o '"[^"]\+

Using certdb.com
While crt.sh gets the data from CT logs only where "legit" CA submit the certs to a log; CertDB is based on
the scanning the IPv4 segment, domains and "finding & analyzing" all the certificates

curl -L -sd "api_key=API-KEY&q=Organization:\"tesla\"&response_type=3" -X POST https://certdb.com/a

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://certdb.com

Finding vulnerable CMS using CT

When setting up some CMSs like Wordpress, Joomla and others, there is a window of time where the installer
has no form of authentication
If the domain supports HTTPS it will end up on a CT log(sometimes in near real time)
If an attacker can search through CT Logs and find such a web application without authentication then he/she
can take over the server

Finding vulnerable CMS using CT

This attack has been demonstrated by Hanno Böck at Defcon 25
He claimed to have found 5,000 WordPress installations using CT logs over a period of 3 months that he could
have potentially taken over
HD Moore also discussed this technique in his talk at BSidesLV 2017

Censys.io

Censys aggregates SSL certificates that are a result of SSL scans on IPv4 address space and also from
Certificate Transparency (CT) logs
This is a good source of domains and also email addresses

https://0xpatrik.com/censys-guide/

Extracting domains/emails from SSL/TLS certs using censys

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://github.com/0xbharath/censys-enumeration

Content Security Policy(CSP)

Content Security Policy(CSP) defines the Content-Security-Policy HTTP header, which allows us to create
a whitelist of sources of trusted content, and instructs the browser to only execute or render resources from
those sources
So basically, Content-Security-Policy header will list a bunch of sources(domains) that might be of interest to
us as an attackers.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Extract domains from CSP headers

https://github.com/0xbharath/domains-from-csp

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://justi.cz/security/2018/05/23/cdn-tar-oops.html

Sender Policy Framework

A Sender Policy Framework(SPF) record and is used to indicate to recieving mail exchanges which hosts are
authorized to send mail for a given domain
Simply put, an SPF record lists all the hosts that are authorised send emails on behalf of a domain

Extract net blocks/domains from SPF record

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://github.com/0xbharath/assets-from-spf

Domain enumeration in DNSSEC

Authenticated Denial of Existence(RFC 7129)

In DNS, when client queries for a non-existent domain, the server must deny the existence of that domain. It is
harder to do that in DNSSEC due to cryptographic signing.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Zone walking NSEC - LDNS
The ldns-walk (part of ldnsutils ) can be used to zone walk DNSSEC signed zone that uses NSEC.

# zone walking with ldnsutils

$ ldns-walk iana.org
iana.org. iana.org. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY
api.iana.org. CNAME RRSIG NSEC
app.iana.org. CNAME RRSIG NSEC
autodiscover.iana.org. CNAME RRSIG NSEC
beta.iana.org. CNAME RRSIG NSEC
data.iana.org. CNAME RRSIG NSEC
dev.iana.org. CNAME RRSIG NSEC
ftp.iana.org. CNAME RRSIG NSEC
^C

Installing ldnsutils

# On Debian/Ubuntu
$ sudo apt-get install ldnsutils

# On Redhat/CentOS
$ sudo yum install ldns
# You may need to do
$ sudo yum install -y epel-release

NSEC3
The NSEC3 record is like an NSEC record, but, NSEC3 provides a signed gap of hashes of domain names.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Returning hashes was intended to prevent zone enumeration(or make it expensive).

231SPNAMH63428R68U7BV359PFPJI2FC.example.com. NSEC3 1 0 3 ABCDEF

NKDO8UKT2STOL6EJRD1EKVD1BQ2688DM A NS SOA TXT AAAA RRSIG DNSKEY NSEC3PARAM

NKDO8UKT2STOL6EJRD1EKVD1BQ2688DM.example.com. NSEC3 1 0 3 ABCDEF

231SPNAMH63428R68U7BV359PFPJI2FC A TXT AAAA RRSIG

Zone walking NSEC3

An attacker can collect all the sub-domain hashes and crack the hashes offline
Tools like nsec3walker, nsec3map help us automate collecting NSEC3 hases and cracking the hashes

Zone walking NSEC3

Zone walking NSEC3 protected zone using nsec3walker:

# Collect NSEC3 hashes of a domain

$ ./collect insecuredns.com > insecuredns.com.collect

# Undo the hashing, expose the sub-domain information.

$ ./unhash < insecuredns.com.collect > insecuredns.com.unhash

Zone walking NSEC3

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 # Checking the number of sucessfully cracked sub-domain hashes
$ cat icann.org.unhash | grep "icann" | wc -l
182

# Listing only the sub-domain part from the unhashed data

$ cat icann.org.unhash | grep "icann" | awk '{print $2;}'
del.icann.org.
access.icann.org.
charts.icann.org.
communications.icann.org.
fellowship.icann.org.
files.icann.org.
forms.icann.org.
mail.icann.org.
maintenance.icann.org.
new.icann.org.
public.icann.org.
research.icann.org.
rs.icann.org.
stream.icann.org.
tally.icann.org.
video.icann.org.
mm.icann.org.
ns.icann.org.
qa.icann.org.
ist.icann.org.
aso.icann.org.
cai.icann.org.
dev.icann.org.
exc.icann.org.
jss.icann.org.
mex.icann.org.
rrs.icann.org.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 syd.icann.org.
upk.icann.org.
vip.icann.org.
crm.icann.org.
dns.icann.org.
liao.icann.org.
redis.icann.org.
svn.icann.org.
admin.icann.org.
orbis.icann.org.
jira.icann.org.
omblog.icann.org.
pptr.icann.org.
splunk.icann.org.
nomcom.icann.org.
rssac.icann.org.
sftp.icann.org.
netscan.icann.org.

Installing nsec3walker

Installation instructions are available at https://dnscurve.org/nsec3walker.html

I used following commands to install nsec3walker on Ubuntu 16.04.
build-essential package is a prerequisite.

# Installing nsec3walker
$ wget https://dnscurve.org/nsec3walker-20101223.tar.gz
$ tar -xzf nsec3walker-20101223.tar.gz
$ cd nsec3walker-20101223
$ make

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Few things that changed with the advent of DevOps
1. Storage
2. Authentication
3. More and more code
4. CI/CD pipelines

Cloud storage
Cloud storage has gotten inexpensive, easy to setup and gained popularity

Especially object/block storage

Object storage is ideal for storing static, unstructured data like audio, video, documents, images and logs as
well as large amounts of text.

i. AWS S3 buckets
ii. Digital Ocean Spaces

What's the catch with object storage?

Due to the nature of object storage, it is a treasure trove of information from an attacker/penetration tester
perspective.
In our experience, given an chance, users will store anything on third-party services, from their passwords in
plain text files to pictures of their pets.

Amazon S3 buckets

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 AWS S3 is an object storage service by Amazon
Buckets allow users to store and serve large amounts of data.

Attack on Accenture(Sep, 2017)- AWS S3 buckets as attack surface

https://www.upguard.com/breaches/cloud-leak-accenture

AWS S3 buckets as attack surface - The trend

AWS S3 buckets as attack surface - The trend

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Hunting for publicly accessible S3 buckets
Users can store Files(Objects) in a Bucket
Each Bucket will get an unique, predictable URL and each file in a Bucket will get an unique URL as well
There are Access controls mechanisms available at both Bucket and Object level.

Hunting for publicly accessible S3 buckets

Good old Google dorks

site:s3.amazonaws.com file:pdf

site:s3.amazonaws.com password

Hunting for publicly accessible S3 buckets

As buckets have predictable URL it is trivial to do a dictionary based attack

Following tools help run a dictionary attack to identify S3 buckets

i. AWSBucketDump
ii. Slurp

Digital Ocean Spaces

Spaces is an object storage service by DigitalOcean
It is similar to AWS S3 buckets

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Spaces API aims to be interoperable with Amazon’s AWS S3 API.

Spaces URL pattern

Users can store Files in a “Space”
Each Space will get an unique, predictable URL
Each file in a Space will get an unique URL as well.
Access controls mechanisms are available at Space and file level.

Hunting for publicly accessible S3 buckets

A Space is typically considered “public” if any user can list the contents of the Space

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 A Space is typically considered “private” if the Space’s contents can only be listed or written by certain users

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Spaces finder
Spaces API is interoperable with Amazon’s S3 API, we tweaked AWSBucketDump to work with DO Spaces
Spaces finder is a tool that can look for publicly accessible DO Spaces using a wordlist, list all the accessible
files on a public Space and download the files.

https://github.com/appsecco/spaces-finder

Spaces finder in action

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://github.com/appsecco/spaces-finder

Authentication
With almost every service exposing an API, keys have become critical in authenticating
API keys are treated as keys to the kingdom
For applications, API keys tend to be achilles heel

https://danielmiessler.com/blog/apis-2fas-achilles-heel/

Code repos for recon

Code repos are a treasure trove during recon

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Code repos can reveal a lot from credentials, potential vulnerabilities to infrastructure details

Github for recon

GitHub is an extremely popular version control and collaboration platform

Code repos on github tend to have all sorts of sensitive information
Github also has a powerful search feature with advanced operators
Github has a very well designed REST API
edoverflow has a neat little guide on GitHub for Bug Bounty Hunters

Things to focus on in Github

There are 4 main sections to look out for here.

Repositories
Code

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Commits(My fav!)
Issues

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Mass Cloning on Github
You can ideally clone all the target organization's repos and analyze them locally
GitHubCloner by @mazen160 comes very handy to automate the process

$ python githubcloner.py --org organization -o /tmp/output

https://gist.github.com/EdOverflow/922549f610b258f459b219a32f92d10b

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Static code analysis
Once the repos are cloned, you can do a static code analysis

There are language specific tools to speed up and automate the process

i. Brakeman for Ruby

ii. Bandit for Python

Finding secrets in code manually

Once you have the repos cloned. You can understand the code, language used and architecture

Start looking for keywords or patterns

- API and key. (Get some more endpoints and find API keys.)
- token
- secret
- vulnerable
- http://

Finding secrets in code in automated fashion

There are various tools available to find juicy information in source code.

1. Truffle Hog
2. git-all-secrets

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Github dorks
Github dorks are the new Google dorks

Github search is quite powerful feature & can be used to find sensitive data on the repos

A collection of Github dorks https://github.com/techgaun/github-dorks/blob/master/github-dorks.txt

Tool to run Github dorks against a repo https://github.com/techgaun/github-dorks

Passive recon using public datasets

There are various projects that gather Internet wide scan data and make it available to researchers and the
security community.
This data includes port scans, DNS data, SSL/TLS cert data and even data breach dumps that they can find.
Find your needle in the haystack.

Why use public data sets for recon?

To reduce dependency on 3rd party APIs and services
To reduce active probing of target infrastructure
More the sources better the coverage
Build your own recon platforms

Let's look at some public datasets

Name Description Price

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Name Description Price

Sonar FDNS, RDNS, UDP, TCP, TLS, HTTP, HTTPS scan data FREE

Censys.io TCP, TLS, HTTP, HTTPS scan data FREE

CT TLS FREE

https://github.com/fathom6/inetdata

Let's look at some public datasets

Name Description Price

CZDS zone files for "new" global TLDs FREE

ARIN American IP registry information FREE

CAIDA PFX2AS IPv4 Daily snapshots of ASN to IPv4 mappings FREE

Let's look at some public datasets

Name Description Price

US Gov US government domain names FREE

UK Gov UK government domain names FREE

RIR Delegations Regional IP allocations FREE

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://github.com/fathom6/inetdata

Let's look at some public datasets

Name Description Price

PremiumDrops DNS zone files for com/net/info/org/biz/xxx/sk/us TLDs $24.95/mo

WWWS.io Domains across many TLDs (~198m) $9/mo

WhoisXMLAPI.com New domain whois data $109/mo

https://github.com/fathom6/inetdata

Rapid7 Forward DNS dataset

Rapid7 publishes its Forward DNS study/dataset on scans.io project(it's a massive dataset, 20+ GB
compressed & 300+ GB uncompressed)
This dataset aims to discover all domains found on the Internet

Hunting sub-domain in FDNS dataset

The data format is a gzip-compressed JSON file so we can use jq utility to extract sub-domains of a specific
domain:

curl --silent -L https://opendata.rapid7.com/sonar.fdns_v2/2018-04-21-1524297601-fdns_any.json.gz

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 cat 2018-04-21-1524297601-fdns_any.json.gz | pigz -dc | grep "\.example\.com" | jq .name

https://opendata.rapid7.com/about/

Hunting sub-domain in FDNS dataset

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 https://github.com/appsecco/bugcrowd-levelup-subdomain-enumeration

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 ICANN.ORG subdomains
Number of unique, resolvable sub-domains each enumeration technique found independently against icann.org

TALK MATERIAL
https://github.com/appsecco/practical-recon-levelup0x02

Take away

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 A gitbook on sub-domain enumeration

https://appsecco.com/books/subdomain-enumeration/

References
- https://www.certificate-transparency.org/ - https://www.cloudflare.com/dns/dnssec/how-dnssec-works/ -
https://www.cloudflare.com/dns/dnssec/dnssec-complexities-and-considerations/ -
http://info.menandmice.com/blog/bid/73645/Take-your-DNSSEC-with-a-grain-of-salt -
https://github.com/rapid7/sonar/wiki/Forward-DNS

Thanks
@0xbharath

© 2019 GitHub, Inc. Terms Privacy Security Status Help

Contact GitHub Pricing API Training Blog About

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD