Download as PDF, PPTX
![python tinfoleak.py pycones -i -s --sdate 2016/01/01 --hashtags --mentions --meta --media [d] --geo
GEOFILE --top 10 -o report.html](https://image.slidesharecdn.com/osintpython-161009091810/75/OSINT-tools-for-security-auditing-with-python-53-2048.jpg)
Uploaded byJose Manuel Ortega Candel
OSINT tools for security auditing with python
This document discusses an OSINT (open-source intelligence) presentation on tools developed with Python for server information, geolocation, metadata, and social media footprinting from Twitter and other sources. It provides information on defining intelligence targets and the types of data to obtain (technical, social, physical, logical). It also lists tools and APIs for geolocation, server information from Censys and Shodan, the Recon-NG toolkit, and OSINT frameworks like Spiderfoot and theHarvester. Python libraries mentioned include BeautifulSoup, Requests, Shodan, and GeoIP. The presentation will be given at PYCONES in October 2016.
More Related Content
![OSINT tools for security auditing [FOSDEM edition]](https://cdn.slidesharecdn.com/ss_thumbnails/osinttoolsforsecurityauditingfosdemedition1-170205120023-thumbnail.jpg?width=640&height=640&fit=bounds)
![[HITCON 2020 CTI Village] Threat Hunting and Campaign Tracking Workshop.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/threathuntingtocampaigntrackingworkshop-hitcon2020ctivillage-201213185811-thumbnail.jpg?width=640&height=640&fit=bounds){kind=tracking}
Similar to OSINT tools for security auditing with python
OSINT tools for security auditing with python
- 1. José Manuel Ortega @jmortegac PYCONES7-9 OCT 2016
- 3.
- 4. ▪ OSINT introduction ▪Server information(Censys,Shodan) ▪ OSINT tools developed with python ▪ Geolocation,Metadata ▪ Twitter,Footprinting,FullContact
- 5. ▪ Define aspecific target and data you wish to obtain ▪ Technical-Accounts,servers,services,software ▪ Social-Social Media,Email,Photos ▪ Physical-Address,Home IP address,Footprinting ▪ Logical-Network,Operational intelligence
- 6. ▪ GeoLocation ▪ IPaddress ▪ Email address ▪ Telephone Number ▪ Usernames in social network profiles ▪ Metadata information from images ▪ Server information & vulnerabilities
- 8.
- 13. ▪ Checking datawith ip address ▪ https://www.shodan.io/host/144.76.246.116
- 15.
- 16. ▪ https://bitbucket.org/LaNMaSteR53/recon-ng ▪ OpenSource OSINT toolkit written in python ▪ Actively maintained ▪ Uses modules and saves all recollected information in databases
- 17. ▪ dnspython -http://www.dnspython.org/ ▪ dicttoxml - https://github.com/quandyfactory/dicttoxml/ ▪ jsonrpclib - https://github.com/joshmarshall/jsonrpclib/ ▪ lxml - http://lxml.de/ ▪ slowaes - https://code.google.com/p/slowaes/ ▪ XlsxWriter - https://github.com/jmcnamara/XlsxWriter/ ▪ Mechanize ▪ PyPDF2 ▪ sqlite3
- 25.
- 27. ▪ httplib ▪ socket ▪requests ▪ shodan
- 29. ▪ pip installosrframework ▪ Developed in python 2.7 ▪ Integrates with maltego transforms ▪ https://pypi.python.org/pypi/osrframework/0.13.2 ▪ https://github.com/i3visio/osrframework
- 30. ▪ BeautifulSoup ▪ Requests ▪Mechanize ▪ pyDNSresolving name servers ▪ python-whoisto recover the whois info from a domain ▪ tweepyfor connecting with Twitter API ▪ Skype4Py for connecting with Skype API ▪ Python-emailahoyfor checking email address ▪ Multiprocessingimport Process, Queue, Pool
- 39. Source Location Notes abuse.chhttp://www.abuse.ch Various malware trackers. AdBlock https://easylist- downloads.adblockplus.org/easylist.txt AdBlock pattern matches AlienVault https://reputation.alienvault.com AlienVault’s IP reputation database. Autoshun.org http://www.autoshun.org Blacklists. AVG Site Safety Report http://www.avgthreatlabas.com Site safety checker. Bing http://www.bing.com Scraping but future version to also use API. Blocklist.de http://lists.blocklist.de Blacklists. Checkusernames.com http://www.checkusernames.com Look up username availability on popular sites. DNS Your configured DNS server. Defaults to your local DNS but can be configured to whatever IP address you supply SpiderFoot. DomainTools http://www.domaintools.com DroneBL http://www.dronebl.org Facebook http://www.facebook.com Scraping but future version to also use API. FreeGeoIP http://freegeoip.net Google http://www.google.com Scraping but future version to also use API. Google+ http://plus.google.com Scraping but future version to also use API. Google Safe Browsing http://www.google.com/safebrowsing Site safety checker. LinkedIn http://www.linkedin.com Scraping but future version to also use API. malc0de.com http://malc0de.com Blacklists. malwaredomainlist.com http://www.malwaredomainlist.com Blacklists.
- 40. Source Location Notes malwaredomains.comhttp://www.malwaredomains.com Blacklists. McAfee SiteAdvisor http://www.siteadvisor.com Site safety checker. NameDroppers http://www.namedroppers.org Nothink.org http://www.nothink.org Blacklists. OpenBL http://www.openbl.org Blacklists. PasteBin http://www.pastebin.com Achieved through Google scraping. PGP Servers http://pgp.mit.edu/pks/ PGP public keys. PhishTank http://www.phishtank.org Identified phishing sites. Project Honeypot http://www.projecthoneypot.org Blacklists. API key needed. RIPE/ARIN http://stat.ripe.net/ Robtex http://www.robtex.com SANS ISC http://isc.sans.edu Internet Storm Center IP reputation database. SHODAN http://www.shodanhq.com API key needed. SORBS http://www.sorbs.net Blacklists. SpamHaus http://www.spamhaus.org Blacklists. ThreatExpert http://www.threatexpert.com Blacklists. TOR Node List http://torstatus.blutmagie.de TotalHash.com http://www.totalhash.com Domains/IPs used by malware. UCEPROTECT http://www.uceprotect.net Blacklists. VirusTotal http://www.virustotal.com Domains/IPs used by malware. API key needed. Whois Various Whois servers for different TLDs. Yahoo http://www.yahoo.com Scraping but future version to also use API. Zone-H http://www.zone-h.org Easy to get black-listed. Log onto the site in a browser from the IP you’re scanning from first and enter the CAPTCHA, then it should be fine.
- 41. ▪ Python 2.7 ▪BeautifulSoup ▪ DNSPython ▪ Socks ▪ Socket ▪ SSL ▪ CherryPy ▪ M2MCrypto ▪ Netaddr ▪ pyPDF
- 42. from bs4 importBeautifulSoup, SoupStrainer
- 46. ▪ PDFPyPDF2,PDFMiner ▪ ImagesPillow,pyexiv2(python2.7),gexiv2(python 3)
- 47.
- 48. ▪ Orb(Python 2.x) •https://github.com/epsylon/orb • python-whois - Python module for retrieving WHOIS information • python-dnspython - DNS toolkit for Python • python-nmap - Python interface to the Nmap port scanner • InstaRecon(Python 2.x) • https://github.com/vergl4s/instarecon • dnspython,ipaddress • ipwhois,python-whois • requests,shodan
- 51. ▪ BeautifulSoup forparsing web information ▪ Requests,urllib3 for synchronous requests ▪ Asyncio,aiohttp for asynchronous requests ▪ Robobrowser,Scrapy for web crawling ▪ PyGeoIP,geoip2,geojson for GeoLocation ▪ python-twitter,tweepy for connecting with twitter ▪ Shodan for obtain information for servers ▪ DNSPython,netaddr for resolving ip address
- 53. python tinfoleak.py pycones-i -s --sdate 2016/01/01 --hashtags --mentions --meta --media [d] --geo GEOFILE --top 10 -o report.html
- 55. ▪ import tweepyTwitterAPI library for Python ▪ from PIL import Image, ExifTags, ImageCmsmetadata from images ▪ import pyexiv2metadata from images ▪ import urllib2requests ▪ from OpenSSL import SSL ▪ from jinja2 import Template, Environment, FileSystemLoaderreport
- 61. ▪ We knowwe have a valid email address ▪ What other profiles are associated with this address? ▪ Go to fullcontact.com for an API key…..
- 67. ▪ https://sourceforge.net/projects/spiderfoot ▪ http://www.edge-security.com/theharvester.php ▪https://developer.shodan.io/api ▪ http://www.clips.ua.ac.be/pattern ▪ http://www.pentest- standard.org/index.php/PTES_Technical_Guidelines#OSINT ▪ http://www.vicenteaguileradiaz.com/tools ▪ https://github.com/automatingosint/osint_public ▪ http://www.automatingosint.com/blog/
- 69.