Python program to Recursively scrape all the URLs of the website Last Updated : 26 Mar, 2020 Comments Improve Suggest changes Like Article Like Report In this tutorial we will see how to we can recursively scrape all the URLs from the website Recursion in computer science is a method of solving a problem where the solution depends on solutions to smaller instances of the same problem. Such problems can generally be solved by iteration, but this needs to identify and index the smaller instances at programming time. Note: For more information, refer to Recursion Modules required and Installation Requests : Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs. pip install requests Beautiful Soup: Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree. pip install beautifulsoup4 Code : Python3 1== from bs4 import BeautifulSoup import requests # lists urls=[] # function created def scrape(site): # getting the request from url r = requests.get(site) # converting the text s = BeautifulSoup(r.text,"html.parser") for i in s.find_all("a"): href = i.attrs['href'] if href.startswith("/"): site = site+href if site not in urls: urls.append(site) print(site) # calling it self scrape(site) # main function if __name__ =="__main__": # website to be scrape site="http://example.webscraping.com//" # calling function scrape(site) Output : Comment More infoAdvertise with us Next Article Python program to Recursively scrape all the URLs of the website R rakshitarora Follow Improve Article Tags : Python Python Programs python-utility Practice Tags : python Similar Reads Python Tutorial | Learn Python Programming Language Python Tutorial – Python is one of the most popular programming languages. It’s simple to use, packed with features and supported by a wide range of libraries and frameworks. Its clean syntax makes it beginner-friendly.Python is:A high-level language, used in web development, data science, automatio 10 min read Python Interview Questions and Answers Python is the most used language in top companies such as Intel, IBM, NASA, Pixar, Netflix, Facebook, JP Morgan Chase, Spotify and many more because of its simplicity and powerful libraries. To crack their Online Assessment and Interview Rounds as a Python developer, we need to master important Pyth 15+ min read Python OOPs Concepts Object Oriented Programming is a fundamental concept in Python, empowering developers to build modular, maintainable, and scalable applications. By understanding the core OOP principles (classes, objects, inheritance, encapsulation, polymorphism, and abstraction), programmers can leverage the full p 11 min read Python Projects - Beginner to Advanced Python is one of the most popular programming languages due to its simplicity, versatility, and supportive community. Whether you’re a beginner eager to learn the basics or an experienced programmer looking to challenge your skills, there are countless Python projects to help you grow.Here’s a list 10 min read Python Exercise with Practice Questions and Solutions Python Exercise for Beginner: Practice makes perfect in everything, and this is especially true when learning Python. If you're a beginner, regularly practicing Python exercises will build your confidence and sharpen your skills. To help you improve, try these Python exercises with solutions to test 9 min read Python Programs Practice with Python program examples is always a good choice to scale up your logical understanding and programming skills and this article will provide you with the best sets of Python code examples.The below Python section contains a wide collection of Python programming examples. These Python co 11 min read Enumerate() in Python enumerate() function adds a counter to each item in a list or other iterable. It turns the iterable into something we can loop through, where each item comes with its number (starting from 0 by default). We can also turn it into a list of (number, item) pairs using list().Let's look at a simple exam 3 min read Python Data Types Python Data types are the classification or categorization of data items. It represents the kind of value that tells what operations can be performed on a particular data. Since everything is an object in Python programming, Python data types are classes and variables are instances (objects) of thes 9 min read Python Introduction Python was created by Guido van Rossum in 1991 and further developed by the Python Software Foundation. It was designed with focus on code readability and its syntax allows us to express concepts in fewer lines of code.Key Features of PythonPython’s simple and readable syntax makes it beginner-frien 3 min read Input and Output in Python Understanding input and output operations is fundamental to Python programming. With the print() function, we can display output in various formats, while the input() function enables interaction with users by gathering input during program execution. Taking input in PythonPython input() function is 8 min read Like