Requesting a URL from a local File in Python

Making requests over the internet is a common operation performed by most automated web applications. Whether a web scraper or a visitor tracker, such operations are performed by any program that makes requests over the internet. In this article, you will learn how to request a URL from a local File using Python.

Requesting a URL from a local File in the Python

webbrowser library would be used to include URL displaying capabilities to the python interpreter. The library could be installed using the following command:

py -m pip install webbrowser

The following file containing URLs on each line will be used for demonstration.

Example 1: Requesting a URL from a local File using webbrowser

The aforementioned file would be read, and all its contents would be stored (delimited by a newline character) as members of a list where each line would contain a URL. Next, a loop would run to go over each of these URLs. In each iteration, a request to the resource located at the particular URL would be made using the webbrowser.open function. This opens the default web browser of the Operating System to display the URL.

import webbrowser

# Opening the file containing URLs in each line
# Extracting the URL from each line and storing it in a list
url_list = open('test.txt').readlines()

# This loop iterates over all the URLs present in the file
for url in url_list:

    # Opening the webbrowser of the OS and displaying the URL
    webbrowser.open(url)

Output:

Firstly the file containing the URLs is opened (in read mode), and all the lines in the file are stored as separate elements of a list. A loop is run on the list, which iterates over all its elements, where the elements are URLs. In each iteration, a request to the specific URL is made, and the response is displayed using the default web browser of the operating system.

The file containing the URLs needs to contain a URL on each line. Otherwise, the code must be modified to reflect such changes in the input file. 

Method 2: Requesting a URL from a local File using requests

Another way of accomplishing the task at hand, without the overhead of running a browser, is by requesting the webpage and storing the response. This allows several operations that could be performed on the data obtained from the URL while not requiring the display of its contents. In the following example, the use of request module would be made to produce the desired effect.

import requests

# Opening the file containing URLs in each line
# Extracting the URL from each line and storing it in a list
url_list = open('test.txt').readlines()

# This loop iterates over all the URLs present in the file
for url in url_list:

    # Making the request for the URL and storing its response
    resp = requests.get(url)

    # Displaying the response code & the content of the response
    print(resp)
    print(resp.content)

Output:

The process of extracting URLs from the file is the same as in the previous example. The only difference lies in the loop body. Inside the loop, the get function makes a request to the URL, and the response is stored in a variable. This variable could be used to obtain a lot of information regarding the URL. Firstly, the response code is displayed, which in our case is 200 (success status response code). After which the contents of the webpage received in response are displayed. If this content is stored within a file with an appropriate extension, it will act as an offline copy of the URL.