KNOWLEDGE INSTITUTE OF TECHNOLOGY

(An Autonomous Institution)

Accredited by NAAC & NBA, Approved by AICTE, New Delhi and Affiliated To Anna University, Chennai

Kakapalayam (PO), Salem - 637 504.

RECORD NOTE BOOK

REG NO.

Certified that this is the bonafide record of work done by

Selvan/Selvi…………………………..…………of the……….
…………............. Semester ARTIFICIAL INTELLIGENCE AND DATA SCIENCE
Branch during the year 2024-2025 in CCS335 – Cloud Computing Laboratory.

Staff-Incharge Head of the Department

____________________________________________________________________________

Submitted for the university Practical Examination on……………………………….

Internal Examiner External Examiner

 List of Experiments
1. Install Virtualbox/VMware Workstation with different flavours of linux or windows OS on top of windows7 or 8.

2. Install a C compiler in the virtual machine created using virtual box and execute Simple Programs

3. Install Google App Engine. Create hello world app and other simple web applications using python/java.

4. Use GAE launcher to launch the web applications.

5. Simulate a cloud scenario using CloudSim and run a scheduling algorithm that is not present in CloudSim.

6. Find a procedure to transfer the files from one virtual machine to another virtual machine.

7. Install Hadoop single node cluster and run simple applications like wordcount.

8. Creating and Executing Your First Container Using Dockers.

9. Run a Container from Docker Hub.

Ex.No: 1
Date :
Install Virtualbox / VMware

Aim:
To Install Virtualbox / VMware Workstation with different flavors of linux or windows OS on
top of windows 7 or 8.

Procedure:
1. Download Link
Link for downloading the software is https://www.vmware.com/products/workstation-pro/workstation- pro-
evaluation.html. Download the software for windows. Good thing is that there is no signupprocess. Click and
download begins. Software is around 541 MB.

2. Download the installer file

It should probably be in the download folder by default, if you have not changed the settings in your browser.
File name should be something like VMware-workstation-full-15.5.1-15018445.exe. This file name can
change depending on the version of the software currently available for download. But for now, till the next
version is available, they will all be VMware Workstation 15 Pro.

3. Locate the downloaded installer file

For demonstration purpose, I have placed the downloaded installer on my desktop. Find the installer on your
system and double click to launch the application.
 VMware workstation 15 pro for windows 10 installer file screenshot.

4. User Access Control (UAC) Warning

Now you should see User Access Control (UAC) dialog box. Click yes to continue.

VMware Workstation 12 Pro installer windows 10 UAC screenshot

Initial Splash screen will appear. Wait for the process to complete.
 VMware Workstation 15 Installation Splash Screen

5. VMware Workstation Setup wizard

Now you will see VMware Workstation setup wizard dialog box. Click next to continue.

6. End User Licence Agreement

This time you should see End User Licence Agreement dialog box. Check “I accept the terms in the Licence
Agreement” box and press next to continue.
 VMware Workstation 15 Installation – End User Licence Agreement

7. Custom Setup options

Select the folder in which you would like to install the application. There is no harm in leaving the defaults as
it is. Also select Enhanced Keyboard Driver check box.
 VMware Workstation 15 Pro installation – select installation folder

8. User Experience Settings

Next you are asked to select “Check for Updates” and “Help improve VMware Workstation Pro”. Do as
you wish. I normally leave it to defaults that is unchecked.

9. Application Shortcuts preference

Next step is to select the place you want the shortcut icons to be placed on your system to launch the
application. Please select both the options, desktop and start menu and click next.
 VMware workstation 15 pro installation shortcut selection checkbox screenshot.

10. Installation begins

Now you see the begin installation dialog box. Click install to start the installation process.

Screenshot for VMware Workstation 15 pro installation begin confirmation dialog box on windows 10.

Below screenshot shows Installation in progress. Wait for this to complete.

 Screenshot for VMware Workstation 15 pro installation process.

At the end you will see installation complete dialog box. Click finish and you are done with the
installation process. You may be asked to restart your computer. Click on Yes to restart.

11. Launch VMware Workstation

After the installation completes, you should see VMware Workstation icon on the desktop. Double click on it
to launch the application.
 Screenshot for VMware Workstation 15 Pro icon on windows 10 desktop.

To Install VirtualBox
1. Visit http://www.virtualbox.org/wiki/downloads,
2. Download VirtualBox platform packages for your OS.
3. Open the Installation Package by double clicking
4. Click continue and finish installing VirtualBox

5. When finished installation, close the window.

To Download Linux
1. Visit the page http://www.ubuntu.com/download/ubuntu/download
2. Choose the Latest version of Ubuntu and 32-bit and click “Start Download”

Install Linux using Virtual Box

1. Run VirtualBox by double-clicking the icon
2. Click “New” button on the top left corner
3. Click “Continue” on the pop-up window
4. Type VM name, select “Linux” for the OS and choose “Ubuntu” for the version.
5. Choose the amount of memory to allocate (suggest choosing between 512 MB to 1024 MB)
6. Click Continue or Next

7. Choose create a new virtual hard disk

8. Click Continue or Next
9. Choose VDI (VirtualBox Disk Image). Click Continue or Next

10. Choose “Dynamically Allocated” click continue. This way, the size of your Virtual Hard Disk will grow as
you use.
11. Click the folder icon and choose the ubuntu iso file you downloaded.
12. Select the size of the Virtual Disk (I recommend choosing 8 GB) and click continue

13. Click Create

Running Linux

1. Choose Ubuntu from left column and click Start

2. Click continue on pop up window

3. Click the folder icon and choose the ubuntu iso file you downloaded and click continue and start

4. Click Install Ubuntu

5. Check “Download updates” and click Forward

6. Choose “Erase disk and install Ubuntu” and click Forward (Don’t worry, it won’t wipe your computer)
7. Click “Install Now” and wait. Maybe grab a snack.
8. When finished, click Restart and press Enter.

Result:
Thus, we installed Virtualbox / VMware Workstation with linux on top of windows 7 or 8.
 Ex.No: 2
Install a C compiler in the virtual machine created
Date : using virtual box and execute Simple Programs

Aim:
To install a C compiler in the virtual machine created using virtual box and execute Simple Programs.

Procedure:
1. Open Terminal (Applications-Accessories-Terminal)

2. Open gedit by typing “gedit &” on terminal (You can also use any other Text Editor application)
3. Type the following on gedit (or any other text editor)

#include<stdio.h>
main()
{
printf("Hello World\n");
}

4. Save this file as “helloworld.c”

5. Type “ls” on Terminal to see all files under current folder
6. Confirm that “helloworld.c” is in the current directory. If not, type cd DIRECTORY_PATH to go to the
directory that has “helloworld.c”
7. Type “gcc helloworld.c” to compile, and type “ls” to confirm that a new executable file “a.out” is created
 8. Type “./a.out” on Terminal to run the program
9. If you see “Hello World” on the next line, you just successfully ran your first C program!

Result:
Thus, we installed a C compiler in the virtual machine created using virtual box and executed Simple
Programs.
 Ex.No: 3
Installing and Running the Google App
Date : Engine On Windows
Aim:

To describe the installation of the Google App Engine Software Development Kit (SDK) on a
Microsoft Windows and running a simple “hello world” application.

Procedure:

The App Engine SDK allows you to run Google App Engine Applications on your local computer. It
simulates the run--‐time environment of the Google App Engine infrastructure.

Pre-Requisites: Python 2.5.4

If you don't already have Python 2.5.4 installed in your computer, download and Install Python 2.5.4 from:

http://www.python.org/download/releases/2.5.4/

Download and Install

You can download the Google App Engine SDK by going to:

http://code.google.com/appengine/downloads.html

and download the appropriate install package.

Download the Windows installer – the simplestthing is to download it toyour Desktop or another folder
that you remember.
Double Click on the GoogleApplicationEngine installer.

Clickthroughtheinstallationwizard, anditshouldinstallthe App Engine. Ifyoudo not have Python 2.5, it will
install Python 2.5 as well.

Once the install is complete you can discard the downloaded installer
MakingyourFirstApplication

Now you need to create a simple application. We could use the “+” option to have the launcher make us an
application – but instead we will do it by hand to get a better sense of what is going on.

Make a folder for your Google App Engine applications. I am going to make the Folder on my Desktop
called “apps” – the path to this folder is:

C:\Documents and Settings\csev\Desktop\apps

And then make a sub--‐folder in within apps called “ae--01--trivial” – the path to this folder would be:

C:\ Documents and Settings \csev\Desktop\apps\ae--01--trivial

Using a text editor such as JEdit (www.jedit.org), create a file called app.yaml in the
ae--01--trivial folder with the following contents:

application: ae-01-trivial
version: 1
runtime: python
api_version: 1
handlers:
- url: /.*
script: index.py

Note: Please do not copy and paste these lines into your text editor – you might end up with strange
characters – simply type them into your editor.

Then create a file in the ae--01--trivial folder called index.py with three lines in it:

print 'Content-Type: text/plain'

print ' '
print 'Hello there Chuck'

Then start the GoogleAppEngine Launcher program that can be found under Applications. Use the File -
-> Add Existing Application command and navigate into the apps directory and select the ae--01--trivial
folder. Once you have added the application, select it so that you can control the application using the
launcher.
Once you have selected your application and press Run. Aftera fewmomentsyour applicationwillstartand
the launcher willshow a little green icon next toyour application. Then press Browse to open a browser
pointing at your application which is running at http://localhost:8080/

Paste http://localhost:8080 into your browser and you should see your application asfollows:

Justforfun,edit the index.py tochange the name“Chuck”to yourown name and press Refresh in the
browser to verify your updates.

Watching the Log

Youcanwatchtheinternallogoftheactionsthatthewebserverisperformingwhen youareinteractingwith
yourapplicationinthebrowser. Selectyourapplicationin the Launcher and press the Logs button to bring
up a log window:
Eachtime you press Refresh inyour browser–you canseeit retrievingthe output with a GET
request.

Dealing With Errors

With two files to edit, there are two general categories of errors that you may encounter. Ifyoumakea
mistakeon the app.yaml file,the AppEngine will not start and your launcher will show a yellow icon
near your application:

To get more detail on what is going wrong, take a look at the log for the application:
In this instance – the mistake is mis-indenting the last line in the app.yaml (line 8).

If you make a syntax error in the index.py file, a Python traceback error will appear in your browser.

The error you need to see is likely to be the last few lines of the output – in this case I made a Python
syntax error on line one of our one-line application.

Whenyoumakeamistakeinthe app.yaml file – youmustthefixthemistakeand attempt to start the

application again.

6
 Ifyoumakeamistakeinafilelikeindex.py, youcansimplyfixthefileandpress refresh in your browser –
there is no need to restart the server.

Shutting Down the Server

To shut down the server, use the Launcher, select your application and press the Stop button.

Result:

To describe the installation of the Google App Engine Software Development Kit (SDK) on a
Microsoft Windows and running a simple “hello world” application.
Ex.No: 4
Create a Website to host on Google App
Date : Engine On Windows
Aim:
To create a website to host on Google App Engine on windows.

Procedure:
You can use Google App Engine to host a static website. Static web pages can contain client-side
technologies such as HTML, CSS, and JavaScript. Hosting your static site on App Engine can cost less than
using a traditional hosting provider, as App Engine provides a free tier.

Sites hosted on App Engine are hosted on the REGION_ID.r.appspot.com subdomain, such as [my-project-
id].uc.r.appspot.com. After you deploy your site, you can map your own domain name to your App
Engine- hosted website.

Before you can host your website on Google App Engine:

Create a new Cloud Console project or retrieve the project ID of an existing project to use: Go to the
Projects page

Tip: You can retrieve a list of your existing project IDs with the gcloud command line tool.

Install and then initialize the Google Cloud SDK: Download the SDK

Creating a website to host on Google App Engine

Basic structure for the project

This guide uses the following structure for the project:

app.yaml: Configure the settings of your App Engine application.

www/: Directory to store all of your static files, such as HTML, CSS, images, and JavaScript.

css/: Directory to store stylesheets.

style.css: Basic stylesheet that formats the look and feel of your site.

images/: Optional directory to store images.

index.html: An HTML file that displays content for your website.

js/: Optional directory to store JavaScript files.

Other asset directories.

Creating the app.yaml file

The app.yaml file is a configuration file that tells App Engine how to map URLs to your static files. In the
following steps, you will add handlers that will load www/index.html when someone visits your website, and
all static files will be stored in and called from the www directory.

Create the app.yaml file in your application's root directory:

Create a directory that has the same name as your project ID. You can find your project ID in the Console.

In directory that you just created, create a file named app.yaml.

Edit the app.yaml file and add the following code to the file:

runtime: python27

api_version: 1

threadsafe: true

handlers:

- url: /

static_files: www/index.html

upload: www/index.html

- url: /(.*)

static_files: www/\1

upload: www/(.*)

More reference information about the app.yaml file can be found in the app.yaml reference documentation.

Creating the index.html file

Create an HTML file that will be served when someone navigates to the root page of your website. Store this
file in your www directory.

<html>

<head>

<title>Hello, world!</title>

<link rel="stylesheet" type="text/css" href="/css/style.css">

</head>

<body>

<h1>Hello, world!</h1>

<p>
 This is a simple static HTML file that will be served from Google App

Engine.

</p>

</body>

</html>

Deploying your application to App Engine

When you deploy your application files, your website will be uploaded to App Engine. To deploy your app,
run the following command from within the root directory of your application where the app.yaml file is
located:

gcloud app deploy

Optional flags:

Include the --project flag to specify an alternate Cloud Console project ID to what you initialized as the
default in the gcloud tool. Example: --project [YOUR_PROJECT_ID]

Include the -v flag to specify a version ID, otherwise one is generated for you. Example: -v
[YOUR_VERSION_ID]

To learn more about deploying your app from the command line, see Deploying a Python 2 App.

Viewing your application

To launch your browser and view the app at https://PROJECT_ID.REGION_ID.r.appspot.com, run the
following command:

gcloud app browse

Result:
Thus, we created a website to host on Google App Engine on windows.
Ex.No: 5
Date : Simulate a cloud scenario using CloudSim

Aim:
To simulate a cloud scenario using CloudSim and to run a scheduling algorithm.

Procedure:
Installation and running the CloudSim Toolkit

CloudSim is written in Java. The knowledge you need to use CloudSim is basic Java programming and some
basics about cloud computing. Knowledge of programming IDEs such as Eclipse or NetBeans is also helpful.
It is a library and, hence, CloudSim does not have to be installed. Normally, you can unpack the downloaded
package in any directory, add it to the Java classpath and it is ready to be used. Please verify whether Java is
available on your system. You just need to unpack the CloudSim file to install. If you want to remove
CloudSim, then remove the whole cloudsim directory. You do not need to compile CloudSim source code.
The JAR files are provided to compile and to run CloudSim applications:

* jars/cloudsim-<VERSION>.jar -- contains the CloudSim class files

* jars/cloudsim-<VERSION>-sources.jar -- contains the CloudSim source code files

* jars/cloudsim-examples-<VERSION>.jar -- contains the CloudSim examples class files

* jars/cloudsim-examples-<VERSION>-sources.jar -- contains the CloudSim examples source code files

To use CloudSim in Eclipse:

The main() method is the pointer from where the execution of this example starts

public static void main(String[] args)

There are eleven steps that are followed in each example with some variation in them, specified as follows:

1. Set the Number of users for the current simulation. This user count is directly proportional to a number
of brokers in the current simulation.

int num_user = 1; // number of cloud users

Calendar calendar = Calendar.getInstance();
boolean trace_flag = false;

2. Initialize the simulation, provided with the current time, number of users and trace
flag. CloudSim.init(num_user, calendar, trace_flag);

3. Create a Datacenter.
Datacenter datacenter0 = createDatacenter("Datacenter_0");
where the createDatacenter() method itself initializes the various datacenter characteristics along with the
host list. This is the most important entity without this there is no way the simulation of hosting the virtual
machine is applicable.

private static Datacenter createDatacenter(String name)

{
List<Host> hostList = new ArrayList<Host>();
List<Pe> peList = new ArrayList<Pe>();
int mips = 1000;
peList.add(new Pe(0, new PeProvisionerSimple(mips)));
int hostId = 0;
int ram = 2048; // host memory (MB)
long storage = 1000000; // host storage
int bw = 10000;
hostList.add(
new Host(
hostId,
new RamProvisionerSimple(ram),
new BwProvisionerSimple(bw),
storage,
peList,
new VmSchedulerTimeShared(peList)
)
);
String arch = "x86";
String os = "Linux";
String vmm = "Xen";
double time_zone = 10.0;
double cost = 3.0;
double costPerMem = 0.05;
double costPerStorage = 0.001;
double costPerBw = 0.0;
LinkedList<Storage> storageList = new LinkedList<Storage>();
DatacenterCharacteristics characteristics = new
DatacenterCharacteristics(arch, os, vmm, hostList,
time_zone, cost, costPerMem,
costPerStorage, costPerBw);
Datacenter datacenter = null;
try {
datacenter = new Datacenter(name, characteristics, new
VmAllocationPolicySimple(hostList),
storageList, 0);
} catch (Exception e) {
e.printStackTrace();
}
return datacenter;
}

4. Create a Datacenter broker.

DatacenterBroker broker = createBroker();
int brokerId = broker.getId();
Where the createBroker() method initializes the entity object from DatacenterBroker class

private static DatacenterBroker createBroker()

{
DatacenterBroker broker = null;
try {
broker = new DatacenterBroker("Broker");
} catch (Exception e) {
e.printStackTrace();
return null;
}
return broker;
}
5. Create a Virtual Machine(s).
vmlist = new ArrayList<Vm>();
int vmid = 0;
int mips = 1000;
long size = 10000;
int ram = 512;
long bw = 1000;
int pesNumber = 1;
String vmm = "Xen";

Vm vm = new Vm(vmid, brokerId, mips, pesNumber, ram, bw, size, vmm, new
CloudletSchedulerTimeShared());

vmlist.add(vm);

6. Submit Virtual Machine to Datacenter

broker. broker.submitVmList(vmlist);

7. Create Cloudlet(s) by specifying their characteristics.

cloudletList = new ArrayList<Cloudlet>();

int id = 0;
long length = 400000;
long fileSize = 300;
long outputSize = 300;
UtilizationModel utilizationModel = new UtilizationModelFull();

Cloudlet cloudlet = new Cloudlet(id, length, pesNumber, fileSize,

outputSize, utilizationModel, utilizationModel,
utilizationModel);

cloudlet.setUserId(brokerId);
cloudlet.setVmId(vmid);

cloudletList.add(cloudlet);
8. Submit Cloudlets to Datacenter
broker.
broker.submitCloudletList(cloudletList);

9. Send call to Start

Simulation.
CloudSim.startSimulation();

10. Once no more event to execute, send the call to Stop

Simulation. CloudSim.stopSimulation();

11. Finally, print the final status of the Simulation.

List<Cloudlet> newList = broker.getCloudletReceivedList();
printCloudletList(newList);
Where printCloudletList() method formats the output to correctly display it on the console.

private static void printCloudletList(List<Cloudlet> list)

{
int size = list.size();
Cloudlet cloudlet;
String indent = " ";
Log.printLine();
Log.printLine("========== OUTPUT ==========");
Log.printLine("Cloudlet ID" + indent + "STATUS" + indent
+ "Data center ID" + indent + "VM ID" + indent + "Time" + indent
+ "Start Time" + indent + "Finish Time");

DecimalFormat dft = new DecimalFormat("###.##");

for (int i = 0; i < size; i++)
{
cloudlet = list.get(i);
Log.print(indent + cloudlet.getCloudletId() + indent +
indent);
if (cloudlet.getCloudletStatus() == Cloudlet.SUCCESS)
{
Log.print("SUCCESS");
Log.printLine(indent + indent +
cloudlet.getResourceId()
+ indent + indent + indent +
cloudlet.getVmId()
+ indent + indent +
dft.format(cloudlet.getActualCPUTime())
+ indent + indent +
dft.format(cloudlet.getExecStartTime())
+ indent + indent +
dft.format(cloudlet.getFinishTime()));
}
}
}
Run the example.
Sample Output from the Existing Example:

Result:
Thus, we simulated a cloud scenario using CloudSim and to run a scheduling algorithm.
Ex.No: 6
Transfer the files from one virtual machine to
Date : another virtual machine

Aim:
To find a procedure to transfer the files from one virtual machine to another virtual machine.

Procedure:
1. You can copy few (or more) lines with copy & paste mechanism. For this you need to share clipboard
between host OS and guest OS, installing Guest Addition on both the virtual machines (probably setting
bidirectional and restarting them). You copy from guest OS in the clipboard that is shared with the host OS.
Then you paste from the host OS to the second guest OS.

2. You can enable drag and drop too with the same method (Click on the machine, settings,
general, advanced, drag and drop: set to bidirectional )

3. You can have common Shared Folders on both virtual machines and use one of the directory shared as
buffer to copy.
Installing Guest Additions you have the possibility to set Shared Folders too. As you put a file in a shared
folder from host OS or from guest OS, is immediately visible to the other. (Keep in mind that can arise
some problems for date/time of the files when there are different clock settings on the different virtual
machines). If you use the same folder shared on more machines you can exchange files directly copying
them in this folder.

4. You can use usual method to copy files between 2 different computer with client-server application. (e.g.
scp with sshd active for linux, winscp... you can get some info about SSH servers e.g. here)
You need an active server (sshd) on the receiving machine and a client on the sending machine. Of course
you need to have the authorization setted (via password or, better, via an automatic authentication method).
Note: many Linux/Ubuntu distribution install sshd by default: you can see if it is running with pgrep sshd
from a shell. You can install with sudo apt-get install openssh-server.

5. You can mount part of the file system of a virtual machine via NFS or SSHFS on the other, or you can
share file and directory with Samba.
You may find interesting the article Sharing files between guest and host without VirtualBox shared folders
with detailed step by step instructions.
You should remember that you are dialing with a little network of machines with different operative systems,
and in particular:
• Each virtual machine has its own operative system running on and acts as a physical machine.
• Each virtual machine is an instance of a program owned by an user in the hosting operative system and
should undergo the restrictions of the user in the hosting OS.
E.g Assume Hastur and Meow are users of the hosting machine, but they did not allow each other to see their
directories (no read/write/execute authorization). When each of them run a virtual machine, for the hosting
OS those virtual machine are two normal programs owned by Hastur and Meow and cannot see the private
directory of the other user. This is a restriction due to the hosting OS. It's easy to overcame it: it's enough to
give authorization to read/write/execute to a directory or to chose a different directory in which both users
can read/write/execute.
• Windows likes mouse and Linux fingers.
I mean I suggest you to enable Drag & drop to be cosy with the Windows machines and the Shared folders
or to be cosy with Linux.
When you will need to be fast with Linux you will feel the need of ssh-keygen and to Generate once SSH
Keys to copy files on/from a remote machine without writing password anymore. In this way it functions
bash auto-completion remotely too!

Result:
Thus, we transferred the files from one virtual machine to another virtual machine.
 Ex.No: 7
Launch virtual machine using trystack (Online
Date : Openstack Demo Version)

Aim:
To find a procedure to launch virtual machine using trystack (Online Openstack Demo Version)

Procedure:
OpenStack is an open-source software cloud computing platform. OpenStack is primarily used for deploying
an infrastructure as a service (IaaS) solution like Amazon Web Service (AWS). In other words, you can make
your own AWS by using OpenStack. If you want to try out OpenStack, TryStack is the easiest and free way to
do it.
In order to try OpenStack in TryStack, you must register yourself by joining TryStack Facebook Group. The
acceptance of group needs a couple days because it’s approved manually. After you have been accepted in the
TryStack Group, you can log in TryStack.

TryStack.org Homepage
I assume that you already join to the Facebook Group and login to the dashboard. After you log in to the
TryStack, you will see the Compute Dashboard like:
OpenStack Compute Dashboard

In this post, I will show you how to run an OpenStack instance. The instance will be accessible through the
internet (have a public IP address). The final topology will like:

Network topology
As you see from the image above, the instance will be connected to a local network and the local network will
be connected to internet.
Step 1: Create Network

1. Go to Network > Networks and then click Create Network.

2. In Network tab, fill Network Name for example internal and then click Next.
3. In Subnet tab,
1. Fill Network Address with appropriate CIDR, for example 192.168.1.0/24. Use private network CIDR
block as the best practice.
2. Select IP Version with appropriate IP version, in this case IPv4.
3. Click Next.
4. In Subnet Details tab, fill DNS Name Servers with 8.8.8.8 (Google DNS) and then click Create.

Step 2: Create Instance

1. Go to Compute > Instances and then click Launch Instance.

2. In Details tab,
1. Fill Instance Name, for example Ubuntu 1.
2. Select Flavor, for example m1.medium.
3. Fill Instance Count with 1.
4. Select Instance Boot Source with Boot from Image.
5. Select Image Name with Ubuntu 14.04 amd64 (243.7 MB) if you want install Ubuntu
14.04 in your virtual machine.
3. In Access & Security tab,
1. Click [+] button of Key Pair to import key pair. This key pair is a public and private key that we
will use to connect to the instance from our machine.
2. In Import Key Pair dialog,
1. Fill Key Pair Name with your machine name (for example Edward-Key).
2. Fill Public Key with your SSH public key (usually is in ~/.ssh/id_rsa.pub). See description in
Import Key Pair dialog box for more information. If you are using Windows, you can use Puttygen to
generate key pair.
3. Click Import key pair.
3. In Security Groups, mark/check default.
4. In Networking tab,
1. In Selected Networks, select network that have been created in Step 1, for example internal.
5. Click Launch.
6. If you want to create multiple instances, you can repeat step 1-5. I created one more instance with instance
name Ubuntu 2.

Step 3: Create Router

I guess you already know what router is. In the step 1, we created our network, but it is isolated. It doesn’t
connect to the internet. To make our network has an internet connection, we need a router that running as the
gateway to the internet.

1. Go to Network > Routers and then click Create Router.

2. Fill Router Name for example router1 and then click Create router.
3. Click on your router name link, for example router1, Router Details page.
4. Click Set Gateway button in upper right:
1. Select External networks with external.
2. Then OK.
5. Click Add Interface button.
1. Select Subnet with the network that you have been created in Step 1.
 2. Click Add interface.
6. Go to Network > Network Topology. You will see the network topology. In the example, there are two
network, i.e. external and internal, those are bridged by a router. There are
instances those are joined to internal network.

Step 4: Configure Floating IP Address

Floating IP address is public IP address. It makes your instance is accessible from the internet. When you
launch your instance, the instance will have a private network IP, but no public IP. In OpenStack, the public
IPs is collected in a pool and managed by admin (in our case is TryStack). You need to request a public
(floating) IP address to be assigned to your instance.
1. Go to Compute > Instance.
2. In one of your instances, click More > Associate Floating IP.
3. In IP Address, click Plus [+].
4. Select Pool to external and then click Allocate IP.
5. Click Associate.
6. Now you will get a public IP, e.g. 8.21.28.120, for your instance.

Step 5: Configure Access & Security

OpenStack has a feature like a firewall. It can whitelist/blacklist your in/out connection. It is called Security Group.
1. Go to Compute > Access & Security and then open Security Groups tab.
2. In default row, click Manage Rules.
3. Click Add Rule, choose ALL ICMP rule to enable ping into your instance, and then click Add.
4. Click Add Rule, choose HTTP rule to open HTTP port (port 80), and then click Add.
5. Click Add Rule, choose SSH rule to open SSH port (port 22), and then click Add.
6. You can open other ports by creating new rules.

Step 6: SSH to Your Instance

Now, you can SSH your instances to the floating IP address that you got in the step 4. If you are using Ubuntu
image, the SSH user will be ubuntu.

Result:
Thus, we launched a virtual machine using trystack (Online Openstack Demo Version)
 Ex.No: 8
Install Hadoop single node cluster and run simple
Date : applications like wordcount

Aim:
To install Hadoop single node cluster and run simple applications like wordcount.

Procedure:

Step 1: Download the Java 8 Package. Save this file in your home directory.
Step 2: Extract the Java Tar File.
Command: tar -xvf jdk-8u101-linux-i586.tar.gz

Fig: Hadoop Installation – Extracting Java Files

Step 3: Download the Hadoop 2.7.3 Package.

Command: wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop- 2.7.3.tar.gz

Fig: Hadoop Installation – Downloading Hadoop

Step 4: Extract the Hadoop tar File.

Command: tar -xvf hadoop-2.7.3.tar.gz

Fig: Hadoop Installation – Extracting Hadoop Files Step 5: Add the Hadoop and Java paths in
the bash file (.bashrc). Open. bashrc file. Now, add Hadoop and Java Path as shown below.
Command: vi .bashrc
 Fig: Hadoop Installation – Setting Environment Variable

Then, save the bash file and close it.

For applying all these changes to the current Terminal, execute the source command.
Command: source .bashrc

Fig: Hadoop Installation – Refreshing environment variables

To make sure that Java and Hadoop have been properly installed on your system and can be accessed through
the Terminal, execute the java -version and hadoop version commands.
Command: java -version

Fig: Hadoop Installation – Checking Java Version

Command: hadoop version

Fig: Hadoop Installation – Checking Hadoop Version

Step 6: Edit the Hadoop Configuration files.
Command: cd hadoop-2.7.3/etc/hadoop/
Command: ls
All the Hadoop configuration files are located in hadoop-2.7.3/etc/hadoop directory as you can see in the
snapshot below:

Fig: Hadoop Installation – Hadoop Configuration Files

Step 7: Open core-site.xml and edit the property mentioned below inside configuration tag:
core-site.xml informs Hadoop daemon where NameNode runs in the cluster. It contains configuration settings
of Hadoop core such as I/O settings that are common to HDFS & MapReduce.
Command: vi core-site.xml

Fig: Hadoop Installation – Configuring core-site.xml

1
<?xml version="1.0" encoding="UTF-8"?>
2
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>fs.default.name</name>
6 <value>hdfs://localhost:9000</value>
</property>
7 </configuration>
8

Step 8: Edit hdfs-site.xml and edit the property mentioned below inside configuration tag:
hdfs-site.xml contains configuration settings of HDFS daemons (i.e. NameNode, DataNode, Secondary
NameNode). It also includes the replication factor and block size of HDFS.

Command: vi hdfs-site.xml
 Fig: Hadoop Installation – Configuring hdfs-site.xml

1
2 <?xml version="1.0" encoding="UTF-8"?>
3 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
4 <property>
5 <name>dfs.replication</name>
6 <value>1</value>
7 </property>
<property>
8
<name>dfs.permission</name>
9 <value>false</value>
10 </property>
11 </configuration>
12

Step 9: Edit the mapred-site.xml file and edit the property mentioned below
inside configuration tag:
mapred-site.xml contains configuration settings of MapReduce application like number of JVM that can run in
parallel, the size of the mapper and the reducer process, CPU cores available for a process, etc.
In some cases, mapred-site.xml file is not available. So, we have to create the mapred- site.xml file using
mapred-site.xml template.

Command: cp mapred-site.xml.template mapred-site.xml

Command: vi mapred-site.xml.
 Fig: Hadoop Installation – Configuring mapred-site.xml

1
<?xml version="1.0" encoding="UTF-8"?>
2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
3 <configuration>
4 <property>
5 <name>mapreduce.framework.name</name>
6 <value>yarn</value>
</property>
7 </configuration>
8

Step 10: Edit yarn-site.xml and edit the property mentioned below inside configuration tag:

yarn-site.xml contains configuration settings of ResourceManager and NodeManager like application memory
management size, the operation needed on program & algorithm, etc.

Command: vi yarn-site.xml

Fig: Hadoop Installation – Configuring yarn-site.xml

 1
2
<?xml version="1.0">
3 <configuration>
4 <property>
5 <name>yarn.nodemanager.aux-services</name>
6 <value>mapreduce_shuffle</value>
</property>
7 <property>
8 <name>yarn.nodemanagxesre.ravuimcaepsr.ednuacmee.>
9 shuffle.class</
1 <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
0 </configuration>
1
1

Step 11: Edit hadoop-env.sh and add the Java Path as mentioned below:

hadoop-env.sh contains the environment variables that are used in the script to run Hadoop like Java home
path, etc.

Command: vi hadoop–env.sh

Fig: Hadoop Installation – Configuring hadoop-env.sh

Step 12: Go to Hadoop home directory and format the NameNode.

Command: cd

Command: cd hadoop-2.7.3

Command: bin/hadoop namenode -format

Fig: Hadoop Installation – Formatting NameNode

This formats the HDFS via NameNode. This command is only executed for the first time. Formatting the file
system means initializing the directory specified by the dfs.name.dir variable.
Never format, up and running Hadoop filesystem. You will lose all your data stored in the HDFS.

Step 13: Once the NameNode is formatted, go to hadoop-2.7.3/sbin directory and start all the daemons.

Command: cd hadoop-2.7.3/sbin

Either you can start all daemons with a single command or do it individually.

Command: ./start-all.sh

The above command is a combination of start-dfs.sh, start-yarn.sh & mr-jobhistory- daemon.sh

Or you can run all the services individually as below:

Start NameNode:

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files stored in the
HDFS and tracks all the file stored across the cluster.

Command: ./hadoop-daemon.sh start namenode

Fig: Hadoop Installation – Starting NameNode

Start DataNode:
On startup, a DataNode connects to the Namenode and it responds to the requests from the Namenode for
different operations.
Command: ./hadoop-daemon.sh start datanode

Fig: Hadoop Installation – Starting DataNode

Start ResourceManager:

ResourceManager is the master that arbitrates all the available cluster resources and thus helps in managing the
distributed applications running on the YARN system. Its work is to manage each NodeManagers and the
each application’s ApplicationMaster.

Command: ./yarn-daemon.sh start resourcemanager

Fig: Hadoop Installation – Starting ResourceManager

Start NodeManager:

The NodeManager in each machine framework is the agent which is responsible for managing containers,
monitoring their resource usage and reporting the same to the ResourceManager.
Command: ./yarn-daemon.sh start nodemanager

Fig: Hadoop Installation – Starting NodeManager

Start JobHistoryServer:

JobHistoryServer is responsible for servicing all job history related requests from client.
Command: ./mr-jobhistory-daemon.sh start historyserver

Step 14: To check that all the Hadoop services are up and running, run the below command.
Command: jps

Fig: Hadoop Installation – Checking Daemons

Step 15: Now open the Mozilla browser and go

to localhost:50070/dfshealth.html to check the NameNode interface.

Fig: Hadoop Installation – Starting WebUI

You have successfully installed a single node Hadoop cluster. To Run Word Count do the following:
WordCount.java Program:

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper

extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(Object key, Text value, Context context

) throws IOException, InterruptedException
{ StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

public static class IntSumReducer

extends Reducer<Text,IntWritable,Text,IntWritable>
{ private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,

Context context
) throws IOException, InterruptedException
{ int sum = 0;
for (IntWritable val : values)
{ sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

public static void main(String[] args) throws Exception

{ Configuration conf = new Configuration();
 Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

To execute Wordcount.java

Assuming environment variables are set as follows:

export JAVA_HOME=/usr/java/default
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

Compile WordCount.java and create a jar:

Command: bin/hadoop com.sun.tools.javac.Main WordCount.java

Command: jar cf wc.jar WordCount*.class
Assuming that:

/user/joe/wordcount/input - input directory in HDFS

/user/joe/wordcount/output - output directory in HDFS
Sample text-files as input:

Command: bin/hadoop fs -ls /user/joe/wordcount/input/

/user/joe/wordcount/input/file01
/user/joe/wordcount/input/file02

Command: bin/hadoop fs -cat /user/joe/wordcount/input/file01

Hello World Bye World

Command: bin/hadoop fs -cat /user/joe/wordcount/input/file02

Hello Hadoop Goodbye Hadoop
Run the application:

Command: bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output

Command for getting output:

Command: bin/hadoop fs -cat /user/joe/wordcount/output/part-r-00000
Output:

Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2

Result:
Thus, we installed Hadoop single node cluster and run simple applications like wordcount.