SAP HANA Smart Data Access using Hadoop/Hive

SAP HANA Smart Data Access using Hadoop/Hive

=================================================================================================

By

Debajit Banerjee

Table of Contents

Introduction about SAP HANA Smart Data Access. Page 02

I.HDP 1.3 for Windows Installation Pre-requisite.. Page 03
II.HDP 1.3 for Windows (Hortonworks Data Platform) Standalone Installation.. Page 13
III.Validation of HDP 1.3 for Windows - Standalone Installation. Page 16
IV.Data Load in Hadoop System : eBook Upload. Page 26
V.Unstructured Data Transformation into Table/View in Hadoop System Page 35
VI.ODBC Driver Installation & Configuration on SAP HANA Server. Page 40
VII.Smart Data Access (Hadoop Data) in SAP HANA.. Page 47

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 1

SAP HANA Smart Data Access using Hadoop/Hive

SAP HANA Smart Data Access

Using the feature of SAP HANA Smart Data Access, it is possible to access remote data, without having to replicate the
data to the SAP HANA database beforehand. The following are supported as sources(till 2013):
Teradata database,
SAP Sybase ASE,
SAP Sybase IQ,
Intel Distribution for Apache Hadoop,
SAP HANA.
SAP HANA handles the data like local tables on the database. Automatic data type conversion makes it possible to map
data types from databases connected via SAP HANA Smart Data Access to SAP HANA data types.

Steps/Procedure :

Hadoop Installation
Data Load in Hadoop system
Activities on Unstructured Data in Hadoop system
ODBC Driver installation & configuration on HANA Server for Hadoop system data access
Smart Data Access in SAP HANA (through SAP HANA Studio), using HADOOP as a remote data source

Assumption SAP HANA System is already up & running.

Scenario / Lab Setup Details :
1) Hadoop Installation Pre-requisite : HDP 1.3 for Windows(Hortonworks Data Platform) - Standalone
2) Hadoop Installation : HDP 1.3 for Windows(Hortonworks Data Platform) - Standalone on Dell Laptop, OS Win7
64bit 8GB)
3) SAP HANA Sever Installation(Lab Server running on VM 24GB Standalone HANA 1.0 SPS 70) SLES 11 SP1
4) Validation of Hadoop Installation
5) Data Load in Hadoop system : eBook Upload
6) Unstructured Data transformation into table/views, so that HANA Server can understand Hadoop data.
7) ODBC Driver installation & configuration on HANA Server
8) Smart Data Access in SAP HANA (through SAP HANA Studio), using Hadoop as a remote data source

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 2

SAP HANA Smart Data Access using Hadoop/Hive

I. HDP 1.3 for Windows Installation Pre-requisite

- On HANA Server -Simba : Apache Hive ODBC Driver Linux 64bit
- On Hadoop System - Microsoft Visual C++ 2010 Redistributable Package (64bit)
- On Hadoop System - Microsoft .NET Framework 4.0
- On Hadoop System - JAVA JDK 1.6/1.7 and PATH, JAVA_HOME environment variables setup
- On Hadoop System - Python 2.7 and PATH environment variable setup

In Linux

In Windows

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 3

SAP HANA Smart Data Access using Hadoop/Hive

MS Visual C++ 2010

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 4

SAP HANA Smart Data Access using Hadoop/Hive

MS .NET Framework 4

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 5

SAP HANA Smart Data Access using Hadoop/Hive

Cancelling it as it gives the option of Repair !!

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 6

SAP HANA Smart Data Access using Hadoop/Hive

Oracle JDK

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 7

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 8

SAP HANA Smart Data Access using Hadoop/Hive

i.
ii.
iii.
iv.
v.
vi.

Open the Control Panel -> System pane and click on Advanced system
settings.
Click on the Advanced tab.
Click the Environment Variables button.
Under System variables, click New.
Enter the Variable Name as JAVA_HOME.
Enter the Variable Value, as the installation path for the Java Development Kit.
For example, if your JDK is installed at C:\Java\jdk1.6.0_31, then you must
provide this path to the Variable Value.

vii.
viii.

Click OK.
Click OK to close the Environment Variables dialog box.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 9

SAP HANA Smart Data Access using Hadoop/Hive

Python

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 10

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 11

SAP HANA Smart Data Access using Hadoop/Hive

Like Oracle JDK above, C:\Python27 also to be set in PATH variable.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 12

SAP HANA Smart Data Access using Hadoop/Hive

II. HDP 1.3 for Windows (Hortonworks Data Platform) Standalone Installation

Now accordingly update the C:\hdp-1.3.0.0-GA\clusterproperties.txt as per following:

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 13

SAP HANA Smart Data Access using Hadoop/Hive

In Command Window(Admin Privilege):

msiexec /i "C:\hdp-1.3.0.0-GA\hdp-1.3.0.0.winpkg.msi" /lv "C:\DEBAJIT\HD\hdp13\hdp.log" HDP_LAYOUT="C:\hdp1.3.0.0-GA\clusterproperties.txt" HDP_DIR="C:\hdp\hadoop" DESTROY_DATA="Yes"

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 14

SAP HANA Smart Data Access using Hadoop/Hive

There are 3 shortcuts created in desktop area.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 15

SAP HANA Smart Data Access using Hadoop/Hive

III. Validation of HDP 1.3 for Windows - Standalone Installation

Now we have to start Hadoop.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 16

SAP HANA Smart Data Access using Hadoop/Hive

Services were not started due to 0 bytes in .xml files(master & regionserver)
Also rest/thrift/thrift2.xml are also of zero bytes.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 17

SAP HANA Smart Data Access using Hadoop/Hive

1) Navigate to the hbase install directory: C:\hdp\hadoop\hbase-0.94.6.1.3.0.0-0380\bin

2) Open the hbase.cmd in a text editor
3) Look for the line that says:
set PATH=%PATH%;%HADOOP_HOME%\bin
4) Delete it or comment it out with a @rem
Now Open a command prompt and navigate to hbase install: C:\hdp\hadoop\hbase-0.94.6.1.3.0.0-0380\bin
Rebuild the .xml files:
hbase.cmd --service master start > master.xml
hbase.cmd --service regionserver start > regionserver.xml
hbase.cmd --service rest > rest.xml
hbase.cmd --service thrift > thrift.xml
hbase.cmd --service thrift2 > thrift2.xml

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 18

SAP HANA Smart Data Access using Hadoop/Hive

Now all the above .xml files having contents.

Stop & Start Hadoop now it is PERFECT. No more failed services.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 19

SAP HANA Smart Data Access using Hadoop/Hive

Hadoop Smoketest

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 20

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 21

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 22

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 23

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 24

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 25

SAP HANA Smart Data Access using Hadoop/Hive

IV. Data Load in Hadoop System : eBook Upload

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 26

SAP HANA Smart Data Access using Hadoop/Hive

Now to check whether Hadoop can read the same or not

It canperfect !!

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 27

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 28

SAP HANA Smart Data Access using Hadoop/Hive

After refresh

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 29

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 30

SAP HANA Smart Data Access using Hadoop/Hive

From the Namenode server, click on Browse the filesystem

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 31

SAP HANA Smart Data Access using Hadoop/Hive

Click on user

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 32

SAP HANA Smart Data Access using Hadoop/Hive

Click on .txt fileone can see the book

If one can click on .out file, then one can see the part file

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 33

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 34

SAP HANA Smart Data Access using Hadoop/Hive

V. Unstructured Data Transformation into Table/View in Hadoop System

Now we have to convert those files to be readable table format for HANA. For that we will use HIVE.

Created a table called debajit_wc for wordcount part file. But right now, it is empty.

Now loading Data.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 35

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 36

SAP HANA Smart Data Access using Hadoop/Hive

Configuration change required in hive-site.xml file.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 37

SAP HANA Smart Data Access using Hadoop/Hive

Just changed from http to thrift servermode.

And then restart Hadoop.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 38

SAP HANA Smart Data Access using Hadoop/Hive

Now we can test whether SAP HANA can connect to Hadoop.

Download the license file from email and deployed. Problem solved.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 39

SAP HANA Smart Data Access using Hadoop/Hive

VI. ODBC Driver Installation & Configuration on SAP HANA Server

Renaming done at WinSCP level.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 40

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 41

SAP HANA Smart Data Access using Hadoop/Hive

Stopping HANA System

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 42

SAP HANA Smart Data Access using Hadoop/Hive

SIMBA Driver

Changed items are as follows:

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 43

SAP HANA Smart Data Access using Hadoop/Hive

UNIXODBC

We have to upgrade it because of compatibility issue with Simba.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 44

SAP HANA Smart Data Access using Hadoop/Hive

ODBC.INI - DSN purpose

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 45

SAP HANA Smart Data Access using Hadoop/Hive

Now added odbc information into customer.sh

So, now the connection is working between HANA Server and Hadoop system from OS level.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 46

SAP HANA Smart Data Access using Hadoop/Hive

VII. Smart Data Access (Hadoop Data) in SAP HANA

SAP HANA Studio

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 47

SAP HANA Smart Data Access using Hadoop/Hive

So, now the connection is working between HANA Server and Hadoop system from SAP HANA Studio.
Creating a schema in HP7

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 48

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 49

SAP HANA Smart Data Access using Hadoop/Hive

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 50

SAP HANA Smart Data Access using Hadoop/Hive

One can do Query and Connection Monitoring when click on Smart Data Access under Provisioning.

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 51

SAP HANA Smart Data Access using Hadoop/Hive

Thats all.

**** END OF DOCUMENT ****

http://debajitb.wix.com/debajitbanerjee | http://debajitb.wix.com/debajitbanerjee/apps/blog

Page 52