Managing Exadata in the Real
          World
         Arup Nanda
What is this About?
 •   How Exadata is different?
 •   Who manages it?
 •   Who does what?
 •   What you need to know about managing it
 •   What must you learn about potential issues




                                                  2
What is Exadata
 • Looks like an appliance
 • But is not an appliance. Why?
   – It contains additional software to make it a better
     database machine
   – The components are individually administered
 • That’s why Oracle calls it a Database Machine
   (DBM)
 • DMA – Database Machine Administrator


                                                           3
iDB

          CPU          iDB


        Memory

        Network

      I/O Controller

          Disk
                             4
Storage Cell Server

             iDB            •   Cells are Sun Blades
                            •   Run Oracle Enterprise
                                Linux
                            •   Software called Exadata
                                Storage Server (ESS)
                                which understands iDB


    Disk1   Disk2   Disk3

                                                          5
Storage Indexes

 Storage Indexes store in memory
 of the Cell Server the areas on
 the disk and the MIN/MAX value
 of the column and whether NULL         Disk1           Disk2      Disk3
 exists. They eliminate disk I/O.

SELECT …                  MIN = 3   MIN = 4      MIN = 1        MIN = 3
FROM TABLE                MAX = 5   MAX = 5      MAX = 2        MAX = 5
WHERE COL1 = 1



          Storage Index                         Disk4
                                                                           6
Put Together: One Full Rack
                    RAC Cluster
       Database                     Database
        Node 1                       Node 8
             Database
               Node 7                           Clients
              InfiniBand     Network Switch     connect to the
                Switch                          database
                                                nodes.
         Cell 1                       Cell 14
            Cell 1
                Cell 1
                   Cell 1




                                                                 7
Disk Layout      •   Disks (hard and flash) are
                     connected to the cells.
                 •   The disks are partitioned at the
       Compute
        Nodes        cell
                 •   Some partitions are presented
                     as filesystems
       Storage
                 •   The rest are used for ASM
         Cell        diskgroups
                 •   All these disks/partitions are
                     presented to the compute nodes




                                                        8
Disk Presentation




                    Node
                           filesystem


             filesystem
  Cell




                                        9
Disk Failures
                    Datafile
                    block1


           Cell 1              Cell 2

         block1                block1




                                        10
One Cluster?

                      One Cluster




 QA1   QA2   QA3     Prod1    Prod2   Prod3   Dev1   Int1


 QA1   QA2   Prod4   Prod1    Prod2   Prod3   Dev1   Int1


 QA1   QA2   QA3     Prod1    Prod2   Prod3   Dev1   Int1

                                                     Dev2
                                                            11
Many Clusters?

       QA Cluster                 Prod Cluster
                                                         Dev    Int


 QA1     QA2        QA3   Prod1      Prod2       Prod3   Dev1   Int1




  QA Cluster                Prod Cluster
                                                         Dev    Int


                                                                       12
Playing Nice
 • Database Resource Manager
 • I/O Resource Manager
 • Cell Fencing
                                      Compute
                QA             Prod   Nodes




                                      Storage Cells




                                                  13
Component Management
                  Linux Commands – vmstat, mpstat, fdisk, etc.
    Compute
     Nodes
                  ASM Commands – SQL*Plus, ASMCMD, ASMCA
                  Database Commands – startup, alter database, etc.
                  Clusterware Commands – CRSCTL, SRVCTL, etc.
   Storage Cell
                  Linux Commands – vmstat, mpstat, fdisk, etc.
                  CellCLI – command line tool to manage the Cell

                      5-part Linux Commands article series
                      http://bit.ly/k4mKQS
                      4-part Exadata Command Reference article series
                      http://bit.ly/lljFl0

                                                                        14
Server Management
 • Sun Blades and Oracle Enterprise Linux
 • Normal Sysadmin Work
   – Shutdown, fdisk, etc.
 • ILOM – Integrated Lights Out Management
 • KVM allows physical access
   – But you can use ILOM for virtual console
 • Needs Pure Linux Skills



                                                15
Database, Cluster Management
 • Cluster
   – crsctl, srvctl
 • ASM
   – asmcmd, SQL*Plus
 • Database
   – srvctl, SQL*Plus




                               16
Storage Management
 • Two ways to manage the
   storage
   – Traditional Storage
     Administrators not quite           iDB
     helpful here
   – Enterprise Manager
   – CellCLI



                                Disk1   Disk2   Disk3


                                                        17
Network Management
• Two types of network
  – Ethernet
  – Infiniband
• Tools
  – ibstatus
  – iblinkinfo
  – verify-topology




                         18
Oracle Provided Tools
 • All tools are found at
    /opt/oracle.SupportTools
 • CheckHWnFWProfile
    – to check the HW profile
 • Directory ibdiagtools
    /opt/MegaRAID/MegaCli/MegaCli64
    # ipmitool -H prolcel01-ilom -U root chassis
      power on
    # imageinfo
    # imagehistory

                                                   19
Skills for Administration
 Skill                    Needed
 System Administrator        10%
 Storage Administrator        0%
 Network Administrator        5%
 Database Administrator      60%
 Cell Administration         25%

                                   DBA
                                   Sys Admin
                                   Network Admin
                                   Cell Admin
                                                   20
Divide and Conquer
  Database            DBA


   Machine    System Admin


   Network   Network Admin


   Storage              ??   DBA
                             Sys Admin
                             Network Admin
                             Cell Admin


                                          21
Combined Skills
  Database                Database
                          Machine
                        Administrator
   Machine                 (DMA)
             New Role
   Network


   Storage                              DBA
                                        Sys Admin
                                        Network Admin
                                        Cell Admin


                                                     22
Resources
 • My Papers
    – 5-part Linux Commands article series
      http://bit.ly/k4mKQS
    – 4-part Exadata Command Reference article series
      http://bit.ly/lljFl0
 • OTN Page on Exadata
    – http://www.oracle.com/technetwork/database/exadata/in
      dex.html
 • Tutorials
    – http://www.oracle.com/technetwork/tutorials/index.html


                                                               23
Pros and Cons of DMA
 • Pros
   – Natural progression from DBA skills
   – Very little non-DBA activities
   – Patching becomes simpler
   – Performance diagnosis becomes easier
 • Cons
   – Skills
   – Security?
      • Appliance model comes in here


                                            24
Backup and DR
 • No SAN connectivity
 • Only NAS                                      Exadata
    – Infiniband
    – Tape , Disk Pool
 • DR                                 Infiniband
    –   No Storage Level Replication
    –   Only Data Guard                          Backup
    –   Supplemental Logging                     Device
    –   Force Logging
    –   http://www.oracle.com/technetwork/database/features/availa
        bility/maa-wp-dr-dbm-130065.pdf
 • Golden Gate

                                                                     25
ETL and Reporting                      OLTP

                  Exadata
                                      Golden
                                      Gate
                  Infiniband

    Informatica                MicroStrategy


                                 Exalytics
                                               26
Summary
• Exadata is an Oracle Database on RAC with a
  specialized storage
• A single role helps in administration: Database
  Machine Administrator (DMA)
• DBA -> DMA is the natural progression
• Use many clusters in the same Exadata frame
• Use IB for ETL, Reporting and Backups
• Data Guard only physical DR solution


                                                    27
Thank You!
             28

Managing Exadata in the Real World

  • 1.
    Managing Exadata inthe Real World Arup Nanda
  • 2.
    What is thisAbout? • How Exadata is different? • Who manages it? • Who does what? • What you need to know about managing it • What must you learn about potential issues 2
  • 3.
    What is Exadata • Looks like an appliance • But is not an appliance. Why? – It contains additional software to make it a better database machine – The components are individually administered • That’s why Oracle calls it a Database Machine (DBM) • DMA – Database Machine Administrator 3
  • 4.
    iDB CPU iDB Memory Network I/O Controller Disk 4
  • 5.
    Storage Cell Server iDB • Cells are Sun Blades • Run Oracle Enterprise Linux • Software called Exadata Storage Server (ESS) which understands iDB Disk1 Disk2 Disk3 5
  • 6.
    Storage Indexes StorageIndexes store in memory of the Cell Server the areas on the disk and the MIN/MAX value of the column and whether NULL Disk1 Disk2 Disk3 exists. They eliminate disk I/O. SELECT … MIN = 3 MIN = 4 MIN = 1 MIN = 3 FROM TABLE MAX = 5 MAX = 5 MAX = 2 MAX = 5 WHERE COL1 = 1 Storage Index Disk4 6
  • 7.
    Put Together: OneFull Rack RAC Cluster Database Database Node 1 Node 8 Database Node 7 Clients InfiniBand Network Switch connect to the Switch database nodes. Cell 1 Cell 14 Cell 1 Cell 1 Cell 1 7
  • 8.
    Disk Layout • Disks (hard and flash) are connected to the cells. • The disks are partitioned at the Compute Nodes cell • Some partitions are presented as filesystems Storage • The rest are used for ASM Cell diskgroups • All these disks/partitions are presented to the compute nodes 8
  • 9.
    Disk Presentation Node filesystem filesystem Cell 9
  • 10.
    Disk Failures Datafile block1 Cell 1 Cell 2 block1 block1 10
  • 11.
    One Cluster? One Cluster QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1 QA1 QA2 Prod4 Prod1 Prod2 Prod3 Dev1 Int1 QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1 Dev2 11
  • 12.
    Many Clusters? QA Cluster Prod Cluster Dev Int QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1 QA Cluster Prod Cluster Dev Int 12
  • 13.
    Playing Nice •Database Resource Manager • I/O Resource Manager • Cell Fencing Compute QA Prod Nodes Storage Cells 13
  • 14.
    Component Management Linux Commands – vmstat, mpstat, fdisk, etc. Compute Nodes ASM Commands – SQL*Plus, ASMCMD, ASMCA Database Commands – startup, alter database, etc. Clusterware Commands – CRSCTL, SRVCTL, etc. Storage Cell Linux Commands – vmstat, mpstat, fdisk, etc. CellCLI – command line tool to manage the Cell 5-part Linux Commands article series http://bit.ly/k4mKQS 4-part Exadata Command Reference article series http://bit.ly/lljFl0 14
  • 15.
    Server Management •Sun Blades and Oracle Enterprise Linux • Normal Sysadmin Work – Shutdown, fdisk, etc. • ILOM – Integrated Lights Out Management • KVM allows physical access – But you can use ILOM for virtual console • Needs Pure Linux Skills 15
  • 16.
    Database, Cluster Management • Cluster – crsctl, srvctl • ASM – asmcmd, SQL*Plus • Database – srvctl, SQL*Plus 16
  • 17.
    Storage Management •Two ways to manage the storage – Traditional Storage Administrators not quite iDB helpful here – Enterprise Manager – CellCLI Disk1 Disk2 Disk3 17
  • 18.
    Network Management • Twotypes of network – Ethernet – Infiniband • Tools – ibstatus – iblinkinfo – verify-topology 18
  • 19.
    Oracle Provided Tools • All tools are found at /opt/oracle.SupportTools • CheckHWnFWProfile – to check the HW profile • Directory ibdiagtools /opt/MegaRAID/MegaCli/MegaCli64 # ipmitool -H prolcel01-ilom -U root chassis power on # imageinfo # imagehistory 19
  • 20.
    Skills for Administration Skill Needed System Administrator 10% Storage Administrator 0% Network Administrator 5% Database Administrator 60% Cell Administration 25% DBA Sys Admin Network Admin Cell Admin 20
  • 21.
    Divide and Conquer Database DBA Machine System Admin Network Network Admin Storage ?? DBA Sys Admin Network Admin Cell Admin 21
  • 22.
    Combined Skills Database Database Machine Administrator Machine (DMA) New Role Network Storage DBA Sys Admin Network Admin Cell Admin 22
  • 23.
    Resources • MyPapers – 5-part Linux Commands article series http://bit.ly/k4mKQS – 4-part Exadata Command Reference article series http://bit.ly/lljFl0 • OTN Page on Exadata – http://www.oracle.com/technetwork/database/exadata/in dex.html • Tutorials – http://www.oracle.com/technetwork/tutorials/index.html 23
  • 24.
    Pros and Consof DMA • Pros – Natural progression from DBA skills – Very little non-DBA activities – Patching becomes simpler – Performance diagnosis becomes easier • Cons – Skills – Security? • Appliance model comes in here 24
  • 25.
    Backup and DR • No SAN connectivity • Only NAS Exadata – Infiniband – Tape , Disk Pool • DR Infiniband – No Storage Level Replication – Only Data Guard Backup – Supplemental Logging Device – Force Logging – http://www.oracle.com/technetwork/database/features/availa bility/maa-wp-dr-dbm-130065.pdf • Golden Gate 25
  • 26.
    ETL and Reporting OLTP Exadata Golden Gate Infiniband Informatica MicroStrategy Exalytics 26
  • 27.
    Summary • Exadata isan Oracle Database on RAC with a specialized storage • A single role helps in administration: Database Machine Administrator (DMA) • DBA -> DMA is the natural progression • Use many clusters in the same Exadata frame • Use IB for ETL, Reporting and Backups • Data Guard only physical DR solution 27
  • 28.