Installing a Clean DB Machine

In past posts, I have written about some of the functionality of Exadata and recently, my colleague Uwe Hesse has added some very fine new posts about specific Exadata features. This post discusses the installation of the Oracle Database Machine at a customer location, as carried out by a combination of Oracle Sun field engineers and Oracle Advanced Customer Services (ACS) engineers.

The planning stages for installation of a Database machine involve the following requirements:

• Environmental setup for space and flooring
• Ventilation, humidity and temperature considerations
• Power requirements
• Network cabling
• Network address assignments for the customer network

Much of this may be found in the Oracle Exadata Database Machine Owner’s Guide for 11g Release 2 which is made available to customers who order the machine.

The Oracle Sun field engineers are involved with the setup of the Database Machine at the customer site, and handle the following types of activities:

• Connecting power to the racks
• Cabling multiple full racks together or half and full racks on the same Infiniband fabric if required
• Configuring the Infiniband switches
• Configuring the Cisco Ethernet switch
• Configuring the Keyboard, Video and Mouse (KVM) hardware if present
• Verifying the configured network topology

Once  the Sun engineers are finished, then the ACS engineers configure the software on all the Database Servers and Storage Servers within the Database machine. In the original V1 Exadata days, each machine was configured for the customer network during the first boot of the machine. This first time boot prompted for the following:

• Name server details
• Time zone details
• NTP server details
• DNS details
• Network Interface IP address assignment for management network on adaptor “eth0“
• Other network details for client network if one is used for accessing the database servers
• Network details for the bonded Infiniband adaptor  “bond0” used for the storage network and cluster interconnect
• Configuration details for the Integrated Lights Out Manager (ILOM)
  • Hostname
  • Network details
  • DNS details
  • NTP details
  • Timezone details

Once this had been completed, the servers could be connected to the customer network. Then the Grid Infrastructure software and database software were installed and configured, followed by the creation of a database on the Database machine.

This interactive method is no longer used, as ACS have developed automated techniques to facilitate installation and configuration. The process, which has evolved over time, begins with the filling in of a configuration spreadsheet in consultation with the customer. Details of this may be found in the Oracle Exadata Database Machine Configuration Worksheets for 11g Release 2 documentation. The ACS engineer fills in details of all the information listed above for network and ILOM using ranges of IP addresses, plus all the other details mentioned.

In addition, the following is done on the spreadsheet:

• Selecting Full, Half or Quarter Rack to match hardware ordered
• Selecting High Performance or High Capacity SAS disks to match hardware ordered
• Entering database node and cell node name prefixes
• Entering rack name prefix
• Entering details for notification of alerts by the cells
  • SMTP details for email notification of alerts
  • SNMP Subscriber details for SNMP traps
• Entering OS user details for the Grid Infrastructure software install
• Entering OS user details for the Oracle Database software install
• Entering the number, names and size of ASM diskgroups to be created
• Entering details of databases to be created

Once the spreadsheet is complete, the config details may be generated as output lower down in the sheet for review. Then the data is stored in a set of files generated by some spreadsheet macros. These files are loaded onto the first database server by the ACS engineer at the customer site and used for configuring the machines in the rack.

The machines are shipped with OS images pre-installed with default IP addresses and subnet masks. This permits the ACS engineer to access the first database node using a PC connected to that network, via the ILOM console. Alternatively for X2-2 configurations, the KVM may be used. The spreadsheet config files are copied to the “/opt/oracle.Supporttools/onecommand” directory and used as input to automation scripts:

• /opt/oracle.Supporttools/firstconf/applyconfig.sh updates the IP addresses for servers in rack and reboots them
• The Ethernet cables from the built in switch are then connected for the management network to each server
• The Ethernet cables from the built in switch are then connected for the ILOM cards on each server
• If a client access network is used, then Ethernet cables from another corporate switch are connected to each server

Now that all the servers are on the required networks, the installation and configuration of the software and creation of the databases may be done. This uses /opt/oracle.Supporttools/onecommand/deploy<nnn>.sh script, where nnn is a number that is release specific and tailoring enhancements may be  made to the process. This command has many steps which are contained in a config file. The ACS engineer may edit files to tailor this generic solution to the customer requirements.

The Oracle Grid Infrastructure is installed with a silent install and configured automatically as is the Oracle Database software. Finally, if a database is created, then this too will be done silently.

My ACS Colleagues inform me that a full rack required about 5 hours to do this including time required for downloading any patches required in addition to the base software releases.

If you are planning on Exadata, then I hope this helps in appreciating the process involved.

Joel

2/2011