Introduction

While working on an Exadata X2-2 running Exadata Storage Software 12.1.2.1.1, I came across an issue with System Disk Replacement. After the System Disk Replacement, the status of the Disk shows "normal - rejected due to system disk imaging failed". Oracle identified this as a BUG in Exadata version 12.1.2.1.0 and 12.1.2.1.1.

Before we see the steps to resolve the System Disk Replacement issue, let's understand about Exadata System area and disk failure briefly.

Exadata System Area

Each Exadata Storage cell consists of twelve (12) physical disks. From the first two disks (disk 0 and disk 1), Oracle uses 29GB of storage to create System area. This system area is used for Operating System, Exadata Storage Software, boot file system, logs and diagnostic files. The System Area is mirrored using Linux software RAID.

The following commands show you on which devices the system and software partitions are created.

[root@oracloudceladm01 ~]# imageinfo | grep partition

Active system partition on device: /dev/md5

Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sdac1

Inactive system partition on device: /dev/md6

Inactive software partition on device: /dev/md8

Rollback to the inactive partitions: Possible

[root@oracloudceladm01 ~]# df -h

Filesystem     Size Used Avail Use% Mounted on

/dev/md5       9.9G 6.2G 3.3G 66% /

tmpfs          12G     0   12G   0% /dev/shm

/dev/md7       2.0G 1.6G 312M 84% /opt/oracle

/dev/md4       116M   40M   71M 36% /boot

/dev/md11       2.3G 298M 1.9G 14% /var/log/oracle

[root@oracloudceladm01 ~]# mdadm -Q -D /dev/md6

/dev/md6:

       Version  : 0.90

Creation Time   : Fri Apr 8 16:54:24 2011

     Raid Level : raid1

     Array Size : 10482304 (10.00 GiB 10.73 GB)

Used Dev Size   : 10482304 (10.00 GiB 10.73 GB)

   Raid Devices : 2

Total Devices   : 2

Preferred Minor : 6

   Persistence  : Superblock is persistent

   Update Time  : Mon Oct 5 09:01:09 2015

         State  : clean

Active Devices  : 2

Working Devices : 2

Failed Devices  : 0

Spare Devices   : 0

           UUID : d57de4ac:d44f8b80:8f1f9533:83c82172

         Events : 0.482

   Number   Major   Minor   RaidDevice State

       0       8       6       0     active sync   /dev/sda6

       1       8       22       1     active sync   /dev/sdb6

From the above output, we can see that the /dev/sda and /dev/sdb physical devices are software mirrored using mdadm utility.

Hard Disk Failure

Hard disk failure is one of the most common hardware failure in Exadata. When a hard disk fails, a Hardware Automatic SR (ASR) is opened - if you are a platinum customer, an Oracle Field engineer contacts customer and visits data center for disk replacement. When a disk is replaced, Lun, cell disk, grid disks are automatically created and grid disks are automatically added to ASM Disk Group.

Most important thing to note here is, if you are running Exadata version 12.1.2.1.0 and 12.1.2.1.1 and replacing a System disk then you are hitting a BUG.

System Disk Replacement Issues

If you are running Exadata storage software 12.1.2.1.0 and 12.1.2.1.1, the system disk replacement process fails. The disk state will be normal but with the message "normal - rejected due to system disk imaging failed". Since the system disk (0 & 1) are mirrored using Linux software RAID there is no impact.

You can run the following commands to get the status of failed system disk.

[root@oracloudceladm01 FixDiskRep]# cellcli -e list physicaldisk where disktype=harddisk

         20:0           E4ADM1         normal - rejected due to system disk imaging failed

         20:1           E0AT2S         normal

         20:2           E0AGXE         normal

         20:3           E0AMBT         normal

         20:4           E0AY4Q         normal

         20:5           E0AVS0         normal

         20:6           E0ATBB         normal

         20:7           E0AXVM         normal

         20:8           E0AYC5         normal

         20:9           E08MH9         normal

         20:10          E0AXRQ         normal

         20:11          E08MN9         normal

[root@oracloudceladm01 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -pdlist -a0 |egrep 'Slot Number|Firmware state'

Slot Number: 0

Firmware state: Unconfigured(good), Spun down

Slot Number: 1

Firmware state: Online, Spun Up

Slot Number: 2

Firmware state: Online, Spun Up

Slot Number: 3

Firmware state: Online, Spun Up

Slot Number: 4

Firmware state: Online, Spun Up

Slot Number: 5

Firmware state: Online, Spun Up

Slot Number: 6

Firmware state: Online, Spun Up

Slot Number: 7

Firmware state: Online, Spun Up

Slot Number: 8

Firmware state: Unconfigured(good), Spun down

Slot Number: 9

Firmware state: Online, Spun Up

Slot Number: 10

Firmware state: Online, Spun Up

Slot Number: 11

Firmware state: Online, Spun Up

Oracle has provided the steps to fix the System Disk Replacement issue.

In this article I will demonstrate the steps on how to fix System disk replacement issue on a live Exadata X2-2 Full Rack Machine.

Environment Details

Steps

1. Download FixDiskRep.zip file attached to MOS note 2003674.1
2. Copy the software to the Storage cell where the System Disk is replaced under /tmp directory.
3. Unzip the FixDiskRep.zip file

[root@oracloudceladm01 ~]# unzip FixDiskRep.zip -d /tmp/FixDiskRep

Archive: FixDiskRep.zip

inflating: /tmp/FixDiskRep/checkdeveachboot

inflating: /tmp/FixDiskRep/common_diskadp_image.sh

inflating: /tmp/FixDiskRep/dozap.sh

inflating: /tmp/FixDiskRep/FixDiskRep.sh

inflating: /tmp/FixDiskRep/image_functions

inflating: /tmp/FixDiskRep/scripts_common_lib.sh

inflating: /tmp/FixDiskRep/README.txt

[root@oracloudceladm01 ~]# cd /tmp/FixDiskRep

[root@oracloudceladm01 FixDiskRep]# ls -l

total 964

-r-xr-xr-x 1 root root 60331 May 29 00:23 checkdeveachboot

-r-xr-xr-x 1 root root 54004 May 13 03:20 common_diskadp_image.sh

-r-xr-xr-x 1 root root   3214 May 13 03:20 dozap.sh

-rwxr-xr-x 1 root root   2994 Jul 9 16:45 FixDiskRep.sh

-rwxrwxr-x 1 root root 827418 May 29 00:00 image_functions

-rw-rw-r-- 1 root root   2701 Jul 8 11:42 README.txt

-r-xr-xr-x 1 root root 21110 May 13 03:20 scripts_common_lib.sh

    4. Read the README.txt carefully.

    5. Validate md5sum

 [root@oracloudceladm01 FixDiskRep]# md5sum *

266264278e981529e33bc046822690bb checkdeveachboot

1994c35e3056946fe13c9be45dff5228 common_diskadp_image.sh

7d50db3cbe42b6b9e09f857c726c9375 dozap.sh

faa56e939155aa35b00b361f48368ae5 FixDiskRep.sh

09917cad53e94b1423a7b013a3f49827 image_functions

1ee7644161a8a5fbaebeb5537a55bfeb README.txt

9dfc4205f880c413e7d955be127e69b5 scripts_common_lib.sh

                       6. Identify the failed System disk

[root@oracloudceladm01 FixDiskRep]# cellcli -e list physicaldisk where disktype=harddisk

         20:0           E4ADM1         normal - rejected due to system disk imaging failed

         20:1           E0AT2S         normal

         20:2           E0AGXE         normal

         20:3           E0AMBT         normal

         20:4           E0AY4Q         normal

         20:5           E0AVS0         normal

         20:6           E0ATBB         normal

         20:7           E0AXVM         normal

         20:8           E0AYC5         normal

         20:9           E08MH9         normal

         20:10          E0AXRQ         normal

         20:11          E08MN9         normal

We can see that disk 0 has failed system disk replacement process.

   7. Drop the System disk for replacement

[root@oracloudceladm01 FixDiskRep]# cellcli -e alter physicaldisk 20:0 drop for replacement

Physical disk 20:0 was dropped for replacement.

                       8. Shutdown the the MS process

[root@oracloudceladm01 FixDiskRep]# cellcli -e alter cell shutdown services MS

Stopping MS services...

The SHUTDOWN of MS services was successful.

                        9. Run the FixDiskRep.sh script

[root@oracloudceladm01 FixDiskRep]# ./FixDiskRep.sh

09917cad53e94b1423a7b013a3f49827 /opt/oracle.cellos/image_functions

266264278e981529e33bc046822690bb /opt/oracle.cellos/validations/init.d/checkdeveachboot

9dfc4205f880c413e7d955be127e69b5 /opt/oracle/cell12.1.2.1.1_LINUX.X64_150316.2/cellsrv/deploy/scripts/unix/hwadapter/diskadp/scripts_common_lib.sh

1994c35e3056946fe13c9be45dff5228 /opt/oracle/cell12.1.2.1.1_LINUX.X64_150316.2/cellsrv/deploy/scripts/unix/hwadapter/diskadp/common_diskadp_image.sh

1994c35e3056946fe13c9be45dff5228 /opt/oracle.cellos/common_diskadp_image.sh

7d50db3cbe42b6b9e09f857c726c9375 /opt/oracle/cell12.1.2.1.1_LINUX.X64_150316.2/cellsrv/deploy/scripts/unix/hwadapter/diskadp/dozap.sh

Expected values

09917cad53e94b1423a7b013a3f49827 image_functions

266264278e981529e33bc046822690bb checkdeveachboot

9dfc4205f880c413e7d955be127e69b5 scripts_common_lib.sh

1994c35e3056946fe13c9be45dff5228 common_diskadp_image.sh

7d50db3cbe42b6b9e09f857c726c9375 dozap.sh

FixDiskRep.sh reports the md5sum of the copied files and should match the list from step 5.

                        10. Start the MS process

[root@oracloudceladm01 FixDiskRep]# cellcli -e alter cell startup services MS

Starting MS services...

The STARTUP of MS services was successful.

                       11. Enable the System Disk

[root@oracloudceladm01 FixDiskRep]# cellcli -e alter physicaldisk 20:0 reenable force

Physical disk 20:0 was reenabled.

If you see this output then the System Disk is enabled. If you see something other than the output above then there are others issue and requires Oracle assistance.

   12. Validate Disk status

Make sure System Disk status is normal

[root@oracloudceladm01 FixDiskRep]# cellcli -e list physicaldisk where disktype=harddisk

         20:0   E4ADM1 normal

         20:1   E0AT2S normal

         20:2   E0AGXE normal

         20:3   E0AMBT normal

         20:4   E0AY4Q normal

         20:5   E0AVS0 normal

         20:6   E0ATBB normal

         20:7   E0AXVM normal

         20:8   E0AYC5 normal

         20:9   E08MH9 normal

         20:10  E0AXRQ normal

         20:11  E08MN9 normal

Make sure Celldisk status is "normal"

[root@oracloudceladm01 FixDiskRep]# cellcli -e list celldisk attributes name,devicename,devicepartition,status where disktype=harddisk

         CD_00_oracloudceladm01         /dev/sdi       /dev/sdi3       normal

         CD_01_oracloudceladm01         /dev/sdb       /dev/sdb3       normal

         CD_02_oracloudceladm01         /dev/sdc       /dev/sdc        normal

         CD_03_oracloudceladm01         /dev/sdd       /dev/sdd        normal

         CD_04_oracloudceladm01         /dev/sde       /dev/sde        normal

         CD_05_oracloudceladm01         /dev/sdf       /dev/sdf        normal

         CD_06_oracloudceladm01         /dev/sdg       /dev/sdg        normal

         CD_07_oracloudceladm01         /dev/sdh       /dev/sdh        normal

         CD_08_oracloudceladm01         /dev/sdi       /dev/sdi        normal

         CD_09_oracloudceladm01         /dev/sdj       /dev/sdj        normal

         CD_10_oracloudceladm01         /dev/sdk       /dev/sdk        normal

         CD_11_oracloudceladm01         /dev/sdl       /dev/sdl        normal

Make sure griddisks are created and active.

[root@oracloudceladm01 FixDiskRep]# cellcli -e list griddisk attributes name,status where disktype=harddisk

         DATA_DG_CD_00_oracloudceladm01       active

         DATA_DG_CD_01_oracloudceladm01       active

         DATA_DG_CD_02_oracloudceladm01       active

         DATA_DG_CD_03_oracloudceladm01       active

         DATA_DG_CD_04_oracloudceladm01       active

         DATA_DG_CD_05_oracloudceladm01       active

         DATA_DG_CD_06_oracloudceladm01       active

         DATA_DG_CD_07_oracloudceladm01       active

         DATA_DG_CD_08_oracloudceladm01       active

         DATA_DG_CD_09_oracloudceladm01       active

         DATA_DG_CD_10_oracloudceladm01       active

         DATA_DG_CD_11_oracloudceladm01       active

         DBFS_DG_CD_02_oracloudceladm01       active

         DBFS_DG_CD_03_oracloudceladm01       active

         DBFS_DG_CD_04_oracloudceladm01       active

         DBFS_DG_CD_05_oracloudceladm01       active

         DBFS_DG_CD_06_oracloudceladm01       active

         DBFS_DG_CD_07_oracloudceladm01       active

         DBFS_DG_CD_08_oracloudceladm01       active

         DBFS_DG_CD_09_oracloudceladm01       active

         DBFS_DG_CD_10_oracloudceladm01       active

         DBFS_DG_CD_11_oracloudceladm01       active

         RECO_DG_CD_00_oracloudceladm01       active

         RECO_DG_CD_01_oracloudceladm01       active

         RECO_DG_CD_02_oracloudceladm01       active

         RECO_DG_CD_03_oracloudceladm01       active

         RECO_DG_CD_04_oracloudceladm01       active

         RECO_DG_CD_05_oracloudceladm01       active

         RECO_DG_CD_06_oracloudceladm01       active

         RECO_DG_CD_07_oracloudceladm01       active

         RECO_DG_CD_08_oracloudceladm01       active

         RECO_DG_CD_09_oracloudceladm01       active

         RECO_DG_CD_10_oracloudceladm01       active

         RECO_DG_CD_11_oracloudceladm01       active

Make sure md devices State is Clean.

[root@oracloudceladm01 FixDiskRep]# for mddev in `cat /proc/mdstat|grep md|awk '{print $1}'`; do mdadm -Q --detail /dev/${mddev}|egrep 'md|State :'; done

/dev/md4:

         State : clean

/dev/md8:

         State : clean, degraded, resyncing (DELAYED)

/dev/md11:

         State : clean, degraded, resyncing (DELAYED)

/dev/md5:

         State : clean, degraded, recovering

/dev/md2:

         State : clean, degraded, resyncing (DELAYED)

/dev/md7:

         State : clean, degraded, resyncing (DELAYED)

/dev/md6:

         State : clean, degraded, resyncing (DELAYED)

/dev/md1:

         State : clean

State is resyncing...

Check the resync status

[root@oracloudceladm01 FixDiskRep]# cat /proc/mdstat

Personalities : [raid1]

md4 : active raid1 sdi1[0] sdb1[1]

     120384 blocks [2/2] [UU]

md8 : active raid1 sdi8[2] sda8[3](F) sdb8[1]

     2096384 blocks [2/1] [_U]

       resync=DELAYED

md11 : active raid1 sdi11[2] sdb11[1]

     2433728 blocks [2/1] [_U]

       resync=DELAYED

md5 : active raid1 sdi5[0] sdb5[1]

     10482304 blocks [2/2] [UU]

md2 : active raid1 sdi9[2] sdb9[1]

   2096384 blocks [2/1] [_U]

       resync=DELAYED

md7 : active raid1 sdi7[2] sdb7[1]

     2096384 blocks [2/1] [_U]

     [===================>.] recovery = 97.7% (2050048/2096384) finish=0.0min speed=51612K/sec

md6 : active raid1 sdi6[0] sda6[2](F) sdb6[1]

     10482304 blocks [2/2] [UU]

md1 : active raid1 sdi10[0] sdb10[1] sda10[2](F)

     714752 blocks [2/2] [UU]

unused devices: <none>

Wait for some time and rerun the command again.

Check the md status again.

[root@oracloudceladm01 FixDiskRep]# for mddev in `cat /proc/mdstat|grep md|awk '{print $1}'`; do mdadm -Q --detail /dev/${mddev}|egrep 'md|State :'; done

/dev/md4:

         State : clean

/dev/md8:

         State : clean

/dev/md11:

         State : clean

/dev/md5:

         State : clean

/dev/md2:

         State : clean

/dev/md7:

         State : clean

/dev/md6:

         State : clean

/dev/md1:

         State : clean

Now the md Status is Clean.

System Disk Replacement issue has been resolved successfully.

Important Note

It could be possible that "cellcli -e alter physicaldisk 20:0 reenable force" can fail if the disk is not inserted/replaced properly, the replaced System Disk itself is bad or the lun is not created automatically after System Disk Replacement. Work with Oracle Engineer to fix the problem before running the steps provided.

MOS Notes

• Exadata Storage Software 12.1.2.1.0 and 12.1.2.1.1 System Disk Replacement Issues (Doc ID 2003674.1)
• Exadata Critical Issues (Doc ID 1270094.1)

Conclusion

In this article we have learned about System area and how to fix the System Disk Replacement issue on a live Exadata Database Machine X2-2. This issue is not just on Exadata X2-2; we may encounter it on any Exadata model running Exadata Storage Software version 12.1.2.1.0 and 12.1.2.1.1.

Introduction

Since Exadata Storage Software version 11.2.2.4.2 dbnodeupdate.sh utility is used for upgrading Exadata Compute Nodes. Exadata Storage Software version 12.1.2.2.0 includes a new enhancement which allows us to upgrade the Exadata Compute Nodes using "patchmgr" utility in addition to dbnodeupdate.sh utility. “patchmgr” utility can be used for upgrading, rollback and backup Oracle Exadata database servers running ESS releases later than 11.2.2.4.2, Oracle Exadata Virtual Server nodes and Oracle Exadata Virtual Machines in addition to running dbnodeupdate.sh standalone.

With this new enhancement, Oracle wants to use a common utility for upgrading Storage cells, Compute nodes and Infiniband Switches. This is a good move as this will avoid any kind of confusion in patching process.

About dbnodeupdate.sh utility

The dbnodeupdate.sh utility was introduced in ESS release 11.2.2.4.2 for updating Exadata Compute Nodes. It is available through patch 16486998. The dbnodeupdate.sh utility automates the process of upgrading the operating system and firmware on a Compute node. To upgrade Compute Nodes you should run dbnodeupdate.sh utility separately on each Compute Nodes. dbnodeupdate.sh utility supports all Exadata hardware version V1, V2, X2, X3, X4 & X5.

About patchmgr utility

The patchmgr utility can be used for upgrading, rollback and backup Exadata Compute Nodes in addition to dbnodeupdate.sh utility. patchmgr and dbnodeupdate.sh are available in the new dbserver.patch.zip file through patch 1553103.1. patchmgr can run single command to update multiple nodes without having to run dbnodeupdate.sh separately on each node. You can create file containing the list of Compute Nodes to be upgraded similar to upgrading Exadata Storage Cells. patchmgr utility can be used for upgrading Compute Nodes in a rolling or non-rolling fashion. Non-Rolling is default.

Launch patchmgr from the compute node that is not upgrading itself. That node needs to be upgraded later by launching patchmgr from a node that has already been upgraded or simply use dbnodeupdate.sh to upgrade that node.

In this article I will demonstrate how to perform pre check, backup and upgrade Exadata Compute Nodes using patchmgr utility.

Advantages of using patchmgr utility

1. A single patchmgr command can upgrade multiple Compute Nodes.
2. You can upgrade the Compute Nodes in rolling and non-rolling fashion using patchmgr.
3. No need to run dbnodeupdate.sh separately on each node.

MOS Notes

• 1553103.1: dbnodeupdate.sh: Exadata Database Server Patching using the DB Node Update Utility
• 888828.1: Exadata Database Machine and Exadata Storage Server Supported Versions
• 1070954.1: Oracle Exadata Database Machine exachk or HealthCheck

Note: Always run Exachk before and after patching.

Software Download

Download the following patches required for Upgrading Compute Nodes. Copy them to Exadata Compute Nodes one.

• Patch 21825906 : Exadata Compute Node 12.1.2.2.0 OL6 Base Repository ISO
• Patch 21634633 : patchmgr and dbnodeupdate.sh utilities

Environment Details

Current Image version

Execute the “imageinfo” command on one of the Compute nodes to identify the current Exadata Image version

[root@oraclouddbadm01 ~]# imageinfo

Kernel version: 2.6.39-400.248.3.el6uek.x86_64 #1 SMP Wed Mar 11 18:04:34 PDT 2015 x86_64

Image version: 12.1.2.1.1.150316.2

Image activated: 2015-04-27 20:52:07 -0500

Image status: success

System partition on device: /dev/mapper/VGExaDb-LVDbSys1

Prerequisites or Assumption

• root user access for compute node.
• root user equivalence must be setup between compute nodes.

(root)#dcli -g dbs_group -l root –k

• Shutdown database(s) and Clusterware stack.

(oracle)$ srvctl stop database -d dbm01

(root)# $GRID_HOME/crsctl stop cluster –all

(root)# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl stop crs’

• dbs_group file containing compute node name or IP address per line
• patchmgr and dbnodeupdate.sh utility downloaded and stage into a directory

(root)# pwd

/u01/app/oracle/patches/ESS_121220

• unmount any active nfs mount points.

(root)# dcli -g dbs_group -l root ‘umount /nfs_mount’

Replace nfs_mount with the actual mount point name

• ISO image method will be used for Compute Node Upgrade.

Preliminary Steps

• unzip the Patch 21634633 containing patchmgr and dbnodeupdate.sh utility

[root@oraclouddbadm01 ~]# cd /u01/app/oracle/patches/ESS_121220

[root@oraclouddbadm01 ESS_121220]# ls -ltr

total 2537072

-rw-r--r-- 1 root root   78513331 Sep 27 04:48 p21634633_121220_Linux-x86-64.zip

-rw-r--r-- 1 root root 966861781 Sep 27 06:45 p21825906_121220_Linux-x86-64.zip

[root@oraclouddbadm01 ESS_121220]# unzip p21634633_121220_Linux-x86-64.zip

Archive: p21634633_121220_Linux-x86-64.zip

   creating: dbserver_patch_5.150922/

inflating: dbserver_patch_5.150922/ExaXMLNode.pm

   creating: dbserver_patch_5.150922/plugins/

inflating: dbserver_patch_5.150922/plugins/010-check_17854520.sh

inflating: dbserver_patch_5.150922/plugins/000-check_dummy_perl

inflating: dbserver_patch_5.150922/plugins/000-check_dummy_bash

inflating: dbserver_patch_5.150922/exadata.img.env

inflating: dbserver_patch_5.150922/imageLogger

inflating: dbserver_patch_5.150922/ExadataImageNotification.pl

inflating: dbserver_patch_5.150922/ipconf.pl

inflating: dbserver_patch_5.150922/exadata.img.hw

inflating: dbserver_patch_5.150922/11_1_3_2_0.pm

inflating: dbserver_patch_5.150922/patchmgr

inflating: dbserver_patch_5.150922/md5sum_files.lst

inflating: dbserver_patch_5.150922/sgutil.pm

inflating: dbserver_patch_5.150922/11_2_1_2_2.pm

inflating: dbserver_patch_5.150922/11_2_1_2_1.pm

   creating: dbserver_patch_5.150922/ibdiagtools/

inflating: dbserver_patch_5.150922/ibdiagtools/setup-ssh

inflating: dbserver_patch_5.150922/ibdiagtools/topology-zfs

extracting: dbserver_patch_5.150922/ibdiagtools/VERSION_FILE

inflating: dbserver_patch_5.150922/ibdiagtools/checkbadlinks.pl

inflating: dbserver_patch_5.150922/ibdiagtools/ibping_test

inflating: dbserver_patch_5.150922/ibdiagtools/cable_check.pl

inflating: dbserver_patch_5.150922/ibdiagtools/tar_ibdiagtools

inflating: dbserver_patch_5.150922/ibdiagtools/SampleOutputs.txt

inflating: dbserver_patch_5.150922/ibdiagtools/dcli

inflating: dbserver_patch_5.150922/ibdiagtools/infinicheck

   creating: dbserver_patch_5.150922/ibdiagtools/topologies/

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/VerifyTopologyUtility.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Group.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Rack.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Switch.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/verifylib.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Node.pm

extracting: dbserver_patch_5.150922/ibdiagtools/xmonib.sh

inflating: dbserver_patch_5.150922/ibdiagtools/monitord

   creating: dbserver_patch_5.150922/ibdiagtools/netcheck/

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/SolarisAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/OSAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/LinuxAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteScriptGenerator.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/runDiagnostics.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/CommonUtils.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/spawnProc.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteLauncher.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteConfig.pm

inflating: dbserver_patch_5.150922/ibdiagtools/verify-topology

inflating: dbserver_patch_5.150922/exadataLogger.pm

inflating: dbserver_patch_5.150922/dcli

inflating: dbserver_patch_5.150922/README.txt

inflating: dbserver_patch_5.150922/11_2_1_1_0.pm

   creating: dbserver_patch_5.150922/linux.db.rpms/

inflating: dbserver_patch_5.150922/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm

extracting: dbserver_patch_5.150922/dbnodeupdate.zip

inflating: dbserver_patch_5.150922/preconfig.pm

inflating: dbserver_patch_5.150922/11_2_2_1_0.pm

inflating: dbserver_patch_5.150922/ExadataSendNotification.pm

[root@oraclouddbadm01 ESS_121220]# ls -l

total 2537076

drwxrwxr-x 5 root root       4096 Sep 22 07:49 dbserver_patch_5.150922

-rw-r--r-- 1 root root 1550028016 Sep 27 07:39 p20131726_121220_Linux-x86-64.zip

-rw-r--r-- 1 root root   78513331 Sep 27 04:48 p21634633_121220_Linux-x86-64.zip

-rw-r--r-- 1 root root 966861781 Sep 27 06:45 p21825906_121220_Linux-x86-64.zip

• Copy the Base Repository ISO image to the dbserver_patch_5.150992 directory.

[root@oraclouddbadm01 ~]# cd /u01/app/oracle/patches/ESS_121220

[root@oraclouddbadm01 ~]# cp p21825906_121220_Linux-x86-64.zip dbserver_patch_5.150922/

• Copy dbs_group file to dbserver_patch_5.150992 directory and remove the Compute Node 1 entry from it. This will be useful while executing patchmgr commands.

[root@oraclouddbadm01 ~]# cd /u01/app/oracle/patches/ESS_121220/dbserver_patch_5.150922

[root@oraclouddbadm01 dbserver_patch_5.150922]# cp ~/dbs_group .

[root@oraclouddbadm01 dbserver_patch_5.150922]# vi dbs_group

[root@oraclouddbadm01 dbserver_patch_5.150922]# cat dbs_group

oraclouddbadm02

oraclouddbadm03

oraclouddbadm04

oraclouddbadm05

oraclouddbadm06

oraclouddbadm07

oraclouddbadm08

I will be launching patchmgr from node 1 to upgrade Compute Nodes, so I can’t Upgrade node 1 at the same time. Once nodes 2 – 8 are upgraded, I will use one of the updated nodes to execute patchmgr and upgrade node 1. See know issues section for more details.

As part of Compute node Upgrade, the server gets rebooted, so we can’t upgrade the Compute node from where we launch the patchmgr utility.

• Make sure root file system has sufficient free space available. If there is no sufficient space then perform cleanup otherwise patchmgr command (precheck) will fail.

[root@oraclouddbadm01 ~]# dcli -g dbs_group -l root 'df -h /'

• Verify root SSH user equivalence is setup.

[root@oraclouddbadm01 ~]# dcli -g dbs_group -l root 'uptime'

Getting help from patchmgr

Issue the patchmgr command with -h flag to list all the options that you can use with patchmgr. All the options are displayed with a brief explanation.

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -h

Usage:

./patchmgr -dbnode database_node_list_file

           -dbnode_upgrade   -dbnode_loc patch_file_name [-dbnode_version version] [-unkey] |

           -dbnode_precheck   -dbnode_loc patch_file_name [-dbnode_version version] [-unkey] |

           -dbnode_rollback                                                        [-unkey] |

           -dbnode_backup     -dbnode_loc patch_file_name [-dbnode_version version] [-unkey]

./patchmgr -h

OPTIONS

   -h

         Displays this screen

DB Node upgrade

   The following options are supported for db node update:

   -dbnode

         Specifies the name of the database node list file. The file has one

         hostname or ip per line.

   -dbnode_upgrade

         Upgrades the database nodes specified in the host list.

   -dbnode_precheck

         Runs the pre-upgrade validation checks in the database nodes

         specified in the host list.

   -dbnode_backup

         Perform backup only the database nodes specified in the host list.

   -dbnode_rollback

         Rollback the database nodes specified in the host list.

   -dbnode_loc

         Baseurl (http or zipped iso file for the repository)

   -dbnode_version

         The patch version. ( The default is 12.1.2.3.0.150922. )

   -smtp_from "addr"

       The address patchmgr notification sent from.

         sendmail daemon must be started to send email notification.

   -smtp_to "addr1 addr2 addr3 ..."

         The address(es) patchmgr notification sent to.

         sendmail daemon must be started to send email notification.

   -smtp_set_envelope_sender

         Use the same from address in Return-Path: mail header.

   -unkey

         Removes passwordless SSH access to the database nodes before exit.

Note: At the time of writing the patch “21634633” had a BUG. The –rolling option is not displayed when you execute the “patchmgr –h” command. But the good side is that, you can still use the –rolling option to perform the rolling upgrades. I guess Oracle will fix the issue and release an updated version of the patch soon.

Rolling and Non-Rolling

Non-Rolling: The pre-check, backup, upgrade and reboot are done in parallel. The default is Non-Rolling.

Rolling: Done one Compute Node at a time. Rollback is always done in rolling fashion even if -rolling option is not specified.

Running pre-check using patchmgr

Before upgrading Exadata Compute Nodes, it is recommended to perform a pre-check. If the pre-check returns successful you can proceed with the Compute Node upgrade. Use the following patchmgr command to perform a pre-check. If there are issues patchmgr pre-check command will fail and you must fix the issue before running the pre-check command again. Patchmgr pre-check command performs the following checks:

• Free space
• SSH equivalence for root user
• ESS version check
• Copy dbnodeupdate.zip file to all Compute Nodes
• Extract the dbnodeupdate.zip file on all Compute Nodes
• Copy patch file to all Compute Nodes
• Copy dbnodeupdate.log file to /var/log/cellos on all Compute Nodes

Non-rolling

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917

2015-10-04 01:58:48 -0500 :Working: DO: Initiate validation check on 7 database node(s).

2015-10-04 01:58:48 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm02.

2015-10-04 01:58:49 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 01:58:50 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 01:58:50 -0500 :Working: DO: pre version check on oraclouddbadm02.

2015-10-04 01:58:51 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm02.

2015-10-04 01:58:51 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 01:59:02 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 01:59:02 -0500 :Working: DO: Copy patch file to oraclouddbadm02.

2015-10-04 01:59:13 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm02.

2015-10-04 01:59:13 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm02.

2015-10-04 02:01:34 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.log directory on the local machine.

2015-10-04 02:01:35 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.041015015914.diag directory on the local machine.

2015-10-04 02:01:35 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm02.

2015-10-04 02:01:36 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm03.

2015-10-04 02:01:36 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:01:38 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:01:38 -0500 :Working: DO: pre version check on oraclouddbadm03.

2015-10-04 02:01:39 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm03.

2015-10-04 02:01:39 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm03 and then extract the file.

2015-10-04 02:01:50 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm03 and then extract the file.

2015-10-04 02:01:50 -0500 :Working: DO: Copy patch file to oraclouddbadm03.

2015-10-04 02:02:01 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm03.

2015-10-04 02:02:01 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm03.

2015-10-04 02:04:22 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.log directory on the local machine.

2015-10-04 02:04:23 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.041015020202.diag directory on the local machine.

2015-10-04 02:04:23 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm03.

2015-10-04 02:04:24 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm04.

2015-10-04 02:04:24 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:04:26 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:04:26 -0500 :Working: DO: pre version check on oraclouddbadm04.

2015-10-04 02:04:27 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm04.

2015-10-04 02:04:27 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm04 and then extract the file.

2015-10-04 02:04:38 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm04 and then extract the file.

2015-10-04 02:04:38 -0500 :Working: DO: Copy patch file to oraclouddbadm04.

2015-10-04 02:04:49 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm04.

2015-10-04 02:04:49 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm04.

2015-10-04 02:07:10 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.log directory on the local machine.

2015-10-04 02:07:11 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.041015020450.diag directory on the local machine.

2015-10-04 02:07:11 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm04.

2015-10-04 02:07:12 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm05.

2015-10-04 02:07:12 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:07:14 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:07:14 -0500 :Working: DO: pre version check on oraclouddbadm05.

2015-10-04 02:07:15 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm05.

2015-10-04 02:07:15 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm05 and then extract the file.

2015-10-04 02:07:26 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm05 and then extract the file.

2015-10-04 02:07:26 -0500 :Working: DO: Copy patch file to oraclouddbadm05.

2015-10-04 02:07:37 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm05.

2015-10-04 02:07:37 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm05.

2015-10-04 02:10:08 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.log directory on the local machine.

2015-10-04 02:10:09 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.041015020737.diag directory on the local machine.

2015-10-04 02:10:09 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm05.

2015-10-04 02:10:10 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm06.

2015-10-04 02:10:10 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:10:12 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:10:12 -0500 :Working: DO: pre version check on oraclouddbadm06.

2015-10-04 02:10:13 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm06.

2015-10-04 02:10:13 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm06 and then extract the file.

2015-10-04 02:10:24 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm06 and then extract the file.

2015-10-04 02:10:24 -0500 :Working: DO: Copy patch file to oraclouddbadm06.

2015-10-04 02:10:35 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm06.

2015-10-04 02:10:35 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm06.

2015-10-04 02:13:06 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.log directory on the local machine.

2015-10-04 02:13:07 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.041015021036.diag directory on the local machine.

2015-10-04 02:13:08 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm06.

2015-10-04 02:13:09 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm07.

2015-10-04 02:13:09 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:13:10 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:13:10 -0500 :Working: DO: pre version check on oraclouddbadm07.

2015-10-04 02:13:11 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm07.

2015-10-04 02:13:11 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm07 and then extract the file.

2015-10-04 02:13:22 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm07 and then extract the file.

2015-10-04 02:13:22 -0500 :Working: DO: Copy patch file to oraclouddbadm07.

2015-10-04 02:13:33 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm07.

2015-10-04 02:13:33 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm07.

2015-10-04 02:15:54 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.log directory on the local machine.

2015-10-04 02:15:55 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.041015021334.diag directory on the local machine.

2015-10-04 02:15:56 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm07.

2015-10-04 02:15:57 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm08.

2015-10-04 02:15:57 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 02:15:58 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 02:15:58 -0500 :Working: DO: pre version check on oraclouddbadm08.

2015-10-04 02:15:59 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm08.

2015-10-04 02:15:59 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm08 and then extract the file.

2015-10-04 02:16:10 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm08 and then extract the file.

2015-10-04 02:16:10 -0500 :Working: DO: Copy patch file to oraclouddbadm08.

2015-10-04 02:16:21 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm08.

2015-10-04 02:16:21 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm08.

2015-10-04 02:18:53 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.log directory on the local machine.

2015-10-04 02:18:53 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.041015021622.diag directory on the local machine.

2015-10-04 02:18:54 -0500 :SUCCESS: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm08.

SUMMARY FROM DATABASE NODE PRECHECK CHECKS:

Database node oraclouddbadm02 has precheck state: SUCCESS.

Database node oraclouddbadm03 has precheck state: SUCCESS.

Database node oraclouddbadm04 has precheck state: SUCCESS.

Database node oraclouddbadm05 has precheck state: SUCCESS.

Database node oraclouddbadm06 has precheck state: SUCCESS.

Database node oraclouddbadm07 has precheck state: SUCCESS.

Database node oraclouddbadm08 has precheck state: SUCCESS.

2015-10-04 02:18:55 -0500 :SUCCESS: DONE: Initiate validation check on database node(s).

Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 –rolling

Running backup using patchmgr

It is highly recommended to perform a file system backup before upgrading the Exadata Compute Nodes. Patchmgr utility can be used to perform a file system backup. Use the following command to perform a file system backup using patchmgr.

Non-Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_backup -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917

2015-10-04 03:40:52 -0500 :Working: DO: Initiate backup on 7 database node(s).

2015-10-04 03:40:52 -0500 :Working: DO: Initiate backup on database node oraclouddbadm02.

2015-10-04 03:40:52 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 03:40:53 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 03:40:53 -0500 :Working: DO: pre version check on oraclouddbadm02.

2015-10-04 03:40:54 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm02.

2015-10-04 03:40:54 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 03:41:06 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 03:41:06 -0500 :Working: DO: Copy patch file to oraclouddbadm02.

2015-10-04 03:41:16 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm02.

2015-10-04 03:41:16 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm02.

2015-10-04 03:52:23 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.log directory on the local machine.

2015-10-04 03:52:29 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.041015034117.diag directory on the local machine.

2015-10-04 03:52:30 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm02.

2015-10-04 03:52:31 -0500 :Working: DO: Initiate backup on database node oraclouddbadm03.

2015-10-04 03:52:31 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 03:52:32 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 03:52:32 -0500 :Working: DO: pre version check on oraclouddbadm03.

2015-10-04 03:52:33 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm03.

2015-10-04 03:52:33 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm03 and then extract the file.

2015-10-04 03:52:45 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm03 and then extract the file.

2015-10-04 03:52:45 -0500 :Working: DO: Copy patch file to oraclouddbadm03.

2015-10-04 03:52:55 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm03.

2015-10-04 03:52:55 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm03.

2015-10-04 04:04:13 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.log directory on the local machine.

2015-10-04 04:04:18 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.041015035256.diag directory on the local machine.

2015-10-04 04:04:19 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm03.

2015-10-04 04:04:20 -0500 :Working: DO: Initiate backup on database node oraclouddbadm04.

2015-10-04 04:04:20 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 04:04:21 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 04:04:21 -0500 :Working: DO: pre version check on oraclouddbadm04.

2015-10-04 04:04:22 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm04.

2015-10-04 04:04:22 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm04 and then extract the file.

2015-10-04 04:04:34 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm04 and then extract the file.

2015-10-04 04:04:34 -0500 :Working: DO: Copy patch file to oraclouddbadm04.

2015-10-04 04:04:44 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm04.

2015-10-04 04:04:44 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm04.

2015-10-04 04:16:02 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.log directory on the local machine.

2015-10-04 04:16:10 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.041015040445.diag directory on the local machine.

2015-10-04 04:16:11 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm04.

2015-10-04 04:16:12 -0500 :Working: DO: Initiate backup on database node oraclouddbadm05.

2015-10-04 04:16:12 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 04:16:13 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 04:16:13 -0500 :Working: DO: pre version check on oraclouddbadm05.

2015-10-04 04:16:14 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm05.

2015-10-04 04:16:14 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm05 and then extract the file.

2015-10-04 04:16:26 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm05 and then extract the file.

2015-10-04 04:16:26 -0500 :Working: DO: Copy patch file to oraclouddbadm05.

2015-10-04 04:16:36 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm05.

2015-10-04 04:16:36 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm05.

2015-10-04 04:27:44 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.log directory on the local machine.

2015-10-04 04:27:49 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.041015041637.diag directory on the local machine.

2015-10-04 04:27:50 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm05.

2015-10-04 04:27:51 -0500 :Working: DO: Initiate backup on database node oraclouddbadm06.

2015-10-04 04:27:51 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 04:27:52 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 04:27:52 -0500 :Working: DO: pre version check on oraclouddbadm06.

2015-10-04 04:27:53 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm06.

2015-10-04 04:27:53 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm06 and then extract the file.

2015-10-04 04:28:05 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm06 and then extract the file.

2015-10-04 04:28:05 -0500 :Working: DO: Copy patch file to oraclouddbadm06.

2015-10-04 04:28:15 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm06.

2015-10-04 04:28:15 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm06.

2015-10-04 04:39:23 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.log directory on the local machine.

2015-10-04 04:39:28 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.041015042816.diag directory on the local machine.

2015-10-04 04:39:28 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm06.

2015-10-04 04:39:30 -0500 :Working: DO: Initiate backup on database node oraclouddbadm07.

2015-10-04 04:39:30 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 04:39:31 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 04:39:31 -0500 :Working: DO: pre version check on oraclouddbadm07.

2015-10-04 04:39:32 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm07.

2015-10-04 04:39:32 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm07 and then extract the file.

2015-10-04 04:39:43 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm07 and then extract the file.

2015-10-04 04:39:43 -0500 :Working: DO: Copy patch file to oraclouddbadm07.

2015-10-04 04:39:54 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm07.

2015-10-04 04:39:54 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm07.

2015-10-04 04:52:02 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.log directory on the local machine.

2015-10-04 04:52:09 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.041015043955.diag directory on the local machine.

2015-10-04 04:52:10 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm07.

2015-10-04 04:52:11 -0500 :Working: DO: Initiate backup on database node oraclouddbadm08.

2015-10-04 04:52:11 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 04:52:13 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 04:52:13 -0500 :Working: DO: pre version check on oraclouddbadm08.

2015-10-04 04:52:14 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm08.

2015-10-04 04:52:14 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm08 and then extract the file.

2015-10-04 04:52:25 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm08 and then extract the file.

2015-10-04 04:52:25 -0500 :Working: DO: Copy patch file to oraclouddbadm08.

2015-10-04 04:52:36 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm08.

2015-10-04 04:52:36 -0500 :Working: DO: dbnodeupdate.sh backup is running on database node oraclouddbadm08.

2015-10-04 05:03:44 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.log directory on the local machine.

2015-10-04 05:03:49 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.041015045237.diag directory on the local machine.

2015-10-04 05:03:50 -0500 :SUCCESS: DONE: Initiate backup on database node oraclouddbadm08.

2015-10-04 05:03:51 -0500 :SUCCESS: DONE: Initiate backup on database node(s).

Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_backup -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -rolling

Sending notification using patchmgr

If you don’t want to sit in front of your desktop/laptop while you are running precheck/backup/upgrade but want to be notified of the progress then -smtp_from and -smtp_to options can send e-mail notifications on progress.

Non-Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_backup -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_upgrade -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com

Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com -rolling

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_backup -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com -rolling

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_upgrade -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from “oraclouddbadm01” -smtp_to xyz@domain.com -rolling

Upgrading Compute Nodes using patchmgr

Finally we are ready to upgrade the Compute nodes. To upgrade the Compute nodes, execute the following patchmgr command. First we will upgrade the Compute node 2 – 8 and then use one of updated nodes to upgrade Compute node 1.

Non-Rolling:

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode dbs_group -dbnode_upgrade -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from "oraclouddbadm01" -smtp_to "abc@mydomain.com"

2015-10-27 05:13:01 -0500 :Working: DO: Initiate update on 7 database node(s).

2015-10-27 05:13:01 -0500 :Working: DO: Initiate upgrade on Database Node(s).

2015-10-27 05:13:02 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-27 05:13:12 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-27 05:13:12 -0500 :Working: DO: pre version check on all nodes.

2015-10-27 05:13:13 -0500 :SUCCESS: DONE: pre version check on all nodes.

2015-10-27 05:13:13 -0500 :Working: DO: Copy dbnodeupdate.zip to all nodes and then extract the file.

2015-10-27 05:13:24 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to all nodes and then extract the file.

2015-10-27 05:13:24 -0500 :Working: DO: Copy patch file to all nodes.

2015-10-27 05:14:25 -0500 :SUCCESS: DONE: Copy patch file to all nodes.

2015-10-27 05:14:25 -0500 :Working: DO: Get the information about the stack status on all nodes.

2015-10-27 05:14:28 -0500 :SUCCESS: DONE: Get the information about the stack status on all nodes.

2015-10-27 05:14:29 -0500 :Working: DO: dbnodeupdate.sh first upgrade step is running on database all nodes.

2015-10-27 05:37:04 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:10 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:11 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:17 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:18 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:26 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:27 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:33 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:34 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:40 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:41 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:49 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:49 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.log directory on the local machine.

2015-10-27 05:37:55 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.271015051429.diag directory on the local machine.

2015-10-27 05:37:56 -0500 Node oraclouddbadm02 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm03 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm04 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm05 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm06 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm07 is ready to reboot.

2015-10-27 05:37:56 -0500 Node oraclouddbadm08 is ready to reboot.

2015-10-27 05:37:56 -0500 :SUCCESS: DONE: Initiate upgrade on Database Node(s).

2015-10-27 05:37:57 -0500 :Working: DO: Initiate reboot on Database Node(s).

2015-10-27 05:37:57 -0500 :         DO: Waiting to ensure database node is down.

2015-10-27 05:41:08 -0500 :         DONE: Waiting to ensure database node is down before reboot.

2015-10-27 05:41:08 -0500 :         DO: Waiting to ensure database node is up after reboot.

2015-10-27 05:45:48 -0500 :         DONE: Waiting to ensure database node is up after reboot.

2015-10-27 05:45:48 -0500 :        DO: Waiting to connect to database node with SSH. During Linux upgrades this can take some time.

2015-10-27 06:10:10 -0500 :         DONE: Waiting to connect to database node with SSH. During Linux upgrades this can take some time.

2015-10-27 06:10:11 -0500 :SUCCESS: DONE: Initiate reboot on Database Node(s).

2015-10-27 06:10:11 -0500 :Working: DO: Initiate complete step from dbnodeupdate.sh on Database Node.

2015-10-27 06:17:04 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:10 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:11 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:17 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm03_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:18 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:26 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm04_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:27 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:33 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm05_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:34 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:40 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm06_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:41 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:49 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm07_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:49 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.log directory on the local machine.

2015-10-27 06:17:55 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm08_dbnodeupdate.271015061012.diag directory on the local machine.

2015-10-27 06:17:56 -0500 :SUCCESS: DONE: Initiate complete step from dbnodeupdate.sh on Database Node

2015-10-27 06:18:45 -0500 :Working: DO: post version check on all nodes.

2015-10-27 06:18:47 -0500 :SUCCESS: DONE: post version check on all nodes.

2015-10-27 06:18:47 -0500 :Working: DO: Check the stack status after upgrade on all nodes.

2015-10-27 06:18:49 -0500 :SUCCESS: DONE: Check the stack status after upgrade on all nodes.

To upgrade Compute node 1, I am going to execute the following command from Compute node 8.

[root@oraclouddbadm01 ESS_121220]# ssh oraclouddbadm08 ls -l /u01/app/oracle/patches/ESS_121220

total 76756

-rw-r--r-- 1 root root 78513331 Oct 4 01:06 p21634633_121220_Linux-x86-64.zip

[root@oraclouddbadm01 ESS_121220]# scp p21825906_121220_Linux-x86-64.zip oraclouddbadm08:/u01/app/oracle/patches/ESS_121220

p21825906_121220_Linux-x86-64.zip                                                                                                                                  100% 922MB 70.9MB/s   00:13

[root@oraclouddbadm01 ESS_121220]# ssh oraclouddbadm08

Last login: Tue Oct 27 06:55:53 2015 from 10.194.167.28

[root@oraclouddbadm08 ~]# cd /u01/app/oracle/patches/ESS_121220

[root@oraclouddbadm08 ESS_121220]# ls -ltr

total 1021888

-rw-r--r-- 1 root root 78513331 Oct 4 01:06 p21634633_121220_Linux-x86-64.zip

-rw-r--r-- 1 root root 966861781 Oct 27 07:01 p21825906_121220_Linux-x86-64.zip

[root@oraclouddbadm08 ESS_121220]# unzip p21634633_121220_Linux-x86-64.zip

Archive: p21634633_121220_Linux-x86-64.zip

   creating: dbserver_patch_5.150922/

inflating: dbserver_patch_5.150922/ExaXMLNode.pm

   creating: dbserver_patch_5.150922/plugins/

inflating: dbserver_patch_5.150922/plugins/010-check_17854520.sh

inflating: dbserver_patch_5.150922/plugins/000-check_dummy_perl

inflating: dbserver_patch_5.150922/plugins/000-check_dummy_bash

inflating: dbserver_patch_5.150922/exadata.img.env

inflating: dbserver_patch_5.150922/imageLogger

inflating: dbserver_patch_5.150922/ExadataImageNotification.pl

inflating: dbserver_patch_5.150922/ipconf.pl

inflating: dbserver_patch_5.150922/exadata.img.hw

inflating: dbserver_patch_5.150922/11_1_3_2_0.pm

inflating: dbserver_patch_5.150922/patchmgr

inflating: dbserver_patch_5.150922/md5sum_files.lst

inflating: dbserver_patch_5.150922/sgutil.pm

inflating: dbserver_patch_5.150922/11_2_1_2_2.pm

inflating: dbserver_patch_5.150922/11_2_1_2_1.pm

   creating: dbserver_patch_5.150922/ibdiagtools/

inflating: dbserver_patch_5.150922/ibdiagtools/setup-ssh

inflating: dbserver_patch_5.150922/ibdiagtools/topology-zfs

extracting: dbserver_patch_5.150922/ibdiagtools/VERSION_FILE

inflating: dbserver_patch_5.150922/ibdiagtools/checkbadlinks.pl

inflating: dbserver_patch_5.150922/ibdiagtools/ibping_test

inflating: dbserver_patch_5.150922/ibdiagtools/cable_check.pl

inflating: dbserver_patch_5.150922/ibdiagtools/tar_ibdiagtools

inflating: dbserver_patch_5.150922/ibdiagtools/SampleOutputs.txt

inflating: dbserver_patch_5.150922/ibdiagtools/dcli

inflating: dbserver_patch_5.150922/ibdiagtools/infinicheck

   creating: dbserver_patch_5.150922/ibdiagtools/topologies/

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/VerifyTopologyUtility.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Group.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Rack.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Switch.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/verifylib.pm

inflating: dbserver_patch_5.150922/ibdiagtools/topologies/Node.pm

extracting: dbserver_patch_5.150922/ibdiagtools/xmonib.sh

inflating: dbserver_patch_5.150922/ibdiagtools/monitord

   creating: dbserver_patch_5.150922/ibdiagtools/netcheck/

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/SolarisAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/OSAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/LinuxAdapter.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteScriptGenerator.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/runDiagnostics.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/CommonUtils.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/spawnProc.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteLauncher.pm

inflating: dbserver_patch_5.150922/ibdiagtools/netcheck/remoteConfig.pm

inflating: dbserver_patch_5.150922/ibdiagtools/verify-topology

inflating: dbserver_patch_5.150922/exadataLogger.pm

inflating: dbserver_patch_5.150922/dcli

inflating: dbserver_patch_5.150922/README.txt

inflating: dbserver_patch_5.150922/11_2_1_1_0.pm

   creating: dbserver_patch_5.150922/linux.db.rpms/

inflating: dbserver_patch_5.150922/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm

extracting: dbserver_patch_5.150922/dbnodeupdate.zip

inflating: dbserver_patch_5.150922/preconfig.pm

inflating: dbserver_patch_5.150922/11_2_2_1_0.pm

inflating: dbserver_patch_5.150922/ExadataSendNotification.pm

[root@oraclouddbadm08 ESS_121220]# cp p21825906_121220_Linux-x86-64.zip dbserver_patch_5.150922/

[root@oraclouddbadm08 ESS_121220]# cd dbserver_patch_5.150922/

[root@oraclouddbadm08 ESS_121220]# vi dbs_group

[root@oraclouddbadm08 dbserver_patch_5.150922]# cat dbs_group

oraclouddbadm01

[root@oraclouddbadm08 dbserver_patch_5.150922]# ./patchmgr -dbnode dbs_group -dbnode_upgrade -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917 -smtp_from "oraclouddbadm08" -smtp_to "abc@mydomain.com"

2015-10-27 07:14:25 -0500 :Working: DO: Initiate update on 1 database node(s).

2015-10-27 07:14:25 -0500 :Working: DO: Initiate upgrade on Database Node(s).

2015-10-27 07:14:25 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-27 07:14:26 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-27 07:14:26 -0500 :Working: DO: pre version check on all nodes.

2015-10-27 07:14:27 -0500 :SUCCESS: DONE: pre version check on all nodes.

2015-10-27 07:14:27 -0500 :Working: DO: Copy dbnodeupdate.zip to all nodes and then extract the file.

2015-10-27 07:14:39 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to all nodes and then extract the file.

2015-10-27 07:14:39 -0500 :Working: DO: Copy patch file to all nodes.

2015-10-27 07:14:49 -0500 :SUCCESS: DONE: Copy patch file to all nodes.

2015-10-27 07:14:49 -0500 :Working: DO: Get the information about the stack status on all nodes.

2015-10-27 07:14:51 -0500 :SUCCESS: DONE: Get the information about the stack status on all nodes.

2015-10-27 07:14:51 -0500 :Working: DO: dbnodeupdate.sh first upgrade step is running on database all nodes.

2015-10-27 07:39:07 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm01_dbnodeupdate.log directory on the local machine.

2015-10-27 07:39:15 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm01_dbnodeupdate.271015071452.diag directory on the local machine.

2015-10-27 07:39:16 -0500 Node oraclouddbadm01 is ready to reboot.

2015-10-27 07:39:16 -0500 :SUCCESS: DONE: Initiate upgrade on Database Node(s).

2015-10-27 07:39:17 -0500 :Working: DO: Initiate reboot on Database Node(s).

2015-10-27 07:39:17 -0500 :         DO: Waiting to ensure database node is down.

2015-10-27 07:41:22 -0500 :         DONE: Waiting to ensure database node is down before reboot.

2015-10-27 07:41:22 -0500 :         DO: Waiting to ensure database node is up after reboot.

2015-10-27 07:45:30 -0500 :         DONE: Waiting to ensure database node is up after reboot.

2015-10-27 07:45:30 -0500 :         DO: Waiting to connect to database node with SSH. During Linux upgrades this can take some time.

2015-10-27 08:10:10 -0500 :         DONE: Waiting to connect to database node with SSH. During Linux upgrades this can take some time.

2015-10-27 08:10:11 -0500 :SUCCESS: DONE: Initiate reboot on Database Node(s).

2015-10-27 08:10:11 -0500 :Working: DO: Initiate complete step from dbnodeupdate.sh on Database Node.

2015-10-27 08:17:36 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm01_dbnodeupdate.log directory on the local machine.

2015-10-27 08:17:45 -0500 Copy dbnodeupdate.sh diagnostic file to the /var/log/cellos/oraclouddbadm01_dbnodeupdate.271015081012.diag directory on the local machine.

2015-10-27 08:17:45 -0500 :SUCCESS: DONE: Initiate complete step from dbnodeupdate.sh on Database Node oraclouddbadm01.

2015-10-27 08:17:45 -0500 :Working: DO: post version check on oraclouddbadm01.

2015-10-27 08:17:47 -0500 :SUCCESS: DONE: post version check on oraclouddbadm01.

2015-10-27 08:17:47 -0500 :Working: DO: Check the stack status after upgrade on oraclouddbadm01.

2015-10-27 08:17:49 -0500 :SUCCESS: DONE: Check the stack status after upgrade on oraclouddbadm01.

This completes the Exadata Compute nodes upgrade.

Post Upgrade Image Version

Execute the following to verify the image version post Compute node upgrade.

[root@oraclouddbadm01 ~]# imageinfo

Kernel version: 2.6.39-400.264.1.el6uek.x86_64 #1 SMP Wed Aug 26 16:42:25 PDT 2015 x86_64

Image version: 12.1.2.2.0.150917

Image activated: 2015-10-27 08:12:04 -0500

Image status: success

System partition on device: /dev/mapper/VGExaDb-LVDbSys1

Important Note

• If you open any of the log file created under “/var/log/cellos” directory, for example “oraclouddbadm02_dbnodeupdate.log” you will find that the patchmgr invokes the dbnodeupdate.sh utility to perform the actual work such as pre-check, backup, and upgrade and rollback. Patchmgr utility essentially gives an option to run the commands on multiple nodes instead of running dbnodeupdate.sh utility separately on each Compute Nodes.
• Launch patchmgr from the compute node that is not upgrading itself. That node needs to be upgraded later by running patchmgr from a node that has already been upgraded or by running dbnodeupdate.sh standalone on that node itself.
• When using the ISO file for the update, it is recommended that you put the ISO file in the same directory where you have dbnodeupdate.zip.
• Non-rolling is default
• Use -rolling switch to perform rolling pre-check, backup and upgrade.
• Always perform file system backup before upgrade

Known Issues

• If you run patchmgr utility from the Compute Node which is listed in the dbs_group file then the patchmgr command will fail. Here I ran the patchmgr pre-check command from Compute node 1 which is listed in dbs_group file and it failed. 

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode dbs_group -dbnode_precheck -dbnode_loc

/u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip

2015-10-01 09:32:03 -0500 :Working: DO: Initiate validation check on 8 database node(s).

2015-10-01 09:32:03 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm01.

2015-10-01 09:32:03 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

ERROR This operation is not allowed because the patchmgr utility was launched

ERROR from server oraclouddbadm01, which is listed in the server list file dbs_group.

ERROR Either remove oraclouddbadm01 from the server list file,

ERROR or run patchmgr from a node that is not listed in the server list file.

2015-10-01 09:32:03 -0500 :FAILED: Details in files /var/log/cellos/<dbnode_name>_dbnodeupdate.log

/u01/app/oracle/patches/ESS_121220/dbserver_patch_5.150922/patchmgr.stdout,

/u01/app/oracle/patches/ESS_121220/dbserver_patch_5.150922/patchmgr.stderr

2015-10-01 09:32:03 -0500 :FAILED: DONE: Check that the database node has SSH equivalence for the root user.

Solution: Remove the Compute Node from the dbs_group file from where you ran the patchmgr command. In my case I removed the node 1 and reran the command it was successful.

• If you run patchmgr command without specifying the “-dbnode_version” option the default version will be “12.1.2.3.0.150922”. This is not the version I am upgrading to.

[root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip

2015-10-04 01:50:52 -0500 :Working: DO: Initiate validation check on 7 database node(s).

2015-10-04 01:50:52 -0500 :Working: DO: Initiate pre-upgrade validation check on database node oraclouddbadm02.

2015-10-04 01:50:53 -0500 :Working: DO: Check that the database node has enough space and SSH equivalence for the root user.

2015-10-04 01:50:54 -0500 :SUCCESS: DONE: Check that the database node has SSH equivalence for the root user.

2015-10-04 01:50:54 -0500 :Working: DO: pre version check on oraclouddbadm02.

2015-10-04 01:50:55 -0500 :SUCCESS: DONE: pre version check on oraclouddbadm02.

2015-10-04 01:50:55 -0500 :Working: DO: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 01:51:06 -0500 :SUCCESS: DONE: Copy dbnodeupdate.zip to oraclouddbadm02 and then extract the file.

2015-10-04 01:51:06 -0500 :Working: DO: Copy patch file to oraclouddbadm02.

2015-10-04 01:51:17 -0500 :SUCCESS: DONE: Copy patch file to oraclouddbadm02.

2015-10-04 01:51:17 -0500 :Working: DO: dbnodeupdate.sh precheck is running on database node oraclouddbadm02.

2015-10-04 01:52:58 -0500 Copy dbnodeupdate.log to the /var/log/cellos/oraclouddbadm02_dbnodeupdate.log directory on the local machine.

2015-10-04 01:52:58 -0500 :ERROR: DONE: Initiate pre-upgrade validation check on database node oraclouddbadm02.

SUMMARY OF ERRORS:

oraclouddbadm02: ERROR: Given 'to release' (12.1.2.3.0.150922) does not match found release (12.1.2.2.0.150917), exiting

SUMMARY OF ERRORS:

Solution: Use the option “-dbnode_version” to specify the version you are upgrading to, to resolve the issue.

 root@oraclouddbadm01 dbserver_patch_5.150922]# ./patchmgr -dbnode ~/dbs_group -dbnode_precheck -dbnode_loc /u01/app/oracle/patches/ESS_121220/p21825906_121220_Linux-x86-64.zip -dbnode_version 12.1.2.2.0.150917

Conclusion

In this article we have learned a new method of Upgrading Compute Nodes using patchmgr utility. The patchmgr utility is easy to use and can minimize the Upgrade time by running the commands in parallel on Compute Nodes. A single utility patchmgr can be used to Upgrade Storage Cells, Compute Nodes and Infiniband Switches.

Introduction

I have been working on an Exadata X4-2 which hosts a large production OLTP database. After the database migration to Exadata, application team started their testing. During the testing, they noticed that initially performance was little slow but it improved towards the end of the testing. From the AWR reports, I noticed that application performing too many “cell single block physical read” which is equivalent to “db file sequential read” on Non-Exadata database, which indicates index scan and also longer IO waits. This indicates that Exadata Smart Scan can’t help us in this situation. We decided to cache the tables and indexes directly into cache to improve the performance. While working, I noticed that the flash cache was used completely by the database. Existing 9.6TB of Exadata flash cache on X4 Quarter Rack was NOT sufficient to cache the large data sets. So we decided to look into the Exadata Flash Cache compression feature. This feature simply doubles the flash cache capacity and store more data.

 Exadata Flash Cache is secret for OLTP databases running on Exadata Database Machine.

About Exadata Flash Cache Compression

Exadata Flash Cache Compression was introduced in Exadata Storage Software version 11.2.3.3.0, which dynamically increases the logical capacity of the flash cache by transparently compressing user’s data as it is loaded into the Flash Cache. Data automatically compressed as it is written to flash cache and decompressed when it is read out of flash cache.

Exadata flash cache compression benefit varies based on the data. Uncompressed tables & indexes see higher compression ratio when compressed to OLTP compressed tables & indexes. Exadata Hybrid compressed tables have minimal benefited from flash cache compression. In this article I will demonstrate how to enable Exadata Flash Cache Compression using two methods:

1. Rolling method &
2. Non-Rolling method

Advantages

• Stores up to 2 times more data in flash cache.
• Improved performance for large datasets.
• No performance overhead
• Compression and decompression operations are transparent to the application and databases.
• Higher compression ratio for uncompressed tables & indexes

Requirement

• Exadata Storage Software version 11.2.3.3.0 and above
• Support Exadata Storage Servers X3 and X4 only
• Requires Oracle Advanced Compression option to enable flash cache compression.

 Note: Exadata Flash Cache Compression is discontinued for Exadata Storage Server X5.

Environment Details

 Current Flash Cache sizes on all Storage cells as follows:

Prerequisites

• root user access for compute node and storage cells.
• root user equivalence must be setup between compute nodes and storage cells.

#dcli -g cell_group -l root –k

• Database and Grid Infrastructure shutdown not required.

Enable Exadata Flash Cache Compression – Rolling Method

In this method we will enable flash cache compression one storage cell at a time.

Step 1: Login to the storage cell as root user and connect to “cellcli” prompt.

Step 2: Determine the current Flash Cache size on the Storage cell.

Step 3: Flush the dirty blocks from Flash Cache to Grid diks.

 This will take a while to complete depending on the amount of data being flushed from Flash Cache to Grid disks. See the Monitoring section on how to monitor the flush operation.

Step 4: Drop the flash cache

Step 5: Drop the flash log

Step 6: Drop the cell disks on the Flash disk

Step 7: Enable Exadata Flash Cache Compression

Step 8: Create the cell disks on flash disks.

Step 9: Create the Flash log

Step 10: Create the Flash Cache

Step 11: Verify Exadata Flash Cache Compression is enabled

Step 12: Determine the new size after Exadata Flash Cache Compression is enabled.

 From the above command output, the flash cache is doubled after enabling Exadata Flash Cache Compression.

Step 13: Verify the overall Storage cell details

Enable Exadata Flash Cache Compression – Non-Rolling Method

In Non-Rolling method we will enable Exadata Flash Cache Compression on remaining 2 Storage Cells at the same time. Exadata dcli utility comes handy to accomplish this task.

Step 1: Create a cell_group2 file under /root directory consists of remaining 2 storage cell one per line.

Login to the compute node 1 as root user and execute the following commands.

Step 2: Determine the current Flash Cache size on the 2 Storage Cells

Step 3: Flush the dirty blocks from Flash Cache to Grid diks.

 This will take a while to complete. See the monitoring section below on how to monitor the flush operation.

Step 4: Drop the flash cache

Step 5: Drop the flash log

Step 6: Drop the cell disks on the Flash disk

Step 7: Enable Exadata Flash Cache Compression

Step 8: Create the cell disks on flash disks.

Step 9: Create the Flash log

Step 10: Create the Flash Cache

Step 11: Verify Exadata Flash Cache Compression is enabled

Step 12: Determine the new size after Exadata Flash Cache Compression is enabled.

From the above command output, the flash cache is doubled after enabling Exadata Flash Cache Compression on storage cell 02 and 03.

Monitoring

These commands are useful to monitor or validate flush operation executed in step 2.

• While the flush operation is running, open another session for the storage cell to monitor flush operation

 If the flushstatus shows “working” the flush operation is still running whereas the flushstatus “complete” means the flush operation was completed successfully.

• After a successful flush operation number of dirty buffers will be zero. Use the following command to validate.

Known Issues

If you try to enable Exadata Flash Cache Compression for an unsupported Exadata Hardware type or Exadata software version, you would see an error similar to below:

Conclusion

 In this article we have learned how to enable Exadata Flash Cache Compression and how it helps us to store more data in Flash Cache without any overhead.

References

1. EXADATA Flash Cache Compression - FAQ (Doc ID 1664257.1)
2. Exadata Manuals:http://docs.oracle.com/cd/E50790_01/doc/doc.121/e51951/storage.htm#DBMMN22018

Written by RaviKumar YV

Introduction:

Oracle databases need to be upgraded to latest version to introduce new features and also get required bug fixes released in higher environment. Database upgrade is unavoidable to any organization.

Oracle database upgrade has multiple methods with different requirements. Proper method has to be chosen which suits to the business needs. The most hindering part is downtime demanded by any upgrade method. Traditional export/import will stipulate huge downtime and also most commonly used DBUA or Manual upgrade require more downtime to upgrade metadata and also it has need of perfect prerequisites match.

To achieve Database upgrade in lesser time, we can make use of Logical standby in upgrade process. Here oracle high availability setup will be created to existing database. User will experience downtime only during switchovers in High availability setup. Logical standby can be removed once upgrade is completed.

Serial steps for Database upgrade

• Create physical standby database using Data Guard setup for the existing database. Physical standby database will be same version as Production database version.

• Convert the Physical standby database to Logical standby database. Physical standby database should be same version as primary database, hence it is converted to Logical standby database.

• Upgrade the Logical standby database to Oracle 12c (12.1.0.2.0). Production database version will not be changed.

• Perform switchover the database roles.

• Production database will be now Logical standby database. Flashback it to physical standby database.

• Shutdown Production database and start it with Oracle 12c (12.1.0.2.0) database version since Physical standby database require same database version.

• Recover the database. It will apply the archive logs which got generated during logical standby database upgrade

• Switchover again. It will make production database to latest version Oracle12c (12.1.0.2.0) as primary database.

Environment Setup:

• Source Database version: Oracle 11g (11.2.0.4.0)

• Target database version: Oracle 12c (12.1.0.2.0)

• Source database name: v112

• Temporary Logical standby database name: v112s

Step1: Connect to v112 database and set initialization parameters to get role of Primary database

Log_archive_config=’dg_config=(v112,v112s)

Log_archive_dest_2=’ service="v112s" LGWR ASYNC db_unique_name="v112s" valid_for=(online_logfile,primary_role)’

Log_archive_dest_state_2=’enable’

Fal_client=v112

Fal_server=v112s

db_recovery_file_dest='/u01/app/oracle/arch'

db_recovery_file_dest_size=7g

Step2: Create initialization parameter file for v112s with basic parameter and add below parameters

db_file_name_convert='/u01/app/oracle/oradata/v112/','/u01/app/oracle/oradata/v112s/'

db_name='v112'

db_unique_name='v112s'

fal_client='v112s'

fal_server='v112'

log_archive_config='dg_config=(v112,v112s)'

log_archive_dest_1='Location=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch'

log_archive_dest_2='service="v112" LGWR ASYNC db_unique_name="v112" valid_for=(online_logfile,primary_role)'

log_archive_dest_state_1='enable'

log_archive_dest_state_2='defer'

log_file_name_convert='/u01/app/oracle/oradata/v112/','/u01/app/oracle/oradata/v112s/'

db_recovery_file_dest='/u01/app/oracle/standby_log'

db_recovery_file_dest_size=7g

Step3: Create new password file for v112s or copy v112 password file and rename it to orapwv112s

Step4: Start the source database (v112) instance to nomount state

SQL>startup nomount pfile=initv112s.ora

ORACLE instance started.

Total System Global Area 221331456 bytes

Fixed Size 2251856 bytes

Variable Size 163578800 bytes

Database Buffers 50331648 bytes

Redo Buffers 5169152 bytes

Step5: Create static listener entry for v112s

(SID_DESC =

(GLOBAL_DBNAME = v112s)

(ORACLE_HOME =/u01/app/oracle/product/11.2.0/dbhome_1)

(SID_NAME = v112s)

)

Step6: Add tnsnames.ora file for source database (v112) and temporary logical standby database (v112s)

V112 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = <hostname>)(PORT = <port>))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = v112)

)

)

v112s =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = <hostname>)(PORT = <port>))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = v112s)

)

)

Step7: Perform Active duplicate command to create standby database from v112 (Standby database can be created in multiple ways, we follow here 11g new feature ‘Active database duplication’)

rman target sys/sys@v112 auxiliary sys/sys@v112s

connected to target database: V112 (DBID=2404776337)

connected to auxiliary database: V112 (not mounted)

RMAN> duplicate target database for standby from active database;

Starting Duplicate Db at 25-SEP-15

using target database control file instead of recovery catalog

allocated channel: ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: SID=19 device type=DISK

.

.

datafile 3 switched to datafile copy

input datafile copy RECID=7 STAMP=891334791 file name=/u01/app/oracle/oradata/v112s/undotbs01.dbf

datafile 4 switched to datafile copy

input datafile copy RECID=8 STAMP=891334791 file name=/u01/app/oracle/oradata/v112s/users01.dbf

Finished Duplicate Db at 25-SEP-15

Step8: Open the database in read only state

Step9: At standby database, create standby redo log file, it will speed up log apply process

SQL> alter database add standby logfile '/u01/app/oracle/oradata/v112s/standby01.log' size 50m;

Database altered.

SQL> alter database add standby logfile '/u01/app/oracle/oradata/v112s/standby02.log' size 50m;

Database altered.

SQL> alter database add standby logfile '/u01/app/oracle/oradata/v112s/standby03.log' size 50m;

Database altered.

Step10: Activate media recovery. It will perform log apply at standby

SQL> recover managed standby database disconnect;

Media recovery complete.

Step11: Enable Flashback on Primary database and standby database

SQL> alter database flashback on;

SQL> select flashback_on from v$database;

Step12: Create restore point at Primary database and standby database

Primary Database:

SQL> create restore point before_upgrade guarantee flashback database;

Restore point created.

Standby Database:

SQL> recover managed standby database cancel;

Media recovery complete.

SQL> recover managed standby database disconnect;

Media recovery complete.

Conversion steps for Logical standby database

Step13: Stop the media recovery at standby database

SQL> recover managed standby database cancel;

Media recovery complete.

Step14: Build data dictionary at primary database for logical standby database

SQL> execute dbms_logstdby.build;

PL/SQL procedure successfully completed.

Step15: Convert physical standby database to logical standby database (ensure database is not in open state)

SQL> alter database recover to logical standby keep identity;

Database altered.

Step16: Restart the database, once it gets started logical standby role will be enabled

SQL> shut immediate

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

SQL>startup

ORACLE instance started.

Total System Global Area 221331456 bytes

Fixed Size 2251856 bytes

Variable Size 163578800 bytes

Database Buffers 50331648 bytes

Redo Buffers 5169152 bytes

Database mounted.

Database opened.

SQL> select database_role from v$database;

DATABASE_ROLE

----------------------------

LOGICAL STANDBY

Step17: Start the log shipping process at logical standby database

SQL> alter database start logical standby apply immediate;

Database altered.

Logminer process would have been started in the background

LOGMINER: Parameters summary for session# = 1

LOGMINER: Number of processes = 3, Transaction Chunk Size = 201

LOGMINER: Memory Size = 30M, Checkpoint interval = 150M

LOGMINER: SpillScn 0, ResetLogScn 0

Also you can notice archive log sequence got reset

SQL> archive log list

Database log mode Archive Mode

Automatic archival Enabled

Archive destination /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch

Oldest online log sequence 2

Next log sequence to archive 4

Current log sequence 4

Step18: Check if the data dictionary build is completed by executing below query

SQL> select * from v$logstdby_state where state = 'LOADING DICTIONARY';

no rows selected

Upgrade Logical standby database to Oracle 12c (12.1.0.2.0) database version

Step19: Perform pre-requisites upgrade check by executing preupgrade.sql script. Implement the recommendations provided by Preupgrade script.

SQL> @/u01/app/oracle/product/12.1.0/rdbms/admin/preupgrd.sql

Check the preupgrade logs

1. Review results of the pre-upgrade checks:

/u01/app/oracle/cfgtoollogs/v112s/preupgrade/preupgrade.log

2. Execute in the SOURCE environment BEFORE upgrade:

/u01/app/oracle/cfgtoollogs/v112s/preupgrade/preupgrade_fixups.sql

3. Execute in the NEW environment AFTER upgrade:

/u01/app/oracle/cfgtoollogs/v112s/preupgrade/postupgrade_fixups.sql

Step20: Upgrade can be done either using DBUA or Manual upgrade. Here we followed manual upgrade shutdown logical standby database at Oracle 11g (11.2.0.4).

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

Step21: Set environment variables to Oracle 12c (12.1.0.2.0) home

[oracle@Server dbs]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/

[oracle@Server dbs]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@Server dbs]$ export ORACLE_SID=v112s

Step22: Start the database in upgrade mode using Oracle 12c (12.1.0.2.0) home

1. Copy init file to Oracle 12c (12.1.0.2.0) home

    

2. Create password file for v112s instance at Oracle 12c (12.1.0.2.0) home

    

[oracle@Server bin]$ cp /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initv112s.ora /u01/app/oracle/product/12.1.0/dbs/

SQL> startup upgrade

ORACLE instance started.

Total System Global Area 222298112 bytes

Fixed Size 2922760 bytes

Variable Size 163579640 bytes

Database Buffers 50331648 bytes

Redo Buffers 5464064 bytes

Database mounted.

Database opened.

SQL> exit

Step23: Start the manual upgrade process using catctl.pl as below

[oracle@Server admin]$ /u01/app/oracle/product/12.1.0/perl/bin/perl catctl.pl -n 4 -l /u01/app/oracle/log catupgrd.sql

Alert log entries:

Argument list for [catctl.pl]

SQL Process Count n = 4

.

.

catctl.pl version: 12.1.0.2.0

Oracle Base = /u01/app/oracle

Analyzing file catupgrd.sql

Log files in /u01/app/oracle/log

catcon: ALL catcon-related output will be written to /u01/app/oracle/log/catupgrd_catcon_16472.lst

catcon: See /u01/app/oracle/log/catupgrd*.log files for output generated by scripts

catcon: See /u01/app/oracle/log/catupgrd_*.lst files for spool files, if any

Number of Cpus = 1

SQL Process Count = 4

.

.

SQL> Rem =====================================================================

SQL> Rem Run component status as the last output

SQL> Rem Note: NO DDL STATEMENTS. DO NOT RECOMMEND ANY SQL BEYOND THIS POINT.

SQL> Rem Note: ACTIONS_END must stay here to get the correct upgrade time.

SQL> Rem =====================================================================

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

LOG FILES: (catupgrd*.log)

Grand Total Upgrade Time: [0d:1h:12m:13s]

Step24: Check the upgraded database

col comp_id format a10

col comp_name format a30

col version format a10

col status format a8

SQL> select substr(comp_id,1,15) comp_id,

substr(comp_name,1,30) comp_name,

substr(version,1,10) version,status

from dba_registry

COMP_ID COMP_NAME VERSION STATUS

--------------------------------------------------------------------------------------------------------------------------

XDB Oracle XML Database 12.1.0.2.0 VALID

CATALOG Oracle Database Catalog Views 12.1.0.2.0 VALID

CATPROC Oracle Database Packages and T 12.1.0.2.0 VALID

Step25: Start the log apply process at Logical standby which is now Oracle 12c (12.1.0.2.0) version. Primary still in Oracle 11g (11.2.0.4.0) version

SQL> alter database start logical standby apply;

Database altered.

Switchover from Logical standby to Primary

Step26: Switchover the roles. Now primary database will become Oracle 11g (11.2.0.4.0) Logical standby and other side will be Oracle 12c (12.1.0.2.0) primary database

1. At primary database execute below query

SQL> alter database commit to switchover to logical standby;

Database altered.
 

2. At standby database execute below query
 

SQL> alter database commit to switchover to primary;

Database altered.

3. Primary database become Standby database
 

SQL> select database_role from v$database;

DATABASE_ROLE

-----------------------------

LOGICAL STANDBY

SQL> show parameter db_unique_name

NAME TYPE VALUE

--------------------------------------------------------------------------------------------------

db_unique_name string v112
 

4. Standby database become Primary database
 

SQL> select database_role from v$database;

DATABASE_ROLE

---------------------------

PRIMARY

SQL> show parameter db_unique_name

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

db_unique_name string v112s

Step26: We cannot directly convert Logical standby database to physical standby database, hence flashback to guarantee restore point which was created prior to upgrade

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount

ORACLE instance started.

Total System Global Area 221331456 bytes

Fixed Size 2251856 bytes

Variable Size 163578800 bytes

Database Buffers 50331648 bytes

Redo Buffers 5169152 bytes

Database mounted.

SQL> flashback database to restore point before_upgrade;

Flashback complete.

SQL> alter database convert to physical standby;

Database altered.

Step27: Now primary database becomes Oracle 11g (11.2.0.4.0) physical standby database. i.e Logical to physical conversion

Step28: Shutdown database and start Physical standby database with Oracle 12c (12.1.0.2.0) binary

SQL> startup mount pfile=initv112.ora

ORACLE instance started.

Total System Global Area 222298112 bytes

Fixed Size 2922760 bytes

Variable Size 163579640 bytes

Database Buffers 50331648 bytes

Redo Buffers 5464064 bytes

Database mounted.

SQL> select database_role from v$database;

DATABASE_ROLE

------------------------------

PHYSICAL STANDBY

Step29: Enable Log shipping

SQL> alter database recover managed standby database using current logfile through next switchover disconnect;

Database altered.

All archive logs generated during DB upgrade will be applied to standby

Step30: After applying all logs standby database will be upgraded to Oracle 12c (12.1.0.2.0) version Perform switchover from physical standby to primary database

Step31: Perform switchover from Primary database to standby database

Current Primary database
 

SQL> alter database commit to switchover to physical standby with session shutdown;

Database altered.

It will convert primary database to physical standby database and shutdown the database.  It can be used in future.
 

Current standby database
 

SQL> alter database commit to switchover to primary;

Database altered.

SQL> alter database open;

Database altered.

SQL> select OPEN_MODE, PROTECTION_MODE from v$database;

OPEN_MODE PROTECTION_MODE

-----------------------------------------------------------------

READ WRITE MAXIMUM PERFORMANCE

Check the DB component status

SQL> col comp_id format a10

SQL> col comp_name format a30

SQL> col version format a10

SQL> col status format a8

SQL> select substr(comp_id,1,15) comp_id,

substr(comp_name,1,30) comp_name,

substr(version,1,10) version,status

from dba_registry;

COMP_ID COMP_NAME VERSION STATUS

---------------------------------------------------------------------------------------------------------------------

XDB Oracle XML Database 12.1.0.2.0 VALID

CATALOG Oracle Database Catalog Views 12.1.0.2.0 VALID

CATPROC Oracle Database Packages and T 12.1.0.2.0 VALID

Database has been upgraded to Oracle 12c (12.1.0.2.0) version and DB components are valid. Client failover settings can be configured in such a way that it will connect to primary always as below. Client can make connections to Logical standby when Primary database is in standby mode.

V112 = 
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=TCP)(HOST=<primary database>)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=<Logical standby database>)(PORT=1521))
)
(CONNECT_DATA = (SERVICE_NAME= <service name>))
)

Summary:

Using temporary logical standby database can be upgraded from Oracle 11g to Oracle 12c in lesser time. Downtime occurred only during switchover from Primary database to Logical standby database and switchback to Primary database role.

Before the version 12.1.0.2, when a CDB is started, all PDBs remains in mounted mode. Except the PDB$SEED, that remains in Read only mode.

To solve this, you can create a trigger to open the PDBs.

CREATE OR REPLACE TRIGGER ABRIR_TODOS_PDBS 

  AFTER STARTUP ON DATABASE 

BEGIN 

   EXECUTE IMMEDIATE 'ALTER PLUGGABLE DATABASE ALL OPEN'; 

END;

The 12.1.0.2 patchset has introduced the ability to preserve the startup state of PDBs. 

[oracle@oracle01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 8 16:26:37 2014 

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics

and Real Application Testing options

SQL> SELECT name, open_mode FROM v$pdbs;

NAME                              OPEN_MODE

------------------------------    ----------

PDB$SEED                          READ ONLY

PDB1                              READ WRITE

PDB2                              MOUNTED

PDB3                              MOUNTED

Now, we can save the state:

ALTER PLUGGABLE DATABASE pdb1 SAVE STATE;

Verifying the saved states:

[oracle@oracle01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 8 16:32:37 2014 

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics

and Real Application Testing options

SQL> SELECT con_name, instance_name, state FROM dba_pdb_saved_states;

CON_NAME             INSTANCE_NAME        STATE

-------------------- -------------------- --------------

PDB1                 cdb1                 OPEN

Restarting the CDB:

[oracle@oracle01 ~]$ sqlplus / as sysdba 

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 8 16:36:35 2014

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics

and Real Application Testing options

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup

ORACLE instance started.

Total System Global Area  626327552 bytes

Fixed Size                        2291472 bytes

Variable Size                276826352 bytes

Database Buffers         339738624 bytes

Redo Buffers                   7471104 bytes

Database mounted.

Database opened.

SQL> SELECT name, open_mode FROM v$pdbs;

NAME                              OPEN_MODE

------------------------------    ----------

PDB$SEED                          READ ONLY

PDB1                              READ WRITE

PDB2                              MOUNTED

PDB3                              MOUNTED

To discard the PDB state:

ALTER PLUGGABLE DATABASE pdb1 DISCARD STATE;

[oracle@oracle01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Sep 8 16:38:37 2014

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics

and Real Application Testing options

SQL> SELECT con_name, instance_name, state FROM dba_pdb_saved_states;

no rows selected

Introduction

In this article we will see how to configure Enterprise Manager Security specific to OMS, Securing OMS is the key role to do as all the servers, databases, accounts information will be available in Oracle management repository which is heart of Enterprise manager (or) cloud control. We also go through with the enabling security for the OMS and also agent with step by step procedure

Securing EM

Likewise the security of database by creating wallet in similar we can secure the entire enterprise manager, the components that can be secured are Infrastructure and operating system that EM configured, securing oracle management repository, the agents which are configured to the EM, securing communication between the components and operational security from OMR database level such as privilege, role management, groups and system and configuring auditing so on.

When it comes to infrastructure and operating system security, there are few key components to be considered, for example in order to access the server using “oracle’ account, it’s recommended to login to oracle through the personal account. So that direct access to the server can be avoided, like that various options available and same time need to take care of unsecure devices and better to consider disabling them. Same time it is highly recommended to setup MOS for oracle updates in order to apply critical patch updates and also OS level patches.

In order to ensure for secure communications we should consider to securing OMS as well, we have to perform securing both the OMS and also dependent agents so that agent can be reachable and communicated with the OMS.

Apart from the above securing methods, on top of that additionally we need to enable security at database level. They are Oracle advanced security like data redeaction, transparent data encryption. Auditing sys user activities, restricting use of SYSMAN user.

Enabling security for Oracle Management Service

To secure the OMS the basic rule is to secure the OMS, in order to do that after securing OMS we have to perform by stopping the OMS, so downtime of OMS should be acceptable in this case. By that it generates the root key inside the OMR which is used to distribute wallets which has digital certificates for the agents. After securing the OMS we should be able to use Enterprise Manager using HTTPS channel. Securing OMS is very short and simple procedure

1)     $OMS_HOME/bin/emctl stop oms

2)     $OMS_HOME/bin/emctl secure oms

3)     $OMS_HOME/bin/emctl start oms

After successful startup of OMS, EM can be accessed using HTTPS channel as (ex: https://em12c.localdomain:7801/em).

After able to connect to the Cloud control url using HTTPS we can confirm/assume that OMS is secured.  We can also check the status of OMS including secured or not using the below command.  

[oracle@em12c ~]$ emctl status oms -details -sysman_pwd welcome1

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Console Server Host        : em12c.localdomain

HTTP Console Port          : 7789

HTTPS Console Port         : 7801

HTTP Upload Port           : 4890

HTTPS Upload Port          : 4901

EM Instance Home           : /u01/app/oracle/gc_inst/em/EMGC_OMS1

OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log

OMS is not configured with SLB or virtual hostname

Agent Upload is locked.

OMS Console is locked.

Active CA ID: 1

Console URL: https://em12c.localdomain:7801/em

Upload URL: https://em12c.localdomain:4901/empbs/upload

 WLS Domain Information

Domain Name            : GCDomain

Admin Server Host      : em12c.localdomain

Admin Server HTTPS Port: 7102

Admin Server is RUNNING

 Managed Server Information

Managed Server Instance Name: EMGC_OMS1

Managed Server Instance Host: em12c.localdomain

WebTier is Up

Oracle Management Server is Up

After securing the OMS, we need to create certification authority using emctl, usually certification authority (CA) created if CA is expiring or to change key strength. Whenever CA is created every time it will be assigned with unique number. By default the certification authority 1 will be created during the installation of EM, so for every new creation of CA the next number will be allocated such as 2, 3 and the key strength may not require to have same strength.  The existing CA 1 can be viewed using the below command and various details are available below including the key strength.

[oracle@em12c ~]$ emcli get_ca_info -ca_id="1" -details

Info about CA with ID: 1

CA is configured

Signature algorithm : sha512

Key strength : 1024

DN: CN=em12c.localdomain, C=US, ST=CA, L=EnterpriseManager on em12c.localdomain, OU=EnterpriseManager on em12c.localdomain, O=EnterpriseManager on em12c.localdomain

Serial# : 75736055487891743580

Valid From: Wed Oct 24 10:01:12 EDT 2012

Valid Till: Sun Oct 23 10:01:12 EDT 2022

Number of Agents registered with CA ID 1 is 4

192.168.0.10:3872

db12c.localdomain:3872

em12c.localdomain:3872

host1.localdomain:3872

[oracle@em12c ~]$

 To create of view to trust the certification chain we have to create CA, it can be performed using the emcli command as below by mentioning the home location and the username we are using to setup.  

[oracle@em12c ~]$ mkdir -p /home/oracle/cert_auth

[oracle@em12c ~]$ emcli setup -url=https://em12c.localdomain:7801/em -username=sysman -dir=/home/oracle/cert_auth

Oracle Enterprise Manager 12c 3.

Copyright (c) 1996, 2013 Oracle Corporation and/or its affiliates. All rights reserved.

Enter password

Warning: This certificate has not been identified as trusted in the local trust store

--------------------------------------

[

[

  Version: V3

  Subject: CN=em12c.localdomain, C=US, ST=CA, L=EnterpriseManager on em12c.localdomain, OU=EnterpriseManager on em12c.localdomain, O=EnterpriseManager on em12c.localdomain

  Signature Algorithm: SHA512withRSA, OID = 1.2.840.113549.1.1.13

  Key:  Sun RSA public key, 1024 bits

  modulus: 133627010103079426106767978733765070671256452755264124298364177515855963402396209268550264027014910629979186232646076391897498654631460970590402851272991410948484798398337141209797689184905537665702466156563546606727372497472894081972082766339094634591866887572507346786958080223041346139625330884681339231267

  public exponent: 65537

  Validity: [From: Wed Oct 24 10:01:12 EDT 2012,

               To: Sun Oct 23 10:01:12 EDT 2022]

  Issuer: CN=em12c.localdomain, C=US, ST=CA, L=EnterpriseManager on em12c.localdomain, OU=EnterpriseManager on em12c.localdomain, O=EnterpriseManager on em12c.localdomain

  SerialNumber: [    041b0c85 330d0fa3 5c]

Certificate Extensions: 2

[1]: ObjectId: 2.5.29.19 Criticality=true

BasicConstraints:[

  CA:true

  PathLen:2147483647

]

[2]: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

  DigitalSignature

  Non_repudiation

  Key_CertSign

  Crl_Sign

]

]

  Algorithm: [SHA512withRSA]

  Signature:

0000: 0E 2B 35 C6 F3 99 A5 11   1E 56 23 41 25 DC F9 41  .+5......V#A%..A

0010: 83 16 2E 1D 22 4B 02 21   B6 0C 31 81 0D E2 6E B8  ...."K.!..1...n.

0020: 7B 65 67 E4 50 36 6A 1F   C8 B0 D2 A4 10 DD 86 DB  .eg.P6j.........

0030: 33 CF 0E 3E E6 6A 96 37   9D 38 18 15 4C BD 87 1E  3..>.j.7.8..L...

0040: C3 EE 14 58 9D AD A3 83   2A 46 33 99 37 E8 55 43  ...X....*F3.7.UC

0050: 0B 82 F0 A7 64 84 95 32   39 C8 00 76 9D B7 52 C9  ....d..29..v..R.

0060: 40 A7 94 38 24 CC EE 01   37 3C 2B C8 1E FD 3E 5A  @..8$...7<+...>Z

0070: C9 AA D4 B0 ED B6 B2 F3   34 5B DA BC 24 F3 10 59  ........4[..$..Y

]

--------------------------------------

Warning: This certificate could be compromised

Warning: This certificate has not been identified as trusted in the local trust store

--------------------------------------

[

[

  Version: V3

  Subject: CN=em12c.localdomain

  Signature Algorithm: SHA512withRSA, OID = 1.2.840.113549.1.1.13

  Key:  Sun RSA public key, 1024 bits

  modulus: 112368975731894125914161537621417272129612339366145866018201999698606351882724580621563935552262439838814097775099738716022935964599562271114030308566143956134180271738280849095747824057799964636516344438690254947507575679885195443085855831701212602259616566202725828803312983711909337862922623119576842004049

  public exponent: 65537

  Validity: [From: Wed Jul 03 01:13:48 EDT 2013,

               To: Sun Jul 02 01:13:48 EDT 2023]

  Issuer: CN=em12c.localdomain, C=US, ST=CA, L=EnterpriseManager on em12c.localdomain, OU=EnterpriseManager on em12c.localdomain, O=EnterpriseManager on em12c.localdomain

  SerialNumber: [   -461beffa 00b97ef6]

Certificate Extensions: 3

[1]: ObjectId: 2.5.29.19 Criticality=true

BasicConstraints:[

  CA:false

  PathLen: undefined

]

[2]: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

  DigitalSignature

  Non_repudiation

  Key_Encipherment

  Key_Agreement

]

[3]: ObjectId: 2.5.29.14 Criticality=false

SubjectKeyIdentifier [

KeyIdentifier [

0000: 83 F4 BB 20 B5 6A B0 4B   8E A3 05 BC 06 A8 46 02  ... .j.K......F.

0010: 5E A3 EB 00                                        ^...

]

]

]

  Algorithm: [SHA512withRSA]

  Signature:

0000: 90 A8 37 59 D1 18 94 99   5A DF 05 85 37 A8 E4 82  ..7Y....Z...7...

0010: 58 69 52 33 5E 82 F1 37   07 C8 0A C1 05 11 FD A0  XiR3^..7........

0020: 10 D5 02 C6 81 DE C2 E3   F2 CC B2 C1 23 31 B4 A6  ............#1..

0030: EB 6D 86 2D 4D 27 9B 78   C6 4A 40 04 07 0D AB 47  .m.-M'.x.J@....G

0040: 7F 4E B9 0D 48 68 9B 17   F7 7A AA 1A 9C D6 D8 AC  .N..Hh...z......

0050: 03 8C 38 6B A6 07 61 02   4F C1 8D A8 0E 51 6C 94  ..8k..a.O....Ql.

0060: 01 56 FC 7C CB 49 E9 03   64 93 EB 9E FD 81 60 F9  .V...I..d.....`.

0070: 14 43 09 6D 0B 6A 72 09   CE 1B 68 30 18 C9 E0 6B  .C.m.jr...h0...k

]

--------------------------------------

Do you trust the certificate chain? [yes/no] yes

Emcli setup successful

[oracle@em12c ~]$

After setup of the EM, we can create new certification authority using the below command.

[oracle@em12c ~]$ emctl secure createca -key_strength 2048

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Creating CA... Started.

Enter Enterprise Manager Root (SYSMAN) Password :

Successfully created CA with ID 2

[oracle@em12c ~]$ 

The new created certification authority again using emctl command, but we have to mention the CA unique ID.

[oracle@em12c ~]$ emcli get_ca_info -ca_id="2" -details

Info about CA with ID: 2

CA is configured

Signature algorithm : sha512

Key strength : 2048

DN: CN=em12c.localdomain, C=US, ST=CA, L=CA2, OU=EnterpriseManager on em12c.localdomain, O=EnterpriseManager on em12c.localdomain

Serial# : -3843635707979125701

Valid From: Thu Oct 29 19:33:06 EDT 2015

Valid Till: Mon Oct 27 19:33:06 EDT 2025

There are no Agents registered with CA ID 2

[oracle@em12c ~]$

After creating the new CA, it is necessary to maintain Oracle wallet which contains the credentials and trusted certificates and private keys. During the installation of EM the preexisting wallet will be available by default. So whenever we create new CA we have to recreate the wallet. It can be created using below command and we need to provide the passwords of weblogic server.

[oracle@em12c ~]$ emctl secure create_admin_creds_wallet -admin_pwd welcome1 -nodemgr_pwd welcome1

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Creating Admin Credentials Wallet...

Created Admin Credentials Wallet successfully

Run this command on all OMS's in your environment

[oracle@em12c ~]$

If we have more than one OMS then this command has to be performed on all the OMS.

[oracle@em12c ~]$ emctl status oms -details

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

Console Server Host        : em12c.localdomain

HTTP Console Port          : 7789

HTTPS Console Port         : 7801

HTTP Upload Port           : 4890

HTTPS Upload Port          : 4901

EM Instance Home           : /u01/app/oracle/gc_inst/em/EMGC_OMS1

OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log

OMS is not configured with SLB or virtual hostname

Agent Upload is locked.

OMS Console is locked.

Active CA ID: 2

Console URL: https://em12c.localdomain:7801/em

Upload URL: https://em12c.localdomain:4901/empbs/upload

WLS Domain Information

Domain Name            : GCDomain

Admin Server Host      : em12c.localdomain

Admin Server HTTPS Port: 7102

Admin Server is RUNNING

Managed Server Information

Managed Server Instance Name: EMGC_OMS1

Managed Server Instance Host: em12c.localdomain

WebTier is Up

Oracle Management Server is Up

[oracle@em12c ~]$

We can refer the emctl.log for more information on the status of the connectivity with node manager.

2015-10-31 20:45:32,106 [main] INFO  commands.EmbeddedWLSTRunner flush.112 - Not connected to Node Manager

2015-10-31 20:45:32,108 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -

2015-10-31 20:45:32,108 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -

2015-10-31 20:45:32,109 [main] INFO  oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet

2015-10-31 20:45:32,148 [main] INFO  oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet

2015-10-31 20:45:32,152 [main] INFO  commands.EmbeddedWLSTRunner flush.112 - Connecting to Node Manager ...

2015-10-31 20:45:32,153 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -

2015-10-31 20:45:32,153 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -

2015-10-31 20:45:32,470 [main] INFO  commands.EmbeddedWLSTRunner flush.112 - Successfully Connected to Node Manager.

Securing Agent

Securing OMS will not be sufficed; we have to secure all the agents dependent to the OMS. This can be done after exporting the environment for the AGENT_HOME using below command.

[oracle@em12c ~]$ emctl secure agent -emdWalletSrcUrl https://em12c.localdomain:7801/em

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Agent is already stopped...   Done.

Securing agent...   Started.

Enter Agent Registration Password :

EMD gensudoprops completed successfully

Securing agent...   Successful.

[oracle@em12c ~]$

After successful securing the agent, we can check the status of agent using “-secure” option.

[oracle@em12c config]$ emctl status agent -secure

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

Checking the security status of the Agent at location set in /u01/app/oracle/agent/agent_inst/sysman/config/emd.properties...  Done.

Agent is secure at HTTPS Port 3872.

Checking the security status of the OMS at https://em12c.localdomain:7801/empbs/upload/...  Done.

OMS is secure on HTTPS Port 4901

[oracle@em12c config]$

We can see that the Agent is secured with OMS and with HTTPS.  After that enforce to upload agent using the traditional command.

[oracle@em12c ~]$ emctl upload agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

EMD upload error:full upload has failed: Upload Recent Sending Statistics

________________________________

Channel=severity send[ms]=21118 fsz=489

Channel=severity send[ms]=21136 fsz=484

..........................................................

Avg send time=1496.172 milliseconds

Backoff Event List

________________________________

Upload Failure List

_______________________________

Connection Event List

_____________________________

Upload timed out before completion.

Number of files to upload before the uploadNow call: 253, total size (MB): 0.4314556

Remaining number of files to upload: 169, total size (MB): 0.33713245 (TIMEOUT)

[oracle@em12c ~]$

[oracle@em12c ~]$ emctl upload agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

EMD upload completed successfully

[oracle@em12c ~]$

We can see that agent is uploaded successfully after securing the OMS and Agent as well. 

Securing Agent using Cloud Control

Securing the agents can be either done using command line command and also from EM GUI, There are more features are involved by using cloud control, like we can secure multiple agents at a time, after login to the cloud control from the summary page follow Setup à manage cloud control à agents.

After following the navigation we can view the agents whichever registered with OMS .

Then select the agent to be secured and we can provide the name of the task.

 From the parameters tab we need to provide the password of console in order to provide appropriate authorization to submit the job. 

After that we need to click the submit button hence the job will be submitted, we can check the status from the status message. 

It takes 60-70 seconds more or less in order to resecure each agent and we can determine the status whether succeeded or not. 

Summary

We’ve seen how to trust the certification chain and creating new certification authority to create the wallet and how to view the various CA Unique ID’s and also securing agents to the OMS with the new CA in command line and also using the cloud control.

Reference

http://docs.oracle.com/cd/E29505_01/doc.1111/e24473/security.htm#BABFBGEH

This video provides an overview of the Library window in Toad Data Point from Dell Software.

This video provides an overview of notifications in Toad Intelligence Central from Dell Software.

Toad for MySQL 7.7

• Online Help (html) - Complete and detailed help for users of Toad for MySQL.
• Release Notes (html) - Find a list of resolved issues, what's new in this release, and the system requirements.
• Installation Guide (html) - Instructions for installing this product.

Toad for MySQL 7.5

• Release Notes (html) - Find a list of resolved issues, what's new in this release, and the system requirements.
• Installation Guide (html) - Instructions for installing this product.

Toad for MySQL 7.3

• Release Notes (html)
• Installation Guide (html)

Toad for MySQL 7.2

• Release Notes (html)
• Installation Guide (html)

You can also find product documentation for earlier versions of Toad for MySQL at: http://documents.software.dell.com/Toad for MySQL/.

 
Released: 1 December 2015

Easily share information across your team with Toad™ Intelligence Central. This centralized repository for sharing Toad files and other data enables efficient, well-governed collaboration. Whether you’re gathering and prepping data in Toad Data Point, or sharing PL/SQL code and code analysis reports in Toad for Oracle, you can centrally publish your work in Toad Intelligence Central. This makes it quick and easy to share and view queries, files, objects and datasets — right within the tool. With this server-side component, you’ll reduce the risk of reporting failures, security threats and data loss while ensuring data governance.

POST QUESTION / COMMENT

Do you have a question or comment about this freeware?  Post it to the product forum:

Go to Forum

Toad Intelligence Central 2.7  Release Notes

Toad Intelligence Central 2.7 Deployment Guide

Toad Intelligence Central 2.7 Quick Start Guide

Written by Deepak Vohra

The debian distributions for Oracle database 10g XE include only the i386 architecture packages and not the amd64 architecture packages. The MultiarchSpec provides for cross-architecture installation of binary packages in the package manager dpkg and the packaging tool apt. In this article we shall install the i386 binaries on a amd64 architecture Ubuntu, which has added support for installing packages from multiple architectures on a single system since the 11.04 version. We shall be using Ubuntu 12.04. This article has the following sections.

Finding the Platform Architecture

Installing Oracle database XE i386 binaries

Configuring the Oracle database XE

Finding the Platform Architecture

To find what architecture does a Ubuntu platform support run the command uname –i; for example the supported architecture is x86_64.

The architecture supported by dpkg on a platform may be obtained with the following command.

dpkg –print-architecture

The foreign architecture supported may be obtained with the following command.

dpkg –print-foreign-architecture

The output on Ubuntu 12.04 installed on a x86_64 platform shows the architecture to be amd64 and foreign architecture to be i386.

The foreign architecture supported is stored in the file /etc/dpkg/dpkg.cfg.d/multiarch.

Having ascertained that we are installing i386 binaries on a amd64 platform that supports i386 as a foreign architecture, we shall install Oracle database XE i386 binaries next.

Installing Oracle database XE i386 binaries

First, fetch the i386 debian distributions for the oracle-xe-universal and libaio, which oracle-xe-universal has a dependency on using the following command.

wget -c http://oss.oracle.com/debian/dists/unstable/main/binary-i386/libaio_0.3.104-1_i386.deb http://oss.oracle.com/debian/dists/unstable/non-free/binary-i386/oracle-xe-universal_10.2.0.1-1.1_i386.deb

The libaio:i386 debian distribution gets downloaded.

The oracle-xe-universal:i386 debian distribution also gets downloaded.

Next, install the libaio:i386 debian binaries on the amd64 architecture. For the cross-architecture installation we need to include --force-architecture in the dpkg command.

sudo dpkg -i --force-architecture libaio_0.3.104-1_i386.deb

The libaio:i386 package gets installed.

Install the oracle-xe-universal:i386 package using the following command.

sudo dpkg -i --force-architecture oracle-xe-universal_10.2.0.1-1.1_i386.deb

If the system does not meet the minimum requirements for swap space the following message gets output indicating that a minimum swap space of 1024 MB is required.

Increase the swap space with the following commands.

sudo su

swapoff -a

cd /host/ubuntu/disks/

mv swap.disk swap.disk.bak

dd if=/dev/zero of=swap.disk bs=1024 count=1024000

mkswap swap.disk

swapon -a

free –m

The output shows that the swap space has been increased.

Rerun the dpkg command for the oracle-xe-universal:i386 package. The output indicates that oracle-xe-universal:i386 depends on libc6 and bc packages. The oracle-xe-universal:i386 package does not get installed due to the missing dependencies.

Ubuntu 12.04 on a amd64 platform does provide a libc6 package, but it is for the amd64 architecture. Output information about the libc:amd64 package using the dpkg-query command.

dpkg-query –s libc6:amd64

The output shows that the Architecture of the package is amd64. MultiArch- aware packages include the Multi-Arch header, which is set to “Same” for the libc6:amd64 package. Multi-Arch: same indicates that the package may be co-installed with a same-name package for a different architecture, libc6:i386 for the libc6:amd64 package, but must not be used to satisfy the dependency of any package of a different architecture from itself. Therefore, we won’t be able to substitute the libc6:i386 with the libc6:amd64 package. If the Multi-Arch were set to “foreign” the libc6:amd64 could have been used instead of libc6:i386, but it is not a feasible provision.

For correcting the dependencies run the following command twice in a sequence, which builds a dependency tree from the package list and installs the required packages.

sudo apt-get –f install

The oracle-xe-universal:i386 package dependencies bc:i386 and libc6:i386 are listed to be installed.

As the output indicates the libc6:i386, bc:i386 dependencies get setup, and the oracle-xe-universal:i386 package gets installed.

The importance of Multi-Arch: same type package becomes apparent if a same-name package for a different architecture is also required as a dependency for some other package. The bc:i386 package is not multiarch-aware (does not include the Multi-Arch header) as indicated in the package information obtained using the dpkg-query command.

If the bc:amd64 package were also required and installed, the bc:i386 package would get removed and because the oracle-xe-universal:i386 package has a dependency on the bc:i386 package the Oracle database XE would instance would stop and the oracle-xe-universal:i386 package would also get removed as shown below.

It should also be noted that we did not install the dependencies libc6:i386 and bc:i386 directly using is the sudo apt-get command for the dependencies as these dependencies have further dependency on other packages. If the libc6:i386 dependency is installed directly, it lists some other dependencies and does not get installed as shown below.

Similarly, the bc:i386 package has dependency on other packages including the libc6:i386 package.

It is best to build the dependency tree for the packages and use the sudo apt-get –f install command to install the missing dependencies. If not certain why a package has been installed run the aptitude why command for the package. For example the aptitude why libc6:i386 command outputs the dependence of oracle-xe-universal:i386 on libc6:i386.

Configuring the Oracle database XE

Having installed the oracle-xe-universal:i386 package and the required dependencies we shall configure the database next. Run the following command to configure the Oracle Database XE.

sudo /etc/init.d/oracle-xe configure

A prompt for specifying the HTTP port for Oracle Application Express gets displayed. If no value is provided, the default port of 8080 is used. A prompt for specifying the port for the database listener gets displayed with the default port as 1521. Specify the password for the SYS and SYSTEM users. Select “y” for starting database on boot. The Oracle Net Listener gets started. The database gets configured. The database instance gets started.

Next, specify the environment variables in the .bash_profile, which is opened using a vi editor. Use the following command to open the .bash_profile.

$ vi ~/$HOME/.bash_profile

We shall be using the following environment variables for ORACLE_HOME, PATH, ORACLE_SID and NLS_LANG; other variables may be specified if required.

export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server

export PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_HOME

export ORACLE_SID=XE

export NLS_LANG=’$ORACLE_HOME/bin/nls_lang.sh’

export PATH

Specify the environment variables and values as shown below in the vi editor.

The Application Express may be accessed using url http://127.0.0.1:8080/apex/. Specify the Username and Password and click on Login.

The Oracle database XE Application Express page gets displayed.

The configuration files for the Oracle database XE may be listed with the dpkg-query command for oracle-xe-universal:i386 package.

In this article we learnt about installing Oracle database XE i386 binaries on amd64 architecture Ubuntu 12.04 OS. The article illustrates the cross-architecture support for packages, provided by Ubuntu 12.04 (11.04 and later), and the procedure discussed may be used for cross-architecture installation of binaries for another version of Oracle Database also.

Product Features

• Key product features with screenshots

System requirements

Before installing Toad - Mac Edition 1.2, ensure that your system meets the following minimum hardware and software requirements.

Software Requirements

Hardware Requirements

I'm trying to connect to Sybase IQ 16 with Toad for SAP Solutions 3 from my laptop as a client.

After setting up all connection manager fields I’m getting a massage which states... <Database error> <Database server not found>

Do i need an ODBC connection on my laptop (client side?)

Note: with same connection settings Aqua Data Studio connects without any problem.

Introduction

Recently Oracle updated the MOS note “Common Exadata security scan findings (Doc ID 1405320.1)" related to Java vulnerabilities. In the October 2015 CPU, there were several Java vulnerabilities addressed. Many of these also affected Exadata images 12.1.2.x.x. Some of the vulnerabilities were very severe (CVSS score 10.0). 

Here is a snippet of the information found in “Common Exadata security scan findings (Doc ID 1405320.1)":

This Critical Patch Update contains 25 new security fixes for Oracle Java SE.  24 of these vulnerabilities may be remotely exploitable without authentication, i.e., may be exploited over a network without the need for a username and password. Some of the vulnerabilities were very severe (CVSS score 10.0). 

The Java vulnerabilities affects the Exadata image versions 12.1.2.1.0 and later.

The Exadata database server and Exadata Storage cells running Exadata image version 12.1.2.1.0 includes the JDK 7 package installed as an RPM. To update the RPM, first obtain the latest version by referring to Note 1439822.1. Make sure you look for the JDK 7. DO NOT use JDK 8 or later.

About Management Server (MS) process

•  It helps execute CELLCLI commands
• It provides Java interface to Enterprise Manager 12c Plugins.
• You can't execute CELLCI commands if MS process is down.

Managing MS process:

To stop MS process

# cellcli -e alter cell shutdown services ms

To start MS process

# cellcli -e alter cell startup services ms

To restart MS process

# cellcli -e alter cell restart services ms

In this article I am going to demonstrate how to update JDK version on an Exadata X2-2 running ESS version 12.1.2.2.0.

Assumption

• root user password for Compute nodes and Storage Cells
• root user equivalence must be setup between compute nodes and storage cells.
• No outage is required for database and storage cell services.

Environment

Procedure to update JDK on Exadata Compute nodes and Storage cells

Unless otherwise stated all the steps will be executed as 'root' user.

Step I - Download and copy latest JDK 7 package to Exadata Compute node and Storage cell

• Using the note [1439822.1]: All Java SE Downloads on MOS download the latest JDK 7 package

Here is the directly link

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=116578518457933&id=1439822.1&_adf.ctrl-state=15es4f8nb6_53

• As root user, login to Exadata Compute node 1 and perform the following steps

[root@oraclouddbadm01 jdk]# mkdir -p /u01/app/oracle/software/jdk

[root@oraclouddbadm01 jdk]# cd /u01/app/oracle/software/jdk

• Using WinScp or similar utility copy the JDK 7 package to Exadata Compute node

[root@oraclouddbadm01 jdk]# ls -ltr p21984104_17091_Linux-x86-64.zip

Step II - Update JDK on Exadata Storage cells

• Stop MS process on all Exadata Storage cells

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'cellcli -e alter cell shutdown services ms'

oracloudceladm01:

oracloudceladm01: Stopping MS services...

oracloudceladm01: The SHUTDOWN of MS services was successful.

oracloudceladm02:

oracloudceladm02: Stopping MS services...

oracloudceladm02: The SHUTDOWN of MS services was successful.

oracloudceladm03:

oracloudceladm03: Stopping MS services...

oracloudceladm03: The SHUTDOWN of MS services was successful.

oracloudceladm04:

oracloudceladm04: Stopping MS services...

oracloudceladm04: The SHUTDOWN of MS services was successful.

oracloudceladm05:

oracloudceladm05: Stopping MS services...

oracloudceladm05: The SHUTDOWN of MS services was successful.

oracloudceladm06:

oracloudceladm06: Stopping MS services...

oracloudceladm06: The SHUTDOWN of MS services was successful.

oracloudceladm07:

oracloudceladm07: Stopping MS services...

oracloudceladm07: The SHUTDOWN of MS services was successful.

oracloudceladm08:

oracloudceladm08: Stopping MS services...

oracloudceladm08: The SHUTDOWN of MS services was successful.

oracloudceladm09:

oracloudceladm09: Stopping MS services...

oracloudceladm09: The SHUTDOWN of MS services was successful.

oracloudceladm10:

oracloudceladm10: Stopping MS services...

oracloudceladm10: The SHUTDOWN of MS services was successful.

oracloudceladm11:

oracloudceladm11: Stopping MS services...

oracloudceladm11: The SHUTDOWN of MS services was successful.

oracloudceladm12:

oracloudceladm12: Stopping MS services...

oracloudceladm12: The SHUTDOWN of MS services was successful.

oracloudceladm13:

oracloudceladm13: Stopping MS services...

oracloudceladm13: The SHUTDOWN of MS services was successful.

oracloudceladm14:

oracloudceladm14: Stopping MS services...

oracloudceladm14: The SHUTDOWN of MS services was successful.

• Unzip the JDK package

[root@oraclouddbadm01 ~]# cd /u01/app/oracle/software/jdk

[root@oraclouddbadm01 jdk]# ls -ltr p21984104_17091_Linux-x86-64.zip

-rw-r--r-- 1 root root 466264412 Nov 16 09:20 p21984104_17091_Linux-x86-64.zip

[root@oraclouddbadm01 jdk]# unzip p21984104_17091_Linux-x86-64.zip

Archive:  p21984104_17091_Linux-x86-64.zip

  inflating: jdk-7u91-linux-x64.rpm

  inflating: jdk-7u91-linux-x64.tar.gz

  inflating: jre-7u91-linux-x64.rpm

  inflating: jre-7u91-linux-x64.tar.gz

  inflating: readme.txt

  inflating: server-jre-7u91-linux-x64.tar.gz

[root@oraclouddbadm01 jdk]# ls -ltr

total 914016

-rw-r--r-- 1 root root       970 Oct 27 16:52 readme.txt

-r--r--r-- 1 root root 138164723 Oct 27 16:53 jdk-7u91-linux-x64.rpm

-r--r--r-- 1 root root  33765685 Oct 27 16:53 jre-7u91-linux-x64.rpm

-r--r--r-- 1 root root 153623439 Oct 27 16:53 jdk-7u91-linux-x64.tar.gz

-r--r--r-- 1 root root  47188352 Oct 27 16:53 jre-7u91-linux-x64.tar.gz

-r--r--r-- 1 root root  95979441 Oct 27 16:53 server-jre-7u91-linux-x64.tar.gz

-rw-r--r-- 1 root root 466264412 Nov 16 09:20 p21984104_17091_Linux-x86-64.zip

After unzipping the patch, locate the JDK RPM which will be named like "jdk-<version>-linux-x64.rpm" (i.e. jdk-7u91-linux-x64.rpm)

Copy only this rpm file over to the all storage cells. It can be staged in a temporary directory like /tmp.

[root@oraclouddbadm01 jdk]# dcli -l root -g ~/cell_group -f /u01/app/oracle/software/jdk/jdk-7u91-linux-x64.rpm -d /tmp

[root@oraclouddbadm01 jdk]# dcli -l root -g ~/cell_group ls -l /tmp/jdk*

oracloudceladm01: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm02: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm03: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm04: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm05: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm06: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm07: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm08: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm09: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm10: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm11: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm12: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm13: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

oracloudceladm14: -r--r--r-- 1 root root 138164723 Nov 16 12:51 /tmp/jdk-7u91-linux-x64.rpm

• Identify the currently installed JDK version

[root@oraclouddbadm01 jdk]# dcli -l root -g ~/cell_group 'rpm -q jdk'

oracloudceladm01: jdk-1.7.0_85-fcs.x86_64

oracloudceladm02: jdk-1.7.0_85-fcs.x86_64

oracloudceladm03: jdk-1.7.0_85-fcs.x86_64

oracloudceladm04: jdk-1.7.0_85-fcs.x86_64

oracloudceladm05: jdk-1.7.0_85-fcs.x86_64

oracloudceladm06: jdk-1.7.0_85-fcs.x86_64

oracloudceladm07: jdk-1.7.0_85-fcs.x86_64

oracloudceladm08: jdk-1.7.0_85-fcs.x86_64

oracloudceladm09: jdk-1.7.0_85-fcs.x86_64

oracloudceladm10: jdk-1.7.0_85-fcs.x86_64

oracloudceladm11: jdk-1.7.0_85-fcs.x86_64

oracloudceladm12: jdk-1.7.0_85-fcs.x86_64

oracloudceladm13: jdk-1.7.0_85-fcs.x86_64

oracloudceladm14: jdk-1.7.0_85-fcs.x86_64

Currently installed JDK version is jdk-1.7.0_85

• Update the JDK patch on all the storage cells.

[root@oraclouddbadm01 jdk]# dcli -l root -g ~/cell_group 'rpm -Uvh /tmp/jdk-7u91-linux-x64.rpm'

oracloudceladm01: Preparing... ##################################################

oracloudceladm01: jdk          ##################################################

oracloudceladm01: Unpacking JAR files...

oracloudceladm01: rt.jar...

oracloudceladm01: jsse.jar...

oracloudceladm01: charsets.jar...

oracloudceladm01: tools.jar...

oracloudceladm01: localedata.jar...

oracloudceladm01: jfxrt.jar...

oracloudceladm02: Preparing... ##################################################

oracloudceladm02: jdk          ##################################################

oracloudceladm02: Unpacking JAR files...

oracloudceladm02: rt.jar...

oracloudceladm02: jsse.jar...

oracloudceladm02: charsets.jar...

oracloudceladm02: tools.jar...

oracloudceladm02: localedata.jar...

oracloudceladm02: jfxrt.jar...

...

...

...

oracloudceladm14: Preparing... ##################################################

oracloudceladm14: jdk          ##################################################

oracloudceladm14: Unpacking JAR files...

oracloudceladm14: rt.jar...

oracloudceladm14: jsse.jar...

oracloudceladm14: charsets.jar...

oracloudceladm14: tools.jar...

oracloudceladm14: localedata.jar...

oracloudceladm14: jfxrt.jar...

[root@oraclouddbadm01 jdk]#

• Verify the JDK package is updated

[root@oraclouddbadm01 jdk]# dcli -l root -g ~/cell_group 'rpm -q jdk'

oracloudceladm01: jdk-1.7.0_91-fcs.x86_64

oracloudceladm02: jdk-1.7.0_91-fcs.x86_64

oracloudceladm03: jdk-1.7.0_91-fcs.x86_64

oracloudceladm04: jdk-1.7.0_91-fcs.x86_64

oracloudceladm05: jdk-1.7.0_91-fcs.x86_64

oracloudceladm06: jdk-1.7.0_91-fcs.x86_64

oracloudceladm07: jdk-1.7.0_91-fcs.x86_64

oracloudceladm08: jdk-1.7.0_91-fcs.x86_64

oracloudceladm09: jdk-1.7.0_91-fcs.x86_64

oracloudceladm10: jdk-1.7.0_91-fcs.x86_64

oracloudceladm11: jdk-1.7.0_91-fcs.x86_64

oracloudceladm12: jdk-1.7.0_91-fcs.x86_64

oracloudceladm13: jdk-1.7.0_91-fcs.x86_64

oracloudceladm14: jdk-1.7.0_91-fcs.x86_64

JDK version is updated to jdk-1.7.0_91

• Remove the RPM file from /tmp directory

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'rm -f /tmp/jdk-7u91-linux-x64.rpm'

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'ls -l /tmp/jdk-7u91-linux-x64.rpm'

oracloudceladm01: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm02: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm03: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm04: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm05: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm06: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm07: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm08: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm09: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm10: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm11: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm12: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm13: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

oracloudceladm14: ls: cannot access /tmp/jdk-7u91-linux-x64.rpm: No such file or directory

• Redeploy MS so it uses the updated JDK package

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'cellcli -e alter cell shutdown services ms'

oracloudceladm01:

oracloudceladm01: Stopping MS services...

oracloudceladm01: The SHUTDOWN of MS services was successful.

oracloudceladm02:

oracloudceladm02: Stopping MS services...

oracloudceladm02: The SHUTDOWN of MS services was successful.

oracloudceladm03:

oracloudceladm03: Stopping MS services...

oracloudceladm03: The SHUTDOWN of MS services was successful.

oracloudceladm04:

oracloudceladm04: Stopping MS services...

oracloudceladm04: The SHUTDOWN of MS services was successful.

oracloudceladm05:

oracloudceladm05: Stopping MS services...

oracloudceladm05: The SHUTDOWN of MS services was successful.

oracloudceladm06:

oracloudceladm06: Stopping MS services...

oracloudceladm06: The SHUTDOWN of MS services was successful.

oracloudceladm07:

oracloudceladm07: Stopping MS services...

oracloudceladm07: The SHUTDOWN of MS services was successful.

oracloudceladm08:

oracloudceladm08: Stopping MS services...

oracloudceladm08: The SHUTDOWN of MS services was successful.

oracloudceladm09:

oracloudceladm09: Stopping MS services...

oracloudceladm09: The SHUTDOWN of MS services was successful.

oracloudceladm10:

oracloudceladm10: Stopping MS services...

oracloudceladm10: The SHUTDOWN of MS services was successful.

oracloudceladm11:

oracloudceladm11: Stopping MS services...

oracloudceladm11: The SHUTDOWN of MS services was successful.

oracloudceladm12:

oracloudceladm12: Stopping MS services...

oracloudceladm12: The SHUTDOWN of MS services was successful.

oracloudceladm13:

oracloudceladm13: Stopping MS services...

oracloudceladm13: The SHUTDOWN of MS services was successful.

oracloudceladm14:

oracloudceladm14: Stopping MS services...

oracloudceladm14: The SHUTDOWN of MS services was successful.

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'ls -l /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy'

oracloudceladm01: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm02: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm03: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm04: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm05: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm06: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm07: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm08: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm09: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm10: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm11: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm12: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm13: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

oracloudceladm14: -rwxr-xr-x 1 root root 22060 Sep 17 18:14 /opt/oracle/cell/cellsrv/deploy/scripts/unix/setup_dynamicDeploy

[root@oraclouddbadm01 jdk]# dcli -g ~/cell_group -l root 'cd /opt/oracle/cell/cellsrv/deploy/scripts/unix; sh setup_dynamicDeploy'

oracloudceladm01: unzipping wls

oracloudceladm01:

oracloudceladm01: CLASSPATH=/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/default//lib/tools.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic_sp.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/features/weblogic.server.modules_10.3.6.0.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/webservices.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/utils/config/10.3/config-launch.jar::/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbynet.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbyclient.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbytools.jar::

oracloudceladm01:

oracloudceladm01: Initializing WebLogic Scripting Tool (WLST) ...

oracloudceladm01:

oracloudceladm01: Welcome to WebLogic Server Administration Scripting Shell

oracloudceladm01:

oracloudceladm01: Type help() for help on available commands

oracloudceladm01:

oracloudceladm01:

oracloudceladm01:

oracloudceladm01: Exiting WebLogic Scripting Tool.

oracloudceladm01:

oracloudceladm01: /opt/oracle/cell/cellsrv/deploy/scripts/unix

oracloudceladm01: MS now has a new self-signed identity and certificate.

oracloudceladm01: sleep until wls is ready ...

oracloudceladm01:

oracloudceladm01: CLASSPATH=/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/jdk1.7.0_91/lib/tools.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic_sp.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/features/weblogic.server.modules_10.3.6.0.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/webservices.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/jdk1.7.0_91/lib/tools.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic_sp.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/weblogic.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/features/weblogic.server.modules_10.3.6.0.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/webservices.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbyclient.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/xqrl.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbyclient.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/server/lib/xqrl.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/utils/config/10.3/config-launch.jar::/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbynet.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbyclient.jar:/opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/deploy/wls/wlserver_10.3/common/derby/lib/derbytools.jar::

oracloudceladm01:

oracloudceladm01: Initializing WebLogic Scripting Tool (WLST) ...

oracloudceladm01:

oracloudceladm01: Welcome to WebLogic Server Administration Scripting Shell

oracloudceladm01:

oracloudceladm01: Type help() for help on available commands

oracloudceladm01:

oracloudceladm01: Connecting to t3://localhost:8888 with userid weblogic ...

oracloudceladm01: Successfully connected to Admin Server 'msServer' that belongs to domain 'msdomain'.

oracloudceladm01:

oracloudceladm01: Warning: An insecure protocol was used to connect to the

oracloudceladm01: server. To ensure on-the-wire security, the SSL port or

oracloudceladm01: Admin port should be used instead.

oracloudceladm01:

oracloudceladm01: Location changed to edit tree. This is a writable tree with

oracloudceladm01: DomainMBean as the root. To make changes you will need to start

oracloudceladm01: an edit session via startEdit().

oracloudceladm01:

oracloudceladm01: For more help, use help(edit)

oracloudceladm01:

oracloudceladm01: Starting an edit session ...

oracloudceladm01: Started edit session, please be sure to save and activate your

oracloudceladm01: changes once you are done.

oracloudceladm01: Activating all your changes, this may take a while ...

oracloudceladm01: The edit lock associated with this edit session is released

oracloudceladm01: once the activation is completed.

oracloudceladm01:

oracloudceladm01: The following non-dynamic attribute(s) have been changed on MBeans

oracloudceladm01: that require server re-start:

oracloudceladm01: MBean Changed : Security:Name=myrealmMSUserAuthenticator

oracloudceladm01: Attributes changed : ControlFlag

oracloudceladm01:

oracloudceladm01: MBean Changed : Security:Name=myrealm

oracloudceladm01: Attributes changed : AuthenticationProviders

oracloudceladm01:

oracloudceladm01: Activation completed

oracloudceladm01:

oracloudceladm01: weblogic.Deployer invoked with options:  -verbose -name MS -source /opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/lib/CliService_CliServicePortImpl.war -targets msServer -user weblogic -adminURL t3://localhost:8888 -deploy

oracloudceladm01: <Nov 16, 2015 1:10:15 PM CST> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, MS [archive: /opt/oracle/cell12.1.2.2.0_LINUX.X64_150917/cellsrv/lib/CliService_CliServicePortImpl.war], to msServer .>

oracloudceladm01: Task 0 initiated: [Deployer:149026]deploy application MS on msServer.

oracloudceladm01: Task 0 completed: [Deployer:149026]deploy application MS on msServer.

oracloudceladm01: Target state: deploy completed on Server msServer

oracloudceladm01: java.lang.Exception: [Deployer:149169]Requires server restart for completion.

oracloudceladm01:

oracloudceladm01: