jump to navigation

ORA-15063:ASM discovered an insufficient number of disks for diskgroup …. 16 November 2009

Posted by David Alejo Marcos in ASM.
Tags:
trackback

Last Friday we started a task to move our local standby database from FC to FATA disks. The goal is to reduce costs as this standby database is only used to refresh QA/DEV environments and not as a proper DR.

We have three databases on this server:

1.- prod1 – local standby database

2.- whqaint – QA Integration database

3.- whtest – test database (50GB size)

The first two (prod1 and whqaint) are a copy of our production database (1.6TB) running on EVA. Whtest was created as a proof of concept (POC) for some testing.

All of them run on 10.2.0.4 and ASM, and all of them have different diskgroups on ASM (data, datb, dat_c).

We started by shutting down all three database, the asm and asmlib.

After the snap was done and the server restarted (to refresh the SCSi bus) I started asmlib.

All looked ok as all disk where listed but, when I queried one of the disk, is when all started to go wrong:

[root@xxxxxxxx init.d]# ./oracleasm querydisk DATA1
Disk “DATA1” is labeled for disk “DATA4”

Well,  something was wrong, but did not know what. The first call was to stop asmlib, disable asmlib, enable asmlib, start asmlib and perform a scandisk.

After these steps were done, the label matched with the Disk. Also a quick check on /dev/oracleasm/disks showed all disks, so far so good.

Once I started the asm, is when I knew it will be a long day:

ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “FRB”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “FRA”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “FRA_C”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATB”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA_C”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “CTRLLOG”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “CTRLLGG”
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “CNTRL_C”

After speaking with the Storage Managed and the SA, all of them confirmed the LUNs were presented and all should be good.

I had a list of the devices before and after the snap, so I could see that it was not the same, the WWIDs were also different, meaning oracleasm should have scanned and recognized the new devices, but it was not the case.

Once thing we have to remember is that, for some reason, oracleasm LVM volumnes.

Oracleasm scans /proc/partitions table to detect block devices with ASM headers.

LVM managed devices are presented to the kernel as /dev/mapper/*, while /proc/partitions are /dev/dm-*

This was the first problem. The good thing is that by scanning those disk again, we managed to fix one of the problems:

[root@xxxxxxxx init.d]# ./oracleasm querydisk data1
Disk “DATA1” is a valid ASM disk

At this point the only way forward I could think of was reinstalling asmlib, something I did not feel comfortable.

I had a look to our ASM spfile:

*.asm_diskgroups=’DATA’,’FRA’,’CTRLLOG’,’CTRLLGG’,’DATB’,’FRB’,’DATA_C’,’CNTRL_C’,’FRA_C’
*.asm_diskstring=’/dev/oracleasm/disks/’

As we were having problems identifying disks on asm_diskgroups, I decided to comment all three entries and start asm. ASM started without problems (as expected).

I then followed a try-and-error approach. Basically, leave asm_diskstring and +ASM.asm_diskgroups commented, create a copy of asm_diskgroups with only one diskgroup (DATA):

#*.asm_diskgroups=’DATA’,’FRA’,’CTRLLOG’,’CTRLLGG’,’DATB’,’FRB’,’DATA_C’,’CNTRL_C’,’FRA_C’

*.asm_diskgroups=’DATA’

#*.asm_diskstring=’/dev/oracleasm/disks/’
And try to start ASM again. And it worked!

To cut a long story short… I kept adding diskgroups until I found the ones failing (DATA_C, CNTRL_C and FRA_C)

Those diskgroups are the ones we created with local disks and, by default, they are mapped as /dev/dm-* (remember when I mentioned oracleasm only scans /proc/partitions and does not scan /dev/dm-*…).

Removing those diskgroups from the init+ASM.ora file allowed us to start both databases (local scandby and qaint).

Devices can be mounted in different ways, Linux (not sure about other platforms) adds those entries to /dev/dm-* by default…

Something to remember ….

Comments»

1. Manish Nashikkar - 9 February 2011

Thanks a lot, U saved my day….

Regards,
Manish Nashikkar.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: