High CPU caused by SYS (E000/EMNC background process) 15 July 2010Posted by David Alejo Marcos in Grid Control, Oracle 11.2, Tuning.
Tags: Grid Control, Oracle 11.2, Tuning
One thing I have learned about being a production DBA is that our SLAs to fix problems are pretty much non-existent and troubleshooting skills are a “good thing to have”.
Today we had a problem on one of our most critical systems. This is a real-time database used by many clients and poor performance means our clients misses their own SLAs.
Several calls and many alerts regarding high CPU on the production box indicated something was not running as expected on production. I had a quick look and it did not look healthy:
Note.- the lower load was after I solved the problem.
I did also have a look to the sys session (E000) to see the load and gather as much information as I could:
EMON is used by streams, bearing in mind we do not use streams technology as a such, the only explanation was Oracle AQ.
Another clue to the solution was the ASH report, indicating Oracle was spending too much time on the following event “Wait for EMON to process ntfns”:
The EMON process was spinning, consuming resources needed for other processes. The only solution we have at this point in time (at least until Oracle releases the new PSU) is to kill the process.
The background process responsible is called EMNC, and this process spawns other process labelled E000,…
First of all, we need the PID of the process running on the server
[oracle@xxxxxx ~]$ ps -ef | grep -i emn
oracle 5952 1 0 Jul03 ? 00:00:00 ora_emnc_MFE
oracle 6667 5963 0 08:03 pts/8 00:00:00 grep -i emn
once we have the process, we proceed to kill it:
[oracle@xxxxx ~]$ kill -9 5952
and we check it re-spawns:
[oracle@xxxxxxx ~]$ ps -ef | grep -i emn
oracle 6706 5963 0 08:04 pts/8 00:00:00 grep -i emn
[oracle@xxxxx ~]$ ps -ef | grep -i emn
oracle 6739 1 2 08:04 ? 00:00:00 ora_emnc_MFE
oracle 6955 5963 0 08:04 pts/8 00:00:00 grep -i emn
Once the process was killed the load on the database decreased dramatically and our alerts cleared.
Note.- Sometimes killing emnc process is not enough and spawned process will also need to be terminated:
The rule of thumb is to check emnc process has restarted, if after 1 min it has not, the proceed to terminal e00? processes:
[oracle@ellison ~]$ ps -ef | grep -i e00
oracle 23537 1 7 07:34 ? 00:00:00 ora_e000_MFE
oracle 23539 1 7 07:34 ? 00:00:00 ora_e001_MFE
oracle 23541 1 32 07:34 ? 00:00:00 ora_e002_MFE
oracle 23543 1 13 07:34 ? 00:00:00 ora_e003_MFE
oracle 23545 1 54 07:34 ? 00:00:00 ora_e004_MFE
oracle 23547 5963 0 07:34 pts/8 00:00:00 grep -i e00
[oracle@xxxxxxx ~]$ kill -9 23537 23539 23541 23543 23545
as always, comments are welcome.