Saturday, April 27, 2013

ORA-07445: exception encountered: core dump [ddfnetCFull()+2040] [SIGSEGV] [Address not mapped to o bject] [0xFFFFFFFF7B4800A0] [] []



ORA-07445: exception encountered: core dump [ddfnetCFull()+2040] [SIGSEGV] [Address not mapped to o bject] [0xFFFFFFFF7B4800A0] [] []

Problem Description: 

Getting the following error while inserting the records over the dblink:

ORA-07445: exception encountered: core dump [ddfnetCFull()+2040] [SIGSEGV] [Address not mapped to o
bject] [0xFFFFFFFF7B4800A0] [] []


Symptoms:

Database Alert Files shows:


Errors in file /oradg2/orasys/admin/adp2/udump/adp_ora_6440.trc:
ORA-07445: exception encountered: core dump [ddfnetCFull()+2040] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFF7B1800A0] [] []


Database Trace File shows:


*** SERVICE NAME:(SYS$USERS) 2009-12-22 07:29:48.796
*** SESSION ID:(2256.34220) 2009-12-22 07:29:48.796
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0xffffffff7b4800a0, PC: [0x102450e98, ddfnetCFull()+2040]
*** 2009-12-22 07:29:48.840
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [ddfnetCFull()+2040] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFF7B4800A0] [] []
Current SQL statement for this session:


INSERT INTO Table_Name From Table_Name@DBLINK


Explanation:


Problem caused due to in-doubt transactions exists in the database.

Solution:

purging pending transactions would rectify  ora-7445


1. Identify the id of the transaction:

SQL> SELECT LOCAL_TRAN_ID, GLOBAL_TRAN_ID FROM DBA_2PC_PENDING;

If there are entries returned by the above query then,

2. Try to commit/ rollback these failed transactions :

To Commit:

SQL>commit force;

To Rollback :

SQL> rollback force;

3. If still no success, then you need to purge these transactions:

SQL> alter system enable distributed recovery;

SQL> alter system set "_smu_debug_mode" = 4;
SQL> EXECUTE DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY('');
SQL> COMMIT;

4. Confirm that the transaction has been purged:

SQL> SELECT LOCAL_TRAN_ID, GLOBAL_TRAN_ID FROM DBA_2PC_PENDING;


Oracle Buffer Busy Wait Event



Oracle Buffer Busy Wait Event

#LR14217
The buffer busy wait event happens when a session tries to access a block in the buffer cache but it can't because the buffer is busy, because another session is modifying the block and the contents of the block are in flux. To guarantee that the reader has a coherent image of the block with either all of the changes or none of the changes, the session modifying the block marks the block header with a flag letting other users know a change is taking place and to wait until the complete change is applied.
What causes the buffer busy wait event
The two main cases where the buffer busy wait occurs are:
  • Another session is reading the block into the buffer - this specific case has been split out into a separate read by other session wait event in 10g and higher.
  • Another session holds the buffer in an incompatible mode to our request.
While the block is being changed, the block is marked as unreadable by others. The changes that are being made should last under a few hundredths of a second, for example, a disk read should be under 20 milliseconds and a block modification should be under one millisecond. Therefore it will take a lot of buffer busy waits to cause a problem, but some examples of this are:
  • Hot block issue, such as the first block on the free list of a table, with high concurrent inserts. All users will insert into that block at the same time, until it fills up, then users start inserting into the next free block on the list, and so on.
  • Multiple users running an inefficient SQL statement performing a full table scan on the same large table at the same time. One user will read the block off disk, and the other users will wait on buffer busy waits (or read by other session in 10g and higher) for the physical I/O to complete.
Getting more information about buffer busy waits
To get more information about the SQL statement being executed and the block being waited for, trace the session or gather data from V$SESSION V$SESSION_WAIT (or just V$SESSION in 10g and higher):
SELECT s.sql_hash_value, sw.p1 file#, sw.p2 block#, sw.p3 reason
FROM v$session_wait sw, v$session s
WHERE sw.event = 'buffer busy waits'
AND sw.sid = s.sid
  • P1 - file number of the data file containing the block being waited for
  • P2 - block number of the block being waited for
  • P3 - this wait is called from many different sections of Oracle code and each uses their own reason code which differ among versions
To determine the object being waited for, use the P1 (file_number) and P2 (block_number) information from the above query:

SELECT  owner , segment_name , segment_type
FROM  dba_extents
WHERE  file_id = &FileNumber
AND  &BlockNumber BETWEEN block_id AND block_id + blocks -1;

Another query that can be very useful is finding the objects in the entire Oracle database that are suffering from "buffer busy waits". The following query gives the top 10 segments:

SELECT * FROM (
   SELECT owner, object_name, subobject_name, object_type,
          tablespace_name, value
   FROM v$segment_statistics
   WHERE statistic_name='buffer busy waits'
   ORDER BY value DESC)
WHERE ROWNUM <=10

Fixing buffer busy waits

Once the database object is known, consider the following causes of contention and their solutions.
  • Undo Header - If using Automatic Undo Management (AUM), increase the size of the undo tablespace. If not using AUM, add more rollback segments.
  • Undo Block - If using AUM, increase size of the undo tablespace. If not using AUM, increase rollback segment sizes.
  • Data Block- Data blocks are the blocks that hold the row data in a table or index. The typical problem is that multiple sessions are requesting a block that is either not in cache or in an incompatible mode (read by other session in 10g and higher).
    • Tune inefficient queries that read too many blocks into the buffer cache. These queries could flush out blocks that may be useful for other sessions in the buffer cache. By tuning queries, the number of blocks that need to be read into the cache is minimized, reducing aging out of the existing "good" blocks in the cache.
    • Resolve Hot Blocks - If the queries above consistently return the same block or set of blocks, this is considered a hot block scenario. Delete some of the hot rows and insert them back into the table. Most of the time, the rows will be placed in a different block. The DBA may need to adjust PCTFREE and/or PCTUSED to ensure the rows are placed into a different block. Also talk with the developers to understand why a set of blocks are hot.
    • Place Table in Memory - Cache the table or keep the table in the KEEP POOL. When multiple sessions are requesting the blocks that reside in the disk, it takes too much time for a session to read it into the buffer cache. Other session(s) that need the same block will register 'buffer busy wait'. If the block is already in buffer cache, however, this possibility is eliminated. Another alternative is to increase the buffer cache size. A larger buffer cache means less I/O from disk. This reduces situations where one session is reading a block from the disk subsystem and other sessions are waiting for the block.
    • Fix Low Cardinality Indexes - Look for ways to reduce the number of low cardinality indexes, i.e. an index with a low number of unique values that could result in excessive block reads. This can especially be problematic when concurrent DML operates on table with low cardinality indexes and cause contention on a few index blocks.
  • Data Segment Header- Each segment has one header block that contains segment information, e.g. free and available block details and the highwater mark. At times, this block can be a point of contention when multiple sessions are attempting to insert/delete into/from the same table.
    • Adjust PCTFREE/PCTUSED or use ASSM - When sessions insert/delete rows into/from a block, the block must be taken out of the freelist if the PCTFREE threshold reached. When sessions delete rows from a block, the block will be put back in the freelist if PCTUSED threshold is reached. If there are a lot of blocks coming out of the freelist or going into it, all those sessions have to make that update in the freelist map in the segment header. A solution to this problem is to create multiple freelists. This will allow different insert streams to use different freelists and thus update different freelist maps. This reduces contention on the segment header block. You should also look into optimizing the PCTUSED/PCTFREE parameters so that the blocks don't go in and out of the freelists frequently. Another solution is to use ASSM which avoids the use of freelists all together.
    • Increase Extent Size - If extents are too small, Oracle must constantly allocate new extents causing contention in the extent map

WARNING: inbound connection timed out (ORA-3136)



 WARNING: inbound connection timed out (ORA-3136) 


The "WARNING: inbound connection timed out (ORA-3136)" in the alert log indicates that the client was not able to complete it's authentication within the period of time specified by parameter SQLNET.INBOUND_CONNECT_TIMEOUT.

You may also witness ORA-12170 without timeout error on the database server sqlnet.log file.This entry would also have the client address which failed to get authenticated. Some applications or JDBC thin driver applications may not have these details.

From 10.2 onwards the default value of this parameter is 60 seconds, hence if the client is not able authenticate within 60 secs , the warning would appear in the alert log and the client connection will be terminated.

This timeout restriction was introduced to combat Denial of Service (DoS) attack whereby malicious clients attempt to flood database servers with connect requests that consumes resources.


Cause:

There can be three main reasons for this error

  1. Server gets a connection request from a malicious client which is not supposed to connect to the database , in which case the error thrown is the correct behavior. You can get the client address for which the error was thrown via sqlnet log file.
  2. The server receives a valid client connection request but the client takes a long time to authenticate more than the default 60 seconds.
  3. The DB server is heavily loaded due to which it cannot finish the client logon within the timeout specified.

Diagnosis: 

The default value of 60 seconds is good enough in most conditions for the database server to authenticate a client connection. If its taking longer period, then its worth checking all the below points before going for the workaround:

1. Check whether local connection on the database server is successful & quick.

2. If local connections are quick ,then check for underlying network delay with the help of your network administrator.

3. Check whether your Database performance has degraded by anyway.

4. Check alert log for any critical errors for eg, ORA-600 or ORA-7445 and get them resolved first.

Resolution:
 
These critical errors might have triggered the slowness of the database server.

As a workaround to avoid only this warning messages, you can set the parameters SQLNET.INBOUND_CONNECT_TIMEOUT and INBOUND_CONNECT_TIMEOUT_listenername
to the value more than 60.


In server side sqlnet.ora file add SQLNET.INBOUND_CONNECT_TIMEOUT

SQLNET.INBOUND_CONNECT_TIMEOUT = 120

In listener.ora file INBOUND_CONNECT_TIMEOUT_listenername

INBOUND_CONNECT_TIMEOUT_LISTENER = 110

From Oracle version 10.2.0.3 onwards the default value of INBOUND_CONNECT_TIMEOUT_ is 60 seconds. For previous releases it is zero by default.