Excellent article.We have a DSS application where we run few selects and huge number of inserts.Deletes are far few in between or none.We keep seeing huge ENQ TX index contention on our global indexes(non partitioned).
I am not sure if the indexes are right side heavy as the application inserts random request_ids on the index.in AWR reports we see Branch and leaf node splits happening on the index.Do you think coaelescing the index can be one of the approaches we can take.

Vik

What version of Oracle ? 10.2.0.4 on solaris 10.

Is the drop in response time intermittent, completely random, or consistent ?
It is consistent since we deleted data from the table.

Are the free buffer waits only on the primary key indexes, or on other indexes on the table ? Primary key table is on sequence but we don’t see any waits on it rather the waits are on other big indexes.

Are you not seeing “gc buffer busy waits”, or even “buffer busy waits” ?
we do see both of these waits on the indexes.

Were the deletes for old (i.e. low key value) data, or were they across the entire range of sequence values ?
Deletes were to purge old data. That data won’t be inserted any time soon or may be never at all.

Are any of the indexes reverse key (which some people do with sequence-based indexes)?
No we don’t use reverse key indexes at all.

Are the sequences cache/nocache, order/noorder. If they are cached is it the default cache or a large cache ?
Sequences are cached with cahce value more than 10000.

Can you confirm you really did mean all your indexes are set with INITRANS = 100 ? (Just in case you meant the tables had initrans = 100, or only the sequence-based indexes)?
Please see below:-

stats after index rebuild:-

SQL> select INDEX_NAME,INI_TRANS,MAX_TRANS,NUM_ROWS,AVG_LEAF_BLOCKS_PER_KEY,AVG_DATA_BLOCKS_PER_KEY from dba_indexes where table_name='CENRESPONSEDETAIL_1';

INDEX_NAME                      INI_TRANS  MAX_TRANS   NUM_ROWS AVG_LEAF_BLOCKS_PER_KEY AVG_DATA_BLOCKS_PER_KEY
------------------------------ ---------- ---------- ---------- ----------------------- -----------------------
PK_CENRESPONSEDETAIL_1                100        255 3451932737                       1                       1
IDX_CRD1_1                            100        255 3506713096                      88                   19759
IDX_CRD1_2                            100        255 2157529757                      19                    2736
IDX_CRD1_3                            100        255 3471412663                       1                       1
IDX_CRD1_4                            100        255 3503166835                       1                       1
IDX_CRD1_5                            100        255 3498456992                       1                       1
IDX_CRD1_6                            100        255 3504149465                       1                       1

7 rows selected.

SQL> select TABLE_NAME,PCT_FREE,INI_TRANS,MAX_TRANS,NUM_ROWS,BLOCKS ,EMPTY_BLOCKS,AVG_ROW_LEN from dba_tables where table_name='CENRESPONSEDETAIL_1';

TABLE_NAME                       PCT_FREE  INI_TRANS  MAX_TRANS   NUM_ROWS     BLOCKS EMPTY_BLOCKS AVG_ROW_LEN
------------------------------ ---------- ---------- ---------- ---------- ---------- ------------ -----------
CENRESPONSEDETAIL_1                    10        100        255 3475661637   36810214        42559          51

SQL> select segment_name,bytes/(1048576*1024) from dba_segments where segment_name='CENRESPONSEDETAIL_1';

SEGMENT_NAME                                                                      BYTES/(1048576*1024)
--------------------------------------------------------------------------------- --------------------
CENRESPONSEDETAIL_1                                                                          285.15625

SQL> select segment_name,bytes/(1048576*1024) from dba_segments where segment_name in (select INDEX_NAME from dba_indexes where table_name='CENRESPONSEDETAIL_1');

SEGMENT_NAME                                                                      BYTES/(1048576*1024)
--------------------------------------------------------------------------------- --------------------
PK_CENRESPONSEDETAIL_1                                                                      113.769531
IDX_CRD1_1                                                                                  90.8203125
IDX_CRD1_2                                                                                  102.539063
IDX_CRD1_3                                                                                  167.480469
IDX_CRD1_4                                                                                  170.410156
IDX_CRD1_5                                                                                  167.480469
IDX_CRD1_6                                                                                  181.152344

7 rows selected.
=======================================
Same stats before the index rebuild
========================================
SQL> select segment_name,bytes/(1048576*1024) from dba_segments where segment_name in (select INDEX_NAME from dba_indexes where table_name='CENRESPONSEDETAIL_1');
SEGMENT_NAME			BYTES/(1048576*1024)    
------------------------        --------------------    
PK_CENRESPONSEDETAIL_1          112.792969
IDX_CRD1_1		        163.085938
IDX_CRD1_2		        101.5625
IDX_CRD1_3                      165.527344
IDX_CRD1_4		        257.8125
IDX_CRD1_5	                166.992188
IDX_CRD1_6                      186.035156

7 rows selected.

SQL> select index_name, LEAF_BLOCKS,DISTINCT_KEYS,AVG_LEAF_BLOCKS_PER_KEY,AVG_DATA_BLOCKS_PER_KEY,CLUSTERING_FACTOR,NUM_ROWS from dba_indexes where table_name='CENRESPONSEDETAIL_1';

INDEX_NAME                LEAF_BLOCKS DISTINCT_KEYS AVG_LEAF_BLOCKS_PER_KEY AVG_DATA_BLOCKS_PER_KEY CLUSTERING_FACTOR   NUM_ROWS
------------------------- ----------- ------------- ----------------------- ----------------------- ----------------- ----------
PK_CENRESPONSEDETAIL_1       14575087    3451932737                       1                       1         505733620 3451932737
IDX_CRD1_1                   21283897        130738                     162                   20111        2629293623 3482095807
IDX_CRD1_2                   13245500        685024                      19                    2736        1874251820 2157529757
IDX_CRD1_3                   21507130     282013977                       1                       1         480053723 3471412663
IDX_CRD1_4                   33518250    1833751750                       1                       1         661135787 3468067940
IDX_CRD1_5                   21789717    2745574118                       1                       1        1419234050 3481621950
IDX_CRD1_6                   24245970    1857577218                       1                       1        1372502833 3482156097

7 rows selected.   

Did you do the index rebuilds immediately after the deletes, or did you rebuild the indexes because performance got worse and you thought a rebuild would help ?
We did the index rebuild after 2 weeks of deleting the data when performance didn’t improve.

Do you have any idea which indexes are subject to most (or longest) FB waits and ITL waits ?
Yes, IDX_CRD1_4 ,IDX_CRD1_5 and IDX_CRD1_6

Have you checked v$segstat for object level statistics ?

I guess you mean v$segment_statistics. Yes that’s where we found ITL and other waits.

Are you using a the keep and recycle caches ?

no we are not using keep or recycle pool.

There’s not really enough detail in your comments to make a good diagnosis, and it would be easy to mis-interpret what you’ve said.

What version of Oracle ?

Is the drop in response time intermittent, completely random, or consistent ?

Are the free buffer waits only on the primary key indexes, or on other indexes on the table ?

Are you not seeing “gc buffer busy waits”, or even “buffer busy waits” ?

Were the deletes for old (i.e. low key value) data, or were they across the entire range of sequence values ?

Are any of the indexes reverse key (which some people do with sequence-based indexes)?

Are the sequences cache/nocache, order/noorder. If they are cached is it the default cache or a large cache ?

Can you confirm you really did mean all your indexes are set with INITRANS = 100 ? (Just in case you meant the tables had initrans = 100, or only the sequence-based indexes)?

Did you do the index rebuilds immediately after the deletes, or did you rebuild the indexes because performance got worse and you thought a rebuild would help ?

Do you have any idea which indexes are subject to most (or longest) FB waits and ITL waits ?

Have you checked v$segstat for object level statistics ?

Are you using a the keep and recycle caches ?

We are a OLTP shop and use sequences for primary keys so most of our indexes are right hand. Our environment is 2 node RAC on Solaris and we use ASM/ASSM. Recently we deleted some data from one of our OLTP tables and started seeing huge drop in website response time. I see lots of node splits, ITL waits and Free buffer waits on the indexes for this table. The inittans is set to 100 and maxtrans is set to 255 with pctfree 10%. We did rebuild the indexes after the deletion. One of the transaction having a problem includes insert into this table followed by update. The avg row size is around 25 bytes for most of the indexes on this table. So my question is – what else can we do to reduce the no. of node splits. Would rebuilding the index with higher pctfree would help.

Thanks
Parvesh