Early on when I started on this issue I did do some index dumps and could clearly see that we had lots of blocks with only 1 or 2 records after our mass delete jobs. I have provided Oracle Support with this information as well as oradump files while the problem is occurring, but they don’t seem to be able to find anything wrong so far.

If you are interested in seeing if you are experiencing a high ‘failed probes on index block reclamation’ event run the query below.

select SS.snap_id,
SS.stat_name,
TO_CHAR(S.BEGIN_INTERVAL_TIME, ‘DAY’) DAY,
S.BEGIN_INTERVAL_TIME,
S.END_INTERVAL_TIME,
SS.value,
SS.value – LAG(SS.VALUE, 1, ss.value) OVER (ORDER BY SS.SNAP_ID) AS DIFF
from DBA_HIST_SYSSTAT SS,
DBA_HIST_SNAPSHOT S
where S.SNAP_ID = SS.SNAP_ID
AND SS.stat_NAME = ‘failed probes on index block reclamation’
ORDER BY SS.SNAP_ID ;

It’s not a good idea to use “analyze index validate structure” on a production index as the command locks the underlying table. It’s also not very useful for your test case, since all it gives is average stats, and you need more detail. For testing, you might like to check the effects of a treedump (on your little test case first) as this will tell you which blocks are still part of the index structure, and how many rows are in each.

I believe that there is a correlation between the enq TX – index contention wait event and a spike in the number of ‘failed probes on index block reclamation. I have specifically asked Oracle to explain why there is a spike in the ‘failed probes on index block reclamation’ during the same time frame as the enq TX index contention wait event, but they have not answered my question.

I was hoping that some investigation by Oracle Support into the failed probes metric might get someone on the right track to discovering the bug. That hasn’t happened though.

As you can see from this posting, and from a related posting mentioned in comment 3 above, there are at least two reasons why you could see “Enq TX: index contention” if you do large deletes; and the advice to deal with the two different problems depends very strongly on which problem you are seeing.

I hope that the Oracle support consultant explained why the coalesce might help – and pointed out the bits of information in your SR that took him to that conclusion.

As far as your example is concerned the most important point is that it’s irrelevant if it isn’t a good match for your production scenario – do you really delete every other row from a large sequential section of your data; do you even get to a position where a number of deletions over a period of days leaves your data in that pattern ?

A couple of comments on your results:

The first “no change” after deleting 25% of the data is probably because you hadn’t issued a commit at that point.

The “free blocks increased” after the first coalesce doesn’t match the perfect scenario – which would have been about 30 at full, and about 10 at FS1 – but the timing for bits being set and cleared in ASSM is an ongoing problem with Oracle and changes with version. We hope for some blocks to be cleared, though, because you have made a number of adjacent leaf blocks half empty, so some of those should be emptied and their contents combined with the next leaf block (i.e. instead of 20 half full blocks you hope to see 10 empty and 10 full).

The “free blocks increased again”. You’ve deleted the rest of the top half of the index and committed (and the bitmaps have been updated). So where you have approximatly 40 full leaf blocks you now have 20 full and 20 empty.

The “after coalesce, nothing changed”. Because you didn’t have any partly empty leaf blocks that could be the subject of a coalesce.

After rebuild – you had a significant difference in the FS2 and Full blocks: but I’d be inclined to put that down to the weakness of the ASSM bitmap updates combined with the way in which the pctfree setting for indexes works – not to mention a couple of little oddities that show up when you’re dealing with small indexes. If you want to analyse exactly what’s going on you need to do a load of block dumps of the bitmap blocks and the index blocks as you work.

To answer your questions:
The free blocks increase after the coalesce because that’s what the coalesce command is designed to do: free up blocks which can be made free. More significantly, though, it’s supposed to take completely empty blocks out of the index structure and the nasty thing about your example is that it suggests that none of the index leaf blocks has become empty when clearly they should have been. You need to do some block dumps to find out whether this is just a defect in the bitmap report, or whether the blocks really are still in the index structure even though they are empty.

The coalesce MAY resolve the contention if the contention is caused by the behaviour described in my posting. But if the coalesce is leaving empty blocks in the index structure then it MAY have no beneficial effect whatsoever.

The coalesce command does not lock the table.

  19:46:45 SQL> create table test(t1 int) tablespace datatb;

 Table created.
SQL> create index ind_t1 on test(t1);

  Index created.

Elapsed: 00:00:00.01
19:47:08 SQL> begin
19:47:17   2    for i in 1..20000 loop
19:47:17   3      insert into test values(i);
19:47:17   4      commit;
19:47:17   5      end loop;
19:47:17   6      end;
19:47:18   7  /

PL/SQL procedure successfully completed.
SQL> set serveroutput on;
SQL> declare
  2 
  3     l_fs1_bytes number;
  4     l_fs2_bytes number;
  5     l_fs3_bytes number;
  6     l_fs4_bytes number;
  7     l_fs1_blocks number;
  8     l_fs2_blocks number;
  9     l_fs3_blocks number;
 10     l_fs4_blocks number;
 11     l_full_bytes number;
 12     l_full_blocks number;
 13     l_unformatted_bytes number;
 14     l_unformatted_blocks number;
 15  begin
 16     dbms_space.space_usage(
 17        segment_owner      => user,
 18        segment_name       => 'IND_T1',
 19        segment_type       => 'INDEX',
 20        fs1_bytes          => l_fs1_bytes,
 21        fs1_blocks         => l_fs1_blocks,
 22        fs2_bytes          => l_fs2_bytes,
 23        fs2_blocks         => l_fs2_blocks,
 24        fs3_bytes          => l_fs3_bytes,
 25        fs3_blocks         => l_fs3_blocks,
 26        fs4_bytes          => l_fs4_bytes,
 27        fs4_blocks         => l_fs4_blocks,
 28        full_bytes         => l_full_bytes,
 29        full_blocks        => l_full_blocks,
 30        unformatted_blocks => l_unformatted_blocks,
 31        unformatted_bytes  => l_unformatted_bytes
 32     );
 33     dbms_output.put_line(' FS1 Blocks = '||l_fs1_blocks||' Bytes = '||l_fs1_bytes);
 34     dbms_output.put_line(' FS2 Blocks = '||l_fs2_blocks||' Bytes = '||l_fs2_bytes);
 35     dbms_output.put_line(' FS3 Blocks = '||l_fs3_blocks||' Bytes = '||l_fs3_bytes);
 36     dbms_output.put_line(' FS4 Blocks = '||l_fs4_blocks||' Bytes = '||l_fs4_bytes);
 37     dbms_output.put_line('Full Blocks = '||l_full_blocks||' Bytes = '||l_full_bytes);
 38  end;
 39  /
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 4 Bytes = 32768
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 39 Bytes = 319488

PL/SQL procedure successfully completed.

Full blocks number is 39 ; fs2 is 4

SQL> delete test where t1>10000 and (mod(t1,2)!=0);

5000 rows deleted.
Now :
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 4 Bytes = 32768
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 39 Bytes = 319488
nothing changed .

SQL> alter index ind_t1 coalesce;  

Index altered.
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 13 Bytes = 106496
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 30 Bytes = 245760

the free blocks increased

SQL> delete test where t1>10000;

5000 rows deleted.

SQL> commit;

Commit complete.

Now :
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 23 Bytes = 188416
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 20 Bytes = 163840
after full block delete ,the free blocks increased again.

SQL> alter index ind_t1 coalesce;

Index altered.

after coalesc , nothing changed:
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 23 Bytes = 188416
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 20 Bytes = 163840

but rebuild:
SQL> alter index ind_t1 rebuild;

Index altered.
FS1 Blocks = 0 Bytes = 0
FS2 Blocks = 1 Bytes = 8192
FS3 Blocks = 0 Bytes = 0
FS4 Blocks = 0 Bytes = 0
Full Blocks = 21 Bytes = 172032

what i want to ask, why the free blocks increased after coalesce? Can coalesce really resolve tx:index contention? If coalesce will lock table?