In the past I have enthused mightily about the benefits of the approximate NDV mechanism and the benefit of using auto_sample_size to collect statistics in 11g; however, as so often happens with Oracle features, there’s a down-side or boundary condition, or edge case. I’ve already picked this up once as an addendum to an earlier blog note on virtual stats, which linked to an article on OTN describing how the time taken to collect stats on a table increased dramatically after the addition of an index – where the index had this definition:
create bitmap index i_s_rmp_eval_csc_msg_actions on
s_rmp_evaluation_csc_message (
decode(instr(xml_message_text,' '),0,0,1)
)
;
As you might guess from the column name, this is an index based on an XML column, which is stored as a CLOB.
In a similar vein, I showed you a few days ago an old example I had of indexing a CLOB column with a call to dbms_lob.getlength(). Both index examples suffer from the same problem – to support the index Oracle creates a hidden (virtual) column on the table that can be used to hold statistics about the values of the function; actual calculated values for the function call are stored in the index but not on the table itself – but it’s important that the optimizer has the statistics about the non-existent column values.
So what happens when Oracle collects table statistics – if you’ve enable the approximate NDV feature Oracle does a 100% sample, which means it has to call the function for every single row in the table. You will appreciate that the decode(instr()) function on the LOB column is going to read every single LOB in turn from the table – it’s not surprising that the time taken to calculate stats on the table jumped from a few minutes to a couple of hours. What did surprise me was that my call to dbms_lob.getlength() also seemed to read every lob in my example rather than reading the “LOB Locator” data that’s stored in the row – one day I’ll take a look into why that happened.
Looking at these examples it’s probably safe to conclude that if you really need to index some very small piece of “flag” information from a LOB it’s probably best to store it as a real column on the table – perhaps populated through a trigger so you don’t have to trust every single piece of front-end code to keep it up to date. (It would be quite nice if Oracle gave us the option for a “derived” column – i.e. one that could be defined in the same sort of way as a virtual column, with the difference that it should be stored in the table.)
So virtual columns based on LOBs can create a performance problem for the approximate NDV mechanism; but the story doesn’t stop there because there’s another “less commonly used” feature of Oracle that introduces a different threat – with no workaround – it’s the index organized table (IOT). Here’s a basic example:
create table iot1 (
id1 number(7,0),
id2 number(7,0),
v1 varchar2(10),
v2 varchar2(10),
padding varchar2(500),
constraint iot1_pk primary key(id1, id2)
)
organization index
including id2
overflow
;
insert into iot1
with generator as (
select --+ materialize
rownum id
from dual
connect by
level <= 1e4
)
select
mod(rownum,20) id1,
trunc(rownum,100) id2,
to_char(mod(rownum,20)) v1,
to_char(trunc(rownum,100)) v2,
rpad('x',500,'x') padding
from
generator v1,
generator v2
where
rownum <= 1e5
;
commit;
alter system flush buffer_cache;
alter session set events '10046 trace name context forever';
begin
dbms_stats.gather_table_stats(
ownname => user,
tabname =>'IOT1',
method_opt => 'for all columns size 1'
);
end;
/
alter session set events '10046 trace name context off';
You’ll notice I’ve created the table then inserted the data – if I did a “create table as select” Oracle would have sorted the data before inserting it, and that would have helped to hide the problem I’m trying to demonstrate. As it is my overflow segment is very badly ordered relative to the “top” (i.e. index) segment – in fact I can see after I’ve collected stats on the table that the clustering_factor on the index is 100,000 – an exact match for the rows in the table.
Running 11.2.0.4, with a 1MB uniform extent, freelist management, and 8KB block size the index segment held 279 leaf blocks, the overflow segment (reported in view user_tables as SYS_IOT_OVER_81594) held 7,144 data blocks.
So what interesting things do we find in a 10046 trace file after gathering stats – here are the key details from the tkprof results:
SQL ID: 7ak95sy9m1s4f Plan Hash: 1508788224
select /*+ full(t) no_parallel(t) no_parallel_index(t) dbms_stats
cursor_sharing_exact use_weak_name_resl dynamic_sampling(0) no_monitoring
no_substrb_pad */to_char(count("ID1")),to_char(substrb(dump(min("ID1"),16,
0,32),1,120)),to_char(substrb(dump(max("ID1"),16,0,32),1,120)),
to_char(count("ID2")),to_char(substrb(dump(min("ID2"),16,0,32),1,120)),
to_char(substrb(dump(max("ID2"),16,0,32),1,120)),to_char(count("V1")),
to_char(substrb(dump(min("V1"),16,0,32),1,120)),
to_char(substrb(dump(max("V1"),16,0,32),1,120)),to_char(count("V2")),
to_char(substrb(dump(min("V2"),16,0,32),1,120)),
to_char(substrb(dump(max("V2"),16,0,32),1,120)),to_char(count("PADDING")),
to_char(substrb(dump(min("PADDING"),16,0,32),1,120)),
to_char(substrb(dump(max("PADDING"),16,0,32),1,120))
from
"TEST_USER"."IOT1" t /* NDV,NIL,NIL,NDV,NIL,NIL,NDV,NIL,NIL,NDV,NIL,NIL,NDV,
NIL,NIL*/
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 0.37 0.37 7423 107705 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 0.37 0.37 7423 107705 0 1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 62 (recursive depth: 1)
Number of plan statistics captured: 1
Rows (1st) Rows (avg) Rows (max) Row Source Operation
---------- ---------- ---------- ---------------------------------------------------
1 1 1 SORT AGGREGATE (cr=107705 pr=7423 pw=0 time=377008 us)
100000 100000 100000 APPROXIMATE NDV AGGREGATE (cr=107705 pr=7423 pw=0 time=426437 us cost=10 size=23944 card=82)
100000 100000 100000 INDEX FAST FULL SCAN IOT1_PK (cr=107705 pr=7423 pw=0 time=298380 us cost=10 size=23944 card=82)(object id 85913)
********************************************************************************
SQL ID: 1ca2ug8s3mm5z Plan Hash: 2571749554
select /*+ no_parallel_index(t, "IOT1_PK") dbms_stats cursor_sharing_exact
use_weak_name_resl dynamic_sampling(0) no_monitoring no_substrb_pad
no_expand index(t,"IOT1_PK") */ count(*) as nrw,count(distinct
sys_op_lbid(85913,'L',t.rowid)) as nlb,null as ndk,
sys_op_countchg(sys_op_lbid(85913,'O',"V1"),1) as clf
from
"TEST_USER"."IOT1" t where "ID1" is not null or "ID2" is not null
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 0.16 0.16 0 100280 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 0.16 0.16 0 100280 0 1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 62 (recursive depth: 1)
Number of plan statistics captured: 1
Rows (1st) Rows (avg) Rows (max) Row Source Operation
---------- ---------- ---------- ---------------------------------------------------
1 1 1 SORT GROUP BY (cr=100280 pr=0 pw=0 time=162739 us)
100000 100000 100000 INDEX FULL SCAN IOT1_PK (cr=100280 pr=0 pw=0 time=164597 us cost=6 size=5900000 card=100000)(object id 85913)
The first query collects table and column stats, and we can see that the approximate NDV method has been used because of the trailing text: /* NDV,NIL,NIL,NDV,NIL,NIL,NDV,NIL,NIL,NDV,NIL,NIL,NDV,NIL,NIL*/. In this statement the hint /*+ full(t) */ has been interpreted to mean an index fast full scan, which is what we see in the execution plan. Although there are only 279 blocks in the index and 7,144 blocks in the overflow we’ve done a little over 100,000 buffer visits because for every index entry in the IOT top we’ve done a “fetch by rowid” into the overflow segment (the session stats records these as “table fetch continued row”). Luckily I had a small table so all those visits were buffer gets; on a very large table it’s quite possible that a significant fraction of those buffer gets will turn into single block physical reads.
Not only have we done one buffer visit per row to allow us to calculate the approximate NDV for the table columns, we’ve done the same all over again so that we can calculate the clustering_factor of the index. This is a little surprising since the “rowid” for an item in the overflow section is stored in the index segment but (as you can see in the second query in the tkprof output) Oracle has used column v1 (the first in the overflow segment) in the call to the sys_op_countchg() function where the equivalent call for an ordinary index would use t.rowid so, presumably, the code HAS to access the overflow segment. The really strange thing about this is that the same SQL statement has a call to sys_op_lbid() which uses the (not supposed to exist in IOTs) rowid – so it looks as if it ought to be possible for sys_op_countchg() to do the same.
So – big warning on upgrading to 11g: if you’ve got IOTs with overflows and you switch to auto_sample_size and enable approximate NDV then the time taken to gather stats on those IOTs may (depending to a large extent on the data clustering) take much longer than it used to.
