The title is the name of an Oracle hint that came into existence in Oracle 10.2.0.3 and made an appearance recently in a question on the rarely used “My Oracle Support” Community forum (you’ll need a MOS account to be able to read the original). I wouldn’t have found it but the author also emailed me the link asking if I could take a look at it. (If you want to ask me for help – without paying me, that is – then posting a public question in the Oracle (ODC) General Database or SQL forums and emailing me a private link is the strategy most likely to get an answer, by the way.)
The question was about a very simple query using a straightforward index – with a quirky change of plan after upgrading from 10.2.0.3 to 12.2.0.1. Setting the optimizer_features_enable to ‘10.2.0.3’ in the 12.2.0.1 system re-introduced the 10g execution plan. Here’s the query:
SELECT t1.*
FROM DW1.t1
WHERE t1.C1 = '0001'
AND t1.C2 IN ('P', 'F', 'C')
AND t1.C3 IN (
'18110034450001',
'18110034450101',
'18110034450201',
'18110034450301',
'18110034450401',
'18110034450501'
);
Information supplied: t1 holds about 500 million rows at roughly 20 rows per block, the primary key index is (c1, c2, c3, c4), there are just a few values for each of c1, c2 and c4, while c3 is “nearly unique” (which, for clarity, was expanded to “the number of distinct values of c3 is virtually the same as the number of rows in the table”).
At the moment we don’t have any information about histograms and we don’t known whether or not “nearly unique” might still allow a few values of c3 to have a large number of duplicates, so that’s something we might want to follow up on later.
Here are the execution plans – the fast one (from 10g) first, then the slow (12c) plan – and you should look carefully at the predicate section of the two plans:
10g (pulled from memory with rowsource execution statistics enabled)
--------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads |
--------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 6 |00:00:00.01 | 58 | 5 |
| 1 | INLIST ITERATOR | | 1 | | 6 |00:00:00.01 | 58 | 5 |
| 2 | TABLE ACCESS BY INDEX ROWID| T1 | 18 | 5 | 6 |00:00:00.01 | 58 | 5 |
|* 3 | INDEX RANGE SCAN | PK_T1 | 18 | 5 | 6 |00:00:00.01 | 52 | 4 |
--------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("T1"."C1"='0001' AND (("T1"."C2"='C' OR "T1"."C2"='F' OR
"T1"."C2"='P')) AND (("C3"='18110034450001' OR "C3"='18110034450101' OR
"C3"='18110034450201' OR "C3"='18110034450301' OR "C3"='18110034450401' OR
"C3"='18110034450501')))
12c (from explain plan)
---------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 359 | 7 (0)| 00:00:01 |
| 1 | INLIST ITERATOR | | | | | |
| 2 | TABLE ACCESS BY INDEX ROWID BATCHED| T1 | 1 | 359 | 7 (0)| 00:00:01 |
|* 3 | INDEX RANGE SCAN | PK_T1 | 1 | | 6 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("T1"."C1"='0001' AND ("T1"."C2"='C' OR "T1"."C2"='F' OR
"T1"."C2"='P'))
filter("C3"='18110034450001' OR "C3"='18110034450101' OR
"C3"='18110034450201' OR "C3"='18110034450301' OR
"C3"='18110034450401' OR "C3"='18110034450501')
When comparing plans it’s better, of course, to present the same sources from the two systems, it’s not entirely helpful to have the generated plan from explain plan in one version and a run-time plan with stats in the other – given the choice I’d like to see the run-time from both. Despite this, I felt fairly confident that the prediction would match the run-time for 12c and that I could at least guess the “starts” figure for 12c.
The important thing to notice is the way that the access predicate in 10g has split into an access predicate followed by a filter predicate in 12c. So 12c is going to iterate three times (once for each of the values ‘C’, ‘F’, ‘P’) and then walk a potentially huge linked list of index leaf blocks looking for 6 values of c3, while 10g is going to probe the index 18 times (3 combinations of c2 x six combinations of c3) to find “nearly unique” rows which means probably one leaf block per probe.
The 12c plan was taking minutes to run, the 10g plan was taking less than a second. The difference in execution time was probably the effect of the 12c plan ranging through (literally) thousands of index leaf blocks.
There are many bugs and anomalies relating to in-list iteration and index range scans and cardinality calculations – here’s a quick sample of v$system_fix_control in 12.2.0.1:
select optimizer_feature_enable ofe, sql_feature, bugno, description
from v$system_fix_control
where
optimizer_feature_enable between '10.2.0.4' and '12.2.0.1'
and ( sql_feature like '%CBO%'
or sql_feature like '%CARDINALITY%'
)
and ( lower(description) like '%list%'
or lower(description) like '%iterat%'
or lower(description) like '%multi%col%'
)
order by optimizer_feature_enable, sql_feature, bugno
;
OFE SQL_FEATURE BUGNO DESCRIPTION
---------- --------------------------- ---------- ----------------------------------------------------------------
10.2.0.4 QKSFM_CBO_5259048 5259048 undo unused inlist
QKSFM_CBO_5634346 5634346 Relax equality operator restrictions for multicolumn inlists
10.2.0.5 QKSFM_CBO_7148689 7148689 Allow fix of bug 2218788 for in-list predicates
11.1.0.6 QKSFM_CBO_5139520 5139520 kkoDMcos: For PWJ on list dimension, use part/subpart bits
11.2.0.1 QKSFM_CBO_6818410 6818410 eliminate redundant inlist predicates
11.2.0.2 QKSFM_CBO_9069046 9069046 amend histogram column tracking for multicolumn stats
11.2.0.3 QKSFM_CARDINALITY_11876260 11876260 use index filter inlists with extended statistics
QKSFM_CBO_10134677 10134677 No selectivity for transitive inlist predicate from equijoin
QKSFM_CBO_11834739 11834739 adjust NDV for list partition key column after pruning
QKSFM_CBO_11853331 11853331 amend index cost compare with inlists as filters
QKSFM_CBO_12591120 12591120 check inlist out-of-range values with extended statistics
11.2.0.4 QKSFM_CARDINALITY_12828479 12828479 use dynamic sampling cardinality for multi-column join key check
QKSFM_CARDINALITY_12864791 12864791 adjust for NULLs once for multiple inequalities on nullable colu
QKSFM_CARDINALITY_13362020 13362020 fix selectivity for skip scan filter with multi column stats
QKSFM_CARDINALITY_14723910 14723910 limit multi column group selectivity due to NDV of inlist column
QKSFM_CARDINALITY_6873091 6873091 trim histograms based on in-list predicates
QKSFM_CBO_13850256 13850256 correct estimates for transitive inlist predicate with equijoin
12.2.0.1 QKSFM_CARDINALITY_19847091 19847091 selectivity caching for inlists
QKSFM_CARDINALITY_22533539 22533539 multi-column join sanity checks for table functions
QKSFM_CARDINALITY_23019286 23019286 Fix cdn estimation with multi column stats on fixed data types
QKSFM_CARDINALITY_23102649 23102649 correction to inlist element counting with constant expressions
QKSFM_CBO_17973658 17973658 allow partition pruning due to multi-inlist iterator
QKSFM_CBO_21057343 21057343 order predicate list
QKSFM_CBO_22272439 22272439 correction to inlist element counting with bind variables
There are also a number of system parameters relating to inlists that are new (or have changed values) in 12.2.0.1 when compared with 10.2.0.3 – but I’m not going to go into those right now.
I was sufficiently curious about this anomaly that I emailed the OP to say I would be happy to take a look at the 10053 trace files for the query – the files probably weren’t going to be very large given that it was only a single table query – but in the end it turned out that I solved the problem before he’d had time to email them. (Warning – don’t email me a 10053 file on spec; if I want one I’ll ask for it.)
Based on the description I created an initial model of the problem – it took about 10 minutes to code:
rem Tested on 12.2.0.1, 18.3.0.1
drop table t1 purge;
create table t1 (
c1 varchar2(4) not null,
c2 varchar2(1) not null,
c3 varchar2(15) not null,
c4 varchar2(4) not null,
v1 varchar2(250)
)
;
insert into t1
with g as (
select rownum id
from dual
connect by level <= 1e4 -- > hint to avoid wordpress format issue
)
select
'0001',
chr(65 + mod(rownum,11)),
'18110034'||lpad(1+100*rownum,7,'0'),
lpad(mod(rownum,9),4,'0'),
rpad('x',250,'x')
from
g,g
where
rownum <= 1e5 -- > hint to avoid wordpress format issue
;
create unique index t1_i1 on t1(c1, c2, c3, c4);
begin
dbms_stats.gather_table_stats(
null,
't1',
method_opt => 'for all columns size 1'
);
end;
/
alter session set statistics_level = all;
set serveroutput off
prompt ==========================
prompt Default optimizer features
prompt ==========================
select
/*+ optimizer_features_enable('12.2.0.1') */
t1.*
FROM t1
WHERE
t1.c1 = '0001'
AND t1.c2 in ('H', 'F', 'C')
AND t1.c3 in (
'18110034450001',
'18110034450101',
'18110034450201',
'18110034450301',
'18110034450401',
'18110034450501'
)
;
select * from table(dbms_xplan.display_cursor(null,null,'cost allstats last'));
select
/*+ optimizer_features_enable('10.2.0.3') */
t1.*
FROM t1
WHERE
t1.c1 = '0001'
AND t1.c2 in ('H', 'F', 'C')
AND t1.c3 in (
'18110034450001',
'18110034450101',
'18110034450201',
'18110034450301',
'18110034450401',
'18110034450501'
)
;
select * from table(dbms_xplan.display_cursor(null,null,'cost allstats last'));
alter session set statistics_level = all;
set serveroutput off
The two queries produced the same plan – regardless of the setting for optimizer_features_enable – it was the plan originally used by the OP’s 10g setting:
-------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time | Buffers |
-------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 20 (100)| 0 |00:00:00.01 | 35 |
| 1 | INLIST ITERATOR | | 1 | | | 0 |00:00:00.01 | 35 |
| 2 | TABLE ACCESS BY INDEX ROWID| T1 | 18 | 2 | 20 (0)| 0 |00:00:00.01 | 35 |
|* 3 | INDEX RANGE SCAN | T1_I1 | 18 | 2 | 19 (0)| 0 |00:00:00.01 | 35 |
-------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("T1"."C1"='0001' AND (("T1"."C2"='C' OR "T1"."C2"='F' OR "T1"."C2"='H')) AND
(("T1"."C3"='18110034450001' OR "T1"."C3"='18110034450101' OR "T1"."C3"='18110034450201' OR
"T1"."C3"='18110034450301' OR "T1"."C3"='18110034450401' OR "T1"."C3"='18110034450501')))
There was one important difference between the 10g and the 12c plans – in 10g the cost of the table access (hence the cost of the total query) was 20; in 12c it jumped to 28 – maybe there’s a change in the arithmetic for costing the iterator, and maybe that’s sufficient to cause a problem.
Before going further it’s worth checking what the costs would look like (and, indeed, if the plan is possible in both versions) if we force Oracle into the “bad” plan. That’s where we finally get to the hint in the title of this piece. If I add the hint /*+ num_index_keys(t1 t1_i1 2) */ what’s going to happen ? (Technically I’ve included a hint to use the index, and specified the query block name to make sure Oracle doesn’t decide to switch to a tablescan):
select
/*+
optimizer_features_enable('12.2.0.1')
index_rs_asc(@sel$1 t1@sel$1 (t1.c1 t1.c2 t1.c3 t1.c4))
num_index_keys(@sel$1 t1@sel$1 t1_i1 2)
*/
t1.*
FROM t1
WHERE
t1.c1 = '0001'
AND t1.c2 in ('H', 'F', 'C')
AND t1.c3 in (
'18110034450001',
'18110034450101',
'18110034450201',
'18110034450301',
'18110034450401',
'18110034450501'
)
;
------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time | Buffers | Reads |
------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 150 (100)| 0 |00:00:00.01 | 154 | 1 |
| 1 | INLIST ITERATOR | | 1 | | | 0 |00:00:00.01 | 154 | 1 |
| 2 | TABLE ACCESS BY INDEX ROWID BATCHED| T1 | 3 | 18 | 150 (2)| 0 |00:00:00.01 | 154 | 1 |
|* 3 | INDEX RANGE SCAN | T1_I1 | 3 | 18 | 142 (3)| 0 |00:00:00.01 | 154 | 1 |
------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("T1"."C1"='0001' AND (("T1"."C2"='C' OR "T1"."C2"='F' OR "T1"."C2"='H')))
filter(("T1"."C3"='18110034450001' OR "T1"."C3"='18110034450101' OR "T1"."C3"='18110034450201' OR
"T1"."C3"='18110034450301' OR "T1"."C3"='18110034450401' OR "T1"."C3"='18110034450501'))
This was the plan from 12.2.0.1 – and again the plan for 10.2.0.3 was identical except for costs which became 140 for the index range scan and 141 for the table access. At first sight it looks like 10g may be using the total selectivity of the entire query as the scaling factor for the index clustering_factor to find the table cost while 12c uses the cost of accessing the table for one iteration (rounding up) before multiplying by the number of iterations.
Having observed this detail I thought I’d do a quick test of what happened by default if I requested 145 distinct values of c3. Both versions defaulted to the access/filter path rather than the pure access path – but again there was a difference in costs. The 10g index cost was 140 with a table access cost of 158, while 12c had an index cost of 179 and a table cost of 372. So both versions switch plans at some point – do they switch at the same point ? Reader, I could not resist temptation, so I ran a test loop. With my data set the 12c version switched paths at 61 values in the in-list and 10g switched at 53 values –
Conclusion: there’s been a change in the selectivity calculations for the use of in-list iterators, which leads to a change in costs, which can lead to a change in plans; the OP was just unlucky with his data set and stats. Possibly there’s something about his data or stats that makes the switch appear with a much smaller in-list than mine.
Footnote:
When I responded to the thread on MOSC with the suggestion that the problem was in part due to statistics and might be affected by out of date stats (or a histogram on the (low-frequency) c2 column) the OP noted that stats hadn’t been gathered since some time in August – and found that the 12c path changed to the efficient (10g) one after re-gathering stats on the table.
