Oracle Scratchpad

September 30, 2015


Filed under: CBO,Histograms,Oracle,Statistics — Jonathan Lewis @ 5:10 pm BST Sep 30,2015

Here’s a live one from OTN – here are a couple of extracts from the problem statement:

We’re experiencing an issue where it seems that the query plan changes from day to day for a particular procedure that runs once a night.
It’s resulting in a performance variance of 10 second completion time vs 20 minutes (nothing in between).
It started occurring about 2 months ago and now it’s becoming more prevalent where the bad query plan is coming up more often.
I noticed that the query plans vary for a simple query.
We do run gather statistics every night. (DBMS_STATS.GATHER_SCHEMA_STATS (ownname=>sys_context( ‘userenv’, ‘current_schema’ ), estimate_percent => 1);)

The query and two execution plans look like this:

select count(*) from cs_bucket_member_v2 where bucket_type='P' and sec_id > 0 and order_id=0;

| Id  | Operation                    | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT             |                     |     1 |    12 |   155   (0)| 00:00:02 |
|   1 |  SORT AGGREGATE              |                     |     1 |    12 |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| CS_BUCKET_MEMBER_V2 |  1148 | 13776 |   155   (0)| 00:00:02 |
|*  3 |    INDEX RANGE SCAN          | CS_BUCKET_MEMBER_N1 |  1272 |       |     3   (0)| 00:00:01 |

Predicate Information (identified by operation id):
   2 - filter("BUCKET_TYPE"='P' AND "SEC_ID">0)
   3 - access("ORDER_ID"=0)

| Id  | Operation          | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT   |                     |     1 |    12 | 11215   (2)| 00:01:41 |
|   1 |  SORT AGGREGATE    |                     |     1 |    12 |            |          |
|*  2 |   TABLE ACCESS FULL| CS_BUCKET_MEMBER_V2 |  1522K|    17M| 11215   (2)| 00:01:41 |

Predicate Information (identified by operation id):
   2 - filter("ORDER_ID"=0 AND "SEC_ID">0 AND "BUCKET_TYPE"='P')

There are a couple of bits of information that would be useful – such as the database version, the number of rows in the table, the number of distinct values in each column, and whether any of the columns have histograms – but there are a couple of reasonable guesses that we might make about the problem. Notice particularly that the number of rows estimated from the index ranges scan is 1272 and only a small volume is then eliminated by the table filter predicates on sec_id and bucket_type. This suggests that the optimizer has information that tells it that most of the rows in the table have sec_id > 0 and bucket_type = ‘P’, and you might note that that suggests that there’s a histogram on bucket_type.

Rather than stating the most obvious guesses about the problem, though, I’ll start by creating a data set and emulating the problem, starting from an empty schema on

create table t1
with generator as (
        select  --+ materialize
                rownum id 
        from dual
        connect by 
                level <= 1e4
        rownum                  sec_id,
                when mod(rownum,1000) = 0
                        then 'X'
                        else 'P'
        end                     bucket_type,
                when rownum < 1e6 - 50000 
                        then mod(rownum-1,1e5)
                        else 1000
        end                     order_id,
        lpad(rownum,10,'0')     id_vc,
        rpad('x',100,'x')       padding
        generator       v1,
        generator       v2
        rownum <= 1e6

create index t1_i1 on t1(order_id) nologging; 

select count(*) from t1 where order_id = 1000 and bucket_type = 'P' and sec_id > 1000;

The column names in the table match those needed by the query, and the bucket_p column has a very skewed distribution that will eliminate very little data; the sec_id column is also not going to eliminate data, but it’s very evenly distributed with no large gaps so not a good candidate for a histogram in any case. The order_id has 50,000 rows out of 1,000,000 (5%) set of a single value, and most of those special rows are at the end of the table – it’s a pretty good candidate for a histogram (if Oracle spots it, and if we actually write queries to access that data).

I’ve run a query that references all three columns so that the default method_opt of “for all columns size auto” will apply to them when I gather stats. So here’s the code that gathers stats and checks the resulting execution plans, first for “auto_sample_size” then for the 1% used by the OP:

set autotrace traceonly explain

/*              estimate_percent => 1, */
                ownname          => user

select count(*) from t1 where order_id = 1000 and bucket_type = 'P' and sec_id > 1000;

                estimate_percent => 1,
                ownname          => user

select count(*) from t1 where order_id = 1000 and bucket_type = 'P' and sec_id > 1000;

set autotrace off

And here are the two plans – in the same order:

| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT   |      |     1 |    12 |  2333   (4)| 00:00:12 |
|   1 |  SORT AGGREGATE    |      |     1 |    12 |            |          |
|*  2 |   TABLE ACCESS FULL| T1   | 51063 |   598K|  2333   (4)| 00:00:12 |

Predicate Information (identified by operation id):
   2 - filter("ORDER_ID"=1000 AND "SEC_ID">1000 AND "BUCKET_TYPE"='P')

| Id  | Operation                    | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT             |       |     1 |    12 |    23   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE              |       |     1 |    12 |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| T1    |    20 |   240 |    23   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | T1_I1 |    20 |       |     3   (0)| 00:00:01 |

Predicate Information (identified by operation id):  
   2 - filter("SEC_ID">1000 AND "BUCKET_TYPE"='P')
   3 - access("ORDER_ID"=1000)

[Update: Following on from a question in the comments, I’ve expanded this section, and wandered a little off-topic]

I don’t know why, but with a 1% sample (which really did sample 10,000 rows) the optimizer didn’t spot the need for a histogram on order_id, but with the auto_sample_size (which sampled 5,500 – yes, half as many rows) the optimizer spotted the need for the histogram. Checking the trace files the only difference visible in the sample SQL was the presence in the 1% sample of the id_vc and padding columns which were not present in the auto_sample_size trace.

According to the manuals when the method_opt is “for all columns size auto”, then

“Oracle determines the columns on which to collect histograms based on data distribution and the workload of the columns.”

There is nothing in the manuals to suggest that there is a deliberate link between the auto_sample_size and estimate_percent, and there is room for ambiguity in how we interpret this bit of text in the manual so the difference in the SQL used and the effects thereof requires (a) some hand-waving, and/or (b) lots more experimentation.  At the moment I’m prepared to go for hand-waving:

Hypothesis 1: auto_sample_size did not sample the id_vc and padding columns because the (100%) sample taken had given Oracle enough information to decide that the data distribution of those columns was not skewed enough to merit further consideration; but it sampled the three columns that had been used in a fashion that might be helped by a histogram. This sampling spotted the benefit of a histogram on order_id and bucket_type but decided that sec_id didn’t need a histogram

Hypothesis 2: the 1% sample got pretty close to the same results in its estimates of number of distinct values for id_vc and padding as the (100%) auto_sample_size, but still decided to do a sampled test for the data distribution (the manual seems to suggest that the histograms will only be considered if there has been some use of the columns in predicates, but doesn’t explicitly preclude the possibility of creating the histogram on the basis of just the data distribution). After doing the 1% sample to analyze the data for suitability of a histogram the result suggested that only the histogram on bucket_type would be beneficial.  (In fact, after the first sample Oracle took a second 1% histogram sample on just the order_id before deciding that it a histogram on order_id wasn’t needed.)

Bottom line on this: I don’t know if the auto_sample_size “accidentally” eliminated a couple of columns from histogram sampling and if a larger fixed sample size (say 50%, or even 100%) might result in Oracle eliminating a few columns from the histogram; or maybe the code path for histogram samples with auto_sample_size in place is actually a different code path. The only thing I can say is that the two sets of events that appeared from my demonstration don’t seem to be entirely self-consistent, but it would probably take most of a day doing experiments to narrow down the variation in behaviour to a few “best guess” ideas of what’s going on behind the scenes – though unwrapping the code might lead to a more accurate answer more quickly.


Histograms are tricky things – and you can only make things worse in 11g by NOT using the auto_sample_size.


Based on previous experience – my “obvious” guess about the OP’s data was that there was a special-case value for order_id, that the rows for that value were fairly well clustered, probably towards the end of the table, and constituted a small percentage of the table, and that the rest of the data reported “a few” rows per value. That’s why I built the model you see above.


  1. Thanks for the article. We see behavior like this from time to time and are currently reviewing our stats gathering process for improvements. I’ll keep this in mind.

    One thing, for the two stats gathering runs, they’re both running with the same settings. Should the second one have “estimate_percent => auto_sample_size”?

    Comment by RJolley — September 30, 2015 @ 7:55 pm BST Sep 30,2015 | Reply

    • RJolley,

      Thanks for the comment – if you search for histograms on the blog you’ll find several ways in which histograms can make like difficult. 12c is much better with Frequency and Top-N histograms being run on 100% samples, but even in 12c Oracle tends to use a small sample when the hybrid (enhancement over the height-balanced) histogram is needed, and you can be very unlucky.

      It’s actually a little hard to see (at least on my current screen) but the estimate_percent is commented out in the first call using the “–” notation and that’s why it falls back to the default auto_sample_size. I’m about to change it to the /* */ notation to improve visibility.

      Comment by Jonathan Lewis — September 30, 2015 @ 8:50 pm BST Sep 30,2015 | Reply

      • Ah, ok, that makes sense. Knew I should’ve cut and paste it into notepad++ to see what the differences were. Thanks for the update.

        Comment by RJolley — September 30, 2015 @ 9:46 pm BST Sep 30,2015 | Reply

  2. Hi,
    I have one doubt that by default method_opt is FOR ALL COLUMNS SIZE AUTO, will it not check for which column we need histogram.
    If we had given estimate_percent 1 or any value but method_opt is default so will it not check histogram for all columns .

    Comment by PP — October 1, 2015 @ 9:40 pm BST Oct 1,2015 | Reply

    • PP

      I don’t think there’s anything in the manuals that connects estimate_percent (size of sample) to method_opt. You’re correct that “for all columns size auto” tells Oracle to figure out for itself which columns need histograms according to their data content AND their usage pattern – which is why I ran a query using the three columns I was interested in before collecting stats.

      I guess that (for 11g) using the auto_sample_size – which equates to 100% – for the basic stats MAY allow Oracle to recognise that some columns definitely don’t need histograms and therefore not include them in the histogram sample query, while a 1% sample may not allow Oracle to take this short cut.

      Comment by Jonathan Lewis — October 1, 2015 @ 10:47 pm BST Oct 1,2015 | Reply

      • Hi,
        If i will use estimate percent 20 or 50 or 1 and method_opt auto, so will it go for histogram for necessary columns and ignore unwanted columns means will got for shot cut or not..?
        If I am using both parameter one with oracle recommended and other with our value.

        Comment by PP — October 1, 2015 @ 11:44 pm BST Oct 1,2015 | Reply

  3. Hi,

    In general, it is possible for histograms to appear or disappear with SIZE AUTO/SKEWONLY depending on the estimate_percent setting, just because different rows are sampled that may or may not look skewed enough to justify a histogram. But in this case the histogram on ORDER_ID is being rejected for a different reason that I think is a defect. I’ve filed Bug 21947352 for the issue.

    Mark Jefferys – Oracle Support

    Comment by Mark Jefferys — October 5, 2015 @ 9:54 pm BST Oct 5,2015 | Reply

  4. Mark,

    Thanks for raising the bug and letting me know.

    I see that in the bug you’ve changed the definition of order_id to show that the anomaly appears even when the special value is scattered evenly across the entire table. I have to say that I’m a little embarrassed by the fact that when my first test behaved exactly as I had thought it would I didn’t think of checking a couple of less extreme cases.

    Comment by Jonathan Lewis — October 5, 2015 @ 11:01 pm BST Oct 5,2015 | Reply

    • The reason I changed order_id was because otherwise the clustering causes the index path to perform much better than the FTS plan, and I wanted the case more clear-cut. Besides, the values being scattered seemed more likely.

      Mark Jefferys – Oracle Support

      Comment by Mark Jefferys — October 5, 2015 @ 11:25 pm BST Oct 5,2015 | Reply

RSS feed for comments on this post. TrackBack URI

Comments and related questions are welcome.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by