My apologies for providing incorrect and incomplete information. Naming convention of the index did not use this column name and I did not notice the index used in the query actually contains this column. You are right this is unique index and which contains three columns (the leading column in the index is the one having histograms issue). I verified the info from the index and columns and actually data in the column is not skew because the distinct_keys in the index and num_distinct in the column are approximately the same as below, the other columns have only 2 and 0 distinct values

SQL> select column_name,num_distinct from dba_tab_cols where table_name='EVE' and column_name='IDENTITY';

COLUMN_NAME                    NUM_DISTINCT
------------------------------ ------------
IDENTITY                         	24749753

SQL> select  index_name,num_rows,distinct_keys from dba_indexes where table_name='EVE' and index_name='UI_EVE';

INDEX_NAME                       NUM_ROWS DISTINCT_KEYS
------------------------------ ---------- -------------
UI_EVE              		25670860      25670860

As you mentioned because first 32 chars in the column values are same so it was reporting incorrect histograms.I decided to delete histograms on this column and now the query is using the Index scan in place FTS.

Thanks for your inputs.

Amit

You’ve got more than 2M distinct values according to the stats; what’s 1/2,000,000 ?

2. The selectivity optimizer is trying to calculate is (80-0.5)/254=0.312992 which is calculating the cardinality as 20297719*0.312992=6353026.22 but I am not sure why it is trying to deduct 0.5 from the popular bucket count. I got the same results for other 10053 trace files for this query

I made the mistake of assuming the column stats and the index access path information you gave me were somehow related – I suppose I should have guessed that an index named UI_Col1 didn’t contain a column named Col2 but I rather discounted that possibility as a typo because I didn’t think your columns were really called Col1 and Col2 anyway. Why would the number of popular buckets for one column have anything to do with the index selectivity of an index which doesn’t include that column ?

3. And the actual problem, should there be no histogram on this column or how to deal with such kind of histogram issue. Predicate used in the query (with skewed histogram) is not a unique column, the access path shown is for RangeScan on other unique index column UI_COL1 which the optimizer is trying to evaluate for other strings not having 32 chars

I’m going to guess that there are two predicates (at least), one on Col1 and one on Col2. But how I’m supposed to work out an answer to the question when you haven’t given me any clues about what the predicates look like, or what the index looks like ?

My query is doing FTS because it is getting the cardinality as 1/3 of records from the table and Index RangeScan for other values in no-popular buckets. But I am trying to understand:

1. Why it is reporting NewDensity:0.000000
2. The selectivity optimizer is trying to calculate is (80-0.5)/254=0.312992 which is calculating the cardinality as 20297719*0.312992=6353026.22 but I am not sure why it is trying to deduct 0.5 from the popular bucket count. I got the same results for other 10053 trace files for this query
3. And the actual problem, should there be no histogram on this column or how to deal with such kind of histogram issue. Predicate used in the query (with skewed histogram) is not a unique column, the access path shown is for RangeScan on other unique index column UI_COL1 which the optimizer is trying to evaluate for other strings not having 32 chars

I have corrected the comment above that you are using as the source of the formula. When I wrote it, I was thinking about cardinality rather than selectivity (and the selectivity for ‘column = constant’ is the value that the optimizer reports as the NewDensity).

But even when you know what the formula is really supposed to give you, you still get a strange answer – and that’s because there is something very wrong with your statistics. Note that the table has 20,297,719 rows and the column has 20,297,719 distinct values. Despite this, the histogram claims that there are 80 buckets out of 254 that all represent the same value … which means the NDV ought to be roughly 2/3 of the size that it is.

I note, then, that your index is called UI_Col1 – which suggests you have a naming convention that prefixes unique indexes with the letters UI, and that means something even more bizarre is happening with the stats. Given the path is a RangeScan, of course, the ix_sel in the Access path calculation doesn’t have to have anything to do with the NewDensity from the table stats. On the other hand the value of 0.31 (combined with the 80 buckets of popular values) suggests that your query may be targetting a range that includes the value that Oracle thinks is the popular value (viz, a little less than one third of the rows … 80/254 = 0.3149).

At this point I think I’m going to guess that Col1 is a character based column of more than 32 bytes and a large number of rows have a value that starts with the same 32 bytes. I say this because it’s one way to reproduce your anomaly – the code to calculate the NDV can count the actual distinct values, the code to generate the histogram limits itself (with some complications) to the first 32 bytes.

I’ll try to write up a short note to demonstrate the NewDensity calculation in a couple of days time.

I am using 11.2 and from the formula (total number of rows – number of rows for popular values) / (total number of distinct values – number of popular values). I tried to calculate the selectivity for my query but not getting the correct value as depicted in the 10053 trace. Below is my 10053 trace output:

Column (#2): 
    NewDensity:0.000000, OldDensity:0.000000 BktCnt:254, PopBktCnt:80, PopValCnt:1, NDV:20297719
  Column (#2): Col2 (
    AvgLen: 36 NDV: 20297719 Nulls: 0 Density: 0.000000
    Histogram: HtBal  #Bkts: 254  UncompBkts: 254  EndPtVals: 176
  Table: Tab1  Alias: UNIT1_
    Card: Original: 20297719.000000  Rounded: 6353026  Computed: 6353026.22  Non Adjusted: 6353026.22

after putting values in the formula, the selectivity comes out to be (20297719- (20297719/254)*80)/(20297719-1)=0.6850394. Though the selectivity depicted in the trace is

Access Path: index (RangeScan)
    Index: UI_Col1
    resc_io: 5766117.00  resc_cpu: 43557791448
    ix_sel: 0.312992  ix_sel_with_filters: 0.312992 
    Cost: 5772791.97  Resp: 5772791.97  Degree: 1

Please tell me where I am wrong in my calculation.

Thanks
Amit

I wouldn’t be 100% sure of that. It’s possible, for example, that there are cases where the optimizer code hasn’t been made completely consistent. I would still prefer to set a density to “the figure I want Oracle to believe” even if I thought it should be ignored.