Thanks for the contribution. Unfortunately it got messed up by a rogue “less than”.
If you can email me the entire source as a text-file attachment I’ll copy it into your comment.

The variation between your prediction and Oracle’s prediction is due to scaling by Oracle to cater for the difference between “total rows in table” and “total rows covered by histogram”.

Oracle version 10.2.0.5 on HP-UX.

SQL: alter session set "_fix_control"='5483301:off' ;
Session altered.


declare

    srec                      dbms_stats.statrec;
    a_bucket                  dbms_stats.numarray;
    v_tot_rows_not_in_freq    number;
    v_tot                     number;
    v_density                 number;
    v_distinct_key_not_in_freq   number;
    v_cutoff                  number :=150     -- set here your variable
begin
     -- create the histogram, maximum 254 buckets but may be lower following v_cutoff_value
     select cpt, USERIDENTITY_ID bulk collect into srec.bkvals, a_bucket from (
     select
          cpt, USERIDENTITY_ID
     from ( select count(*) cpt, USERIDENTITY_ID
                              from IBS6_EB_OWNER.USERMESSAGE
                                   group by USERIDENTITY_ID order by count(*) desc
          )
     where rownum  150  order by USERIDENTITY_ID
    );

    -- Calculate now the density for all values which are not in the histo.
    -- the density is calculated from all rows not in the frequency histogram
    -- so density =   (tot rows not in freq / distinct key not in freq)/tot row in table)
    -- the cut off value to be taken into the histogram is set by default to 150, adapt following needs
    
    -- total rows not in frequency:

    with v as (select useridentity_id
            from ( select useridentity_id
                             from MYTABLE_OWNER.CUSTOMER
                                 group by USERIDENTITY_ID having count(*) > v_cutoff  order by count(*) desc)
             where rownum  'MYTABLE_OWNER',
        tabname     => 'CUSTOMER',
        colname     => 'CUST_ID',
        density     => v_density,
         srec        => srec );
end;
/

Rows in table                  : 459699
distint Keys in histogram      : 74
Tot Rows keys covered by histo : 454026
Distinct Keys not in histogram : 250
Rows not in histogram          : 5673
Density for keys not in histo  : 0.0000493627

Produces this histogram:

TABLE_NAME   COLUMN_NAME        Bck nbr ENDPOINT_VALUE Frequency
------------ --------------- ---------- -------------- ----------
USERMESSAGE  USERIDENTITY_ID        303             11        303
                                   1990             12       1687
                                   2443             19        453
                                   5165             21       2722
                                   5417             24        252
                                   5945             27        528
                                   6105             33        160
                                   6381             85        276
                                  23950            148      17569   <-- See SQL1
                                  24371            149        421   <-- see SQL2
                                                                    <-- see SQL3 value 169 not in histogram
                                  27792            180       3421
                                 107833            201      80041
                                 108440            216        607
                                 109053            221        613
                                 110623            223       1570
                                 110833            422        210
                                 111030            429        197
                                 111222            440        192
                                 151751            441      40529
.
.
(74 rows)

SQL_ID  g3b5j7qfy7u5d, child number 0
-------------------------------------
SELECT /*+ gather_plan_statistics */ COUNT(*) FROM usermessage um, useridentity ui  ,
usermessagetype umt WHERE um.isconsumed = 0 AND ui.useridentity_id = um.useridentity_id  AND
umt.usermessagetype_id = um.usermessagetype_id AND ui.useridentity_id = 148

Plan hash value: 3616009165

-----------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name            | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                 |      1 |        |      1 |00:00:00.07 |    2362 |
|   1 |  SORT AGGREGATE               |                 |      1 |      1 |      1 |00:00:00.07 |    2362 |
|   2 |   NESTED LOOPS                |                 |      1 |   7505 |   8559 |00:00:00.06 |    2362 |
|*  3 |    INDEX UNIQUE SCAN          | USERIDENTITY_PK |      1 |      1 |      1 |00:00:00.01 |       2 |
|*  4 |    TABLE ACCESS BY INDEX ROWID| USERMESSAGE     |      1 |   7505 |   8559 |00:00:00.06 |    2360 |
|*  5 |     INDEX RANGE SCAN          | USERMESSAGE_IDX |      1 |  17792 |  17569 |00:00:00.01 |      46 |
-----------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - access("UI"."USERIDENTITY_ID"=148)
   4 - filter("UM"."ISCONSUMED"=0)
   5 - access("UM"."USERIDENTITY_ID"=148)

Expected cardinality is very close. I wonder why it modified the 17569 to 17792

  COUNT(*)
----------
        33


SQL_ID  81dtb814kvvy9, child number 0
-------------------------------------
SELECT /*+ gather_plan_statistics */ COUNT(*) FROM usermessage um, useridentity ui  ,
usermessagetype umt WHERE um.isconsumed = 0 AND ui.useridentity_id = um.useridentity_id  AND
umt.usermessagetype_id = um.usermessagetype_id AND ui.useridentity_id = 149

Plan hash value: 3616009165

-----------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name            | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                 |      1 |        |      1 |00:00:00.01 |     231 |
|   1 |  SORT AGGREGATE               |                 |      1 |      1 |      1 |00:00:00.01 |     231 |
|   2 |   NESTED LOOPS                |                 |      1 |    180 |     33 |00:00:00.01 |     231 |
|*  3 |    INDEX UNIQUE SCAN          | USERIDENTITY_PK |      1 |      1 |      1 |00:00:00.01 |       2 |
|*  4 |    TABLE ACCESS BY INDEX ROWID| USERMESSAGE     |      1 |    180 |     33 |00:00:00.01 |     229 |
|*  5 |     INDEX RANGE SCAN          | USERMESSAGE_IDX |      1 |    426 |    421 |00:00:00.01 |       5 |
-----------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - access("UI"."USERIDENTITY_ID"=149)
   4 - filter("UM"."ISCONSUMED"=0)
   5 - access("UM"."USERIDENTITY_ID"=149)

Not bad

  COUNT(*)
----------
         9

SQL_ID  0z8t787a3t8an, child number 0
-------------------------------------
SELECT /*+ gather_plan_statistics */ COUNT(*) FROM usermessage um, useridentity ui  ,
usermessagetype umt WHERE um.isconsumed = 0 AND ui.useridentity_id = um.useridentity_id  AND
umt.usermessagetype_id = um.usermessagetype_id AND ui.useridentity_id = 169

Plan hash value: 3616009165

-----------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name            | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                 |      1 |        |      1 |00:00:00.01 |      11 |
|   1 |  SORT AGGREGATE               |                 |      1 |      1 |      1 |00:00:00.01 |      11 |
|   2 |   NESTED LOOPS                |                 |      1 |     10 |      9 |00:00:00.01 |      11 |
|*  3 |    INDEX UNIQUE SCAN          | USERIDENTITY_PK |      1 |      1 |      1 |00:00:00.01 |       2 |
|*  4 |    TABLE ACCESS BY INDEX ROWID| USERMESSAGE     |      1 |     10 |      9 |00:00:00.01 |       9 |
|*  5 |     INDEX RANGE SCAN          | USERMESSAGE_IDX |      1 |     23 |      9 |00:00:00.01 |       3 |
-----------------------------------------------------------------------------------------------------------

23 rows this is the default density : 5673/250 = 22.692 round up to 23.

I think that is essential.

Regards Hans-Peter

I noticed that the dbms_stats is influenced someway by how columns are used in where clause predicates.
I did the following
1. create table
2. gather stats : distinct keys and density set to 1
3. query the table with col ‘N’ and col := :b1 and col2 =
4. gather stats : distinct key set to 1 and density 1/*num_rows

I must have missed that in the documents and on the net somehow.

Regards Hans-Peter

Randolf,
Thanks for the links – your post was initially marked as spam because it had too many links in it. (By the way, Alberto sent me an email last night questioning whether he was the one that had mentioned this change to me. I was fairly sure it was, but now I think it might have been you that first brought it to my attention. Apologies for the mis-attribution if that’s the case).

Hemant,
Thanks for the comment, your post also got marked as spam.

Timur,
Thanks for posting the example – a bit got lost because you missed one of the “less thans” when editing the text. I’ve edited it with my best guess about the text that got dropped.

All –
Is this the quoted bug ? Yes and No.
I don’t think the “bug” was really a bug, just an optimizer assumption of reasonable behaviour that nobody questioned for several years. Personally I was quite amused when another bug number appeared about one week after the 10.2.0.4 patchset came out explaining how to reverse the effect – so both behaviours appear as bugs on Metalink.