But, if DPR fts/ffs are just replaced by fts/ffs into SGA database cache, it looks like I should watch the following pretty closely to see if I’m really getting the benefit I want:
1. Midpoint vs near-endpoint buffer cache LRU insertion. Midpoint insertion would allow a larger time window for near-concurrent queries to find the data still in cache. Insertion of fts/ffs blocks near the buffer cache LRU might not lower the total workload IO or the risk of QFULL much.
2. change of touch count (x$bh.tch). If the ffs/fts blocks don’t have their touch count increased by subsequent touches after the three second timer goes by, they’ll age out and again might not provide as much cache benefit as originally expected.
3. forget-bit.
4. Updates to last touch time (x$bh.tim)

In addition to watching total accumulated IO, short and long table scan counts, ffs counts… that’s a lot to watch in comparative tests. Makes me think in my case it may be better to work with optimizer_index_cost_adj, optimizer_index_caching, and/or table_cached_blocks to replace fts and ffs with more targeted index use with index blocks and index-selected table blocks just going to the normal database buffer cache LRU insertion point, with their touch count handled just like everything else.

I’d like to design a good system test against a full-scale test workload, comparing baseline, modifications to _small_table_threshold, and modifications to optimizer_index_cost_adj. The comparisons would be batch execution time (of course), workload total IO (data volume), workload total IOPs, long table scans, short table scans, index ffs, DPR traffic, midpoint database buffer cache insertions, near-endpoint database buffer cache insertions, and total count database cache buffers with forget-bit set (if this is still used in 11.2.0.3).

Thanks for the comment and links.

I didn’t think of it when I was writing the note, but the behaviour for tablescans (and ndex fast full scans) is probably linked directly to the 25% target that I describe for the auxiliary replacement list (REPL_AUX) in Oracle Core pp. 107- 109. I haven’t worked through it in detail, but I suspect – rough outline only – that:

any tablescan under the 2% limit goes to repl_main (mid-point)
any tablescan (index ffs) over the 2% figure goes directly to repl_aux
any tablescan (index ffs) less than the 10% figure gets its touch count incremented if the blocks are still in repl_aux when the scan is repeated (which means they can eventually be promoted to the hot end of repl_main).
any tablescan (index ffs) that is larger than repl_aux automatically cycles itself out of repl_aux as the scan is repeated.

“going direct to repl_aux” may actually turn out to mean blocks first go into repl_main but are relinked almost immediately (on the next physical read request) to repl_aux as a side effect of the free buffer scan.

Guessing (hypothesising) again – I don’t think multiple working data sets and database writers would make any difference to the 25% magic number, because every working data set has its own repl_aux which hovers around 25% of the set, and the tablescan will (statistically) spread its data evenly across all the working data sets. (That’s why the object queue table x$kcboqh ends up with multiple entries per obj# – it’s one row per object per working data set of the relevant buffer pool).

- Alex Fatkulin’s has written a little script for test the effect of object caching too: http://afatkulin.blogspot.ca/2009/01/11g-adaptive-direct-path-reads-what-is.html
- Regarding buffer cache warmup – there’s a nice statistic called “physical reads prefetch warmup” which should tell whether prefetching happened due to buffer cache being “empty”. There’s an example of this happening here: http://blog.tanelpoder.com/2012/05/02/advanced-oracle-troubleshooting-guide-part-10-index-unique-scan-doing-multiblock-reads/

Also, regarding the 25% of buffer cache threshold, I think one more thing what may affect this is that how many working sets you have in the buffer pool and how many DBWRs servicing them… I haven’t tested any of this but there’s a chance that these factors affect the 25% threshold logic as well…

I’ve replied to this on the OTN thread, but I didn’t say anything about Oracle choosing to do the direct path read just after startup. The public information about this is still under investigation but I recall seeing some notes somewhere about Oracle’s decision being based in part on the number (or percentage) of blocks from the object that are already in the cache. When you start up there’s nothing from the object in the cache, so it maximises the chance of Oracle picking the direct path option.

I made some comments, and showed some arithmetic, about the effect of an empty cache on the speed of repeating the tablescan using db file scattered, so I won’t repeat that bit here.

Very interesting reads. I follow up the thread that I opened in OTN under title “serial table scan with direct path read compared to db file scattered read”, http://forums.oracle.com/forums/thread.jspa?messageID=10081807

I have a table 1.7 Million rows with 1,943,824 blocks (no index) and a default buffer cache of 8320MB. That is 8320*1024*1024/8192 = 1,064,960 8K available blocks. At 25% max this table can have up to 1,064,960 * 0.25 = 266,240 blocks in memory or around 14% of the table. So roughly in 7 cycles the whole table can be scanned in.

When the buffer is empty after reboot, it prefers to use “direct path read” (DPR) as opposed to full serial table scan (FTS). So there must be some costing estimates that says DPR is cheaper than FTS. I suspect this may be the size of the underlying table larger than 2% of small_table_threshold as shown below).

The full stats are as follows:

Parameter                      buffer cache size/MB
------------------------------ --------------------
buffer_cache                                  8,320

Small table threshold at 2% of buffer cache size/MB Small table block limit
--------------------------------------------------- -----------------------
                                                166                  21,299

My table details

TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- ------------ ------------- ------------ ----------------- -------------
TDASH                               1,729,204             8    1,943,824               805        13,714

Table block size/threhold limit
-------------------------------
                             91

When I run a test load on this table as

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'test_with_tdash_ssdtester_noindex';
DECLARE
        type array is table of tdash%ROWTYPE index by binary_integer;
        l_data array;
        l_rec tdash%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;
 
        DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true );
        FOR rs IN 1 .. 100
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM tdash WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;

It takes 6,520 seconds to finish.

When I force it not to use DPR, it chooses db file scattered read and it finishes in 4,299 seconds. Details are posted in the link

So I am a bit confused.

cheers,

Mich

That would make sense! I was wondering what kind of algorithm Oracle could use to efficiently cache a table that was larger than the buffer cache, but using this (accidental?) algorithm would be an explanation.

What I’m most interested in is the parameters that control the caching mechanisms.
My main concern is a customers 10.2.0.4 is not caching even with an empty buffer cache when on my in house system it is caching with tables up to 80% of the buffer cache size. (I’m on LINUX and customer is on Solaris)
In both cases _small_table_threshold is the default.

The use case I’m looking to solve is a customer going from depending on the UNIX file system cache to depending on the SGA alone as they are switching to Direct File I/O.
On their system (Solaris 10.2.0.4 empty cache) the caching only happens well below 80%, more like 10-25% (I didn’t have the liberty to run extensive tests and nail down the value). I’m trying to figure out the options for making caching happen with bigger tables. Customer won’t use the cache table option because they want the new configuration to run just like the old without any extra administration. They are willing to change init params and maybe _small_table_threshold is the only way, but on my LINUX 10.2.0.4 I have the default _small_table_threshold and I’m seeing caching up with tables up to 80% of buffer cache size (start with an empty buffer cache) and I’m wondering “is there some other mechanism controlling caching besides _small_table_threshold?”
Customer wants the same “feel” with direct I/O as without. Without Direct I/O they could read a table once via FTS and then upon second read, the response time would be as if it was cached even though Oracle reports a high level of physical reads because the reads are coming from the UNIX file system cache.
Now, with Direct IO and even if I have them increase the size of their buffer even beyond the size of this table, my concern is they won’t be caching the table. I can have them change _small_table_threshold but I’m wondering if there is another mechanism which seems to be the case on my 10.2.0.4 on LINUX.
The other alternative is something on the customers database is preventing the caching of larger tables even after an “alter system flush buffer_cache”

PS my test script came from:
http://dioncho.wordpress.com/2009/04/22/strong-doubt-on-small-table_small_table_threshold/
Charles Hooper pointed me to your entry here, from his blog at:
http://hoopercharles.wordpress.com/2010/06/17/_small_table_threshold-parameter-and-buffer-cache-what-is-wrong-with-this-quote/
Glenn Fawcett has a good blog on the overhead of using the UNIX file system cache verses Oracle buffer cache
http://blogs.sun.com/glennf/entry/where_do_you_cache_oracle

One of the boundary conditions on the buffer cache behaviour appears when the entire cache is “free” – that’s why I started my experiment with an index range scan to load a very large fraction of the cache before I started doing any tablescans.

In your 10.2.0.4 case the whole cache was free, which means there are no useful data blocks to protect, so the code seems to have been allowed to use the whole cache on the tablescan.

Your first 11.2.0.1 experiment (table above/below 10%) suggests that Oracle Corp. has changed the boundary case so that it always uses the 10% that would apply to a “full” cache, even if the whole cache is free.

The second 11.2.0.1 experiment (_small_table_threshold set to 10% of cache_size) suggests that the algorithm I suggested relating to the 10% limit still holds at 10% – and NOT at 5 * _small_table_threshold.

With the very large _small_table_threshold, it would be useful to check how many blocks from the table were in the cache after the tablescan – it might be 10% of the cache. (You didn’t say what the cache size was in this test, was it about 110,000, or was it back to the original 220,000).

A possibly explanation for the 5x buffer cache anomaly might be that with a starting empty cache you fill the cache as the tablescan starts, and then keep recycling the bottom 25% of the cache from that point onwards.

  drop table cacher;
  create table cacher(c1 char(2000), c2 char(2000), c3 char(2000)) nologging;
  insert /*+ append */ into cacher
    select 'x', 'x', 'x'
    from dual
       connect by level <= 220000     ;
  EXEC DBMS_STATS.gather_table_stats(NULL, 'CACHER');
  commit;
  alter system flush buffer_cache;
  set autot on stat
  select count(*) from cacher;
  select count(*) from cacher;
  select count(*) from cacher;
  set autot off;

with parameters
_db_block_buffers 273102
_small_table_threshold 5462

80% of buffer cache Statistics with stats gathered
----------------------------------------------------------
     220033  consistent gets
     220005  physical reads
----------------------------------------------------------
     220023  consistent gets
          0  physical reads
----------------------------------------------------------
     220023  consistent gets
          0  physical reads

On my windows box 11.2.0.1 the caching seems to start out at 10% of the buffer cache, then if I change small_table_threshold, that value seems to be what is used for the caching limit:

with parameters
_db_block_buffers 51220 25% = 12805,10% = 5122
_small_table_threshold 1024

 drop table cacher;
  create table cacher(c1 char(2000), c2 char(2000), c3 char(2000)) nologging;
  insert /*+ append */ into cacher
    select 'x', 'x', 'x'
    from dual
    -- connect by level <= 5120    -- caches
       connect by level <= 5122    -- doesn't cache ;
  commit;
  alter system flush buffer_cache;
  select count(*) from cacher;
  select count(*) from cacher;
  set autot on stat
  select count(*) from cacher;
  set autot off;

I get these results

  3rd count(*) Statistics  10% of buffer cache size
  ----------------------------------------------------------
         5126  consistent gets
         5122  physical reads

now if I increase small table threshold and bounce the database:

alter system set "_small_table_threshold"=10240 scope=spfile;
startup force

I can now cache more on full table scan:

Statistics with size just below small table threshold
----------------------------------------------------------
      10091  consistent gets
      10004  physical reads
----------------------------------------------------------
      10011  consistent gets
          0  physical reads
----------------------------------------------------------
      10011  consistent gets
          0  physical reads
Statistics for FTS with size just above small table threshold
----------------------------------------------------------
      11101  consistent gets
      11288  physical reads
----------------------------------------------------------
      11004  consistent gets
      11000  physical reads
----------------------------------------------------------
      11004  consistent gets
      11000  physical reads

Also if I set small table threshold greater than the size of the buffer cache I get no caching with a table bigger than the buffer cache:

alter system set "_small_table_threshold"=1100000 scope=spfile;
startup force;
Statistics
-----------------------------------------------------
     102545  consistent gets
     102522  physical reads
-----------------------------------------------------
     102460  consistent gets
     102425  physical reads
-----------------------------------------------------
     102460  consistent gets
     102425  physical reads

where as on LINUX I get surpringly helpful caching with a table 5x the buffer cache:

_db_block_buffers 273102
_small_table_threshold 5462

500% bigger buffer cache
———————————————————-
1350131 consistent gets
1350002 physical reads
———————————————————-
1350037 consistent gets
1177881 physical reads
———————————————————-
1350037 consistent gets
1177856 physical reads