Thanks, it’s always useful to have observations like this about new behaviour.

I compared read performance of 10g and 11g some weeks ago and found more or less accidentally the discussed behaviour.
An FTS on a big table answered after 3 minutes in 10g but after 1 minute in 11g due to the direct path reads. Executing the statement a second time it anwered after 1 second in 10g because the blocks were cached but again after 1 minute in 11g. At the first glance it seemed to be a strange behaviour if not a bug. But probably it is a feature to avoid flushing buffer cache by large FTSs.

I performed some tests to influence the behaviour. I tried switching off parallel execution, setting _serial_direct_read=false and changing db_file_multiblock_read_count, without any result. What I’m not tested until now is to change parameters that influence the file IO like filesystemio_option and so on.

Then I changed the size of the table and observed that the turning point is approximately at 10% of the buffer cache. If the table is bigger then a FTS makes direct path read otherwise db file scattered read.

Frank Haney

I haven’t investigated what triggers the choice – but there must be some type of cost-based decision since any direct reads have to start with an object checkpoint to write any dirty blocks for that object back to disc before the read starts.

I’ve rerun the test again, more carefully this time, and the results indicate that you are probably right about the waits being caused by reading from the filesystem although they are not traced.

When I make sure the file is indeed cached in memory by measuring subsequent runs but the first, cpu and elapsed time are almost the same :

From V$SQL:

ELAPSED_TIME 59704747
CPU_TIME 59704231
USER_IO_WAIT_TIME 356

When I said that iostat was not showing any read activity, I was apparently watching the subsequent runs, not the first one due to an error in my script. Still, even with the first run, the filesystem read waits are not accounted for at all in the trace file and only partially in the V$SQL counters as can be seen in my first message.

Thank you.

Val

I just ran a quick test.
No waits of any kind recorded for reading from the filesystem – and there really should have been some, even if they were virtually zero. Just 2 waits for direct path writes, although 10,000 blocks written by direct path.

The former could be indicative of an uninstrumented wait, the latter of a possible cause of highly concurrent CPU usage as the async processes take over the writes.

I was getting close to suggesting that 2Gb at 50Mb/s is roughly the 44 seconds – but that doesn’t match with your comment about iostat :(

I did look through the raw file, but except the small number of waits that tkprof already saw, there was nothing else. The active session history table snapshots always showed ‘ON CPU’ with no other waits.

The 20 million rows were coming from an external table ‘p_et’ (a flat file 2 GB in size) which explains why there were so few reads. The file itself was most likely heavily cached in memory due to previous runs, so there was almost no IO judging by iostat. Essentially, as I understand the insert amounted to moving memory blocks between the OS file cache and SGA (db buffer).

The timeslice preemption is a good idea, but at the time there was no other process running on the four-cpu 8GB box, so it’s unlikely that the 44 seconds can be explained by that.

I encountered quite a few situation like that where I could not explain a similar difference, but this one is easily reproducible and ‘clean’ in the sense that almost no IO was happening, no concurrent activity was going on, and yet substantial time was spent !

Thanks.

It’s not possible to make a solid statement about where the lost time went – there may simply be some uninstrmented waits going on. Have you read through the trace file itself to check for any clues ?

One possibility is your run-state. You have 20,000,000 rows inserted with only 4 direct path writes and 21 single block reads !

Possibly you did a lot more direct I/O (reads for a tablescan, writes for the append) which turned into asynch I/O, which consumed a lot of CPU. Your lost time might be your process being runnable but not running because it had used up its time-slice and the operating system had descheduled it and was allowing the asynch drivers to do their thing.

You could check what’s going on in v$osstat for the OS_CPU_WAIT_TIME while this insert was running. I’m not sure it’s 100% meaningful (see http://jonathanlewis.wordpress.com/2007/11/01/awr-oddity/#comments ) but it may give you a clue.

You could also use something like ‘top’ at the operating system to see where the CPU time is going.