The original question came up in (at least) two places on OTN – this is the one I saw and responded to.

Your comments about me saying “AWR” when really I should have been saying “9i” are correct, of course. There is some OS information in 10g, though the presentation is different; in fact if you enable it you can get some OS stats appearing in the 8i and 9i Instance Activity.

My inference from the db file read figures is that since they’re so fast they must be coming from the file system cache – and if that’s the case then there’s going to be a big chunk of CPU being recorded against the OS time for file system manipulation. (I’ve assumed, of course, that the huge excess of memory, and the low version of Oracle, are good indicators of a simple set up.).

I think I made the point about confusing cause and effect in the OTN thread, but at this point I think it ought to be fairly clear that which is which. If anyone doesn’t get your point though, it’s worth mentioning that if you have a bit of a latch sleep problem then if everyone starts chewing up all the CPU (and without necessarily doing anything related to latches) then the latch sleeps get worse because it takes longer for a sleeper that’s woken up to get to the top of the run queue because of all the processes in the queue ahead of it that are using up their time slices.

Good alternative explanation for how we could lose about 8 Billion LIOs that took place in this time interval. The OP might benefit from running off the next two or three (or more) hours reports as well.

Predicate capture (statspack doesn’t do it): at this point, with such extreme problems, I’m only really interested in the big picture – are today’s (or this interval’s) execution plans the ones we’ve been seeing for weeks, or is today a very special day. I’ll worry about WHY there’s been a change after I’ve determined that a (massive) change has occurred.

Regarding SGA size: it may not be in the thread you saw, but the sga_max_size is set to 1.7GB, and the db_cache_size and shared_pool_size are set (and reported) as 300MB – it looks as if someone has allowed for manual adjustment up to the legal 32-bit max for the platform, but has not actually allocated it. So there seems to be about 1GB of memory doing nothing that ought to have been in the db cache to reduce the constant copying from file system to Oracle cache. But it is possible that an increase might make the wrong thing go faster – with the side effect (see “top of run queue” comment) that the thing we’re worried about might slow down even more.

In fact there’s a follow-up post from the same person comparing two consecutive days. I’m trying to find time to write some comments about it as “Statspack 13a”. It’s very educational.

The point of that comment was that the instance activity reports something like 14.4 Billion LIOs, so somewhere we’ve “lost” 8 or 9 billion. This means that solving the 5B problem might be nice (and it might even be easy) but there’s still a huge amount of work hiding somewhere that we’re still going to have to address.

Maybe the big one that we can see all happened in the seconde hour of the report, and there’s a similarly sized query that took place in the first hour. It’s always worth thinking about the things that we can’t see, and wondering whether there’s a way in which we can find them.

Thanks for the comment – it’s too easy to lose track of which bits appear in which version (though this was a 9i system, of course). The only bit I don’t forget is that Statspack 10 reported the event histograms that didn’t get into the AWR until 11 ;)

In many ways this boils down to the question: “Who knows what the system is supposed to do?”

Intuitively we feel that a query that takes 10,000,000 buffer gets and runs hundreds of times per day in an OLTP system has to be wrong – either it should only be running rarely, or it should be taking thousands of LIOs rather than millions. There are special cases, of course, and there’s a very large grey area where the reource usage looks suspicious but not necessarily outrageous. When in doubt, we need to be able to get a good answer to the question: “does this seem reasonable?”, and I often find that the answer to that question is that nobody knows, and you have to go through a lot of work trying to reverse engineer the requirement and the intent of the programmer.

and the machine had 12 CPUs

Where did you get this number? Original thread doesn’t show available cores count.

Working from the top down, the first thing we notice is that 2M logical I/Os per second is quite busy

Just to translate it into bytes makes sense: that’s around 16GiB/s, which could tell you more if you happen to know chipset – maybe that’s close to its limit.

Since this isn’t the AWR, of course, we don’t have the OS Stats available, so we can’t see if the rest of the machines CPU is being used up outside the database – but looking at the db file read figures I’ll bet it is

Could you please clarify what is it in the db file read figures which makes you think something else is eating up CPU outside the DB? BTW Statspack in 11g provides OS stats (10g doesn’t AFAIR), so it’s not only the tool that is an issue.

If you have extreme CPU usage in an Oracle system, then latch contention problems are very likely to show up – and that’s our next highest figure – AWR would probably tells us which latch but, again based on the db file read figures, I’ll guess it’s the cache buffers chains latches.

1. That’s not AWR which will tells the latch name, but modern Oracle kernel which has special names for specific latch free events, so Statspack in the 10g or 11g will show CBC latch in the Top 5 as well.
2. It’s hard to decide what is the cause and what is the consequences – latching is causing CPU saturation or extreme CPU usage causes latch waits to appear in Top 5. In this case I think it is the 1st scenario, i.e. too much block gets cause too much CPU usage due to too much latch acquisitions.

so there may be a couple more big ones that have fallen out of the library cache in the interim, perhap an hourly report on the previous hour would catch them

That is a very likely situation, especially with relatively small shared pool. Another possibility is that something is still being executed, and in this case Oracle kernel in 9i could not help Statspack in identifying a query which is still running at the end snapshot.

Trouble is, though, that unless you set statspack to run at level 6 you don’t capture the execution plans – unlike the AWR which capture them by default

… but it doesn’t help in some situation as it doesn’t capture predicates (Bug 7493519: ACCESS_PREDICATES AND FILTER_PREDICATES MISSING IN DBA_SQL_PLAN_HIST, fixed in 12c). I don’t remember if Statspack gathers the predicates correctly under level 6 – if it is, then Statspack is preferable tool for this task.

As a quick and dirty which might help a bit – put more of the sga_max_size into the buffer cache – it doesn’t happen automatically in 9.2 – and that may reduce the interaction between the O/S and Oracle and save a bit of latching and CPU.

If it is indeed 32-bit binaries then I’m not sure it’s possible to allocate more than 2.7G to SGA. If it is 64-bit then SGA increase should also be combined with the switching to direct IO and possibly to the huge pages if it’s Linux (hostname suggests so). But without fixing SQL it’s possible to make things worse: removing IO completely, the app will (try to) use more CPU, resulting in potentially worse latch contention and more problems.

What looks a bit suspicious to me in the report are Parallel operations: there were 3/3 DDL/DML statements parallelized, 54 DFO trees parallelized, 48 queries parallelized. I’m not sure if these numbers are consistent – but I think that even 9 PX slaves on a 12 core system could utilize 75% of the CPU if they catch the wrong plan. It could be related to a DDL on the index, perhaps, changing its degree although this is just a guess. Anyway, I think it’s worth looking if Parallel Execution is acceptable on this system or not.

Happy new year.

Could you please explain further your thoughts when you’ve wrote the following sentence :
“and even then that’s 40% of the total – so there may be a couple more big ones that have fallen out of the library cache in the interim, perhap an hourly report on the previous hour would catch them”

Do you think 5 billion of LR should represent more than 40% of the total?

Not generally , since Version 10 captures Statspack also OS stats in table stats$osstat.

Best Regards,
Petr

A single SQL was in its 3rd day of uninterrupted execution. Finally got permission from the business to shoot it and fix the blessed thing.

Typical j2ee “framework” disaster: auto-generated, 3.8Mrow-table joined to 4Mrow-GTT, with multiple subqueries on both predicates AND select column list coming from equally monster-joins. None of them with the slightest chance of ever using an index.
What I call a “legacy query”: our grandchildren will see its results.

The execution plan in EM was a very long, sorry list of nested over nested over nested loops of million row FTS…
But because the buffer cache was quite large, the whole lot fit in cache. Hence, a LOT of CPU being used and negligible reads.
And of course: lots of buffer cache waits and even much more lots of CPU waits!

Which prompted the supplier’s “Oracle guru” to *categorically* claim “you are starved of CPU – add more of it”.
Of course, that got shot in the bud.
After a little judicious re-coding and trimming of totally unnecessary nested subqueries, the whole lot finished in around 10 minutes.

More CPU. Yeah! Right…
(save us from “j2ee Oracle gurus”!…)