Oracle Scratchpad

November 25, 2009

Counting

Filed under: Uncategorized — Jonathan Lewis @ 7:10 pm UTC Nov 25,2009

In one of those little coincidences that seem to happen so often a question came up on the comp.databases.oracle. server news group that has an answer in a blog note I’d written just a few of days earlier . The question was simply asking for a way of counting the number of rows in each partition of a partitioned table.

Of course, there are lots of ways of doing this, depending on how much time you want to write the code, how efficient you want the code to be, how generic you want the code to be, and so on; but here’s a little script that I wrote in about five minutes based on the ‘partition marker’ that appeared in recent versions of Oracle.

select
	obj.object_name,
	obj.subobject_name,
	rct.row_count
from
	(
	select
		/*+ no_merge */
		object_name,
		subobject_name,
		data_object_id
	from
		user_objects
	where
		object_type like 'TABLE%'
	and	subobject_name is not null
	)	obj,
	(
	select
		/*+ no_merge */
		dbms_mview.pmarker(rowid)	data_object_id,
		count(*) 			row_count
	from
		pt_range        -- Your table name goes here.
	group by
		dbms_mview.pmarker(rowid)
	)	rct
where
	rct.data_object_id = obj.data_object_id
order by
	obj.object_name,
	obj.subobject_name
;

It’s very simple, and also happens to demonstrate a good use of the /*+ no_merge */ hint. (I could have used subquery factoring with the /*+ materialize */ hint to similar effect).

My partitioned table is called pt_range  and I’ve had to include its name in the query. The query is then written in two halves; the first half finds the names and data_object_ids of all the partitions (or subpartitions) of tables in the user’s schema (and I could have put in a restriction on object_name at that point, of course). The second half counts rows by data_object_id (which is what the dbms_mview.pmarker(rowid) function returns. Then it’s simply a question of joining the two result sets (using an outer join if you want to list partitions with no rows) and reporting the desired columns in the order you like.

I’ve used the no_merge hint to allow Oracle to do the aggregation early – at the most efficient point possible; the alternative would be to do the join and then aggregate on all the other columns reported, but would be a waste of resources. The no_merge hint is probably redundant in the query against the user_objects view – but for a quick hack it’s quite sensible to block the optimizer from trying to get too clever with the code you’re writing.

If you don’t approve of using the dbms_mview.pmarker() function (because of the reliance on the undocumented fact that it returns the data_object_id) you could use the dbms_rowid.rowid_object() function which also returns the data_object_id (despite its name) for a given rowid.

The code isn’t fast, though – so you might want to go one better and trying use a substr() on the rowid to extract the encoded data_object_id, aggregate on that, and then write your own function to turn the encoded form into a decimal form.

Bear in mind, though, whatever you do Oracle will have to scan the entire table and call whichever function you apply on every single row, and then aggregate across the entire result set. So even though the code looks quite clean and simple, it’s not efficient.

If efficiency combined with flexibility is important you might be better off writing a (pipelined) pl/sql function that takes a table-name as an input and loops through each partition in turn with a simple count(*).

 

Flash Cache

Filed under: Uncategorized — Jonathan Lewis @ 6:50 pm UTC Nov 25,2009

You may have heard about “flash cache” for Oracle 11.2 and Exadata – the clever trick where the code can use SSD (solid state disc) for data that’s “fairly popular” but not quite popular enough to be kept in the data cache by the LRU. The code works to keep a copy of the more useful data on the faster medium to avoid re-reading it from disk.

The technology was originally restricted to Exadata2, but as Guy Harrison reports, a recent patch allows it to run on Oracle Enterprise Linux even when you don’t have Exadata.

 

Drop Table

Filed under: Uncategorized — Jonathan Lewis @ 1:34 pm UTC Nov 25,2009

Here’s an interesting question raised by Uwe Hesse: “if you drop a table when someone else is querying it, what happens to their query?”

It’s an interesting demonstration of how many features you have to consider before you reach a conclusion – and while you may start by assuming that the question is irrelevant to a production system, when you think a little more carefully that’s not necessarily the case.

 

November 21, 2009

ora_hash function

Filed under: Uncategorized — Jonathan Lewis @ 10:59 am UTC Nov 21,2009

A few weeks ago I wrote a note demonstrating the way in which Oracle’s strategy for hash partitioning is engineered to give an even data distribution when the number of partitions is a power of two. In one of the follow-up comments, Christo Kutrovsky pre-empted my planned follow-up by mentioning the hashing function ora_hash() that appeared in 10g.

This function takes up to three parameters: the input value (which can be of any base type), the maximum hash bucket value (the minimum value is implicitly zero) and a seed value (default zero) for the hash function. So, for example, ora_hash(‘fred’,1023) will return an integer value between zero and 1023.

The ora_hash() function is similar to the packaged function dbms_utility.get_hash_value({expression},{min},{max}) that appeared in 8i, although the older function only handles character expressions (using an implicit conversion for dates and numbers), requires you to specify the minimum and maximum bucket values, and doesn’t allow you to introduce a seed value.

In fact, for character inputs, the ora_hash() and get_hash_value() functions can be mapped to each other very easily by the relationship:

dbms_utility.get_hash_value(string,1,N) = 1 + ora_hash(string,N,0)

So, for example, we can run a little query like:


select
	1 + ora_hash('fred',1023,0)			ora_hash_plus_one,
	dbms_utility.get_hash_value('fred',1,1024)	get_hash_value
from
	dual
/

ORA_HASH_PLUS_ONE GET_HASH_VALUE
----------------- --------------
              904            904

One important feature of the ora_hash() function is that it seems to be the function used internally – with a zero seed – to determine which partition a row belongs to in a hash partitioned table. Of course, as I pointed out in the previous article, you have to tailor the use of the function to the number of partitions you have declared – rounding up to the nearest power of two (and subtracting one) for the “max bucket” input, then adding one to the result, then dropping the highest bit of the result if the number of partitions is not a power of two.

Here’s a little demonstration, based on a table with six partitions (i.e. poor choice of partition count), running under 11.1.0.6:

create table t1 (
	id,
	v1
)
partition by hash(v1)
partitions 6
as
select
	rownum,
	object_name
from
	all_objects
where
	rownum <= 32768
;

execute dbms_stats.gather_table_stats(user,'t1')

select	partition_position, partition_name, num_rows
from	user_tab_partitions
where	table_name = 'T1'
order by
	partition_position
;

column partition_name new_value m_pt

select	partition_name
from	user_tab_partitions
where	table_name = 'T1'
and	partition_position = 3
;

select
	ora_hash(v1, 7)				ora_hash,
	dbms_utility.get_hash_value(v1,1,8)	get_hash_value,
	dbms_mview.pmarker(rowid)		pmarker,
	count(*)
from
	t1 partition( &m_pt )
group by
	ora_hash(v1, 7),
	dbms_utility.get_hash_value(v1,1,8),
	dbms_mview.pmarker(rowid)
;

You’ll see that I’ve collected stats on the table then reported each partition’s position, name, and number of rows . This is the result I got from that query:


PARTITION_POSITION PARTITION_NAME         NUM_ROWS
------------------ -------------------- ----------
                 1 SYS_P608                   3989
                 2 SYS_P609                   4055
                 3 SYS_P610                   8356
                 4 SYS_P611                   8320
                 5 SYS_P612                   4083
                 6 SYS_P613                   3965

As you might expect from the previous article, two of my partitions (numbers 3 and 4) are twice the size of the rest. If I were to issue a ‘split partition’ call then partition 3 would split (roughly 50/50) into partitions 3 and 7, and on a second ‘split partition’ call partition 4 would split (again 50/50) into partitions 4 and 8.

To demonstrate the effect of the ora_hash() function and how it relates to hash partitioning I’ve then run a query that references partition 3 by name (using an earlier query to convert a partition position into a partition name).

You’ll notice that my call to ora_hash() uses the value seven for the maximum hash bucket – this is because (a) my ideal partition count is eight (the smallest power of two that is not less than six), and (b) although I want values between 1 and 8 the function will be generating values between zero and 7. Given the relationship between ora_hash() and dbms_utility.get_hash_value() – and given that I’m applying the functions to a character string – I’ve also called the get_hash_value() function to show that its behaviour really is consistent with ora_hash() across the entire data set. Here are the results from the final query:


  ORA_HASH GET_HASH_VALUE    PMARKER   COUNT(*)
---------- -------------- ---------- ----------
         2              3      86154       4194
         6              7      86154       4162

You can see that both functions have split my set of values into two distinct sets in exactly the same way – values that would go into a new partition three on a “split partition”, and values that would go into partition seven. And the split is roughly 50/50 (to within 1%).

As a little bonus, I’ve also included a call to dbms_mview.pmarker() in the query. This was a function that Oracle introduced in 9i (I think) to allow more refresh options for materialized views on partitioned objects when you used partition maintenance operations on them. The function returns an identifier for the partition of a partitioned table that the rowed belongs to. (In fact all it is effectively doing is returning the data_object_id from dba_objects for the containing partition).

Once you’re aware of how ora_hash() works you may be able to find ways to take advantage of it. Occasionally I’ve been asked if it’s possible to work out which partition some data is going to go into – now you know that the answer is yes.

To be investigated: Does the ora_hash() function play any part in the distribution of data across hash clusters ? (Probably not – the manuals make some comments about rounding up the supplied number of hashkeys to the next larger  prime number, which suggests that a different hashing algorithm is used.

To be confirmed: Does the ora_hash() function plan any part in parallel query distribution by hash ? One quick test suggests not – but the method of use in this case may be more subtle than the “obvious” guess that I used in my test.

November 20, 2009

OOW09 Video

Filed under: Uncategorized — Jonathan Lewis @ 7:12 pm UTC Nov 20,2009

I’ve just discovered I’m on YouTube in an impromptu video interview I gave while I was at Oracle World this year. It’s only three minutes, and doesn’t have a lot of technical content, but here’s the URL.

Footnote: Rumour has it that next year the event will be rebranded as My Oracle World – implemented entirely in Flash.

 

November 4, 2009

Anonymous

Filed under: Uncategorized — Jonathan Lewis @ 9:00 pm UTC Nov 4,2009

From time to time I browse the comments that Akismet (the spam detector used by WordPress) has intercepted just to check that a useful comment hasn’t been unjustly labelled as spam; and a couple of days ago I was surprised to find that four comments (from the same person) had been marked as spam even though there was no obvious reason why they had been so marked.

They were clearly rubbish, of course – one expressed suprise that I hadn’t published my resume [sic] (it could do with an acute accent over the final e, but apart from that it’s American for C.V. ), another expressed the opinion that my blog showed I had no experience with ACTUAL [sic] working databases … and so on.

But how had Akismet worked out that these comments were garbage ?

The author had supplied a gmail account name – and I think that comments from the gmail domain have triggered spam exclusion in the past but only when the comment included a hyperlink. Then I thought of tracing the IP address that WordPress has captured – and it turned out to lead back to an “anonymizing proxy server”.

Clever trick that – if it’s coming from someone who’s trying to hide, a comment probably deserves to be binned.

Back to the USA – etc.

Filed under: Uncategorized — Jonathan Lewis @ 7:30 pm UTC Nov 4,2009

California, Bulgaria (yes, I know it’s not in the USA), Virginia and Texas

It’s nice after a long trip to get a few email messages from people who have heard you speak and enjoyed the experience – so thanks to the people who let me know what they thought of the seminars and presentations I’ve done in the last few weeks.

For an idea of how things went – here’s a review from Bob Watkins of one of the full day seminars.

UKOUG Conference Series

The agenda for the “Technical and E-Business Suite”  conference (30th Nov – 2nd Dec) is now available.

I’m doing a 45 minute presentation on  “Writing Optimal SQL” at 14:50 on Monday 30th, and another 45 minutes on “Introducing Partitions” at 13:15 on Tuesday 1st.

There’s also going to be a “round table” – which means everyone gets their chance to talk – on Cost Based Optimisation at 12:35 on Wednesday 3rd that I’ll be chairing.

Future options

And while we’re looking ahead – Oracle Education has started talking about trips to Switzerland in Feb, Germany in March, and Greece in Apr. I’m also talking to Oracle in the APAC region about  a trip to China (Beijing and Shanghai) in March.  Watch this space.

October 31, 2009

Top Ten

Filed under: Uncategorized — Jonathan Lewis @ 10:41 pm UTC Oct 31,2009

Sorry, this isn’t a posting about efficient ways of getting the first 10 rows from a result set – it’s a little note about Oracle Open World. I don’t brag very often, but sometimes it’s hard to resist.

I’ve just received an email about Openworld with the following content:

Go to Oracle OpenWorld On Demand
Get in on sessions you missed, or hear keynotes and executive solution sessions again.
FREE for all full conference, Oracle Develop, and Primavera program attendees.
Only US$245 for all others.

    • Find all keynotes, general sessions, executive solution sessions, and breakout sessions—more than 1,900 presentations—in an online streaming rich media portal
    • Get keynotes with synchronized video, slides, scrolling transcripts, and downloadable MP3s and MP4s
    • Comment and rate presentations online and post your own content
  • For 1,900 presentations, $245 doesn’t seem to be an extreme price – although it might be nicer to be able to get a set of DVDs rather than downloading all those pdfs, mp3s and adobe flash files. But I was registered anyway, so I just clicked on the link, signed in, and got to the opening portal page to find a list of the “Top Ten Presentations”, in this order:

    • Keynote: Extreme Innovation - Scott McNealy, Sun Microsystems, Inc.
    • Keynote: Larry Ellison, Oracle
    • Keynote: Innovation Across the Stack -  Thomas Kurian, Oracle
    • Keynote: Oracle Develop – What Are We Still Doing Wrong? – Thomas Kyte
    • General Session: Oracle Fusion Middleware 11g – Foundation for Innovation -  Hasan Rizvi
    • General Session: Lower Costs with Oracle Database 11g Release 2 - Robbins Jeffrey, Masayuki Karahashi, Andy Mendelsohn
    • Keynote: The Art of the Possible - Charles Phillips and Safra Catz, Oracle
    • How to Hint -  Jonathan Lewis
    • Accelerating Java for Better Enterprise Performance - Charlie Hunt, John Pampuch
    • General Session: Complete, Open, and Integrated Applications for Your Business - Andy Platt, Paco Aubrejuan, Janet Foutty, Anthony Lye,  Steve Miranda, Syed Nadeem, Jewell Richard
    Looking at the list of keynotes and well-known Oracle names above mine, I think I’m currently the top “civilian” (at the moment) – so I’m going to feel just a little smug for the rest of the weekend.

    October 26, 2009

    Philosophy – 8

    Filed under: Uncategorized — Jonathan Lewis @ 8:33 pm UTC Oct 26,2009

    Btree indexes vs. Bitmap indexes – the critical difference

    • A single B-tree index allows you to access a small amount of data very precisely.
    • It is the combination of a subset of the available bitmap indexes that offers the same degree of precision.

    You should not be comparing the effectiveness of a bitmap index with the effectiveness of a b-tree index.

    (Inevitably it’s a little more subtle than this – you may create some low-precision b-tree indexes to avoid foreign key locking issues,the optimizer can combine b-tree indexes, and so on - but if you start from this basis you will have a rational view about how to use bitmap indexes).

    Footnote: remember, also, that bitmap indexes introduce massive concurrency issues and other maintenance overheads.

    October 23, 2009

    Quiz Night

    Filed under: Uncategorized — Jonathan Lewis @ 6:55 pm UTC Oct 23,2009

    Okay, so it’s not night-time, or even early evening, in some parts of the world – but somewhere it’s Friday night, and Friday night is quiz night.

    Now, most people are aware that the clustering_factor of an index measures the ordering (or, if you want to be slightly more accurate, the clustering) of the rows in a table relative to the order of entries in the index. Consequently  if you check the clustering_factor before and after rebuilding an index it cannot change.

    So how do you explan this output – which comes from a clean cut-n-paste, with no selective editing, from an SQL*Plus session running Oracle 10.2.03 with echo on - with no other sessions active on the system:

    
     SQL> execute dbms_stats.gather_index_stats(user,'i1', estimate_percent=>100)
    
    PL/SQL procedure successfully completed.
    
    SQL>
    SQL> select
      2          clustering_factor
      3  from       user_indexes
      4  where      index_name = 'I1'
      5  ;
    
    CLUSTERING_FACTOR
    -----------------
                  396
    
    1 row selected.
    
    SQL>
    SQL> alter index i1 rebuild pctfree 10;
    
    Index altered.
    
    SQL>
    SQL> execute dbms_stats.gather_index_stats(user,'i1', estimate_percent=>100)
    
    PL/SQL procedure successfully completed.
    
    SQL>
    SQL> select
      2          clustering_factor
      3  from       user_indexes
      4  where      index_name = 'I1'
      5  ;
    
    CLUSTERING_FACTOR
    -----------------
                  198
    
    1 row selected.
    

    You have two minutes, starting from NOW.

    October 20, 2009

    Understanding

    Filed under: Uncategorized — Jonathan Lewis @ 7:27 pm UTC Oct 20,2009

    Karen Morton has a few wise words to say about understanding vs. memorization. Definitely worth reading and understanding – and maybe even memorizing.

    October 18, 2009

    Experts

    Filed under: Uncategorized — Jonathan Lewis @ 7:30 pm UTC Oct 18,2009

    Chen Shapira wrote a very nice note about the unconference (and Oracle Closed World) presentation that I did at Oracle Open World this year, loosely titled “The Beginner’s Guide to being an Oracle Expert”. In her comments she has captured a point more clearly than I have ever expressed it: “DBAs are under a lot of pressure not to be experts.”

    I am not a working DBA – yet I spend a lot of my time solving problems for DBAs and helping DBAs do their jobs better. How do I manage this ? By knowing more than they do about the technology they are using***; and I’m in that position because I have the time to investigate and test and experiment – time that a working DBA is usually not given.

    It’s easy for me to make that time, of course, because I am self-employed and create my own timetable – and on my timetable I aim to average one day per week working at home investigating mechanisms and solving problems: call it my R&D budget. I’ll charge you more for the time I spend on your site because I’ve spent time in my “lab” making sure I know the answers to your questions before you even ask them. 

    I have the time to be an expert because I can choose how I use my time; the typical DBA is not so lucky. If you want to be an Oracle expert, time is the most important commodity.

    *** I should point out that there are areas of the technology where I am not an expert – so there have been occasions when people have asked me to do a job and I have suggested they contact a colleague who is an expert in that particular field.

    Footnote: as I glanced through this note just before publishing it, it cross my mind that DBAs are often advised to ignore the wait event “SQL*Net Message from Client” as it’s only “think time” and doesn’t affect performance. When you’re aim is to become an expert “think time” becomes very important, and ends up making a huge impact on performance.

    October 16, 2009

    Correlation

    Filed under: Uncategorized — Jonathan Lewis @ 7:14 pm UTC Oct 16,2009

    One of the “inspirational thoughts” on my opening page is the observation by the late Stephen J. Gould that

    The invalid assumption that correlation implies cause is probably among the two or three most serious and common errors of human reasoning.”

     It’s very easy to equate correlation with causation and take inappprioate action as a result – it’s an example of faulty thinking that I see fairly frequently on forums such as OTN or the Oracle newsgroups.

    If you want to get an insight into the difference between correlation and causation, you ought to read Robyn Sands’ note on “Nonsense Correlation”.

    October 13, 2009

    OpenWorld

    Filed under: Uncategorized — Jonathan Lewis @ 7:12 pm UTC Oct 13,2009

    I’ve just attended one of the most enjoyable sessions I’ve had at any conference. Richard Foote’s unconference session on indexes. It consisted of two slides for the introduction, followed by an open invitation for questions.

    The really nice thing about it was the level of audience participation. There were several interesting questions, of course, but more importantly the audience also offered answers and variations on the questions; so we had some interesting discussions about how specific circumstances might make a “well-known” solution less appropriate than an alternative strategy.

    Topics touched on included:

    • index skip scans – how often to people see it (and is it the right choice)
    • unique indexes on partitioned tables
    • order of index columns
    • choosing between coalesce and rebuild
    • recognising critical indexes
    • the difficulty of getting rid of indexes once someone has created them

    There was some much discussion Richard had virtually no chance to say anything about the music of David Bowie – although a couple of quick questions popped up at the very end of the hour.

    October 12, 2009

    Philosophy – 7

    Filed under: Uncategorized — Jonathan Lewis @ 8:26 pm UTC Oct 12,2009

    There are no “secrets”.

    At least, there are no secrets involved in making a database perform properly. Occasionally a useful new piece of information is uncovered – and if it’s worth knowing it will be documented, discussed, and validated in public.  (It won’t necessarily be documented on Metalink, OTN, or tahiti.oracle.com – but that doesn’t make it a secret.)

    Whenever I’ve seen people doing presentations about “secrets” they’ve usually split their time between stating the obvious,  making mistakes, and offering sweeping generalisations that needed careful justification.

    I have a simple rule of thumb for presentations – the more glamorous, trendy or exciting the title sounds the less likely it is that the presentation will be useful (but that won’t stop me reading the abstract – just in case).

    [Back to Philosophy- 6]

    Next Page »

    Blog at WordPress.com.