I posted this question on twitter earlier on today (It was a thought that crossed my mind during a (terrible) presentation on partitioning that I had to sit through a few weeks ago – no matter how bad a presentation is I always seem to be able to get a couple of interesting questions out of it):
Quiz: if you have a table with a date column date_col and create a virtual column defined as trunc(date_col,’W’) and partition on it – will a query on date_col result in partition elimination?
The answer is yes – at least for the version of Oracle that I happened to have to hand (12c) the next time I had a few minutes spare. Here’s a quick and dirty demo – with data content relevant to the publication date of this blog note so you may need to adjust the code to your current date if you want to run the test.
rem rem Script: pt_virtual_date.sql rem Author: Jonathan Lewis rem Dated: Oct 2013 rem Purpose: drop table transactions purge; create table transactions ( transaction_date date not null, job_id varchar2(10) not null, account_id number(8) not null, transaction_type varchar2(2) not null, transaction_id varchar2(10) not null, amount number(10,2) not null, padding varchar2(100), transaction_week generated always as (trunc(transaction_date,'IW')) ) partition by range (transaction_week) interval (numtodsinterval(7,'day')) ( partition p0 values less than (to_date('30-Sep-2013','dd-mon-yyyy')) ) ; insert into transactions( transaction_date, job_id , account_id , transaction_type, transaction_id , amount , padding ) values(trunc(sysdate)-7, 1, 1, 'IN', 1, 1, rpad('x',100)) ; insert into transactions( transaction_date, job_id , account_id , transaction_type, transaction_id , amount , padding ) values(trunc(sysdate), 2, 2, 'IN', 2, 2, rpad('x',100)) ; insert into transactions( transaction_date, job_id , account_id , transaction_type, transaction_id , amount , padding ) values(trunc(sysdate) + 7, 3, 3, 'IN', 3, 3, rpad('x',100)) ; begin dbms_stats.gather_table_stats( ownname => user, tabname =>'transactions', method_opt => 'for all columns size 1' ); end; /
So transaction_date is a column in the table but the partitioning column is transaction_week, which is defined as trunc(transaction_date,’IW’) which gives me Mondays as the partition boundaries. (I always go back to the manuals for truncating on weeks – there are at least two options (W and IW) and there’s a significant difference in the way they are defined.)
So here are a couple of simple queries – will the optimizer do partition elimination ?
select * from transactions where transaction_date = sysdate; select * from transactions where transaction_date = to_date('17-Oct-2013','dd-mon-yyyy');
And the answer is yes. Note the “partition range single” that appears at operation 1 in both case even though the Pstart and Pstop show KEY/KEY – which means Oracle has to deduce which partitions are relevant at run-time.
------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | ------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 130 | 2 (0)| 00:00:01 | | | | 1 | PARTITION RANGE SINGLE| | 1 | 130 | 2 (0)| 00:00:01 | KEY | KEY | |* 2 | TABLE ACCESS FULL | TRANSACTIONS | 1 | 130 | 2 (0)| 00:00:01 | KEY | KEY | ------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("TRANSACTION_DATE"=SYSDATE@! AND "TRANSACTIONS"."TRANSACTION_WEEK"=TRUNC(SYSDATE@!,'fmiw')) ------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | ------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 130 | 2 (0)| 00:00:01 | | | | 1 | PARTITION RANGE SINGLE| | 1 | 130 | 2 (0)| 00:00:01 | KEY | KEY | |* 2 | TABLE ACCESS FULL | TRANSACTIONS | 1 | 130 | 2 (0)| 00:00:01 | KEY | KEY | ------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("TRANSACTION_DATE"=TO_DATE(' 2013-10-17 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "TRANSACTIONS"."TRANSACTION_WEEK"=TRUNC(TO_DATE(' 2013-10-17 00:00:00', 'syyyy-mm-dd hh24:mi:ss'),'fmiw'))
Pretty clever !
I find it slightly strange that the predicate section also reports the derived predicate on the virtual column – it’s clearly not necessary for this example because we can see that the optimizer has already decided that the query will visit just one partition and we (and, I would have thought, includes the optimizer) know that every row in that partition will have the same value for the virtual column. Possibly its presence is just a side effect of the generic strategy used for more complex predicates and the special-case code hasn’t been included, but maybe there’s something I’ve overlooked that makes it necessary. (It might be interesting to experiment with adding a constraint describing the relationship between transaction_date and transaction_week to see what happens … but that’s left as an exercise.)
Hi Jonathan.
Pretty clever indeed.
This behavior is new in 12c (I’ve just tested it in 11.2).
It reminds me another clever feature in 12c: if we create an index on TRANSACTION_WEEK in your example (with or without the partitioning), and select “where TRANSACTION_DATE = :x”, the index is used:
This is also true if we don’t have the virtual column, and we just create the function based index.
Thanks,
Oren.
Comment by Oren Nakdimon @DBoriented — October 17, 2013 @ 10:06 pm BST Oct 17,2013 |
Oren,
Excellent contribution, thank you very much.
That may be related to the appearance of the apparently redundant predicate. Perhaps this is an extension of generating new predicates though constraints and transitive closure that gives Oracle the option of finding more optimisation options.
Comment by Jonathan Lewis — October 17, 2013 @ 10:11 pm BST Oct 17,2013 |
I also tested in in 11.2.0.3 and generated these plans:
The results posted in the previous comment are the result of adding a global index to your example. If the example is run as-is I get the same results in 11.2.0.3 as you do in 12c.
David Fitzjarrell
Comment by dfitzjarrell — October 17, 2013 @ 10:56 pm BST Oct 17,2013 |
David,
Thanks for the note – I’ve just repeated the test in 11.2.0.4, and the “partition range single” appears in that version too.
Oren,
Did you test an earlier version of 11.2, or can you find some other detail of your configuration (e.g. optimizer parameter) that could explain the difference ?
Comment by Jonathan Lewis — October 17, 2013 @ 11:07 pm BST Oct 17,2013 |
Indeed, Jonathan, it was my ancient 11.2.0.1 database.
David, it seems that you created the index on TRANSACTION_DATE, is that right?
The point I was making is that if you create an index on TRANSACTION_WEEK (the virtual column), the optimizer can use it even for the condition on TRANSACTION_DATE. And I tested it now in more versions: doesn’t work on 11.2.0.1, works as of 11.2.0.2.
It’s fun (yet embarrassing) discovering that features you thought to be new are actually several years old…
Thanks,
Oren.
Comment by Oren Nakdimon @DBoriented — October 18, 2013 @ 5:42 am BST Oct 18,2013 |
Quite recently I ran a little test on 12c and discovered something I hadn’t seen before – so I started re-running the test on older versions of Oracle, and discovered it did the same thing all the way back to at least 8.1.7.4. (I don’t recall what the detail was, it was just one of those tiny but clever touches that’s very impressive when you happen to notice it.)
Comment by Jonathan Lewis — October 18, 2013 @ 2:36 pm BST Oct 18,2013
Two remarks. First here are my tests on 11.2.0.3 where the predicate section doesn’t report the virtual column
Second, I have a sligthly different case in the same database where a table t1 is list partitioned by a virtual column (DAY_IN_MONTH) which is defined as follows
In this typical configuration and table design, I am not getting the same results when selecting from T1 using the CRE_DATE instead of the DAY_IN_MONTH virtual column
But when I use the virtual column the partition elimination occurs as shown below:
And not in passing that the predicate part is not present :-)
Comment by hourim — October 18, 2013 @ 9:31 am BST Oct 18,2013 |
Hi Mohamed,
>>predicate section doesn’t report the virtual column
It seems that dbms_xplan.display_cursor doesn’t display it (not stored in v$sql_plan) when dbms_xplan.display (after explain plan) displays it.
But the PARTITION RANGE SINGLE means that the predicate was considered.
Cheers,
Franck.
Comment by Franck Pachot — October 18, 2013 @ 1:48 pm BST Oct 18,2013 |
Mohamed,
Interesting example of a EXPLAIN PLAN and actual execution being different – even thought it’s not in the predicate section in your first comment. (I don’t expect to see the partition-related predicate if it is an exact match for the entire contents of a single partition – or even for a consecutive range of partitions). Interesting, though, that the predicate is visible when the path switches to indexed access (I tried it for local and global indexes).
Your second point is another example where we feel the optimiser could be just that little bit smarter – especially given the truncate(,’IW’) example – again your requirement is very like Stefan’s.
Why is truncate difference – I did have one thought – there is a parameter: _truncate_optimization_enabled set to TRUE, which looks promising, but it’s enabled back in 10.2.0.1. At which point I rememberer that object truncating got an optimization some time back there where seg$ ended up being updated once instead of once per extent remove.
There’s nothing obvious in the 10053 trace file to suggest why the truncate(,’IW’) works – the extra predicate just suddenly appears in the unparsed subquery. (And yes, it’s in the unparsed text even though it then disappears from the display_cursor() call).
Comment by Jonathan Lewis — October 18, 2013 @ 4:02 pm BST Oct 18,2013 |
Hi Jonathan,
great article. It seems like that the terrible presentation was useful for something anyway :-)) Unfortunately Oracle is not that clever by using virtual columns with data type conversions.
For example – something like that (on 12c because of invisible virtual columns):
Queries with predicates on column JINUM will force an “PARTITION RANGE ALLL” starting by 1 up to 1048575. Queries with predicates on column JINUM2 will use partition pruning. I currently can not think of a valid reason why Oracle is not able to check the input of JINUM for “numeric characters only” and decide between two plans based on that finding. I know that it is better to use the correct data type right from the start, but unfortunately you can not change this in SAP environments due to specific database layer handling :-((
Regards
Stefan
Comment by Stefan Koehler — October 18, 2013 @ 9:52 am BST Oct 18,2013 |
Stefan, you should only “encourage” the optimizer a little bit ;-)
Following your example,
select * from tab1_part where JINUM=’42’
will not cause partition pruning, but
select * from tab1_part where JINUM=42
will.
Comment by Oren Nakdimon @DBoriented — October 18, 2013 @ 10:09 am BST Oct 18,2013 |
Hi Oren,
i would love to, but the predicate data type (in reality binds are used) is generated by SAP DBSL depending on the SAP DDIC definitions and so no chance to change the predicate from “JINUM = VARCHAR2 TYPE” to “JINUM = NUMBER TYPE” … the whole construct is like raping the database of course, but i hoped that the optimizer would be that clever in 12c :-((
Best Regards
Stefan
Comment by Stefan Koehler — October 18, 2013 @ 10:50 am BST Oct 18,2013 |
Just recently, I was looking at approaches to partition a table with a number “date” of format yyyymmdd and I was looking at partitioning by a virtual column of date datatype.
I had distant hopes that the optimizer might just be clever enough to figure out the partition pruning without me having to change a lot of SQL to use the virtual column.
But it wasn’t. No surprise really.
But because you can do interval partitioning on a number – e.g. “interval(1)” for daily or “interval(100)” for monthly – it wasn’t too big a disappointment.
E.g.
Comment by Dom Brooks — October 18, 2013 @ 10:09 am BST Oct 18,2013 |
“… not too big a disappointment.”
Presumably because you did at least partition the data the way you wanted, even though the optimizer couldn’t do partition elimination for existing application code.
Comment by Jonathan Lewis — October 18, 2013 @ 2:38 pm BST Oct 18,2013 |
Right, disappointing but not a showstopper.
Desire to reverse engineer best practice by moving away from number dates to date dates remains unfulfilled.
But the tables could be still be interval partitioned by number, although in hindsight I would have stuck with traditional range partitioning.
Comment by Dom Brooks — October 18, 2013 @ 3:11 pm BST Oct 18,2013 |
Dom,
Just realised that you’re trying to do the same as Stefan, except he’s trying to “turn characters into numbers” and you’re trying to “turn numbers into dates”. Looking at the 10053 Oracle gets SO CLOSE if you add a predicate that reiterates virtual column definition – in his case:
Comment by Jonathan Lewis — October 18, 2013 @ 3:22 pm BST Oct 18,2013 |
I like the interchange of thoughts that this post has generated – but I’m going to have to spend a little time reading the comments before I can catch up.
Comment by Jonathan Lewis — October 18, 2013 @ 2:00 pm BST Oct 18,2013 |
[…] answer before I did, so I won’t repeat it. The posting was prompted by an email I got about the previous posting, suggesting that the apparently redundant predicate might have been generated to avoid exactly this […]
Pingback by Quiz Night | Oracle Scratchpad — October 18, 2013 @ 2:26 pm BST Oct 18,2013 |
[…] stage of the investigation I remembered that Jonathan Lewis has blogged about partitioning using virtual column where I have already presented (see comment n°3) a similar case to what I have been, […]
Pingback by Partition by virtual column | Mohamed Houri’s Oracle Notes — October 1, 2014 @ 4:19 pm BST Oct 1,2014 |
[…] this failure, though, there are cases (as I showed a couple of years ago) where the optimizer in 12c can get clever enough to recognize the connection between a queried […]
Pingback by Virtual Partitions | Oracle Scratchpad — May 23, 2016 @ 1:17 pm BST May 23,2016 |
[…] how the optimizer tries to find as much helpful information as it can from the data dictionary. The earliest note I can find on my blog about this at present is about partition elimination and generated predicates – which […]
Pingback by Generated Predicates | Oracle Scratchpad — March 2, 2022 @ 11:25 am GMT Mar 2,2022 |