Oracle Scratchpad

March 2, 2011

First_Rows

Filed under: CBO,Execution plans,Oracle,Troubleshooting — Jonathan Lewis @ 9:37 pm GMT Mar 2,2011

Browsing through the archive for the Oracle-L listserver a couple of days ago I came across this item dated Feb 2011 where the author was puzzled by Oracle’s choice of index for a query.

He was using 10.2.0.3, and running with the optimizer_mode set to first_rows – which you shouldn’t really be doing with that version of Oracle since Oracle Corp. told us about 10 years ago that “first_rows is avaiable only for backwards compatibility”.

I’ve created a model of their problem to demonstrate the effect. As usual, to make it easier to get a reproducible result, I’ve used locally managed tablespaces with 1MB uniform extents, freelist management, and CPU costing disabled:


create table t1
as
with generator as (
	select	--+ materialize
		rownum id
	from dual
	connect by
		rownum <= 10000
)
select
	to_char(
		to_date('01-Jan-2011','dd-mon-yyyy') +
			trunc((rownum-1)/317),
		'yyyymmdd'
	)			a,
	mod(rownum,317) + 1	b,
	lpad(rownum,10,'0')	small_vc,
	rpad('x',100)		padding
from
	generator	v1,
	generator	v2
where
	rownum <= 100000
;

alter table t1 add constraint t1_pk primary key(a,b);
create index t1_ba on t1(b,a);

The SQL creates 317 rows for a list of dates which have been stored as eight character strings in the form YYYYMMDD. The 317 rows are numbered from 1 to 317, and the data is stored in order of date and number. I’ve created a primary key on (date, number), and I’ve also created an index on (number, date) – the PK has a very good clustering_factor and the other index has a very bad one because of the way I generated the data.

With this data in hand, and after collecing statistics (compute, no histograms), I run the following SQL (and like the OP I am using 10.2.0.3):

alter session set optimizer_mode = first_rows;

set autotrace traceonly explain

select
	small_vc
from
	t1
where
	a = '20110401'
and	b > 10
order by
	a, b
;

select
	/*+ index(t1(a,b)) */
	small_vc
from
	t1
where
	a = '20110401'
and	b > 10
order by
	a, b
;

I’m after 307 consecutive rows of one date – and I want the data sorted by the date and number. With first_rows optimization the default plan is a little surprising. Here are two execution plans for the query – first the plan that the optimizer chose by default, the second when I hinted the SQL to use the primary key – note that neither plan shows a sort operation:

Default execution plan
---------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost  |
---------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |   307 |  7368 |   617 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |   617 |
|*  2 |   INDEX SKIP SCAN           | T1_BA |   307 |       |   309 |
---------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("B">10 AND "A"='20110401')
       filter("A"='20110401' AND "B">10)

Hinted execution plan
---------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost  |
---------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |   307 |  7368 |    10 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    10 |
|*  2 |   INDEX RANGE SCAN          | T1_PK |   307 |       |     2 |
---------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("A"='20110401' AND "B">10)

How strange – it is clearly better to use the primary key index for this query, yet the optimizer doesn’t do it under first_rows optimisation. (It does if you use the slightly more appropriate first_rows(1) – the “new” improved option from 9i).

The first thought you might have when looking at this example is the first_rows has a heuristic (i.e. rule) that says “use an index to avoid sorting at all costs if possible (unless the hidden parametere _sort_elimination_cost_ratio is non-zero)”. But that shouldn’t apply here because both indexes will allow Orace to avoid sorting.

And here’s an even stranger detail: notice that the “order by” clause includes column “a”, which is obviously constant because of the “where” clause. Since it’s constant removing it won’t make any difference to the final output – but look what happens:

select
	small_vc
from
	t1
where
	a = '20110401'
and	b > 10
order by
	b
;

---------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost  |
---------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |   307 |  7368 |    10 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    10 |
|*  2 |   INDEX RANGE SCAN          | T1_PK |   307 |       |     2 |
---------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("A"='20110401' AND "B">10)
       filter("A"='20110401' AND "B">10)

For no obvious reason the optimizer now picks the right index. What’s going on ? Unfortunately I have to say that I don’t know.

When I checked the 10053 trace file there were a few differences for the two “order by” clauses but I couldn’t see anything that gave me any reasonable ideas. The most significant difference was the choice of indexes examined when the optimizer was doing the “recost for order by” bit. When we ordered by a,b the optimizer considered only the t1_ba index (note – the final costs are slightly higher here because in this run I enabled CPU costing to see if that was having an effect, so there’s a little extra for the CPU):

  Access Path: index (skip-scan)
    SS sel: 0.0030744  ANDV (#skips): 308
    SS io: 308.00 vs. index scan io: 321.00
    Skip Scan chosen
  Access Path: index (SkipScan)
    Index: T1_BA
    resc_io: 617.00  resc_cpu: 23882848
    ix_sel: 0.0030744  ix_sel_with_filters: 0.0030744
    Cost: 618.60  Resp: 618.60  Degree: 1
  Best:: AccessPath: IndexRange  Index: T1_BA
         Cost: 618.60  Degree: 1  Resp: 618.60  Card: 307.44  Bytes: 24

when we ordered by b alone the optimizer considered only the t1_pk index:

  Access Path: index (RangeScan)
    Index: T1_PK
    resc_io: 10.00  resc_cpu: 191334
    ix_sel: 0.0030744  ix_sel_with_filters: 0.0030744
    Cost: 10.01  Resp: 10.01  Degree: 1
  Best:: AccessPath: IndexRange  Index: T1_PK
         Cost: 10.01  Degree: 1  Resp: 10.01  Card: 307.44  Bytes: 24

There really seems to be a flaw in the logic behind the choice of index – and there’s an important point to think about here: if it’s a bug it’s probably not going to be fixed. The first_rows option only exists for “backwards compatibility” and things stop being compatible if you change them.

Footnote: Because the cost of the skip scan path in the original run was 617 and the cost of the primary key range scan path was 10 I could make Oracle choose the primary key by setting the parameter _sort_elimination_cost_ratio to a value just less than 617/10 (say 60); but I mention that only as an idle curiosity. You shouldn’t be using first_rows , and if you do use it you shouldn’t be hacking with undocumented parameters to work around the problems it produces.

14 Comments »

  1. Jonathan,

    on a slightly different topic – your usage of the INDEX hint in this particular case is interesting. Do you think Oracle should use T1_BA (I mean obey) because of the order of the columns or it really shouldn’t? The documentation is not clear on this matter, but a simple change in the order of indexes creation works differently (this is 11.2.0.2 with OFE=’10.2.0.3′, but it behaves the same way with setting it to ‘11.2.0.2’):

    create index t1_ba on t1(b,a);
    alter table t1 add constraint t1_pk primary key(a,b);
    ...
    -------------------------------------------------------------------------------------
    | Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
    -------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT            |       |   307 |  7368 |    31   (0)| 00:00:01 |
    |   1 |  TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    31   (0)| 00:00:01 |
    |*  2 |   INDEX SKIP SCAN           | T1_BA |   307 |       |    15   (0)| 00:00:01 |
    -------------------------------------------------------------------------------------
    
    Predicate Information (identified by operation id):
    ---------------------------------------------------
    
       2 - access("B">10 AND "A"='20110401')
           filter("A"='20110401' AND "B">10)
    
    
    Execution Plan
    ----------------------------------------------------------
    Plan hash value: 1071398364
    
    -------------------------------------------------------------------------------------
    | Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
    -------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT            |       |   307 |  7368 |    31   (0)| 00:00:01 |
    |   1 |  TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    31   (0)| 00:00:01 |
    |*  2 |   INDEX SKIP SCAN           | T1_BA |   307 |       |    15   (0)| 00:00:01 |
    -------------------------------------------------------------------------------------
    
    Predicate Information (identified by operation id):
    ---------------------------------------------------
    
       2 - access("B">10 AND "A"='20110401')
           filter("A"='20110401')
    

    So I think it isn’t absolutely safe to use such INDEX hint notation in the demo scripts with multiple indexes that cover the same columns set.

    Comment by Timur Akhmadeev — March 3, 2011 @ 8:14 am GMT Mar 3,2011 | Reply

    • Timur,

      I believe the new format index hint should pick the index that matches the column list in exactly the column order. If no such index exists then it should pick an index that starts with those columns in the right order. If there is no index that starts correctly then the hint is invalid.

      I was under the impression that I had tested this carefully – but I may have forgotten to test the effects of creating the indexes in a different order. And I don’t think I have seen anything in the documentation that confirms my hypothesis.

      One consequence of my assumptions is that a hint /*+ index(t1(a,b)) */ will NOT allow the optimizer to use an index (b,a) if the index (a,b) – or any index starting (a,b, …) exists.

      One thing about your tests – if you do:

      create index t1_ba on t1(b,a);
      alter table t1 add constraint t1_pk primary key (a,b);
      

      Oracle doesn’t create a new index (a,b) because it can protect the primary key with the existing non-unique index (b,a). Is this the reason for the choice of the “wrong” index – the right one doesn’t exist.

      One thing I did find, thanks to a couple of extra tests that your comment made me run, is that the “order by b” version of the query seemed to use whichever index had been created first.

      Comment by Jonathan Lewis — March 3, 2011 @ 9:19 am GMT Mar 3,2011 | Reply

      • Oracle doesn’t create a new index (a,b) because it can protect the primary key with the existing non-unique index (b,a).

        Yep, this is it. I completely forgot about such thing.

        Comment by Timur Akhmadeev — March 3, 2011 @ 9:48 am GMT Mar 3,2011 | Reply

  2. > The first thought you might have when looking at this example is the first_rows has a heuristic
    Indeed. I did.

    On a similar case on OTN a while back, http://forums.oracle.com/forums/message.jspa?messageID=3231089#3231089 ,there was also a suggestion that the naming of the index might make a difference – i.e. first in alphabetical order. I did a quick test similar to above and it didn’t seem to.

    Comment by Dom Brooks — March 3, 2011 @ 12:20 pm GMT Mar 3,2011 | Reply

    • Dom.
      I’d forgotten all about that thread. It’s a pity that it died after I made the suggestion to rename the index. It would have been nice to have confirmation that it (probably) didn’t have an effect.

      Looking at it again, I do wonder if the first_rows algorithm recosts all index-driven paths that bypass the “order by” – but, in this (these) cases, “knows” that the current best path is index-driven and therefore doesn’t include it in the recosting, then forgets to cross-check its cost against the ones it has recosted. It would be a fairly simple logic (or even counting) error to make this mistake.

      Comment by Jonathan Lewis — March 3, 2011 @ 5:53 pm GMT Mar 3,2011 | Reply

  3. Sorry Jonathan,

    Is it possible to let me know why I have the following observations?

    Thanks in advance

    SQL> select * from v$version
      2  where rownum=1;
    
    BANNER                                                                                                      
    -----------------------------------------------------------------------
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production                                        
    
    SQL> alter session set optimizer_features_enable='10.2.0.3';
    
    Session modifiée.
    
    SQL> select index_name, table_name
      2  from user_indexes
      3  where table_name='T1';
    
    INDEX_NAME       TABLE_NAME                                                                     
    ------------------------------                                                  
    T1_PK              T1                                                                             
    T1_BA              T1                                                                             
    
    SQL> alter session set optimizer_mode = first_rows;
    
    Session modifiée.
    
    SQL> select
      2      small_vc
      3  from
      4      t1
      5  where
      6      a = '20110401'
      7  and b > 10
      8  order by
      9      a, b
     10  ;
                                                                                         
    
    307 ligne(s) sélectionnée(s).
    
    SQL> select * from table(dbms_xplan.display_cursor);
    
    PLAN_TABLE_OUTPUT                                                                                           
    ---------------------------------------------------------------------------------------------
    SQL_ID bdfa9jku43jfh,child number 2                                                                         
    -------------------------------------                                                                       
    select     small_vc from     t1 where     a = '20110401' and b > 10                                           
    order by     a,b                                                                                             
                                                                                                                
    Plan hash value: 3756972603                                                                                                                                                                                        
    --------------------------------------------------------------------------------------                      
    | Id  | Operation                    | Name  | Rows  | Bytes | Cost (%CPU)| Time     |                      
    --------------------------------------------------------------------------------------                      
    |   0 | SELECT STATEMENT             |       |       |       |    11 (100)|          |                     
    |   1 |  SORT ORDER BY               |       |   307 |  7368 |    11  (10)| 00:00:01 |                      
    |   2 |   TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    10   (0)| 00:00:01 |                      
    |*  3 |    INDEX RANGE SCAN          | T1_PK |   307 |       |     2   (0)| 00:00:01 |                      
    --------------------------------------------------------------------------------------                                                                                                                               
    Predicate Information(identified by operationid):                                                           
    ---------------------------------------------------                                                                                                                                              
     3 - access("A"='20110401' AND "B">10)                                                                      
                                                                                                                
    SQL> select
      2      /*+ index(t1(a,b)) */
      3      small_vc
      4  from
      5      t1
      6  where
      7      a = '20110401'
      8  and b > 10
      9  order by
     10      a, b
     11  ;
                                                                               
    307 ligne(s) sélectionnée(s).
    
    SQL> select * from table(dbms_xplan.display_cursor);
    
    PLAN_TABLE_OUTPUT                                                                                           
    -----------------------------------------------------------------------------------
    SQL_ID  20ppz5r9gk99q, child number 2                                                                       
    -------------------------------------                                                                       
    select     /*+ index(t1(a,b)) */     small_vc from     t1 where     
    a ='20110401' and b > 10 order by  a,b                                                                                                                                                                                  
    Plan hash value: 3756972603                                                                                                                                                                                         
    --------------------------------------------------------------------------------------                      
    | Id  | Operation                    | Name  | Rows  | Bytes | Cost (%CPU)| Time     |                      
    --------------------------------------------------------------------------------------                      
    |   0 | SELECT STATEMENT             |       |       |       |    11 (100)|        |                                                                                            
    |   1 |  SORT ORDER BY               |       |   307 |  7368 |    11  (10)| 00:00:01 |                      
    |   2 |   TABLE ACCESS BY INDEX ROWID| T1    |   307 |  7368 |    10   (0)| 00:00:01 |                      
    |*  3 |    INDEX RANGE SCAN          | T1_PK |   307 |       |     2   (0)| 00:00:01 |                      
    --------------------------------------------------------------------------------------                      
                                                                                                     
    Predicate Information(identified by operationid):                                                           
    ---------------------------------------------------                                                                                                                                                 
     3 - access("A"='20110401' AND "B">10)
    

    Comment by hourim — March 5, 2011 @ 8:47 am GMT Mar 5,2011 | Reply

    • Since you’re running 11.2.0.1 the first guess would be that Oracle has recognised a bug in earlier versions and fixed it – have you checked the release notes to see if there is anything about “bad choice of index under first_rows” ?

      Although optimizer_features_enable will make the optimizer revert to earlier behaviour, I don’t think there’s any guarantee that it will re-introduce bugs that were present in earlier versions.

      In your case I’d look at the 10053 trace to see if my comment about the “recost” is still true:


        The most significant difference was the choice of indexes examined when the optimizer was doing the “recost for order by” bit. When we ordered by a,b the optimizer considered only the t1_ba index

      Comment by Jonathan Lewis — March 5, 2011 @ 8:57 am GMT Mar 5,2011 | Reply

  4. Dear Jonathan,

    I generated the 10053 trace file and, if I have not missed anything, I can say that there is no recost for order by(using index) when I ordered by a,b
    and there is a recost for order by(using index) when I ordered by b but this recost doesn’t change the original best access path.

    This is the 10053 trace file when I ordered by a, b

    optimizer_features_enable           = 11.2.0.1
    optimizer_mode                      = first_rows
    
    Access path analysis for T1
    ***************************************
    SINGLE TABLE ACCESS PATH 
      Single Table Cardinality Estimation for T1[T1] 
    
      Table: T1  Alias: T1
        Card: Original: 100000.000000  Rounded: 307  Computed: 307.44  Non Adjusted: 307.44
      Access Path: TableScan
        Cost:  506.27  Resp: 506.27  Degree: 0
          Cost_io: 503.00  Cost_cpu: 35196627
          Resp_io: 503.00  Resp_cpu: 35196627
    kkofmx: index filter:"T1"."A"='20110401'
    
      Access Path: index (skip-scan)
        SS sel: 0.003074  ANDV (#skips): 308.000000
        SS io: 308.000000 vs. index scan io: 503.000000
        Skip Scan chosen
      Access Path: index (SkipScan)
        Index: T1_BA
        resc_io: 617.00  resc_cpu: 23882848
        ix_sel: 0.003074  ix_sel_with_filters: 0.003074 
        Cost: 619.22  Resp: 619.22  Degree: 1
    
      Access Path: index (RangeScan)
        Index: T1_PK
        resc_io: 10.00  resc_cpu: 191334
        ix_sel: 0.003074  ix_sel_with_filters: 0.003074 
        Cost: 10.02  Resp: 10.02  Degree: 1
      ****** trying bitmap/domain indexes ******
      ****** finished trying bitmap/domain indexes ******
      Best:: AccessPath: IndexRange
      Index: T1_PK
             Cost: 10.02  Degree: 1  Resp: 10.02  Card: 307.44  Bytes: 0
    ***************************************
    
    

    And this is the 10053 trace file when I ordered by b only

    
    optimizer_features_enable           = 11.2.0.1
    optimizer_mode                      = first_rows
    
    Access path analysis for T1
    ***************************************
    SINGLE TABLE ACCESS PATH 
      Single Table Cardinality Estimation for T1[T1] 
    
      Table: T1  Alias: T1
        Card: Original: 100000.000000  Rounded: 307  Computed: 307.44  Non Adjusted: 307.44
      Access Path: TableScan
        Cost:  506.27  Resp: 506.27  Degree: 0
          Cost_io: 503.00  Cost_cpu: 35196627
          Resp_io: 503.00  Resp_cpu: 35196627
    kkofmx: index filter:"T1"."A"='20110401'
    
      Access Path: index (skip-scan)
        SS sel: 0.003074  ANDV (#skips): 308.000000
        SS io: 308.000000 vs. index scan io: 503.000000
        Skip Scan chosen
      Access Path: index (SkipScan)
        Index: T1_BA
        resc_io: 617.00  resc_cpu: 23882848
        ix_sel: 0.003074  ix_sel_with_filters: 0.003074 
        Cost: 619.22  Resp: 619.22  Degree: 1
    
      Access Path: index (RangeScan)
        Index: T1_PK
        resc_io: 10.00  resc_cpu: 191334
        ix_sel: 0.003074  ix_sel_with_filters: 0.003074 
        Cost: 10.02  Resp: 10.02  Degree: 1
      ****** trying bitmap/domain indexes ******
      ****** finished trying bitmap/domain indexes ******
      Best:: AccessPath: IndexRange
      Index: T1_PK
             Cost: 10.02  Degree: 1  Resp: 10.02  Card: 307.44  Bytes: 0
    
    
    ****** Recost for ORDER BY (using index) ************
    Access path analysis for T1
    ***************************************
    SINGLE TABLE ACCESS PATH 
      Single Table Cardinality Estimation for T1[T1] 
    
      Table: T1  Alias: T1
        Card: Original: 100000.000000  Rounded: 307  Computed: 307.44  Non Adjusted: 307.44
      Access Path: TableScan
        Cost:  506.27  Resp: 506.27  Degree: 0
          Cost_io: 503.00  Cost_cpu: 35196627
          Resp_io: 503.00  Resp_cpu: 35196627
    
      Access Path: index (RangeScan)
        Index: T1_PK
        resc_io: 10.00  resc_cpu: 191334
        ix_sel: 0.003074  ix_sel_with_filters: 0.003074 
        Cost: 10.02  Resp: 10.02  Degree: 1
      Best:: AccessPath: IndexRange
      Index: T1_PK
             Cost: 10.02  Degree: 1  Resp: 10.02  Card: 307.44  Bytes: 24
    
    

    Best Regards

    Comment by hourim — March 6, 2011 @ 7:35 am GMT Mar 6,2011 | Reply

    • The trace file shows us that the logic has changed in 11.2. It’s then an interesting philosophical question whether the code for first_rows analysis has been modified (which seems unlikely if it exists for backwards compatibility only) or whether there is a structure to the optimisation strategy that has changed – thus removing a few possible paths from consideration – and it’s the arithmetic that changes within the boundaries of that structure. (I’d guess the latter.)

      It might be possible to construct a complete answer by doing several more experiments using a larger collection of indexes on longer lists of columns – but that’s not a project I’d want to pursue at present.

      Comment by Jonathan Lewis — March 6, 2011 @ 8:05 pm GMT Mar 6,2011 | Reply

      • I know we shouldn’t be using first_rows anyway but…

        The 11.2.0.1 behaviour reported above by hourim is inconsistent with my 11.2.0.2 at least.

        Running with 11.2.0.2, optimizer_features_enable=11.2.0.2 and optimizer_mode=first_rows, my 10053 trace showed only T1_BA as considered in the recost.

        Having said that, although I thought I was initially able to reproduce the 11.2.0.1 change by way of optimizer_features_enable & optimizer_mode, I now think I must have been careless because subsequent attempts failed.

        Comment by Dom Brooks — March 7, 2011 @ 2:18 pm GMT Mar 7,2011 | Reply

  5. It seems like hit Bug 7430474(FIRST_ROWS_K recost for ORDER BY elimination may not find best plan.)

    Comment by johnwuwuwu — March 15, 2011 @ 8:47 am GMT Mar 15,2011 | Reply

    • Johnwuwuwu,

      Thanks for that.

      Although the title suggests the fix is for the “proper” first_rows_K optimisation, the description suggests that the implementation would have made it apply to the old first_rows as a side-effect (pretty much as I surmised in my comment above.

      Comment by Jonathan Lewis — March 15, 2011 @ 10:18 pm GMT Mar 15,2011 | Reply

  6. You have a very interesting blog and I enjoy reading it. However, I would like to point out that some Oracle users are not using Oracle in anyway close to how it was intended. We have legacy software 40 years old (yes, still sells) that uses a legacy B-tree ISAM type data base. Almost every single SQL statement is to get a single row with all of the columns for the index explicitly filled out and certainty that the index exists. These calls always give the hint first_rows(1) and specify the index. This hint alone increased the performance of our application by factors of 10. Still does under 11g. Now we also do other calls that ask for groups of rows, and since a lot is screen I/O we built in a learning algorithm. If I ask for the next record (e.g. next row, and ISAM thing) then it also hints with first_rows(50) — 50 being tuneable. But, if I keep asking for more the hints go away letting Oracle do it’t things the best it can. This is generally reports. So the first_rows and index hints are VERY valuable and valid when doing something rather crazy like treating Oracle as if it was an ISAM database. Blanket statements by people that hints should not be used, etc etc, drive me nuts.

    Comment by Eric Eberhard — June 30, 2014 @ 10:43 pm GMT Jun 30,2014 | Reply

  7. […] was correct; I’ve written the first_rows option a few times in the past – it was left in for backwards compatibility, and reported as such from 9i […]

    Pingback by First Rows | Oracle Scratchpad — October 27, 2014 @ 7:21 am GMT Oct 27,2014 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 4,520 other followers