Oracle Scratchpad

September 4, 2012

Online Rebuild

Filed under: Index Rebuilds,Indexing,Oracle — Jonathan Lewis @ 5:46 pm BST Sep 4,2012

I’ve commented in the past about the strange stories you can find on the internet about how Oracle works and how sometimes, no matter how daft those stories seem, there might be something behind them. Here’s one such remark I came across a little while ago – published in two or three places this year:

“An index that enforces referential integrity cannot be rebuilt online.”

There are a couple of problems with this statement – first, of course, indexes don’t enforce referential integrity, though they may help to enforce uniqueness, and the so-called “foreign key” index may avoid a locking issue related to referential integrity: that’s splitting hairs a little bit, though, and we can probably guess what the author means by “indexes enforcing referential integrity”.  (An example demonstrating the problem would have been useful, though – it would have saved me from writing this note, and it might save other people from jumping to the wrong conclusion and taking unsuitable action as a consequence.)

So here’s a simple test (run under 11.2.0.3):

create table parent (
	id	number	not null,
	n1	number,
	constraint par_pk primary key(id)
);

create table child (
	id_p	number	not null,
	id_c	number	not null,
	n1	number,
	constraint chi_pk primary key (id_p, id_c),
	constraint chi_par_fk foreign key (id_p) references parent(id)
);

insert into parent values(1,1);
insert into child values (1,1,1);
commit;

alter index par_pk rebuild online;
alter index chi_pk rebuild online;

If you run this test you’ll find that both indexes rebuild online with no problems. The original claim is clearly false. Almost inevitably it wasn’t backed up with an example, or even a detailed explanation, but surely there must have been some reason that made the author make a claim that can so easily be proved false. So what can we do to create a problem ?

How about continuing the test like this:

alter table child drop constraint chi_par_fk;
alter table child drop primary key;

alter table child add constraint chi_pk primary key(id_p, id_c) deferrable;
alter index chi_pk rebuild online;

Note that I’ve got rid of anything to do with referential integrity here – I’m just using the child table as the target for an index and a primary key constraint – but I’m making the constraint deferrable. The attempt to rebuild the index online fails with Oracle error: “ORA-08108: may not build or rebuild this type of index online”. This is a tiny bit interesting, because what “type” of index is it that we’re not allowed to rebuild online ?

The obvious answer to the last question is: maybe it’s something to do with a non-unique index supporting a unique constraint (which is what we have to have if we want a primary key (or unique) constraint to be deferrable. So let’s continue with the experiment:

alter table child drop primary key drop index;

alter table child add constraint chi_pk primary key(id_p, id_c) using index (
	create index chi_pk on child(id_p, id_c)
);
alter index chi_pk rebuild online;

This time the index rebuild is successful – so it doesn’t seem to be the lack of uniqueness (as such) that causes the problem.

Until further notice, I’m working on the hypothesis that the index protecting a deferrable primary key (or unique) constraint (which I’ve also tested – you can modify the code I’ve shown so far if you want to check for yourself) cannot be rebuilt with the online option. It’s possible that there’s some requirement relating to not null constraints in there as well, but I haven’t got around to looking at that yet.

Footnote: this is an investigation that isn’t complete – and that’s okay as far as I’m concerned because:

  • I don’t yet NEED to complete it because I don’t have a problem to solve; but when I see an ORA-08108 on a production system I’ll remember this anomaly and check to see if it’s relevant and do any further investigation that’s needed at that point.
  • I don’t intend to go around generalising a single observation into a misleading statement – I’ll let other people review the test case, consider my current hypothesis, and leave them with the opportunity to extend the investigation as they see fit.

Footnote 2: One reason why people make silly statements like this is that they read the manuals. But sometimes (a) they read out of date manuals and (b) they assume the manuals are correct, even when they make surprisingly unlikely statements. Here’s a quote from the SQL Reference manual for 9.0.1 under the alter index command:

 Restrictions:

  • Parallel DML is not supported during online index building. If you specify ONLINE and then issue parallel DML statements, Oracle returns an error.
  • You cannot specify ONLINE for a bitmap index or a cluster index.
  • You cannot specify ONLINE when rebuilding an index that enforces a referential integrity constraint

Footnote 3: Notwithstanding any comments about bitmap indexes that you can find in the 9.0.1 manuals, I didn’t seem to hit a problem with the online option when I tried it with a bitmap index on a simple heap table in 10.2.0.3, and the trace file showed the standard journal IOT being created and dropped as the rebuild operation took place.

13 Comments »

  1. Jonathan,

    Interesting, thanks for the information.

    I have not read this book yet (I do not plan to read it), but it is currently a fast seller in the Oracle category on Amazon:

    In Amazon’s preview for the book, searching on the keywords “enforces referential integrity”, an errata report should be filed for pages 88 and 90. In addition to essentially stating what you quoted above, the book states that bitmap indexes cannot be rebuilt online.

    Comment by Charles Hooper — September 4, 2012 @ 8:03 pm BST Sep 4,2012 | Reply

    • Sorry for the off topic reply.

      Another errata for the book the I linked to above on page 83 regarding function based indexes (in case anyone likes to file errata reports). The book states that for function based indexes to be considered by the optimizer, the QUERY_REWRITE_ENABLED parameter must NOT be set to FALSE. That was a true statement prior to Oracle Database 10.1, I believe. I am guessing that you might also see a recurrence of that obsolete function based index limitation.

      Comment by Charles Hooper — September 4, 2012 @ 8:40 pm BST Sep 4,2012 | Reply

      • Charles,

        “That was a true statement prior to Oracle Database 10.1″ – specifically it was a limitation of Oracle 8, removed by Oracle 9 (possibly not until 9.2, but I think it was 9.0.)

        Comment by Jonathan Lewis — September 5, 2012 @ 9:39 am BST Sep 5,2012 | Reply

    • Charles,

      Did you leave the word “online” off the end when reporting that last sentence ? Or was “bitmap indexes cannot be rebuilt” really it ?

      [Original comment updated in response to reply below.]

      Comment by Jonathan Lewis — September 5, 2012 @ 9:27 am BST Sep 5,2012 | Reply

      • Jonathan,

        Yes, the “ONLINE” keyword was unintentionally omitted [original now corrected - JPL]. I was attempting to quickly paraphrase the Tip and the bullet point item, and failed to properly proofread my comment before hitting the “Post Comment” button. I noticed the mistake an hour or two after posting the comment, and considered posting a third comment to correct the word omission but decided that I had already posted too much off-topic material in your article. Thank you for pointing out the error.

        I see that WordPress has now replaced the link that I supplied to the Amazon website with a picture of the front cover of the book – I was not expecting that change [Original edited to link URL to text, large image no longer appears - JPL]. Since it now appears that I intended to advertise the book, I took a look at another page of the book using Amazon’s preview. This time I will provide quotes rather than paraphrases (I have addressed very similar quotes in other book reviews, and I am trying hard not to review this book, so I have resisted the temptation to comment about the quotes):
        * “Indexes can only be used to find data that exists within a table. Whenever the NOT EQUAL operators are used in the WHERE clause, indexes on the columns being referenced cannot be used.” (page 63)
        * “You can create your indexes and analyze them in a single step by using the COMPUTE STATISTICS clause of the CREATE INDEX command.” (page 63)
        * “When you use IS NULL or IS NOT NULL in your WHERE clauses, index usage is suppressed because the value of NULL is undefined… Indexed columns that have rows containing a NULL value do not have an entry in the index (except for bitmap indexes…” (page 63)

        Regarding the QUERY_REWRITE_ENABLED parameter and function based indexes, that requirement was still listed in a couple of spots in the 9.2 documentation (but seems to be removed from the 10.1 documentation). That listing, of course, does not imply that the documented requirement is correct:

        http://docs.oracle.com/cd/B10500_01/server.920/a96533/data_acc.htm#2185

        Comment by Charles Hooper — September 5, 2012 @ 10:43 am BST Sep 5,2012 | Reply

        • Charles,

          I’ve updated the original comment to relect your comments above. I have no idea why the URL you supplied turned into a large image of the book, when I pasted it into my browser I got the full Amazon page.

          I’m not surprised you found it so easy to find other errors in the book. I think you’re wise, though, to avoid reading the book and commenting on it – I doubt if you’d learn anything new and the review might even break your previous records.

          I note the page you’ve linked to from the 9.2 manual also describes the query_rewrite_integrity paremeter incorrectly – in fact it looks like the description is a cut-n-paste from a part of the manual set dealing with materialized views anyway. A quick demo from 9.2.0.8 – partial listing, plus final execution plan:

          
          drop index t1_f1;
          
          create or replace function f(i in number) return number
          deterministic
          as
          begin 
          	return i+1;
          end;
          /
          
          create index t1_f1 on t1(f(n1));
          
          execute dbms_stats.gather_table_stats(user,'t1')
          
          alter session set query_rewrite_enabled=false;
          alter session set query_rewrite_integrity=enforced;
          
          set autotrace traceonly explain
          
          select 
          	n1, count(*) 
          from 
          	t1 
          where 
          	f(n1) = 2 
          group by 
          	n1
          ;
          
          set autotrace off
          
          Execution Plan
          ----------------------------------------------------------
             0      SELECT STATEMENT Optimizer=ALL_ROWS (Cost=9 Card=14 Bytes=56)
             1    0   SORT (GROUP BY) (Cost=9 Card=14 Bytes=56)
             2    1     TABLE ACCESS (BY INDEX ROWID) OF 'T1' (Cost=5 Card=15 Bytes=60)
             3    2       INDEX (RANGE SCAN) OF 'T1_F1' (NON-UNIQUE) (Cost=1 Card=15)
          

          Clearly the optimizer is happy to consider a user-defined index even when query rewrite is disabled, and rewrite integrity is enforced.

          Comment by Jonathan Lewis — September 6, 2012 @ 8:12 am BST Sep 6,2012

        • Jonathan,

          Thank you for putting together a short test case that demonstrates that the documentation for Oracle Database 9.2 is incorrect about the functionality of the QUERY_REWRITE_ENABLED and QUERY_REWRITE_INTEGRITY parameters, as those parameters apply to function based indexes in Oracle Database 9.2. It is typically a rare occurrence to find errors in the documentation, yet there are two in that page.

          OK, technically there are (at least) four errors in that page of the documentation (as the documentation applies to 9.2 or later). A couple of Richard Foote’s articles address the other errors, which are sometimes blindly reproduced in various books and discussion threads:
          “You can find indexes that are not being used by using the ALTER INDEX MONITORING USAGE functionality over a period of time that is representative of your workload. If you find that an index has not been used, then drop it.”

          http://richardfoote.wordpress.com/2008/09/12/index-monitoring-and-foreign-keys-caution-kid-a/

          “If all keys are used in WHERE clauses equally often, then ordering these keys from most selective to least selective in the CREATE INDEX statement best improves query performance.”

          http://richardfoote.wordpress.com/2008/02/13/its-less-efficient-to-have-low-cardinality-leading-columns-in-an-index-right/

          Comment by Charles Hooper — September 7, 2012 @ 5:41 pm BST Sep 7,2012

  2. Here is some research about deferable constraint internals http://www.ruoug.org/library/4/index.html.
    I don’t see any logical obstacles to implement online rebuild for the indexes serving deferable PK.

    It might be some difficulties in the implementation as oracle has to maintain some additional temporary index structure keeps object_id + the number of violations for the PK.

    Comment by Valentin Nikotin — September 5, 2012 @ 11:02 am BST Sep 5,2012 | Reply

    • Valentin,

      Thanks for that link – one of the many questions on my todo list was to find out how Oracle keeps track of a large number of changes that break deferrable constraints, so it’s nice to have the answer laid out.

      Comment by Jonathan Lewis — September 6, 2012 @ 8:14 am BST Sep 6,2012 | Reply

      • Hello Jonathan,

        Here are few funny consequences from deferrable constraints implementation. We can notice that oracle stores information about violations privately for the session. So what will happen if answer to the question “do we have violation or not?” depends on action in concurrent session? There will be locks :-)

        If we try to insert the same value in 2 parallel sessions – it will hung. Obviously is not necessary for the deferred constraints logic but is required by implementation:

        -- session 1
        SQL> create table t (id number primary key deferrable initially deferred);
         
        Table created
        SQL> insert into t values (1);
         
        1 row inserted
        -- session 2
        SQL> insert into t values (1);
        
        ... hungs on tx lock
        -- session 1
        SQL> commit;
         
        Commit complete
        -- session 2
        ... 
        1 row inserted
        SQL> commit;
         
        commit
         
        ORA-02091: transaction rolled back
        ORA-00001: unique constraint (TEST_USER.SYS_C007318) violated
        

        If we try to delete the row which is locked before by the same transaction … but unfortunately affects on the violation for other session – it hungs…

        -- session 1
        SQL> create table t (id number primary key deferrable initially deferred);
         
        Table created
         
        SQL> insert into t values (1);
        
        1 row inserted
        
        SQL> commit;
        
        Commit complete
        SQL> select rowid from t;
         
        ROWID
        ------------------
        AAAPZrAAEAAAClmAAA
        
        SQL> insert into t values (1);
         
        1 row inserted
        -- session 2
        SQL> select * from t where rowid = 'AAAPZrAAEAAAClmAAA' for update;
         
                ID
        ----------
                 1
                  
        SQL> delete t where rowid = 'AAAPZrAAEAAAClmAAA';
        
        ... hungs on tx lock
        -- session 1
        SQL> rollback;
         
        Rollback complete
        -- session 2
        ...
        
        1 row deleted
        

        This way for implementation allows us to perform instant successful commit (in the case of violation it takes time to rollback changes before raises error). We just store little structure which is able to give the fast answer “do we have any violations within our transaction”.

        The other possibility is to create MV log with rowid + fast refresh on commit IOT MV. Here we have to store double amount of key column data + rowid and will have long serialized commit (i.e. if we have 2 transaction which have inserted big amount of data simultaneously and start commit – one will wait for another refresh).

        Comment by Valentin Nikotin — September 6, 2012 @ 11:22 pm BST Sep 6,2012 | Reply

        • Valentin,

          Thanks for that.
          The first of the those two examples looked fairly reasonable, but the second one really surprised me (and puzzled me since it wasn’t immediately obvious how the lock wait could appear). I repeated the second test, but dumped the index block immediately after the first session had inserted its duplicate – and the result was another fascinating detail of how this deferrable stuff is implemented:

           Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
          0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
          0x02   0x0005.01a.0000f922  0x00803d17.256b.48  ----    3  fsc 0x0008.00000000
           
          ...
          
          row#0[8018] flag: ---D--, lock: 2, len=6
          col 0; len 2; (2):  c1 02
          col 1; len 0; (0): 
          row#1[8024] flag: ------, lock: 2, len=12
          col 0; len 2; (2):  c1 02
          col 1; len 6; (6):  01 80 0e 8a 00 00
          row#2[8006] flag: ------, lock: 2, len=12
          col 0; len 2; (2):  c1 02
          col 1; len 6; (6):  01 80 0e 8a 00 01
          
          

          The session shows three locked entries in the index after inserting a single row into the table.

          row#1 is the original index entry, and it has been marked as locked (lock: 2)

          row#0 is a copy of the key without a rowid, so it sorts first in order for that key value, but it’s marked as deleted and locked

          row#2 is the index entry for the new, duplicate, key.

          This means that the second session can select the original row by rowid for update – because that locks the TABLE row, but when it tries to do the delete the first index entry for the key value that Oracle hits is the deleted and locked entry above.

          (Since this is a test that started with a new table and index we can see the order of action by the address offsets of the index entries – they’re stacking into the block from the bottom upwards – 8024, 8018, 8006).

          Comment by Jonathan Lewis — September 7, 2012 @ 6:52 am BST Sep 7,2012

        • A couple details how does index structure work:
          it is shared for the pk violations
          it appears as the temp segmet on 257 changes (not violations as mentioned in article)

          test code:

          drop table t1;
          drop table t2;
          drop table t3;
          drop table t4;
          
          create table t1 (id number primary key deferrable initially deferred);
          create table t2 (id number primary key deferrable initially deferred);
          create table t3 (id number not null deferrable initially deferred);
          create table t4 (id number not null deferrable initially deferred);
          
          insert into t1 values (1);
          insert into t2 values (1);
          
          -- 256 changes in "pk" and "not null" supporting structures 
          insert into t1 select 1 from dual connect by level <= 100;
          insert into t2 select 1 from dual connect by level <= 100;
          insert into t3 select null from dual connect by level <= 130;
          insert into t4 select null from dual connect by level <= 131;
          delete from t1 where rownum <= 28;
          delete from t2 where rownum <= 28;
          delete from t3 where rownum <= 126;
          delete from t4 where rownum <= 125;
          
          -- no segment created
          select segblk#, segrfno# from v$tempseg_usage, v$session where saddr = session_addr and sid = userenv('sid');
          
          -- do one more changes
          delete from t1 where rownum = 1;
          delete from t3 where rownum = 1;
          delete from t4 where rownum = 1;
          
          -- segments appears
          select segblk# min_blk, segblk# + blocks - 1 max_blk,  segrfno# from v$tempseg_usage, v$session where saddr = session_addr and sid = userenv('sid');
          

          output:

          SQL> drop table t1;
          
          Table dropped.
          
          SQL> drop table t2;
          
          Table dropped.
          
          SQL> drop table t3;
          
          Table dropped.
          
          SQL> drop table t4;
          
          Table dropped.
          
          SQL> create table t1 (id number primary key deferrable initially deferred);
          
          Table created.
          
          SQL> create table t2 (id number primary key deferrable initially deferred);
          
          Table created.
          
          SQL> create table t3 (id number not null deferrable initially deferred);
          
          Table created.
          
          SQL> create table t4 (id number not null deferrable initially deferred);
          
          Table created.
          
          SQL> insert into t1 values (1);
          
          1 row created.
          
          SQL> insert into t2 values (1);
          
          1 row created.
          
          SQL> -- 256 changes in "pk" and "not null" supporting structures
          SQL> insert into t1 select 1 from dual connect by level <= 100;
          
          100 rows created.
          
          SQL> insert into t2 select 1 from dual connect by level <= 100;
          
          100 rows created.
          
          SQL> insert into t3 select null from dual connect by level <= 130;
          
          130 rows created.
          
          SQL> insert into t4 select null from dual connect by level <= 131;
          
          131 rows created.
          
          SQL> delete from t1 where rownum <= 28;
          
          28 rows deleted.
          
          SQL> delete from t2 where rownum <= 28;
          
          28 rows deleted.
          
          SQL> delete from t3 where rownum <= 126;
          
          126 rows deleted.
          
          SQL> delete from t4 where rownum <= 125;
          
          125 rows deleted.
          
          SQL> -- no segment created
          SQL> select segblk#, segrfno# from v$tempseg_usage, v$session where saddr = session_addr and sid = userenv('sid');
          
          no rows selected
          
          SQL> -- do one more changes
          SQL> delete from t1 where rownum = 1;
          
          1 row deleted.
          
          SQL> delete from t3 where rownum = 1;
          
          1 row deleted.
          
          SQL> delete from t4 where rownum = 1;
          
          1 row deleted.
          
          SQL> -- segments appears
          SQL> select segblk# min_blk, segblk# + blocks - 1 max_blk,  segrfno# from v$tempseg_usage, v$session where saddr = session_addr and sid = userenv('sid');
          
             MIN_BLK    MAX_BLK   SEGRFNO#
          ---------- ---------- ----------
                1792       1919          1
                2048       2175          1
                1920       2047          1
          

          blocks dump:

          ============================================================= t4 not null : 
          
          Block header dump:  0x00400701
           Object id on Block? Y
           seg/obj: 0x400700  csc: 0x00.15301c  itc: 2  flg: -  typ: 2 - INDEX
               fsl: 0  fnx: 0x0 ver: 0x01
           
           Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
          0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
          0x02   0xffff.000.00000000  0x00000000.0000.00  C---    0  scn 0x0000.0015301c
          Leaf block dump
          ===============
          header address 139764988541020=0x7f1d927e785c
          kdxcolev 0
          KDXCOLEV Flags = - - -
          kdxcolok 0
          kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
          kdxconco 1
          kdxcosdc 0
          kdxconro 5
          kdxcofbo 46=0x2e
          kdxcofeo 7967=0x1f1f
          kdxcoavs 7921
          kdxlespl 0
          kdxlende 0
          kdxlenxt 0=0x0
          kdxleprv 0=0x0
          kdxledsz 0
          kdxlebksz 8032
          row#0[8019] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 da 01 00 2b a3 00 7e
          row#1[8006] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 da 01 00 2b a3 00 7f
          row#2[7993] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 da 01 00 2b a3 00 80
          row#3[7980] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 da 01 00 2b a3 00 81
          row#4[7967] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 da 01 00 2b a3 00 82
          ----- end of leaf block dump -----
          
          
          ============================================================= t1 and t2 PK : 
          
          Block header dump:  0x00400781
           Object id on Block? Y
           seg/obj: 0x400780  csc: 0x00.153018  itc: 2  flg: -  typ: 2 - INDEX
               fsl: 0  fnx: 0x0 ver: 0x01
           
           Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
          0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
          0x02   0xffff.000.00000000  0x00000000.0000.00  C---    0  scn 0x0000.00153018
          Leaf block dump
          ===============
          header address 139764988541020=0x7f1d927e785c
          kdxcolev 0
          KDXCOLEV Flags = - - -
          kdxcolok 0
          kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
          kdxconco 1
          kdxcosdc 0
          kdxconro 2
          kdxcofbo 40=0x28
          kdxcofeo 8010=0x1f4a
          kdxcoavs 7970
          kdxlespl 0
          kdxlende 0
          kdxlenxt 0=0x0
          kdxleprv 0=0x0
          kdxledsz 4
          kdxlebksz 8032
          row#0[8021] flag: ------, lock: 0, len=11, data:(4):  47 00 00 00
          col 0; len 4; (4):  d6 f6 00 00
          row#1[8010] flag: ------, lock: 0, len=11, data:(4):  48 00 00 00
          col 0; len 4; (4):  d8 f6 00 00
          ----- end of leaf block dump -----
          
          
          ============================================================= t3 not null : 
          
          Block header dump:  0x00400801
           Object id on Block? Y
           seg/obj: 0x400800  csc: 0x00.15301a  itc: 2  flg: -  typ: 2 - INDEX
               fsl: 0  fnx: 0x0 ver: 0x01
           
           Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
          0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
          0x02   0xffff.000.00000000  0x00000000.0000.00  C---    0  scn 0x0000.0015301a
          Leaf block dump
          ===============
          header address 139764988541020=0x7f1d927e785c
          kdxcolev 0
          KDXCOLEV Flags = - - -
          kdxcolok 0
          kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
          kdxconco 1
          kdxcosdc 0
          kdxconro 3
          kdxcofbo 42=0x2a
          kdxcofeo 7993=0x1f39
          kdxcoavs 7951
          kdxlespl 0
          kdxlende 0
          kdxlenxt 0=0x0
          kdxleprv 0=0x0
          kdxledsz 0
          kdxlebksz 8032
          row#0[8019] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 d9 01 00 2b 9b 00 7f
          row#1[8006] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 d9 01 00 2b 9b 00 80
          row#2[7993] flag: ------, lock: 0, len=13
          col 0; len 10; (10):  00 00 f6 d9 01 00 2b 9b 00 81
          ----- end of leaf block dump -----
          

          Comment by Valentin Nikotin — September 7, 2012 @ 8:11 pm BST Sep 7,2012

  3. […] to find that I had had a conversation about this sort of thing some time ago, triggered by a comment to an earlier post. If you want to read a more thorough investigation of the things that can happen and how deferrable […]

    Pingback by Deferrable RI – 2 | Oracle Scratchpad — July 13, 2014 @ 7:46 pm BST Jul 13,2014 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 4,013 other followers