Oracle Scratchpad

March 8, 2016

Wrong Results

Filed under: Bugs,Hints,Indexing,Oracle,Partitioning — Jonathan Lewis @ 6:57 pm GMT Mar 8,2016

Just in – a post on the Oracle-L mailing lists asks: “Is it a bug if a query returns one answer if you hint a full tablescan and another if you hint an indexed access path?” And my answer is, I think: “Not necessarily”:


SQL> select /*+ full(pt_range)  */ n2 from pt_range where n1 = 1 and n2 = 1;

        N2
----------
         1
SQL> select /*+ index(pt_range pt_i1) */ n2 from pt_range where n1 = 1 and n2 = 1;

        N2
----------
         1
         1

The index is NOT corrupt.

The reason why I’m not sure you should call this a bug is that it is a side effect of putting the database into an incorrect state. You might have guessed from the name that the table is a (range) partitioned table, and I’ve managed to get this effect by doing a partition exchange with the “without validation” option.


create table t1 (
        n1      number(4),
        n2      number(4)
);

insert into t1
select  rownum, rownum
from    all_objects
where   rownum <= 5
;

create table pt_range (
        n1      number(4),
        n2      number(4)
)
partition by range(n1) (
        partition p10 values less than (10),
        partition p20 values less than (20)
)
;

insert into pt_range
select
        rownum, rownum
from
        all_objects
where
        rownum <= 15
;
create index pt_i1 on pt_range(n1,n2);

begin
        dbms_stats.gather_table_stats(
                ownname    => user,
                tabname    => 'T1',
                method_opt => 'for all columns size 1'
        );

        dbms_stats.gather_table_stats(
                ownname    => user,
                tabname    => 'PT_RANGE',
                method_opt => 'for all columns size 1'
        );
end;
/

alter table pt_range
exchange partition p20 with table t1
including indexes
without validation
update indexes
;

The key feature (in this case) is that the query can be answered from the index without reference to the table. When I force a full tablescan Oracle does partition elimination and looks at just one partition; when I force the indexed access path Oracle doesn’t eliminate rows that belong to the wrong partition – though technically it could (because it could identify the target partition by the partition’s data_object_id which is part of the extended rowid stored in global indexes).

Here are the two execution plans (from 11.2.0.4) – notice how the index operation has no partition elimination while the table operation prunes partitions:


select /*+ full(pt_range)  */ n2 from pt_range where n1 = 1 and n2 = 1

---------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
---------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |     2 (100)|          |       |       |
|   1 |  PARTITION RANGE SINGLE|          |     1 |     6 |     2   (0)| 00:00:01 |     1 |     1 |
|*  2 |   TABLE ACCESS FULL    | PT_RANGE |     1 |     6 |     2   (0)| 00:00:01 |     1 |     1 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter(("N1"=1 AND "N2"=1))


select /*+ index(pt_range pt_i1) */ n2 from pt_range where n1 = 1 and n2 = 1

--------------------------------------------------------------------------
| Id  | Operation        | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT |       |       |       |     1 (100)|          |
|*  1 |  INDEX RANGE SCAN| PT_I1 |     1 |     6 |     1   (0)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("N1"=1 AND "N2"=1)


Note: If I had a query that did a table access by (global) index rowid after the index range scan it WOULD do partition elimination and visit just the one partition – never seeing the data in the wrong partition.

So is it a bug ? You told Oracle not to worry about bad data – so how can you complain if it reports bad data.

Harder question – which answer is the “right” one – the answer which shows you all the data matching the query, or the answer which shows you only the data that is in the partition it is supposed to be in ?

32 Comments »

  1. Hi Jonathan –

    for me the answer is clear: there is no bug in this scenario, neither on the exchange side nor on the way the queries are answered.

    As you know very well, Oracle is a powerful and flexible engine that comes with a lot of functionality – “bells and whistles” – to being able to address every possible aspect of our customer’s requirements and therefore the database’s usage, from vanilla straight-forward simple use cases to the very high-end mission critical systems that keep our digital world running.

    With power and flexibility on the kernel side, the breadth and depth of the functionality increases (plus complexity), but the responsibility on the user’s side increases, too. Some of the functionality actually crosses the invisible borderline and gives the users the ultimate power to override and disable the kernel’s fundamental responsibility: data integrity. This is purely done for the sake of extreme performance at the high-end and should never be taken light-heartedly. Exchange partition is in that category, just like rely constraints and other things.

    If you explicitly ask Oracle to not worry about the content of a partition when doing exchange, or the actual state of a constraint, you are consciously taking the full responsibility for all the risks and side effects that come with that decision (I almost sound like a pharmacist, but the analogy is a good one IMHO: heavy drugs aren’t taken lightheartedly either). There is nothing Oracle is to blame for. You’ve got what you asked for.

    Last but not least, to answer what I think the right answer is here: there is none from a database’s perspective. The database does not have enough context to know whether the data that was added with the exchange should have gotten into the system in first place and was just ‘misplaced’, or whether the data should have never been added to begin with. Could we silently add another predicate to the index when doing an index-only access? Sure. Should we? Not so sure. Does it change the situation and answers what query result is correct: not at all. The database still does not have enough context. That’s just another angle to the risks and side effects – and possibly a good nurturing ground for endless philosophical discussions.

    Comment by Hermann Baer — March 8, 2016 @ 7:32 pm GMT Mar 8,2016 | Reply

    • Hermann,

      Thanks for taking the time to supply such a detailed comment.

      I’m happy to agree with every point you’ve made. The analogy I proposed on Oracle-L was that it’s like complaining about wrong results after setting a constraint to RELY novalidate disable, or setting query_rewrite_integrity to trusted.

      Comment by Jonathan Lewis — March 8, 2016 @ 7:47 pm GMT Mar 8,2016 | Reply

      • Hi Jonathan –

        a good blog post deserves good comments .. or at least the attempt of such. Good to hear that we are in full agreement here .. did not really expected it differently, given your indication at the end of the blog ;-)

        Comment by Hermann Baer — March 8, 2016 @ 7:54 pm GMT Mar 8,2016 | Reply

  2. I agree with everything you wrote. I believe the missing feature is that when you manipulate partitions without validation it could mark partitions that might contain data at odds with the partition specification. Then whatever most efficient search was possible if such a partition might otherwise be pruned would be done, with a further provision of a session and system parameter (or even possibly a hint) to skip the fix-up. Without these features, I do not see a way around the “wrong answer” conundrum you have identified that still allows the partition exchange without validation. As documented this is not really a current bug. The bug, if any, is either the behavior of allowing data into the “wrong” partition OR allowing it without flagging the possibility. One is user behavior. The other is a product defect. I won’t beg which viewpoint is correct. Thanks so much for reminding me of this.

    Comment by rsiz — March 8, 2016 @ 9:44 pm GMT Mar 8,2016 | Reply

    • Hi Mark –

      having a flag that identifies partitions that entered the system without Oracle’s validation is something we contemplated years ago, but that somehow got lost. Definitely something I will take on and enter as enhancement request. We should be able to tell you that, just to identify what data has entered the system without us validating it.

      Having said that, I am not convinced we want to/have to also enhance all the variants of partition pruning to know about nonvalidated partitions and that touching them all the time solves a problem. Coming back to my statement about “what answer is correct”, I do not believe that under the above-described scenario you can assume that all data that has entered your system is valid data. If the data is clean, we can use the partition metadata for partition elimination and are correct. If the data is not clean, then it violated at least one fundamental constraint of my system – the data placement in the partitioned table – and that makes it suspicious to begin with.

      Comment by Hermann Baer — March 8, 2016 @ 11:25 pm GMT Mar 8,2016 | Reply

      • Since I was the editor of the Oracle VLDB database enhancement requests from 1990 until it was disbanded about 1995, I know exactly where the contemplation arose. As for constantly touching it, that is not at all required as I see it. IF marked as unvalidated, it simply is never pruned with normal operations. So wherever you generate your list of partitions that are required whenever partition pruning is in play, that flags as a “YES” to the needed partitions list. If the proposed alternative “don’t suppress pruning that might yield bad results” session value is set then ignore the flag. I think that read operation and “OR” logic is all you need. Since this happens at parsing only I don’t see this as a big performance hit and it is the price to be paid for shortcuts on the partition manipulation allowing potentially bad data. A major use case for the partition swapping without validation is when it is definitely known validation won’t fail because of the way the swapped in partition was created (quite possibly as a clean standalone table that had enabled constraints as it was built and folks can’t have any extra overhead at swap-in time). In the past two decades I’m probably a little rusty on the nuances here, but that is the overview. It feeds into the “Scaling to Infinity” approach so well documented by Tim Gorman, and I still think it is worth getting right and not unduly adding overhead in all cases, specifically when it is known with certainty that the unvalidated swapped in bit contains only valid contents. Thanks for the correspondence. I don’t know whether Oracle stashed the VLDB and MOSES notes anywhere. I believe I still have the originals in my attic if they are of interest.

        Comment by rsiz — March 9, 2016 @ 5:03 am GMT Mar 9,2016 | Reply

        • Hi Mark –

          yes, exchange without validation is a pure performance feature – with a big burden on the customer to ensure integrity. That’s what it was designed for. Nothing more, nothing less.

          So when it comes to accessing such partitions – or not – I think we have to agree that we disagree for the time being: with the current conceptual design and philosophy of partitioning there is no right or wrong in terms of whether the data belongs in the system or not.

          Unless we decide to implement a persistent state of validated and non-validated partitioning and support this across the board I personally do not see a need for any change. Exchange partitioning, as said before, was implemented and designed as pure performance feature and not to implement as persistent state of validated and nonvalidated partitions. That a user can cause this is a different story. Call me a nitpicker, but that’s a fundamental conceptual design difference, and the implications on the kernel and what it does is enormous. So validated/nonvalidated partitioning is simply yet another enhancement idea for the future.

          If you have use cases for that functionality then please drop me an email: hermann.baer@oracle.com. And sorry Jonathan from digressing from the initial topic; but I think it’s worth it and in all our interest.

          Comment by Hermann Baer — March 10, 2016 @ 4:15 am GMT Mar 10,2016

  3. Let me paraphrase and carefully restate your initial question from a different perspective Jonathan.

    “Is it a bug if a query returns one answer if the CBO chooses an execution plan that results in a full tablescan and another if the CBO chooses an indexed access path?”

    And your answer is, I think: “…..”

    I think we both know your answer.

    Comment by Michael D O'Shea — March 8, 2016 @ 10:23 pm GMT Mar 8,2016 | Reply

  4. I suspect this will be closed as “not a bug” since you can validate the data you told Oracle not to validate. It doesn’t seem all that far removed from repeatable read, dirty read, race conditions, and a gazillion other examples both in and across engines. You just need to understand what you’ve told the database to do. Says the guy who has to deal with an app on Oracle that gives DEC RMS style error messages about privilege violation that intermittently appear when there are controller errors… sigh.

    Comment by jgarry — March 9, 2016 @ 12:12 am GMT Mar 9,2016 | Reply

  5. While I accept the point of view that “Because you told Oracle to trust you, you can’t complain”, there have been “improvements” added to Oracle RDBMS in recent releases where oracle “cleverly corrects/masks” inefficiencies introduced as a result of human actions. For e.g. pushing/not pushing predicates in the subqueries (depending upon whether the end result will change or not), silently eliminating duplicate table access when the end result is not affected, ignoring hints etc. So I wonder why exception in the above case? For e.g. if, in the above case, if the partition exchange is carried out without validation, if the indexes were made unusable, above example will end up ignoring the hint and will always produce “one result”. I am sure there will be side-effects of it that I might have missed but being able to produce consistent/correct results, irrespective of execution paths, is, in my opinion, utmost important.

    Comment by Narendra — March 9, 2016 @ 8:39 am GMT Mar 9,2016 | Reply

    • > but being able to produce consistent/correct results, irrespective of execution paths, is, in my opinion, utmost important

      Spot on Narenda! Put another way, if you don’t have confidence in the result set produced, you may as well not bother to execute the query. What Jonathan has done here is rationalised why Oracle is doing what it is doing. However what is is doing is wrong, in my opinion.

      Comment by Michael D O'Shea — March 9, 2016 @ 9:06 am GMT Mar 9,2016 | Reply

      • Michael,

        GIGO – if you put bad data into the system why do expect good results to come out ?

        Oracle allows you to declare relational integrity constraints as RELY DISABLE NOVALIDATE. If you declare such constraints and then load bad data into the tables will you blame Oracle if you get inconsistent results ? “Without Validation” is just another example of Oracle allowing to you to take advantage of a type of constraint without the cost of enforcing it.

        Comment by Jonathan Lewis — March 10, 2016 @ 9:13 am GMT Mar 10,2016 | Reply

        • Hi again Jonathan, if I put bad data into the system, I expect the same bad data to come out, not just a subset of it, and even then only sometimes.

          Comment by Michael D O'Shea — March 10, 2016 @ 1:06 pm GMT Mar 10,2016

        • Michael,

          You expect to see the bad data even when you’ve put it in the wrong place and explcitly told Oracle that it should not go looking there ?

          If I were pushed to make a black and white decision about the example I gave, I would say that Oracle shouldn’t have shown me the row that was in the wrong partition when it was using the index-only query.

          Comment by Jonathan Lewis — March 11, 2016 @ 8:02 am GMT Mar 11,2016

    • Just realized that I made a mistake in my earlier response about which result is correct. I was thinking indexed access generated incorrect result but the full table scan execution ended up generating incorrect result. But I still stand by my opinion that oracle should be able to produce consistent results despite the execution path chosen. For e.g. in the above example, as one of the partition has been loaded with data that is not compliant with partition definition, the “parsing” stage of the query should be able to ignore the partition elimination logic (probably using dynamic sampling at higher level ??)

      Comment by Narendra — March 9, 2016 @ 9:43 am GMT Mar 9,2016 | Reply

      • @Narendra:
        Why is the full table scan access with pruning correct? What factual evidence do you have that the data that was misplaced in a wrong partition should not have been added to the table?
        .. and if you had kept your original answer my question would be:
        Why is the index-only access correct? What factual evidence do you have that the data was correct and just unfortunately placed in the wrong partition?

        Comment by Hermann Baer — March 10, 2016 @ 3:59 am GMT Mar 10,2016 | Reply

        • Hermann,

          Not sure if you read my revised response where I have mentioned that full table access with pruning is not producing correct results. I stand by my assertion that the a SQL should always produce consistent (or correct) results no matter what execution path is chosen. As for factual evidence, I am not author of the Oracle RDBMS software so can’t exactly provide evidance (I guess many experts, including Jonathan, have used this argument before and it is a valid one). The UPDATE INDEXES clause was specifically introduced ( I think in 9i and specifically for maintaining global indexes) in order to do the index maintenance as part of partition operations. Tom Kyte mentions in his book “Expert Oracle…” under section 13.3.3 that during the partition operations, UPDATE INDEXES clause effectively deletes and inserts rows from global indexes. If I apply that statement to e above example, the index will be “refreshed” by accessing the table, where it finds 2 rows satisfying the condition (where n1 = 1 and n2 = 1). Both the segments (index and table) do contain the data and Oracle is aware of this (as a result of EXCHANGE PARTITION and UPDATE INDEXES options).

          Comment by Narendra — March 10, 2016 @ 5:32 am GMT Mar 10,2016

        • Narendra,

          I think the point Hermann was making with his request for “factual evidence” was about business requirements, not about internal mechanisms. It goes back to my closing question: if Oracle were to give you the same answer in all cases should it be the one row that’s in the right partition, or should it be both rows ?

          Whichever choice Oracle made the business could argue it was wrong, along the lines of:

            If Oracle reports one row: there are two rows in the table, obviously you should have reported both of them even though one of them had got into the wrong place by accident.

            If Oracle reports two rows: one of those rows is obviously there in error, it should have been eliminated by the software doing the load, and Oracle should have ignored it because it’s not where it should be if we had wanted it in the database.

          Comment by Jonathan Lewis — March 10, 2016 @ 9:30 am GMT Mar 10,2016

    • Narendra,

      Thanks for taking the time to supply a dissenting viewpoint.

      You’ve asked why this case should be different from the examples you’ve supplied; but I don’t think your examples are appropriate analogies. Query transformation, subquery caching, etc. are examples of the optimizer applying rules about the data to minimise the impact of errors in design (of structures or SQL), not errors in data; and putting the data in the wrong partition is a data error not a design error.

      By using the “without validation” option you have promised Oracle that you will enforce the rules it uses to justify some of its optimisation methods. If you break that promise why is it Oracle’s fault if those optimisation methods now produce inconsistent results ?

      Comment by Jonathan Lewis — March 10, 2016 @ 9:09 am GMT Mar 10,2016 | Reply

      • Jonathan,

        Oracle does have features like dynamic sampling that solely works based on data (metadata is still data about data). One can also claim that because user has not updated statistics (aka metadata), why is it Oracle’s fault if the query does not follow best execution path.

        Allow me to put forward a different angle. So Oracle RDBMS is a piece of software which allows its users to interact with the database and provides APIs (like SELECT, INSERT, UPDATE, DELETE etc.) for the same. So we now have 2 APIs (INSERT and EXCHANGE PARTITION) that effectively achieve the same end result (although in different ways) of adding data to a table. So is it a good idea for a piece of good software to provide 2 APIs for the same function, where one is capable of producing invalid data?

        One another place where oracle has managed to achieve similar feature (but probably in better manner) is implementing unique constraints without validation. As far as I remember, a non-validated unique constraint can not be enforced using a unique index.

        Comment by Narendra — March 10, 2016 @ 9:49 am GMT Mar 10,2016 | Reply

        • Narendra,

          I’m not entirely sure what point you’re trying to make in the first paragraph – but assuming it refers to my comment about “rules about the data”, a sample of what happens to be there is not a rule about what is supposed to be there.

          The answer to your second paragraph is yes. It’s a perfectly good idea to supply two such APIs, if one says “we will enforce the rules” and the other says “you promise that you will enforce the rules because you think you can do it more quickly, and we will trust you but you will suffer the consequences if you fail to live up to the promise.”

          What makes you think this has been achieved better ? If you set a unique constraint without validation you can still get inconsistent results if you haven’t cleaned up the data. Oracle has several features where you can choose the faster but dangerous option or the slower but safe option. It all comes down to WHEN you want to save resources and what errors you’re prepared to risk if you don’t enforce constraints in real-time all the time.

          Comment by Jonathan Lewis — March 10, 2016 @ 10:21 am GMT Mar 10,2016

  6. Illustration of a known situation but with an interesting final question followed by some good comments… I’m getting back up on the fence.

    Comment by Dom Brooks — March 9, 2016 @ 9:53 am GMT Mar 9,2016 | Reply

  7. I have designed an “EXCHANGE PARTITION … WITHOUT VALIDATION” routine in the past where we perfectly knew that the data didn’t need validation. That was exchanging from a Live Schema to an Archival Schema with exactly the same partition definition (I had written a blog post too : http://hemantoracledba.blogspot.com/2013/02/moving-partition-to-archival-schema-and.html )
    So there was a use case for WITHOUT VALIDATION for me.

    But I believe that there should be a “flag” or a “marker” or “parameter” that Oracle can use at Query Parse time to decide if it should rely on an Unvalidated Partition.

    Comment by Hemant K Chitale — March 10, 2016 @ 3:22 am GMT Mar 10,2016 | Reply

    • Hemant,

      Thanks for the example; and I agree with your comment that it would be good to have a flag to show that the partition had arrived “without validation”.
      Consider, though, what would happen if the optimizer was not allowed to rely on flagged partitions

        a) None of the queries against your archival schema would be able to use partition elimination because any partition might contain data that matched any query
        b) None of the queries against your archive scheme would be able to use partition-wise joins because you wouldn’t be able to claim that “data in partition X in table A can only match against data in partition X of table B”.

      For the sake of saving a little time in data loading you would have crippled query performance if the optimizer had to cater for the flag.

      Comment by Jonathan Lewis — March 10, 2016 @ 9:18 am GMT Mar 10,2016 | Reply

      • Jonathan,
        I meant to say that let there be a flag or marker to identify an Unvalidated Partition but a parameter that we can use at session / sql (hint) level that says “don’t ignore OR do ignore the flag and treat the Unvalidated Partition accordingly”. Thus, let there be flexibility.

        Comment by Hemant K Chitale — March 11, 2016 @ 6:32 am GMT Mar 11,2016 | Reply

        • Hemant,

          I agree, in principle, and what you’re describing matches Mark Farnham’s (rsiz) suggestion; but I have a feeling that if Oracle implemented the idea I’d find myself visiting major customers and telling them to stop using it ;)

          I’m going to have to do some experiments on recent versions of Oracle, but I remember writing up the details a long time ago of what happens in a partition exchange, and pointing out that WITH validation is likely to be faster than WITHOUT validation: but that probably depends on the indexes, and any changes in the validation code since the last time I tested.

          Comment by Jonathan Lewis — March 11, 2016 @ 8:06 am GMT Mar 11,2016

  8. If Oracle reports two rows: one of those rows is obviously there in error, it should have been eliminated by the software doing the load, and Oracle should have ignored it because it’s not where it should be if we had wanted it in the database.

    The simple explanation for Oracle reporting 2 rows is because the SQL in question is accessing the table with a criteria, which is satisfied by 2 rows. The row in incorrect partition is there because user specifically asked for it (using option provided by oracle) while loading the data.
    A partitioned table being implemented using multiple segments (partitions) does / should not influence the resultset. Partition pruning is an optimizer feature and if optimizer, being a piece of software, is generating different results as a result of its feature, it does look like bug (to me).

    p.s. On a different note, my name is spelled “Narendra” and not “Narenda”… :)

    Comment by Narendra — March 10, 2016 @ 10:14 am GMT Mar 10,2016 | Reply

    • Narendra,

      Sorry about getting the name wrong – an error I’ve now corrected.

      Bear in mind that the thing you’ve quoted is the hypothetical argument from someone who wants to see just one row because they know that their software should have discarded that that row rather than loading it into the database. That being the case it doesn’t really matter what YOUR opinion is, someone else can hold the opinion that this is clearly bad data that the database should ignore.

      Partitioning is defined by a type of constraint on the data that allows the optimizer to minimise the workload. Oracle allows you to enforce the constraint externally because this can be a benefit at data loading time. If you want to see the correct answer after failing to enforce the constraint then there is no point*** in using partitioning and Oracle has to do more work. The question then becomes why should other people suffer because you can’t live up to your promise. (That’s “you generic”, not “you personally”)

      *** I say this on the basis that in my experience an organisation that uses exchange without validation does it on every partition load, so none of their partitions are validated and any data could be in any partition (in theory).

      Comment by Jonathan Lewis — March 10, 2016 @ 10:35 am GMT Mar 10,2016 | Reply

      • Jonathan,

        Not entirely sure I understand your last response. Specifically this statement The question then becomes why should other people suffer because you can’t live up to your promise
        If by “other people” you mean users mining/reading the data and “you” means people loading the data then we are back to GIGO principle (but in the context of oracle having to do more work).
        If you are suggesting that when faced with choice between doing more work and producing inconsistent results (due to user’s actions), oracle is right to go for avoiding less work then that just becomes an opinion.

        Comment by Narendra — March 10, 2016 @ 10:47 am GMT Mar 10,2016 | Reply

        • Small correction to my last response
          If you are suggesting that when faced with choice between doing more work and producing inconsistent results (due to user’s actions), oracle is right to go for avoiding more (my typo using less) work then that just becomes an opinion.

          And with GIGO, I meant oracle having to do more work (which will be an overhead) due to incorrect data being loaded.

          Comment by Narendra — March 10, 2016 @ 10:53 am GMT Mar 10,2016

        • Narendra,

          Read it as:

          “Why should all the sites that use the feature correctly have to suffer bad performance because one site that can’t manage to use the feature properly blames the Oracle software for the side effects produced by their own incompetence.”

          Bottom line – if you don’t want to see the anomaly, and can’t enforce it properly outside the database, tell Oracle to validate the data.

          Comment by Jonathan Lewis — March 10, 2016 @ 10:53 am GMT Mar 10,2016

  9. […] that if the headline is a question then the answer is no. So, following on from a discussion of possible side effects of partition exchange, let’s look at an example which doesn’t involve partitions.  I’ve got a schema […]

    Pingback by Wrong Results ? | Oracle Scratchpad — March 11, 2016 @ 9:18 am GMT Mar 11,2016 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.