Testing for correctness:
[The Philosophy Series]
This is one of the things I have the hardest times convincing people of in the RAD type of environment (and I’m sure the modern equivalents of RAD must have the same issues). I’ve changed one program 3 times in the last two days, for reasons I predicted in arguments over it years ago. You need to temper what the users want with foresight, they tend to be focused too narrowly on function and not focused enough on data. And vice-versa.
As far as the right thing and nothing more, that could get into some deep philosophy. Yes, you can design provably correct software, but that’s really, really expensive at the enterprise level, even non-competitive when selling large applications. Look what’s happened to the ultimate software, the US space program software. Who can maintain it now? People start making silly mistakes like US/metric conversions. Hardware still winds up in out-of-bound conditions. New stuff just isn’t done right.
Comment by joel garry — September 24, 2009 @ 9:00 pm BST Sep 24,2009
I think the key statement here is “… and nothing more”
While it needs to be said, I think most people do test for positive outcomes, and sometimes basic negative ones. However side effects are oft forgotten. And Oracle provides us with more and more functionality/ways to create side effects. Sometimes these are difficult to track, usually from negative responses – but how much effort due you spend chasing potential problems that may not exist?
I think I/we just answered myself.
Comment by Scott Wesley — September 25, 2009 @ 4:46 am BST Sep 25,2009
I think you’ve highlighted the biggest problem – how do you know when you’ve done enough … “you can’t prove a negative”.
One of the big gaps in testing, though – and one that’s quite hard to think through – is the idea of concurrency. We can’t imagine all the things that could go wrong, but we can ask the question “what might other people be doing in the same place at the same time?”. That one question would probably pre-empt quite a lot of the problems I see with Oracle systems.
Comment by Jonathan Lewis — September 25, 2009 @ 5:03 pm BST Sep 25,2009
Some guys put testing on top of the development process, i.e. they write tests before they write single line of code. TDD is all about it.
Comment by Timur Akhmadeev — September 25, 2009 @ 6:24 am BST Sep 25,2009
Yes indeed, I think it’s very easy to fall into the trap of defining tests based on what you know the code that you’ve written is intended to do, rather than on what it is supposed to do. A very good step when software development is to consider first how you will test that the code is correct _before_ writing it. Preferably you’d write the test code first in fact.
Comment by David Aldridge — September 25, 2009 @ 6:45 am BST Sep 25,2009
Testing what you know is supposed to happen — that’s why it’s good to give the program spec to someone else to create the test data. They don’t know what your code is going to do, so they’re more likely to create data that finds your errors.
Comment by Jonathan Lewis — September 25, 2009 @ 4:59 pm BST Sep 25,2009
There’s a package called dbms_space_admin (or something like that) that includes procedures for patching up space management bitmaps.
I was once told, though I can’t tell how accurate the tale was, that the code to fix the bitmaps was one of the first things written when the entire bitmap management stuff was created.
I think that captures the idea of how to think about testing.
Comment by Jonathan Lewis — September 25, 2009 @ 4:57 pm BST Sep 25,2009
“Tests are done to find bugs and not to show their absence”.
The more we find bugs the less we have risk to meet them in Production.
The best way I have found to find more bugs is to run my code against a large volume of data; is to run my code in an overnight batch job with data coming from several sources.
Code is run in a for loop and all errors are logged into a dedicated table without exiting the loop in case of error.
Comment by Mohamed Houri — September 25, 2009 @ 7:51 am BST Sep 25,2009
Null data testing. What happens when you run you program with no data to process? It’s caught me out a couple of times.
I agree with Timur and David,it’s hard to beat the process of writing you test cases (and developing your test data) before you start writing your code. In fact, it’s nice just to see some properly structured test data once in a while… *sigh*
Comment by mwidlake — September 25, 2009 @ 1:48 pm BST Sep 25,2009
RSS feed for comments on this post.
Fill in your details below or click an icon to log in:
You are commenting using your WordPress.com account. ( Log Out / Change )
You are commenting using your Twitter account. ( Log Out / Change )
You are commenting using your Facebook account. ( Log Out / Change )
You are commenting using your Google+ account. ( Log Out / Change )
Connecting to %s
Notify me of new comments via email.
Enter your email address to get email about new posts and
Join 6,803 other followers
Blog at WordPress.com.