“Priors”

I was listening to a podcast recently about “Thinking Clearly” when the presenter gave a very rapid description of a common mistake that occurs when human intuition meets statistical information.

The particular bit of statistical analysis, when you meet its effects in real-life, can be hard to understand so I thought I’d draw a picture that might help people to understand the mechanism of putting a “threat percentage” into the correct context.

Imagine there is a “nasty medical condition X” that affects one in 20 of all men over the age of 55. Now imagine that there is a diagnostic test for the condition that is 90% accurate (meaning it will return the correct result 90 times out of a hundred and the wrong result 10 times out of 100).

You are male, just past your 55th birthday, and your doctor tells you that you’ve tested positive. How terrified should you be or, to put it another way, which is more likely (and by how much): you’ve got X, or you haven’t got X?

The human, intuitive, response is simple: you’ve been told the test is 90% accurate and you’ve tested positive; so your digestive system is probably telling you that it’s almost certain you’ve got X.

The statistician’s approach (after the initial reflexive shock, perhaps) is to apply Bayesian thinking which can be turned into pictures as follows:

• Step 1: What does “one in 20” look like? (left hand square)

• Step 2: What does “90% accurate” look like? (right hand square)

• Step 3: What does the picture look like when we superimpose the individual (independent) probabilities:

The big white block toward the top left tells us about the group of people who are correctly informed that they are X-free; the tiny red block in the bottom right tells us about the really unlucky ones who are told they’re X-free when they actually have X (false negatives).

Now compare the two pink blocks: the vertical rectangle to the right is the group that have X and test positive; the horizontal rectangle at the bottom is the group who don’t have X but test positive anyway (false positives).

The visual impression from this image is that if you’ve been told that you tested positive it’s nearly twice as likely that you are X-free than you are to have X: but let’s put the numbers into the picture to get a better handle on this. I’ll use a population of 10,000 (which, conveniently, I can represent as a square measuring 100 by 100):

In a population of 10,000

• X-free = 95 * (90 + 10) = 9,500 (95%)
• Got X = 5 * (90 + 10) = 500 (5%)

• Correct result given = 90 * (95 + 5) = 9,000 (90%)
• Wrong result given =10 * (95 + 5) = 1,000 (10%)

• X-free and received right result = 8,550 … (95 * 90, top left)
• Got X and received wrong result = 50 … (5 * 10, bottom right)

• Got X and received right result = 450 … (5 * 90, top right)
• X-free and received wrong result = 950 … (95 * 10, bottom left)

Given the underlying population data (“priors”) for this example, we see that a positive result from a test that’s 90% accurate means there’s a probability of 450 / (950 + 450) = 0.32 (32%) that you’ve got X.

Footnote

The result of this very simple hypothetical case is not intuitively obvious to most people; but if you thought it was easy to get to the right answer you might want to look at the Monty Hall problem, which also leads to Bertrand’s Boxes and the Three Prisoners problem.-

Thank You

Regular readers will have noticed that for the last few weeks I’ve been adding a footnote (now deleted) to any new blog posts asking for donations to the Nepal Youth Foundation (UK), with an offer to match (up to a limit) any donations made by my readers.  The page I had set up on JustGiving to collect donations and describe the work of the foundation reached my limit last week, so I sent off my matching cheque on 6th Jan. I got a very prompt email telling me the cheque had arrived the following day, and thanking me for the effort; then yesterday a follow-up thank you letter arrived in the morning post.

I thought I’d share this with you so that you could see that your generosity has been appreciated by an organisation that really cares about the change it can make to the lives of the people who most need their help:

My JustGiving page is still live, but if you feel at any time that you’d like to donate to a worthwhile cause the NYF(UK) has a JustGiving page of its own that you could donate to directly, and for further information about what they do, here’s a link to their main (UK) website. For other countries there’s a link to the US site with information about other international offices.

So long …

… and thanks for any Douglas Adams references.

I decided on Friday that after more than 31 years using the Oracle software and more than 37 years of being self-employed in the computer industry – not to mention a further 8 years being an enthusiastic amateur – the time had finally come for a change of pace.  As from 1st June 2019 I am retired.

I have a couple of long term commitments that I’ll still be doing a little work for over the next few months, and I wouldn’t be surprised if someone occasionally tempts me into doing a few days solving interesting problems.  I’ll still be answering questions on the Oracle Developer Community forum and (if the email gets through) on the Oracle-L listserver , and I’ll still be writing blog notes about Oracle and submitting abstracts to conferences for a few years. Who knows, I may even get round to writing another book – after all, I’ve already downloaded Oracle 19 so I’ll still be experimenting, investigating, and trying to break things Some people retire to do the Times crossword or Sudoku, my retirement pastime is going to be playing with Oracle.

But now I’ve got 3 musical instruments to re-learn, a garden to re-engineer, apple trees to train, and a lot of rowing [U.S. “crew”] time to catch up on so that I can get fit enough to get back to playing squash (which might be an interesting experience now that my eyesight has been fixed). And for relaxation I’ll be sitting in the garden watching the birds, drinking fine wine and reading a good book.

Update (6th June 2019)

Thanks for all the appreciative comments to know. It’s nice to have a record, both here and on the tweet, of the fact that I have made a difference for so many people.

Bait and Switch

Just what you need after a long hike in the Lake District:

Soup

When my mother-in-law comes round to Sunday lunch we often have roast chicken – and a serious error in estimating the requirement for the vegetable bed I was roasting on led to the discovery of home-made soup. (I did warn you that my post-operative posts would be light-weight)

Ingredients

• Remnants of cooked chicken
  • (or 250ml, 1/3rd pint, one cup of  boiled water with a vegetable or chicken stock cube)
• One large carrot
• One small parsnip
• One medium onion
  • The three vegetables should be similar in weight: roughly 200g, a bit less than 1/2 lb.
  • (Or just about any leftover vegetables from a roast dinner – I’ve even used leeks in cheese sauce)
• Greek yoghurt (One tablespoon)
• Choice of herbs and seasoning.

 

Method

• Discard any large areas of fatty skin from the chicken carcase
• Break up the carcase and place in saucepan with 500ml (2 cups, 3/4 pint) water
• Bring to the boil and simmer for about 30 minutes with saucepan lid on.
• Strain into a fresh saucepan, discard the remnants of chicken

 

• Top, tail and peel the carrot, cut into discs
• Top, tail and peel turnip, cut into small chunks
• Peel and dice the onion.
• Mix the stock and vegetables in a fresh saucepan
• add herbs and seasoning
  • (I like a teaspoon of chopped tarragon – fresh from the garden)
• Simmer with lid on for about 20 minutes
  • (until the carrots are softened)

 

• Tip contents into blender
• Blend until smooth.
• Add a rounded tablespoon of plain Greek yoghurt, and blend
• Serve – 2 large or 3 small portions

Despite looking a bit boring before it goes into the blender the soup tends to come out a surprisingly cheerful sunshine yellow thanks to the carrot. If you’ve managed to get the same results as I do then, because of the yoghurt I think, the texture will be almost like a thick foam or very light mousse.

I typically end up making the stock immediately after lunch is over, then keep it in the fridge for a couple of days before making the soup; that does mean you can skim off any excess fat before using the stock for the soup. And if there’s any gravy left over from lunch that’s a bonus to go in the soup.

 

RTFM ?

My entrance at the Polish Oracle User Group conference 2017 has just resurfaced on Twitter. There is a back-story to this which contains an allegorical lesson in using Oracle. As I said in the opening remarks in the subsequent presentation: “The lesson began before I got to the stage”.

Like all the other speakers at POUG2017 I had received an email asking me to name a tune for a playlist. Having decided that Beethoven, Brahms and Tchaikowski were not the sort of composers the organisers had in mind I nominated (with a touch of irony) “Who wants to live forever?” by Queen (despite Richard Foote’s strenuous efforts to turn the Oracle world to David Bowie).

When the conference started I noticed two things: first, that the tunes for the “playlist” were actually being used to accompany speakers to the stage, secondly that the admin staff were all wearing monk-like gowns and hoods. So I asked for my tune to be changed to “The Imperial March” (Darth Vader’s theme) and borrowed the gown from the tallest admin person, with the results you see in the video clip.

So what’s the Oracle allegory ?

First you read the manuals, then you observe how it really works before you go live.

 

Been There

It’s the end of the year and time for a retrospective of some sort so I was thinking of listing the top 10 most popular pages on my blog, but Connor McDonald beat me to it, so I decided to see if I could remember all the countries I’d visited since starting to work with the Oracle software, and here’s the list in alphabetical order:

Antigua
Australia
Austria
Belgium
Bosnia
Brunei
Bulgaria
Canada
China
Croatia
Czech Republic
Denmark
Egypt
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
India
Indonesia
Ireland
Israel
Italy
Japan
Latvia
Lithuania
Malaysia
Netherlands
New Zealand
Norway
Poland
Portugal
Romania
Russia
Scotland (see comment 7)
Serbia
Singapore
Slovakia
Slovenia
South Africa
Spain
Sweden
Switzerland
Thailand
Tunisia
Turkey
UAE (United Arab Emirates)
USA
Wales (see comment 7)

A few of these were holidays rather than work, and I may have forgotten a couple, so if you’ve seen me in your country and it’s not on the list let me know.

The list can be a bit of a nuisance, I had to list “all the countries you’ve visited in the last 10 years” for both the US and Russian visas: the US form only allowed for 5 countries and the Russian one for 40; and the US expected me to list EVERY visit, with dates and city!

In memoriam – 3

My father-in-law died a couple of weeks ago, aged 95. This is the story that he wrote for his children and grandchildren a few years ago describing his experiences as a Naval engineer on the aircraft carrier HMS Indefatigable during the second world war.

ROY‘S NAVAL CAREER

When war broke out on 3rd September 1939 I wanted to join the Navy, and a few days later I saw a  new recruiting office near Southend Pier so I went in and asked how I would be able to join. A Petty Officer looked at me and said “Well, sonny, you will have to wait until you are 18”. I was then only 17 so I continued with my plan to become an engineer. In those days parents either had to pay the full cost of university education or rely on their children gaining scholarships. In my case scholarships were essential. So, concentrating on mathematics, I took Higher School Certificate (A-Levels) in July 1938 and July 1939, but did not gain any scholarships. At that time I was Head Boy at Lindisfarne College and in late September the school was evacuated to North Wales from the Southend area because of fears of bombing and invasion but here the buildings were not well equipped and there was no laboratory. However, the Southend High School remained at Southend and arrangements were made to transfer me there.

In December 1939 I was awarded a Scholarship at Queens’ College, Cambridge. Then in May 1940 when the German blitzkrieg started the High School was evacuated to Mansfield in the Midlands, but there I took the HSC again and as a result gained a State Scholarship and a Southend Borough Major Scholarship, which in total was enough to see me through Cambridge. There I made friends with Denis Campbell, Stuart Glass and Edward Higham. In addition to lectures we went regularly to tutorials with a great character called (Professor) Archie Browne. He had additional duties as Steward of the College, and was responsible for obtaining food supplies and coal for heating, which was very difficult in wartime.

The course was completed in two years and, with blackouts, air raid precautions and other restrictions, social life was limited. I joined the Naval Section of the Cadet Corps and the Home Guard which took up one or two afternoons each week. I remember one exercise where we had to make a mock attack by night on an airfield some ten miles north of Cambridge. The defenders somehow knew that we would attack the SE corner and mustered there, but we made a mistake and went for the NE corner which was undefended, so we theoretically captured that bit of airfield! We had to march there and back, and the blisters lasted for weeks. On another exercise Cambridge was attacked by the Welch Fusiliers, I remember being knocked on the head and falling into a ditch half full of water. I was considered a casualty and allowed to return to college for a hot bath.

July – September 1942         I applied to join the Royal Navy as an engineer officer and had interviews at the Admiralty including medical examinations. As a result I was accepted and appointed a Probationary Temporary Acting Sub.Lieutenant (E) RNVR, and the next step was to purchase my uniform at Gieves in London, including the purple stripe denoting engineering.

October 1942         I reported to Portsmouth Barracks for four weeks training. I wore my uniform for the first time at Warminster in Wiltshire where we were living, and traveled to Portsmouth without any knowledge of how to make or receive naval salutes in public! This was soon rectified at Portsmouth where I joined twenty other trainees for the course which included instruction in naval customs and traditions, rules and regulations, security, and the all-embracing Kings Regulations and Admiralty Instructions. We also had training in small arms firing and endless square bashing under the eagle eye of Chief Petty Officer Sims, who was as tough as old boots.

November 1942 – November 1943         I was posted to John Brown’s Engineering Works at Clydebank with Donald Townend and Ian Richardson for practical marine engineering training. John Brown’s was a huge organisation which built engines as well as ships, and just after we arrived Indefatigable was launched. This was an amazing sight, seeing 30,000 tons of ship slide down the slipway into the river Clyde. Before the war the Queen Mary and the Queen Elizabeth were built on the same slipway.

The three of us were billeted with two or three other naval officers in lodgings at Glasgow where we three shared a room and were looked after by a homely landlady and her staff. Every morning we put on civilian clothes and caught a rickety old tram for a 30 minute journey to Clyde bank. There we worked successively in the Pattern-shop (making wood moulds), Foundry, Boiler-shop (being deafened by riveting), Machine-shop, Fitting-shop, Pipe-shop, Drawing Office and Dockyard. We did actually work, scraping bearings, operating lathes, casting metal, always under the supervision of an experienced workman. During lunch hours we used to climb over Indefat, deafened again by riveting, but we got to know the ship. At that time the yard was completing R class destroyers at the rate of about one every fortnight, and we used to take part in their initial sea trials so gaining experience of firing up boilers and operating turbine plant.

During the summer of 1943 we got to know the permanent RN engineer officers appointed to supervise the fitting out of the ship, including Peter Sandison who looked after the flight deck gear. We were seconded to help the checking of the installation and testing of all kinds of machinery, and in November I was chosen to be officially appointed to Indefat, while the other two went off to other ships.

December 1943 – February 1944         The ship was commissioned on 8th December and taken over by the RN from the yard. After dock trials we steamed down the Clyde and carried out various trials including full power of the 148,000 HP engines, the measured mile speed test (32 knots) off the Isle of Arran, and steering and going astern trials from full ahead. I remember on one occasion the steering gear locked solid at hard-a-starboard while doing full speed. We went round in circles flying two black balls showing we were out of control! Several weeks were spent commissioning and training the crew, taking on stores and ammunition, gunnery practice, testing of radar and flight deck equipment, while some time was spent at sea.

March – June 1944         The first aircraft flew in on 23rd March, and thereafter the squadrons began to arrive. We spent days at sea practising aircraft landings and by the end of June we had a complement of some 75 aircraft including Seafires, Fireflies and Hellcats. When at sea engineer officers kept watch for four hours at a time, the middle watch (midnight to 4 am) and the morning watch (4 am to 8 am) being the worst. During a watch we had to visit each engine room and boiler room, and altogether a total of seventeen machinery spaces where each involved climbing up and down three sets of ladders, as the only passage was via the main deck. The best visit was always to a boiler room, where the Chief Stoker would provide a mug of ‘kai’, a chocolate slab heated in hot water and steam.

In addition each officer had responsibility for a department which included the operation and maintenance of all the equipment in it, and the men carrying out this work. Over the years mine included seven steam generators supplying electricity to the ship, three emergency diesel generators, motor boats, steering gear and auxiliary machinery including the big evaporators for making the ship’s fresh water from seawater. Also every six months each officer took it in turn to run the ship’s laundry for 2000 crew!

At Action Stations if not on watch each engineer officer had a Damage Control section of the ship to look after. Mine was the midships section above one of the engine rooms, and my team consisted of about ten stokers and technicians. We might be stationed there all day with only sandwiches and ‘Spotted Dick’ for lunch!

July – October 1944         Indefat joined the Fleet at Scapa Flow surrounded by battleships, cruisers and destroyers, and spent much time at sea on Russian convoy escort duty going beyond the Arctic Circle. In July we made an attack on the largest German battleship Tirpitz, which was moored in a Norwegian fiord and was always a potential menace to Russian convoys. This operation was called MASCOT and with two other carriers the aircraft carrying out the attack included 44 Barracudas, 18 Hellcats and 12 Fireflies, supported by many Seafires as fighter escorts. The weather was not good with cloud and fog around and although the Tirpitz was damaged it was decided to make another attack in mid-August. Prior to that strikes were made against some installations on the Norwegian coast and then on 18th August we sailed for the second Tirpitz attack called operation GOODWOOD. At this time a valuable convoy was en route to Russia and our job was to protect it from the Tirpitz and Uboats. The convoy did arrive safely.

Indefat aircraft included 12 Barracudas, 12 Fireflies, 12 Hellcats and 32 Seafires, and the ship was accompanied by Formidable, Furious and two small escort carriers, together with destroyers. On the first day one escort carrier was torpedoed and badly damaged, and had to return to Scapa escorted by the second small carrier. Some time later a destroyer was torpedoed and sank, with few survivors. The operation lasted for seven days with the ship at Action Stations most of the time. At one point Indefat seemed to be under serious attack by Uboats, with the ship taking evasive action and shaken by exploding depth charges from nearby destroyers, while it was reported that one torpedo passed under Indefat. GOODWOOD was successful as Tirpitz was hit several times and had to be moved to the port of Tromso for repairs, where she was later sunk by the RAF with their 10 ton Tallboy bombs. Had she remained in the narrow fiord in the lee of the mountains protected by smokescreens they might never have hit her.

Above the Arctic Circle the sun at this time only went below the horizon for a short time, which meant that our ships could be continually kept under observation by German aircraft and Uboats. There were however some fascinating panoramas of sea and sky, and I remember that one evening the ship had to steam into the wind straight for the coast and the spectacular black rugged mountains of Norway loomed up ahead. I vowed that one day I would revisit the area, and so I did with Joan during our Norwegian cruise of 1987.

Our base was Scapa Flow where we returned every few days. Occasionally we went ashore and the main treat was a visit to the NAAFI canteen which provided a large dish of bacon and baked beans. Otherwise we spent time in the wardroom eating, drinking and playing shove ha’penny or bar skittles. One day we played hockey against a team of large and ruthless Wrens, who beat us using their sticks with wild abandon.

In July more engineer officers joined the ship and I knew that one of them would occupy the vacant berth in my double cabin. I anxiously watched them come aboard and liked the look of Brian, and was very glad when he was allocated to my cabin. Then began a friendship which has lasted all our lives.

October – November 1944         We returned to Clydebank in October and made preparations for going to the Far East. Then we steamed down to Portsmouth and went into dry dock for maintenance and cleaning the ship’s bottom. After this we were ready for sailing but before doing so on 16th November the King and the Royal Family came aboard to wish us Good Luck. We were all mustered in our divisions on the flight deck, the King inspected us and then asked for a cup of tea. This caused a flap as all the cooks and stewards were mustered, and it took the duty officer nearly half an hour to find some tea and make it!

December 1944         After leaving Portsmouth we sailed to Ceylon, passing through Gibraltar, the Mediterranean, the Suez Canal, and then across the Indian Ocean arriving at Colombo on 10th December. We stopped off Algiers where our Mess Secretary went ashore and triumphantly came back with a large load of Algerian wine, which turned out to be the most awful plonk! We had Admiral Vian, the fighting Admiral, on board and at Colombo he demanded to be ferried ashore immediately in his Admiral’s Barge. This motor boat arrived on board at Portsmouth just before we sailed and was stowed in one of the hangars, where the engine could not be tested. I was in charge of boats and I insisted that the boat should have a trial run before an official trip. The Admiral was furious and came storming down the Hight deck demanding an explanation, so I stood to attention quaking in my shoes and gave one. He looked me up and down and said “Right, I will give you ten minutes”. Luckily all went well. Strange how one remembers these things!

During the remainder of the month we spent time at Colombo or Trincomalee storing ship, or at sea exercising with other ships of the Fleet. Trincomalee was a beautiful harbour, and I remember Brian and I were thrilled to bring back a pineapple (which we hadn’t seen for years) to our cabin, but when with due ceremony we slit it open it was full of insects!

January 1945         On New Year’s Day we sailed in company with three other carriers, the battleship King George V, and several cruisers and destroyers for air strikes against the Japanese oil refineries at Palembang in Sumatra. The first strike took place on 4 January and about 100 aircraft took part plus 40 Seafires which provided fighter cover. The refineries were damaged but after returning to Trincomalee it w as decided that further strikes would be carried out and they took place on 24 and 29 January. These were major strikes carried out by 144 aircraft for the first and 128 for the second, plus the usual fighter cover. This time the Japanese were well prepared and on several occasions the Fleet came under attack by enemy aircraft. These were fought off by our guns and aircraft, two being shot down close to Indefat. There were many air battles and we lost 41 aircraft together with many of the aircrews. This included several aircraft that were damaged by enemy action and then crashed on deck landing. The worst event was the fate of nine aircrew survivors who had to force land in Sumatra, were made prisoners, taken to Singapore and then later beheaded. The strikes were successful as the refineries produced some 50 of Japanese oil requirements and they were reduced to a standstill, only increasing back to one third capacity by the end of March. After this we steamed south for Australia and crossed the line (the Equator) with King Neptune and his cohorts “coming aboard” on 1st February. I was duly ducked and scrubbed m a makeshift swimming pool.

February 1945         We called in at Fremantle and six days later arrived in Sydney and moored at Wooloomooloo near the Harbour Bridge. The Australians were very hospitable and Brian, Colin and I were “adopted” by the Murray-Jones family with two daughters, Judy and Annabel. They would invite us home for a meal or arrange some tennis or swimming, not that there was much time as we were busy with maintenance and storing for the Pacific. Towards the end of the month we steamed north with the British Pacific Fleet under Admiral Rawlings.

March 1945         After 11 days at sea we arrived at the island of Manus and then went on to Ulithi, another island. This had an enormous harbour and was full of American ships, a total of about 1,400 preparing for the invasion of Japan. Our Fleet then became Task Force 57 operating with the American 3rd Fleet under Admiral Spruance, and consisted of three other Fleet carriers, eleven destroyers and a number of support ships including sloops, frigates, minesweepers, oil tankers and hospital ships. Sailing from Ulithi our first strike took place on 26th March against some of the Japanese islands south of Okinawa where it was estimated that the Japanese had 10,000 aircraft, of which about 4,000 were suicide bombers called Kamikazes.

Then began a series of strike days, each being a long day’s activity for the Fleet, particularly for the ships’ companies of the aircraft carriers. We would go to Action Stations at 0600 and return to Defence Stations at 2000, and periodically a “Flash Red” warning would be broadcast when enemy aircraft approached. Several air battles took place and, throughout the day, the Fleet wheeled and turned in and out of the wind for the carriers to land on and fly off strikes and fighter escorts. When the last aircraft landed on at dusk the air engineering department worked all night to repair, re-arm, and refuel aircraft ready for the next day.

April 1945         On the morning of 1 April we were hit by a Kamikaze which exploded into the flight deck and bridge structure. Because the flight deck had 3″ armour plate the damage was not catastrophic but fourteen of the crew, including the ship’s doctor, were killed and there was a lot of damage to the flight deck barrier gear and bridge communications. I was Damage Control Officer for the area and my team had to remove the casualties and start repairing the damage. I remember the whole area was flooded with hot steam, as the steam-to-ships siren pipework was broken, until I managed to telephone Y boiler room to shut off the master valve.

Peter Sandison’s team did a good job to repair arrester wires and barriers, and the ship was flying off aircraft an hour later, much to the amazement of the American ships and Admirals. The American carriers with light steel decks were very vulnerable and many of their carriers were sunk or badly damaged due to Kamikazes. On 6 and 7 April the Japanese made massive attacks on allied ships with most of them concentrated on American ships to the north of our Fleet. These attacks were made by 600 aircraft, including 355 Kamikazes, and some 380 were shot down but six American ships were sunk and twenty-one damaged. At this time the giant Japanese battleship Yamato came out on a suicide mission and was sunk by American torpedo bombers with a loss of 2,100 men.

Operations continued until the last week of April when our Fleet returned to Leyte island for refitting and oiling, having been at sea continuously for 32 days. By this time sixty support ships had arrived to provide repair and maintenance facilities. During the month I was promoted to Temporary Lieutenant (E) RN and wore my second stripe.

May 1945         On 1st May the Fleet including the carriers Indefatigable, Implacable. Indomitable, Formidable and Victorious left Leyte to resume operations against the Japanese shipping and shore installations, with Action Stations every day except for the odd day when we retired for refuelling by waiting tankers. British ships were essentially designed for Atlantic operations, and consequently there was very little air conditioning to deal with the hot climate of the Pacific, Some of the machinery spaces reached temperatures of 140⁰ F and almost every day one’s boiler suit could be twice soaked with perspiration. After a few weeks one would suffer from prickly heat and would be painted with purple potassium permanganate, so looking like an Ancient Briton! Sleeping at night on the quarter-deck was the most comfortable time. Food was almost all dehydrated or tinned and a staple of the diet of dehydrated potato served in a variety of ways – mashed, cubed, boiled, roast or fried. There were also plenty of tins of egg powder and powdered milk!

On 4 May Formidable was hit by a Kamikaze which caused considerable damage and fires on the flight deck but the ship remained operational. Indomitable was nearly hit by another Kamikaze which was shot down and crashed some thirty feet off the starboard bow. A few days later Formidable was again hit and fires were started in the hangar, and nine aircraft were destroyed. All through this period the enemy pressed home their attacks with great skill and determination, making good use of cloud cover, decoys and variations of height. All five carriers were hit at least once by Kamikazes, but nevertheless our aircraft flew some 2500 sorties, dropped over 500 tons of bombs and destroyed about 60 aircraft, at a loss of 98 aircraft.

June 1945         At the beginning of June we returned to Sydney for vital boiler maintenance, aircraft repairs and other general refitting. This was a welcome relief after 100 days on the ship at sea and again the Murray Jones were very hospitable, so we enjoyed some tennis and swimming off Bondi Beach. Towards the end of June the Fleet sailed north again and resumed operations in co-operation with the American Third Fleet.

July – August 1945         We carried out strikes against the Japanese mainland for the first time, including airfields and installations in the Tokyo area. The routine developed of 4 or 5 days at Action Stations, then a day’s withdrawal for refuelling, and then back again for more strikes. It was a time of Action Stations, watch-keeping, eating and sleeping in a noisy, hot and tiring atmosphere, with some excitement when enemy aircraft appeared. The Flight Deck was again busy from dawn to dusk, sending off bombers and also fighters to protect the Fleet. Unfortunately many did not return, and several had accidents when landing back on. At this stage the whole of the Japanese mainland from north to south was under attack by allied ships, with the Americans concentrating on destroying the remnants of the Japanese navy. The British aircraft bombed industrial targets including shipping, oil storage tanks, railways and factories, and on two occasions the battleship King George V carried out extensive bombardment with her heavy guns.

On 4th August all ships were ordered to withdraw some 300 miles from Japan, and on the 6th the first atomic bomb was dropped on Hiroshima, and then the second on Nagasaki. Further strikes continued until the Japanese finally surrendered on 15th August. The Fleet remained at sea but on the 25th we were hit by a typhoon. The waves were awesome, I remember standing on the flight deck which was 70 ft above normal sea level, and watching waves much higher than this coming towards me. The ship was rolling 35° from one side to the other, but we survived. Three American destroyers capsized, and we saw one American carrier with a large part of its flight deck hanging over its bows, as though it had received a punch on the nose!

September 1945         The Japanese Surrender was signed on the USS Missouri in Tokyo Bay on 2nd September, much to our relief. We remained at sea, and with the American Fleet took part in an enormous “parade” of ships outside Tokyo Bay. Then we spent three days in the Bay, while some of our crew went ashore to find and collect prisoners of war and transport them to hospital ships. The famous Mount Fuji is usually covered in cloud but early one morning the tannoy broadcast that it was visible, and I remember a marvellous view of its snow-capped peak.

After this we steamed back to Sydney arriving towards the end of the month, ready for a respite after 73 days of sea time. It was time to reflect on past events, the worst being during July and August when the Fleet lost over 140 aircraft from all causes, by enemy action or deck landings. Since then there has been a lot of discussion about dropping the atomic bombs and their consequences, but to my mind the following reasons justified the decision.

1. The Americans estimated that there would be around a half-million Allied casualties if the invasion of Japan had taken place later in the year. This did not happen.
2. About 40,000 British and Allied prisoners of war were kept by the Japanese in horrendous conditions and most would probably not have survived another winter. They were rescued.
3. The Japanese had some five thousand aircraft and pilots trained as Kamikazes to be used against an invasion fleet, and we would have been in the forefront of this.

October – December 1945         Indefat remained in Sydney and the crew were allowed a lot of shore leave. The Murray-Jones thoughtfully provided a flat where many of us could stay, including Brian, Colin, Peter Fanghanel and others. One highlight was when all the latter including me made up a party to go ski-ing for a week at Mount Kosciusco. We arrived at the snowline and were then told that the chalet was 12 miles away and could only be reached on skis. Some of us, including me, had not skied before but we were told “Oh, that’s OK, today is Tuesday, and there is a tractor going up on Thursday which could pick you up if you are stuck”!

This was a time of hard work and playas supplies were exhausted, the engines needed refitting, the ship needed cleaning and the typhoon had damaged part of the hull so the ship had to go into dry dock. Some of us were seconded to the dockyard to help out with various jobs and I enjoyed the use of a 500cc motor-bike.

We managed another five days on a sheep farm, again with Brian and Colin. The farm was enormous and the family relied on horses to get around. On the first day we were each provided with a horse, but I viewed this with trepidation. So evidently did the horse, as after 15 minutes he turned round and trotted home, and there was nothing I or anyone else could do to stop him! I decided that I would stick to something with a brake and throttle.

January – March 1946         On 20th January we left Sydney for the journey home. Three days later we arrived at Melbourne where we had a tremendous welcome, with a parade led by the Royal Marines hand marching through the streets to the City Hall where the Governor took the salute accompanied by Admiral Vian. We stayed a week and were well entertained, then steamed across the Australian Bight which was unpleasantly rough to call in at Fremantle for a few hours before setting off for Capetown.

We arrived at Capetown after 17 days at sea. Again we were well looked after with a reception at the Governor’s Residence and an expedition to Table Mountain. This was the highlight of the visit, we took the cable car to the top with marvellous views all round and then came all the way down on foot. On 24th February we left Capetown, arriving at Gibraltar on 11 th March. On the way we passed close to St. Helena and Ascension Island. The Duty Officer went ashore and paid his respects to the Governor, who presented him with a live turtle to make soup! The ship’s butcher did not think much of this so when we left the turtle was returned to the sea, and was last seen swimming happily to the shore.

This part of the trip was pleasant and not too hot, every day there were games of deck hockey on the flight deck using a rope grommet instead of a ball. At Gibraltar we stayed for one day and Peter Fanghanel was the only one of our group who managed to go ashore, he came back with a large case of Tio Pepe sherry.

Finally we arrived at Portsmouth on 16th March and berthed inside the harbour, with crowds lining the Southsea promenade and cheering as we went in. We engineers saw little of this, hut we looked forward to a pint at the St. Enoch’s Hotel and then some leave. I think I had about ten days at Westcliff with Mother and Brenda, it was good to see them again after nearly 2½years.

April – October 1946         The ship sailed again on 25th April with 130 “Bush Brides”, who were brides of Australian servicemen and were going to join their husbands to live in Australia. The voyage again was through the Mediterranean and then a brief stay at Aden. Brian, Colin and I went ashore and we asked some joker the way to the local Club for a drink. “Oh” he said “lt’s that white building up on the hill”. So we trudged up the hill, knocked on the front door which was opened by a smart servant who asked what we wanted. We said we would like a drink, to which he replied that this was the Consul’s Residence. Anyway, the Consul was very decent, gave us more than one drink and we went happily back to the ship.

We arrived back in Sydney on 25th May and left again on the 9th June with over 1,000 service personnel due to be demobilised, including some RAF. We also carried 65 tons of food for Britain and about 18,000 gift parcels of food. From Fremantle the engines worked at full power and lndefat made a record-breaking non-stop trip of 21 days to Portsmouth. Then on 29th July we sailed again to Colombo and repatriated another large number of service personnel. The highlight I remember was a visit to Kandy and the Temple of the Sacred Tooth, where we were guided by Buddhist priests in their saffron yellow robes.

The last major event was a parade by the ship’s company on 19th September through Holborn in London, the borough that had “adopted” us during the war. As one of the officers with the longest-serving time in Indefat I was placed in the front rank, and there is a photo in our album. After the march we were inspected by the Mayor and then had a luncheon in the Town Hall, where our Battle Ensign flown by Indefat during Action Stations was presented to be hung in the Council Chamber. The demobilisation process was slow, but I finally left the ship and the Navy on 1st October 1946, after a wardroom party the night before! I well remember going down the gangway, walking through Portsmouth Dockyard and then out through the Main Gate, ready to face a different kind of life and world.

In Memoriam – 2

This is the second of two items that my mother typed out more than 25 years ago. I had very mixed emotions when reading it but ultimately I felt that it was a reminder that, despite all the nasty incidents and stupid behaviour hyped up by the press and news outlets, people and organisations are generally kinder, gentler and more understanding than they were 60 years ago.

This story is about the birth of my brother who was born with a genetic flaw now known as Trisomy 21 though formerly known as Down’s syndrome or (colloquially, and no longer acceptably) mongolism. It is the latter term that my mother uses as it was the common term at the time of birth and at the time she typed her story.

A child is born.

The history written by Dorothy Kathleen Lewis (1925 – 2017) about the birth of her first son

My pregnancy was normal. The first indication I had that something was wrong was in the delivery room when the baby was born; there was “oh” and silence then whispers. I asked what was wrong but was told nothing. The baby was put in a cot and the doctor came into the room and then he was taken out. I had not seen the baby – just knew that it was a boy. Again when I asked I was told that this was routine. Eventually the baby was brought back and given to me. When I saw him I thought he looked very odd and was so floppy. When I held him upright I could see he was a mongol, but prayed that I was wrong and this would go away. I asked to be told what the matter was with the baby and was told to tell my husband to ask if I was worried – which made me more suspicious.

Visiting was restricted and I did not see my husband until the evening. Fathers were just shown the babies at the nursery door and were not allowed to hold them. My husband was delighted that we had a little boy and I didn’t have the heart to tell him what I feared.

David had difficulty feeding and was put on a bottle at three days, the teat of the bottle made with a big enough hole for the milk to drip into his mouth because he was not sucking. When we went home, still not having been told of his conditions, he was being fed 8 times a day taking just 1.5 ounces per feed. Each feed took an hour to get into him, then at night it was back to bed for 2 hours and a repeat performance.

I took David to the child welfare clinic and again the actions of the people there spoke volumes, the health visitor hurried into the doctor and I was shown in – jumping the queue. (The clinic was held in our church hall which was next to the vicarage and I was very embarrassed that I should be singled out, although it was obvious why.) I asked the doctor what she thought and she said he did look mongoloid, but perhaps I should see the paediatrician where he was born.

At 6 weeks [ed: see photo] I went for my post natal and there was great concern in the waiting room as to how the baby was getting on. None of the other mothers who were there were being asked. I said he still looks like a mongol. My husband was still not aware of David’s condition or my suspicions, I wanted to protect him from the hurt I was feeling, but now I know it was not the kindest thing to do.

I then took David back to the Middlesex hospital and saw the paediatrician, who took him away from me, and whilst I sat at one end of a very large room he had David on a table at the other end with a group of students. I could not hear what they were saying, but when David was brought back to me I was told: “You have a complete vegetable; he will never walk or talk – just lie in his pram and stare up at the sky. The best thing you can do is to put him in a home and forget you ever had a baby.” I was devastated; I couldn’t run away from it any more. He had an enlarged liver and spleen and his spine was curving outwards. When I held him in my arms it was a little like a floppy parcel and there was no buoyancy at all.

When I got home I couldn’t hold back the tears that had been stifled all those weeks and I had to tell my husband. It was dreadful, I think it would have been better had we been able to grieve together in the beginning.

From then on everything David did was a milestone and he brought us a lot of joy. Just before he was five I had to take him to County Hall in London for assessment. That was a nightmare because by this time I had two other children – little boys – it was necessary to take the older of the two with me, a very busy child. We went into a large room and an elderly fussy lady had a lot of questions for David to answer. He was shown pictures and asked what they were. He hid his face from her and was saying the words to me, many of which he already knew, but because he was not answering her they were crossed out. So his assessment was a very low one.

I don’t think it would have made a lot of difference whether he had answered her he so surely wasn’t school material. He had been going to a junior training centre from the age of 3½ because I was expecting Jonathan. A social worker who came to see me at that time asked what sex I would like my third child to be and I said I didn’t mind so long as I had a normal healthy child, and she said that was a funny answer to give – I didn’t think it funny.

People’s reactions were very different 37 years ago [ed: now 64 years]. Once it was made known that David was as he was people who had known me from childhood would cross the street [to avoid speaking to me], they didn’t know what to say. But we didn’t hide him away and when we went on holiday we just said three children and we sometimes got a reaction when we arrived, but David was always well-behaved and everybody loved him. He learned a great deal from his brothers and I thank God he was our first child.

[Dorothy Kathleen Lewis: Banbury 1990]

Footnote

While copying up this story I was prompted to look at a few statistics from the UK’s Office of National Statistics for 1953 (and 50 years later); in particular the stats about child mortality and measles caught my eye.

Infant mortality for England and Wales

Year	Births	Still-births	Died with 1 week	Died within 4 weeks	Died within 1 year
1953	684,372	15,681	10,127	12,088	18,324
2003	621,469	3,612	1,749	2,264	3,306

Don’t forget when you read the mortality figures that the 2003 numbers will include births that could be anything up to 8 weeks premature. I think anything more than about 2 weeks premature would probably have ended up in the still-births column in 1953.

Measles (England and Wales).

Year	Cases Reported	Deaths
1953	545,050	242
2003	2,048	0

Addendum

I’ve received a private email pointing out that some of the cases reported as still-births in 1953 would now be identified as murder, where babies born with obvious viability issues would be smothered at birth – sometimes without the mother even knowing.

 

In Memoriam

My mother died a few weeks ago after a couple of months in terminal care. One of my tasks while she was in care was to go through all her paperwork and while doing so I discovered a couple of stories from her past that she had typed out (remember type-writers?) about 30 years ago. I typed them up on my laptop and printed several copies to hand out for people to read at the tea-party we held for her old friends – of all ages, ranging from 15 to 99 – after the funeral; this seemed to give them a lot of pleasure and they found them so interesting that I decided to share them with a larger audience. So here’s the story, written by my mother in 1983, of her evacuation experience at the start of the 2nd world war when she was a little over 14 years old.

The Summer of 1939.

Reminiscences of Dorothy Kathleen Lewis (1925 – 2017)

There had been a lot of talk about a war coming. Adolf Hitler and his armies had marched into Austria and were threatening Poland. We had all been issued with gas masks – just in case we would need them – and emergency plans had been made to evacuate all children from the big cities.

During the school holidays I was taken by my parents, with my sister, to my mother’s home village of Llangeitho in Cardiganshire. My mother had a cousin who was a retired school teacher and it was arranged with Auntie Jane that if war broke out Peggy and I would be sent to stay with her. I don’t think we were very pleased with the arrangement because to us she was very old-fashioned, not a bit like our mother. We ended our holiday and went back to London to wait for the school term to begin.

On the 1st September we heard that all children from our school whose parents wanted them to be evacuated should assemble at the school gates with a small suitcase and their gas masks. As we had already been told we were going to Llangeitho if the war broke out we stood and watched all our friends walking in crocodile fashion down the street and mothers and fathers crying watching them go. It was a very sad day, but I wished I was going with them. I didn’t like the idea of staying with Auntie Jane. None of these children knew where they were going, just somewhere in the countryside for safety, and they didn’t know who would be looking after them.

Well, on the morning of 3rd September Neville Chamberlain, our prime minister, spoke on the wireless (we now call it a radio) to say that we were at war with Germany. Immediately the sirens went and everyone ran to the shelters. My parents, Peggy, and I went to Baker Street Station, which has very deep platforms. There were hundreds of people with the same thing on their minds. We all took our gas masks with us. After a short time the all-clear went. My father sent a telegram to Auntie Jane to say Peggy and I would be leaving London on the train at 9:25pm that night. Trains did not travel as fast as they do today and we were due to arrive at Pont Llanio Station at 7:30am on Monday morning. Peggy’s friend and her mother (an Italian lady who did not speak very good English) was coming too, also one of the young people from the village who was working in London.

Paddington Station had very dim lights and when we got on the train there were no lights at all. After a little while we children began to feel a bit less afraid and started to tell ghost stories and play memory games. It was fun going to the toilet on the train because there were people sitting in the corridor and so was their luggage. We could not see them and I don’t think we really tried – it was all a game. We were supposed to be sleeping, but we were too excited for that. When it came time to eat our sandwiches we had to taste them before we knew what we were eating. Can you imagine being in a train without any lights, and there were no lights in the streets or houses or on the station platforms that we passed. Names of stations had already been removed in case the country was invaded by the enemy. The belief was that the enemy would not know were he was if there were no road signs etc. No-one thought about them using maps and compasses as they would now. [ed: 1983]

We eventually arrived in a town called Carmarthen where we had to change trains and take a slow train to Pont Llanio where a car would meet us. Our train from Paddington was very late arriving and the slow train had gone. Someone telephoned Pont Llanio station to say we would be late and to send the car back. The train from Carmarthen was a very slow one and my father used to say “you could get out of the train and pick flowers on the bank and get back into the train again”. It really was very slow and chugged its way along the line. We arrived at last in Pont Llanio and then in Llangeitho after a journey of 16 hours. [ed: 4:30 to 5:00 hours driving time, now; 6 hours by public transport] I am sure we must have looked very dirty and untidy. The trains in those days were steam and there would be plenty of coal smuts flying around.

I did not think Auntie Jane would be very pleased to see us and I was soon to find out that I had thought rightly. The first thing she did was to take up the stair carpet in case we wore it out. I don’t know how she thought we would do that because once we came down in the morning we were not allowed to go back upstairs again until we went to bed. [ed: if you’ve read “Carrie’s War” you may recognise the behaviour]  She also did not know that children eat quite a lot too. For breakfast Auntie Jane would boil an egg and cut it in half, so Peggy and I had half each. And the same for our dinner, we would have two small potatoes – and this was before rationing and shortage of food. We had a lot of friends in the village and if it was not for them asking us out to tea and/or supper we would have been very hungry. Peggy went to school in the village, but I was too old [ed: at 14 yrs 4 months] and had nothing to do all day, but soon found a baby I could take out in the pram and that meant I would be asked if I would like a piece of cake and a drink. After a few weeks and a number of letters home things goT a little better because my mother was sending parcels of food to Auntie Jane. I don’t know what arrangements were made money wise; because we were not Government evacuees Auntie Jane would not have been paid by the authorities to keep us.

One of the things we used to do together with two of our friends was to help the local butcher clean out his slaughter-house after he had killed a beast. This meant he then asked us to supper in his old farm-house with a huge Inglenook fireplace. Another of my mother’s friends used to have us in early for a meal and say “don’t tell Auntie Jane or she will not give you anything else to eat”. I often think back on those days and wonder why she was so mean. She had never married and had children, but being a teacher I would have expected her to be more tolerant.

In December of 1939 Peggy wrote a letter home which was full of complaint and left it somewhere where Auntie Jane found it and this letter was sent to my parents with a letter from Auntie Jane asking that we be sent back to London. A lot of the people in the village were very surprised to think that she should think to send us back to London when there were air-raids (these had not started at that time). People were saying we would be going home to be killed, but as for me I would rather take that chance than be left in Llangeitho.

Going back to London wasn’t much fun – the school was closed so once again we were at a loose end. We stayed in London over Christmas and again the government started evacuating school children and in February we joined a group who were leaving London – this time as London School Evacuees. We were sent to Buckingham to a family with a little girl of 2 years. This seemed to be alright and we went to school in the afternoons whilst the local children went to school in the mornings. It got rather uncomfortable there after a while because the man of the house, aged 24, lost his job (I don’t know why) and there were a lot of arguments in the house. His wife did not make herself look smart and he started to pay too much attention to me. Again a letter home that it was time we left there and one morning my father arrived and said: “pack your bags, you’re coming home”. What joy!

I don’t have much memory about this part of being an evacuee except to say I was relieved to be out of that house and back in the safety of my family. Whilst we were in Buckingham there had been bombing in London and Peggy and I were taken to see some of the damage that had been done. I think this was to frighten us so that we would be willing to go away again. I certainly did not like the sirens going and having to stop what we were doing and go to the shelter^[1]. Once again we were on the move and this time I have a very detailed memory of the events.

We were assembled at the school playground all with our cases and gas masks – worried children and even more worried parents and teachers. No one knew where we were going except that we all piled into a double-decker bus. Lots of tears this time because we knew that people were being killed and injured. Would we see our parents again? What was going to happen to us if they were killed? Where would we go, who would look after us? Questions, questions!

We were taken to Marylebone station and put on an underground train. Where was it going? What were mum and dad doing now; were they still blowing their noses? We were not so bothered because we still knew where we were. Next stop Paddington Station and hundreds of children milling about. I remember I was in the Junior Red Cross at that time and a Red Cross nurse saw my badge and came to speak to me. Such a little thing but it meant such a lot and I have never forgotten her kind words, saying I was going somewhere safe and would be alright. Maybe I was crying at the time, I don’t know.

As the train pulled out of Paddington Station we were all trying to get to a window to wave, although we didn’t know anybody and we didn’t know where we were going. Well of all places – we arrived in Banbury. Now my Auntie Kit, my father’s sister, only lived about 2 miles out of Banbury in a village called Bodicote. I knew Banbury well because we had often visited Bodicote. If only she knew I was here. I know we could not stay with her because she had a very small house and she had 4 children already.

Again on another bus and somehow Peggy and I and two other girls from my class got separated from the rest of our school and we were taken to a village called Great Rollright. Peggy and I went to stay with a lady called Mrs. Robinson who kept the village shop, and my two class friends went to a farm.

Mrs. Robinson was a kind lady – she had asked for two boys but somewhere along the line she had two girls instead. It was very strange arriving in Great Rollright. We were all taken into the Church Hall and there were village people there who, to us, had a funny way of speaking. And one after the other they were saying how many children they would take. Mrs. Robinson wasn’t there, so maybe that is why she didn’t get two boys. I thought it was very embarrassing to be standing there with these quaint country people whilst they were deciding whether they wanted one, two, or three children.

Our time with Mrs. Robinson was very happy. Peggy went to the village school and I went to the county school in Chipping Norton, again on a part-time basis. Mrs. Robinson had a pet cockerel which was allowed to roam round the house – I hated that bird and I think it knew it. Its name was Cocky. Every time I wanted to go down the garden to the toilet this bird would follow me and stay outside until I came out again and peck my legs as I raced back up the garden.

There was certainly plenty to eat in this house and we really had an enjoyable time there. We were always sent to bed with a small glass of cider. I never knew then that cider was an alcoholic drink and I thought it was lovely. We didn’t seem any the worse for it anyway.

We got involved with the village church and would have been happy to have stayed there. But doom. One day I came home from school to find my head mistress from London sitting at the table. Unbeknown to us she had been frantically looking for these four girls she had lost at Banbury Station. I don’t know how she had found us, whether she had contacted our parents or hunted through the schools in the area. With a surname like Walklett I don’t think we would have been difficult to find. I don’t think she had been in touch with our parents – what an awful thing to say to them: “I’m very sorry but I may have lost your children”. No, she must have hunted through the school registers.

The upshot of this visit was that she had found somewhere more suitable for us and would we pack our things because she had found somewhere else for us to stay. More tears because we liked Mrs. Robinson, and the village people were all lovely to us and we were invited to their homes. Off we went with Miss Attride in the car to another village called Duns Tew. The strange thing was that none of our school were there, so why were we moved yet again?

This time we stayed with Mr. and Mrs. Beck, his name was Harry and hers was Daisy, but they were Mr. and Mrs. Beck to us. Mr. Beck was a farm hand and he worked with horses. He used to plough the fields of the farm across the road. He must have walked miles in the days he was ploughing. Although I had had many holidays in Wales and Shropshire at haymaking time I knew nothing about ploughing.

Mr. and Mrs. Beck had a young man living with them. He was like their son; although his family lived in the village he had lived with the Becks since he was a baby and they called him their son. His name was Walter. The village was a beautiful place and we lived in No. 73. There were no street names, every house had a name and a number so we were at No. 73 Duns Tew, the last house in the village, a lovely old thatched cottage. There was always a big wood fire in the grate and plenty on the table. Mr. and Mrs. Beck were the nicest people in village.

Peggy now had to go to Steeple Aston School (since moving to Banbury in 1975 I have met the widow of her headmaster there), and I went to a Continuation College which had been evacuated from the East End of London. This was very odd to me – we were taught shorthand, typing, arithmetic, English grammar, French. This was obviously training us for the commercial world. I was much younger than the other girls there but my education was more advanced than theirs so I soon became top of the class. My English was always being complimented. What they didn’t know was that I had a Welsh mother and the Welsh language used very letter in the word. My French was well in advance and my Maths took a flying leap.

I made friends in the class. The class was held in The Hall, North Aston – a country seat. The Hall was so large that there were 9 girls living there and they had servants. The school room was in the Grand Hall and it was so beautiful it seemed a pity to me that there were desks etc. on the polished floor.

In Duns Tew we had one of the masters of the school staying in The Nurseries (which is still there) and every Friday evening the family he stayed with invited those of us in the village to spend the evening in their house and they had a piano so all the war songs were being sung: “Roll out the Barrel”, “We’re going to hang out the washing on the Siegfried line” and many more.

Because the school at North Aston was a long walk I bought a bike, something I had always wanted, and I joined the cycling group. This meant on one day a week we would go for an outing to places like Blenheim Palace [ed: 10 miles away] etc. I became a good cyclist and had plenty of energy when others flagged behind. I certainly made use of my bike.

One particularly happy time was when it snowed in the winter. Yes, we did get snow in London, but not like this. It was white^[2] and where the wind blew it was as high as the hedgerows; I couldn’t believe what I saw. Walter the Beck’s son had a sledge and he showed us where it was good to use it. It was a fantastic time.

 

[Banbury, 1983]

 

 

^[1] One of the stories about my mother that I first heard at her funeral was about the time she persuaded her parents to let her stay at home overnight. At the time the family used to head for the air-raid shelter (i.e. the local underground station) at the end of the day and stay there all night long. My mother hated this and persuaded her parents to let her stay at home in her own bed provided she promised to join them at the air-raid shelter as soon as the air-raid sirens sounded. She was only allowed to do this once – because she managed to sleep through two bombing runs and without being woken by the sirens or the explosions.

^[2]If you’re wondering why white snow is worth mentioning you probably don’t know about the density of London smog at that time.

So Long ACED

… and thanks for all the fish.

Today I removed myself from the OTN ACE program. This isn’t a reflection on anything to do with the ACE program – quite the reverse, in fact – it’s because they’re introducing steps to ensure that the ACE Directors can justify their titles. Unfortunately, as anyone who has gone through (e.g.) ISO 9001 certification can tell you, quality assurance tends to translate into paperwork and ticking boxes – and while I can always find time to write up some interesting feature of Oracle I really find it hard to prioritise time for filling in forms.

In the last 4 months I’ve failed to file my monthly list of relevant activities twice, failed to request funding for two of the international conferences I’ve spoken at, and failed to submit claims against the two for which I had requested and received funding approval – so there really was no hope of me being motivated to collect all the extra details that the new regime requires.

So, best wishes to the ACE program – I’m still happy to do what I’ve been doing for the last 25+ years, and I’ll still be meeting up with ACEDs, but I’ll just be wearing one label less as I do it.

#ThanksOTN

To mark the OTN Appreciation Day I’d like to offer this thought:

“Our favourite feature is execution plans … execution plans and rowsource execution statistics … rowsource execution statistics and execution plans …  our two favourite features are rowsource execution stats and execution plans … and ruthless use of SQL monitoring …. Our three favourite features are rowsource execution stats, execution plans, ruthless use of SQL monitoring and an almost fanatical devotion to the Cost Based Optimizer … Our four … no, no … amongst our features … amongst our favourite features  are such elements as rowsource execution statistics, execution plans …. I’ll come in again.”

With apologies to Monty Python.

 

 

 

Month End

A question about parallel query and cardinality estimates appeared on OTN a little while ago that prompted me to write this note about helping the optimizer do the best job with the least effort.  (A critical point in the correct answer to the original question is that parallel query may lead to “unexpected” dynamic sampling, which can make a huge difference to the choice of execution plans, but that’s another matter.)

The initial cardinality error in the plan came from the following predicate on a “Date dimension” table:

      AR_DATE.CALENDAR_DT   = AR_DATE.MONTH_END_DT 
AND   AR_DATE.CALENDAR_DT  >= ADD_MONTHS(TRUNC(SYSDATE,'fmyy'), -36) 
AND   AR_DATE.MONTH_END_DT >= ADD_MONTHS(TRUNC(SYSDATE,'fmyy'), -36)

In the parallel plan the estimated number of rows on a full tablescan of the table was 742, while on the serial plan the same tablescan produced a cardinality of 1. You will appreciate that having an estimate of 1 (or less) that is nearly three orders of magnitude wrong is likely to lead to a very bad execution plan.

My first thought when I saw this was (based on a purely intuitive interpretation): “there’s one day every month that’s the last day of the month and we’re looking at roughly that last 36 months so we might predict a cardinality of about 36”. That’s still a long way off the 742 estimate and 1,044 actual for the parallel query, but it’s a warning flag that the serial estimate is probably an important error – it’s also an example of the very simple “sanity checking” mental exercises that can accompany almost any execution plan analysis.

My second thought (which happened to be wrong, and would only have been right some time well before version 10.2.0.5) was that the optimizer would treat the add_months() expressions as unknown values and assign a selectivity of 5% to each of the predicates, reducing the combined selectivity to 1/400th of the selectivity it gave to the first predicate. In fact the optimizer evaluates the expressions and would have used the normal (required range / total range) calculation for those two predicates.

It’s the first predicate that I want to examine, though – how does the optimizer calculate a selectivity for it ? Here’s some code to generate sample data to use for testing.

rem
rem     Script:         month_end.sql
rem     Author:         Jonathan Lewis
rem     Dated:          June 2016
rem

create table t1
nologging
as
select
        rownum                                                   id,
        to_date('01-Jan-2010','dd-mon-yyyy') + rownum - 1       calendar_date,
        add_months(
                trunc(to_date('01-Jan-2010','dd-mon-yyyy') + rownum - 1 ,'MM' ),
                1
        ) - 1                                                   month_end_date
from
        dual
connect by
        level <= trunc(sysdate) - to_date('01-jan_2010','dd-mon-yyyy') + 1
;

execute dbms_stats.gather_table_stats(user,'t1',method_opt=>'for all columns size 1')

This clunky bit of code gives me consecutive dates from 1st Jan 2010 up to “today” with the month_end_date column holding the month end date corresponding to the row’s calendar_date. So now we can check what the optimizer makes of the predciate calendar_date = month_end_date:

set autotrace on explain

select count(*) from t1 where calendar_date = month_end_date;

  COUNT(*)
----------
        79

Execution Plan
----------------------------------------------------------
Plan hash value: 3724264953

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |    16 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |    16 |            |          |
|*  2 |   TABLE ACCESS FULL| T1   |     1 |    16 |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("CALENDAR_DATE"="MONTH_END_DATE")

Looking at operation 2 we can see that, in effect, the optimizer has considered two independent predicates “calendar_date = {unknown}” and “month_end_date = {unknown}” and taken the lower of the two selectivities – which means the cardinality estimate is 1 because the calendar_date column is unique across this table.

There are various ways to work around problems like this. One of the simplest would be to tell Oracle to sample this table with the (table-level) hint /*+ dynamic_sampling(t1 1) */; in fact, since this predicate is effectively treated as an examination of two predicates the (cursor-level) hint /*+ dynamic_sampling(4) */ would also cause sampling to take place – note that level 4 or higher is required to trigger sampling for “multiple” predicates on a single table. As a general guideline we always try to minimise the risk of side effects so if this problem were embedded in a much larger query I would prefer the table-level hint over the cursor-level hint.

There are other options, though, that would allow you to bypass sampling – provided you can modify the SQL. The script I used to create this table also included the following statement:

alter table t1 add (
        date_offset1 generated always as (calendar_date - month_end_date) virtual,
        date_flag generated always as (case when calendar_date - month_end_date = 0 then 'Y' end) virtual
);

In 12c I would declare these virtual columns to be invisible to avoid problems with any SQL that didn’t use explicit column lists. For demonstration purposes I’ve set up two options – I can find the rows I want with one of two obvious predicates:

    date_offset1 = 0
    date_flag = 'Y'

In fact there’s a third predicate I could use that doesn’t need to know about the virtual columns:

    calendar_date - month_end_date = 0

Unfortunately I can’t arbitrarily swap the order of the two dates in the last predicate, and the optimizer won’t spot that it is also equivalent to “calendar_date = month_end_date”. Here are a few execution plans – for which the only significant bit is the cardinality estimate of the full tablescans:

select count(*) from t1 where date_flag = 'Y';

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     2 |     4  (25)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |     2 |            |          |
|*  2 |   TABLE ACCESS FULL| T1   |    79 |   158 |     4  (25)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("DATE_FLAG"='Y')



select count(*) from t1 where date_offset1 = 0;

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     4 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |     4 |            |          |
|*  2 |   TABLE ACCESS FULL| T1   |    78 |   312 |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("DATE_OFFSET1"=0)



select count(*) from t1 where calendar_date - month_end_date = 0;

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     4 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |      |     1 |     4 |            |          |
|*  2 |   TABLE ACCESS FULL| T1   |    78 |   312 |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("T1"."DATE_OFFSET1"=0)

It’s interesting to note that the optimizer has transformed the last predicate into the equivalent virtual column expression to do the arithmetic. You might also note that the date_flag option is slightly more accurate, but that’s because it’s based on an expression which is null for the rows we don’t want while the date_offset1 column has a value for every row and a little bit of arithmetical rounding comes into play. You might also note that there’s a small cost difference – which I’d ascribe to the CPU cost that the optimizer has added for the CASE expression being applied on top of the simple date arithmetic.

Of course, whatever else you might play around with when working around a cardinality problem like this, I think the strategic aim for a data warehouse system would be to get a REAL flag column on the table and populate it at data loading time if month-end dates played an important part in the proceedings – though I have to say that the virtual flag column is one I do rather fancy.

 

Centenary

I rarely blog about anything non-technical but after the events last Friday (1st July) I wanted to say something about the pride that I shared with several hundred parents around the country as they saw the effect their offspring created through a living memorial of the terrible waste of life that happened  a hundred years ago on 1st July 1916 when some 70,000 soldiers (a very large fraction of them British) were killed or injured on the first day of the battle of the Somme.

While a memorial service was being held at Thiepval – a monument to 72,000 British (Empire) soldiers who died in the battle of the Somme but have no known grave – 1,500 “ghosts of the Somme” were silently wending their way in small groups through the streets, shopping centres, and train stations of cities across the UK, pausing to rest from time to time and occasionally bursting into the song of the trenches: “We’re here because we’re here”.

Each “ghost” represented a specific solder killed on the first day of the Somme and if you approached one of them to ask what was going on their only response was to look you in the eye and hand you a card stating their name, rank, regiment and, where known, the age at which they died.

Although many of the posts and tweets about the event mention the collaboration and assistance of various theatre groups around the country almost all of the soldiers were simply people who had responded to an advertisement for Project Octagon and had spent time over the previous 5 weekends rehearsing for the event. My son Simon was one of the volunteers who was on the London beat, starting with the morning commuters at Kings Cross then trekking around London all day – in hobnailed leather boots – to end at Waterloo station for the evening commuters.

After hours of walking this was how he appeared at Waterloo at the end of the day:

Like me he normally sports a beard and moustache but he’d shaved the beard off and trimmed the moustache to the style of an older era. The absent, dazed, look is in character for the part but also, I think, an indication of exhaustion, both physical and emotional. I wasn’t even sure he’d realised I was crouching six feet away to take this photo until I asked him about it the following day. When I showed the picture to my wife it brought tears to her eyes to think that 100 years ago that might have been the last sight of her son she’d see before he went off to die – it’s a sentiment that appeared more than once on Twitter through the day.

Shortly before 6:00 pm several more groups converged on Waterloo for a final tableau giving us a rendition of “We’re here because we’re here” that ended in an agonised scream:

It’s a gut-wrenching thought that a group that size would have been killed roughly every 6 minutes, on average, on the first day of the Somme though, realistically, the entire 1,500 that volunteered for the day would probably have died in the first few minutes of the first wave.

Behind the Scenes

There was no announcement of this living memorial so throughout the day people were asking where the soldiers came from and who had organised the event. Finally, at 7:00 in the evening 1418-Now identified themselves as the commissioning body, with Jeremy Deller as the artist in collaboration with Rufus Norris of the National Theatre.

Like any military operation, though, between the generals at the top and the troops at the bottom there was a pyramid of personnel connecting the big picture to the final detail. Under Jeremey Deller and Rufus Norris there was a handful of key characters without whom the day would have been very different. I can’t tell you who they all were but I’m proud to say that one of them was my daughter Anna who, along with a colleague, spent a large fraction of the last 16 months in the role of “Lead Costume Supervisor ” preparing for the day. Under the pair there were several regional costume supervisors, and each costume supervisor was responsible for several dressers who would have to help the volunteers put on the unfamiliar battledress.

Despite working on the project for 16 months Anna told me very little about what was going on until the day was over, and this is a thumbnail sketch (errors and omissions are my fault) of what she’s told me so far.

Amongst other things she selected a list of names from the soldiers who had died on the first day of battle, recording their rank, regiment, battalion and, where known, age. She then had to find out exactly what kit each battalion would have been wearing on the day, allowing for some of the variation that would have appeared within each battalion and catering for the various ranks; then she had to find a supplier who could make the necessary uniforms in a range of sizes that would allow for the variation in the build of the (as yet unknown, unmeasured) volunteers.

As batches of uniforms arrived each one had to be associated with its (historic) owner and supplied with 200 cards with the owner’s details – and it was really important to ensure that the right name was attached to a uniform before the uniforms could be dispatched around the country. Ideally a uniform would arrive at a location and find a volunteer who was the right size to wear it, with the right apparent age to match the card that went with the uniform; inevitably some uniforms had to be moved around the country to match the available volunteers.

The work didn’t stop with the uniforms being in the right place at the right time, of course. There aren’t many people alive who know how to dress in a British Army uniform from 1916 – so Anna and her colleague had to create a number of videos showing the correct way to wear webbing, how to put on puttees, etc. The other problem with the uniforms was that they were brand new – so they had to be “broken down”. That’s a lot of work when you’ve got 1,500 costumes. In part this was carried out by the volunteers themselves who spent some of their rehearsal time wearing the costumes while doing energetic exercises to wear them in and get them a little grubby and sweaty; but it also meant a lot of work for the dressers who were supplied with videos showing them how to rub (the right sort of) dirt into clothes and how to rough them up and wear them down in the right places with wire brushes etc.

One of the bits of the uniform you probably won’t have seen – or even if you saw it you might not have noticed it – was the T-shirt: the army uniform of the day would have been rather sweaty, itchy and uncomfortable on a hot summer’s day, so the soldiers weren’t wearing the real thing. Anna and her colleague designed a T-shirt that looked like the front of the shirt the troops should have worn under their battledress made of a material that was thinner, softer and much more comfortable than the real thing. In the end the day wasn’t as hot as expected so very few volunteers seemed to unbutton their tops – but if they had done so the T-shirts would have appeared to be the real thing.

Walking the Walk.

Apart from the authenticity of the uniforms another major feature of the day was the way that the ghosts made their way around from place to place silently, in single file, with no apparent references to maps (or satnav). Every group had a carefully planned route and timetable and two stage managers wearing brightly coloured backpacks so that they could be seen easily by the soldiers but, since one who walked 50 metres ahead and one 50m behind, were unlikely to be noticed by anyone who wasn’t looking. The stage managers were following carefully planned and timetabled routes – allowing the soldiers to stay in character all the time.

You may have seen pictures of the troops on the various underground trains – that’s just one demonstration of the level of detailed planning that went into the day. With a tight timetable of action and previous communications to station masters and other public officials to ensure that there would be no hold-ups at critical points the lead stage manager could (for example) get to a station guard to warn them of the imminent arrival of a squad, show them the necessary travel cards, and get the gate held open for them. No need for WW1 ghosts to break character by fumbling for electronic travel cards, just a silent parade through an open gate.

Just as Anna was the Lead Costumer Supervisor, there was a Lead Stage Manager with a cascade of local route masters beneath her. She was based in Birmingham and was responsible for working out how to make the timetabling and routing possible, using her home town as the test-bed for the approach, then briefing the regional organizers who applied the methods to prepare routes and handle logistics for their own locations.

End Results

To the people of London and Manchester and Belfast and Swansea and Penzance and Shetland and half a dozen places around the UK, it just happened: hundreds of ghosts of the past appeared and wandered among us. The uniforms were “real”, the journeys from place to place were “spontaneous” and “unguided”, and the ghosts were haunting. To most of us “it just happened” but behind the scenes the effort involved in preparation, and the attention to detail was enormous.

Between the “headline” names at the top of the pyramid and the highly visible troops on the ground at the bottom, it took the coordinated efforts of about 500 people to turn a wonderful idea into a reality that moved millions of people in their daily lives.

 

If you want to see more images and comments about the day you can follow the hashtag #wearehere and there is a collection of instagram images at http://becausewearehere.co.uk/  and if you’re in the London area on 11th July and want to hear more about the instigation and implementation of the day there’s an evening event on at the National Theatre on Monday 11th July featuring Jenny Waldman, Jeremy Deller and Rufus Norris discussing the event with a BBC correspondent.

 

 

Speaker Scores

I published a note this morning that I drafted in January 2015, and I didn’t notice that it had gone back in time to publish itself on the date that I first drafted it – and it’s already been tweeted twice so I can’t move it. So this is a temporary link to pop it to the head of the queue while leaving it where it first appeared.