Oracle Scratchpad

January 18, 2015

Speaker Scores

Filed under: Non-technical — Jonathan Lewis @ 11:47 am BST Jan 18,2015

This is a note I drafted early in 2015 but, apparently, failed to publish. I rediscovered it this morning while searching for something else that I needed for an abstract I was submitting, so I thought I’d post it to see what people thought.  (For reasons I cannot explain the article has retroactively published itself on the date that I first drafted it even though I published it on 10th May 2016.)

There was a brief conversation on the Oak Table List when the UKOUG Tech 14 scores came out about the fact that the UKOUG used a range of 1 – 6 for rating speakers when the rest of the world used 1 – 5. Personally I think 1 – 5 is a bad idea and, as a speaker and an organiser, I think 1 – 6 is better and what I’d really like is 1 – 10. (In fact it’s possible that the UKOUG uses 1 – 6 as a consequence of a remark I made a few years ago when I was on the board of the UKOUG.)

Here’s how I’d argue my case. If you have 1 – 5 then, when supplying a rating, your choices are (nominally):

  1. Awful
  2. Bad
  3. I stayed to the end but don’t have any particular opinion
  4. Good
  5. Fantastic

The range of possibilities is too low: where’s the rating for “a couple of nice points, but not particularly good”, or “I wish I’d gone to something else, but it’s not so bad I want to walk out”. With a range of 1 – 6 you’re denied the possibility of an uninformative default of 3; and have a range of three ranks of “good” and three of “bad”.

Note, also, that I said “nominally” when listing the choices – unfortunately it’s a well-known effect that the extreme ends of a range of ratings tend to be automatically excluded, and the value viewed as meaning “average” is some way above the mid-point of the range (another hand-waving reason there for not having a mid-point value). If you set a range from 1 – 10, the actual values will tend to fall in the range 4 – 9, which gives you 6 values where you can get a proper feeling for how well the presentation was received, and if you get a 10 you know it went down well (which isn’t necessarily the same as it being a good presentation, of course) and if you get less than 4 you know you’ve got something to worry about.

Footnote:

In my dim and distant past I was a school teacher (teaching maths and computer science to the 11 – 18 age group) and the rating system for reports where I taught was two-part: A-E for effort, 1-5 for results. Nominally C-3 meant that your effort and results were typical of your year group. Realistically any pupil who got a C-3 was hugely offended and the average rating given was more like B-2.

(Inevitably there was at least one pupil I taught who spent his time trying to get his report card filled with E-1 for every subject.)

 

Your comments are welcome – and if there are enough then they may be of some assistance to the people who organise and speak at the conferences you attend. In fact, with that thought in mind I’ll even put up a poll on the three ranges I’ve mentioned in this article.

4 Comments »

  1. I realise it provides even less information, by I quite like the “sad, neutral, happy face” or traffic light ratings. It’s the written comments I really take notice of, not the scores.

    Cheers

    Tim…

    Comment by oraclebase — May 10, 2016 @ 2:41 pm BST May 10,2016 | Reply

    • Tim,

      I agree that comments do a much better job of helping the speaker hone their material or presentation skills.

      From a simple scoring perspective reducing the range is an interesting one – I should have thought of it. The psychology probably works as well as the psychology of increasing the range in terms of making the result less ambiguous – there’s also the benefit that at three entries a summary report that shows the counts for the three icons shouldn’t be too fussy for easy comprehension.

      Comment by Jonathan Lewis — May 10, 2016 @ 3:26 pm BST May 10,2016 | Reply

  2. And then there’s us Germans who instinctively use scorings up to 6 in the reverse order – 1 being the best score you can get at school and 6 the worst. Of course only until you aproach Abitur (the last two years at what we call Gymnasium) where you strive for up to 15 points. So without written explanation regarding the meaning of the values I’d even got your first two options wrong.
    But then I wouldn’t like having to decide if a presentation was a 6 or a 7 especially when there are usually multiple properties to score – content, presentation, appearence and whatever else the people preparing the questions consider important.
    So 1-5 it is.

    Comment by Holger — May 11, 2016 @ 7:16 am BST May 11,2016 | Reply

    • Holger,

      Now I’m wondering whether I’ve accidentally been marking down all the excellent presentation I’ve attended over the last couple of years at DOAG !

      Comment by Jonathan Lewis — May 11, 2016 @ 6:45 pm BST May 11,2016 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.