Exam factories

[David Richardson]

That's right - it's a rule that doesn't work … but it's a 'rule' that's very easy to test. One the other hand, the reality is much more difficult to construct a reliable test for (using 'reliable' in this special sense). Time and again in my experience, testing systems have gone for tests with high reliability and low (or non-existent) validity.

John Holt's 'How Children Fail' was compulsory pre-reading for my teacher training course, and there's a lot of good stuff in it still about the absurdity of many of the tests teachers set. The sad thing is that it was written about experiences in the late 1950s - and we still don't seem to have learned!

[Guest Adrian Dingle]

David

It strikes me that your "tea" question is simply a very poor question to put on an exam. Poor exam questions are those which provoke debate and uncertainty. Good exam questions are those that can be graded consistently and unambigously.

[Guest]

David

It strikes me that your "tea" question is simply a very poor question to put on an exam. Poor exam questions are those which provoke debate and uncertainty. Good exam questions are those that can be graded consistently and unambigously.

For instance, "What's the definition of a horse?"

[Guest Adrian Dingle]

Sorry Andy, you've lost me.

[David Richardson]

Adrian

Yes, it's a poor question … but the 'right' question is a very difficult one to examine, since the number of variables which have to be taken into account make it difficult to write fairly simple questions.

(Andy's 'Fact' picture looks like Chapter 2 of Hard Times [the 'empty vessels' chapter], by the way. We had it read out to us on the first day of teacher training college - mostly to make us wary of being too certain!)

I used to think that the complexity and difficulty of testing in a meaningful way was something particular to languages, since they're one of the most complex systems we're ever likely to come across. Then I started working with mathematicians and physicists at university level, and heard them express the same kinds of frustrations at the type of 'knowledge' new students were bringing with them from school.

Physicists, for example, would claim that they had to 'unlearn' most of what the students had learned at school, since the school teachers had tended to construct over-simplified maps of the physical world for testing purposes … which were misleading and just plain wrong.

The mathematicians had similar things to say about the students' fixation with answers, rather than with understanding the language that is mathematics.

You can see from my inputs into this debate that I feel that language students come to me with a similar baggage of largely useless information and misconception about how language works. However, they've been tested good and proper before they come to me, and they've been adjudged to be competent.

[Guest]

(Andy's 'Fact' picture looks like Chapter 2 of Hard Times [the 'empty vessels' chapter], by the way. We had it read out to us on the first day of teacher training college - mostly to make us wary of being too certain!)

You certainly deserve some form of "external reward" for this David - perhaps I can send you a certificate?

[Guest Andrew Moore]

I think that the league tables belong not to the UK, but to England only. Scotland never had them and the Welsh and Northern Irish are, I believe, getting rid of them (or may have done so already).

I can see the value of some objective yardsticks - say that I want to be an athlete, then my times or distances in various events will be significant.

But in the case of many academic subjects, they are arbitrary, culturally biased and often ephemeral. A student can achieve a high grade for writing in English at GCSE by doing things that will, later, earn a low grade at GCE, and would repel any editor. This may be in something simple, such as the notion that lots of qualifiers make the writing better.

Teachers of English have somehow decided that description of character is more important than recapitulation of plot. But this belief has no foundation in logic, nor in the tastes of real readers. Maybe this judgement is sound; but what is not sound is the way that it is accepted uncritically - and used as a criterion for awarding marks.

Exam boards have become aware of, and tried to mitigate, the way (maybe making up for millennia of oppression the other way) in which the English-teaching establishment is biased towards the things that girls do well - so talking about emotions is considered superior to talking about information and processes. And when it comes to literature, narratives that present subjective navel-gazing are deemed superior to those with a pacy story and action. The situation with poetry is even more bent - as a small number of writers more or less earn their keep from producing stuff that no-one would read, were it not made compulsory, feeding the exam system that in turn sustains them.

And what of our national treasure, Shakespeare? His audiences flocked to see live performances, not to read the text in a book. Yet study of the bard is now based on printed copies of the text. So the state insists that the real study of Shakespeare means reading a book, with notes and glosses. And the pupils sensibly decide that they do not like this.

[John Simkin]

Teachers of English have somehow decided that description of character is more important than recapitulation of plot. But this belief has no foundation in logic, nor in the tastes of real readers. Maybe this judgement is sound; but what is not sound is the way that it is accepted uncritically - and used as a criterion for awarding marks..

Great posting. Needs to be developed into an article and sent to Charles Clarke.

[David Richardson]

Good points. The problem with most 'mass' testing systems is that there's only certain kinds of knowledge that they're good at testing … and those might not be the kinds of knowledge we need.

A few years ago I was listening to a presentation of why the medical school at the university of Linköping introduced problem-based learning to train doctors. Conventional medical training places a heavy emphasis on fairly easily-examined courses such as Anatomy. There are only so many bones in the human body and they've all got specific, discrete names. However, conventionally, there was no easy way to test "the doctor's ability to communicate with the patient", so this was given a very low priority, both by medical schools and medical students, since educational institutions tend to teach to the exam, if they've got an exam to teach to.

Now, I'm not arguing that doctors *don't* need to know the names of bones and how they work together. But they definitely *do* need to be able to handle the 'last 30 seconds' - as the patient walks through the door at the end of the session, and blurts out what the *real* problems or symptoms are. On a purely pragmatic basis, the reason it's important for doctors to be able to do this is that otherwise they waste vast amounts of money on unnecessary treatment, and miss the real condition.

The anecdotal evidence I get from UK academics is that the rot is spreading to the university system. Where once UK universities were famous for producing a small number of graduates who'd been thoroughly trained to think, universities are now having major problems with school leavers who've been thoroughly trained to 'get the right answer', but not to think.

[John Simkin]

The anecdotal evidence I get from UK academics is that the rot is spreading to the university system. Where once UK universities were famous for producing a small number of graduates who'd been thoroughly trained to think, universities are now having major problems with school leavers who've been thoroughly trained to 'get the right answer', but not to think.

This is a crucial point. I have noticed this is happening to newly qualified teachers. They are very good at using the “official language” of teaching provided by government documents (the tick box version of teacher training). However, find it very difficult to think for themselves. I am sure this is a result of the removal of educational philosophy from the course. These courses are now much more like the course I took as an apprentice printer than the one I took as a student teacher.

[Rowena Hopkins]

I have to say that, as a Chemistry teacher, much of chemistry teaching and examining in NOT about straightforward factual recall, and this is the reason that so many students fail chemistry so horrendoursly and consider it to be the most difficult subject of all.

Example.

One chemistry practical involved heating a piece of copper metal in a bunsen burner flame. After some time the copper became coated with black copper oxide. We wrote balanced equations on the board showing that the copper reacted with oxygen in the air to produce copper oxide. We dicussed why this happened and why we used the flame.

All well and good.

In the exam the students were shown a picture of a new copper penny and an old copper penny and asked to suggest the identity of the black coating on the old penny.

70% of the students replied 'dirt'.

The examiners wanted them to say 'copper oxide'.

So, we have two choices.

1. We teach the students EVERY SINGLE possible exam question scenario to make sure that all bases are covered, so in the weeks before the exam I should at some point have specifically told them the 'fact' that the coating you find on old copper coins is copper oxide.

2. We teach them how to figure things out, relate events involving similar things, apply logic etc etc

I would opt for number 2, but its hard work and even harder work when the rest of the school (including the science deparment) are teaching individual unrelated facts to be stored in a big pile in their overflowing brains.

I think we'd all be doing each other a huge favour if we did teach skills as well as the required basic facts, it can ony help.

And I did actually think that dirt was a pretty sensible answer.... but sadly it wasn't on the list of acceptable answers.

Rowena

[David Richardson]

You probably know the anecdote about the student taking a Physics exam at Copenhagen University (I don't say it's true - I read it on the Internet!):

The exam question was "how do you measure the height of a skyscraper using a barometer". The student wrote: take a long piece of string, tie it to the barometer, then stand on the roof and lower the barometer down. When it touches the ground, measure the length of the string and the length of the barometer and you've got the height of the building.

He failed - and promptly appealed. At the viva, the examiners told him he was frivolous, but he asked for another chance. OK, they said, you can have 5 minutes to come up with a 'proper' answer. After 4 minutes of silence, the chief examiner asked him if he was going to answer. He said that the problem was he had too many answers, and he was deciding which one to choose.

For example, you could go to the caretaker and say "you can have this fine barometer, if you tell me how high the building is"; or you could drop the barometer off the top and time how long it took before it hit the ground, then work out the distance; or you could be boring and measure the air pressure at the top and the air pressure at the bottom and work it out. The problem with the third alternative, he said, is that it's boring, and I thought we were being taught to think for ourselves here.

The student was Nils Boehr, one of the only Danes to ever win a Nobel prize.

[Guest Adrian Dingle]

David

The anecdote (if true) is a nice one, but certainly isn't an argument against standardized testing and exams. Here's why. Just because Niels Bohr "failed" this test doesn't make him a "failure". People who understand what exams do and don't achieve understand that. All it means is that on that day he didn't produce an answer that the examiners wanted. No more no less. Remember, I've already acknowledged the flaws in the system;

I concede that the examination system is not perfect, but it absolutely and certainly allows an objective comparison of one kid, on one day, to another kid on the same day. Does it give the complete picture of the kids abilities? - Of course not, but it is an objective snapshot.

[Guest Adrian Dingle]

Rowena

I think that you are making the mistake of forgetting that the answer "dirt" is certainly not in the spirit of the exam, nor is it likely to be what the examiners were attempting to solicit as an answer. I can't beleive that you would think that it was!

Now, one counter-argument might be, "But what about the students who thought that it was a reasonable answer?" My answer would be, they are wrong, they would (quite correctly in my opinion) be punished by the exam for thinking so, and the as a result, the exam would have achieved its goal of making distinctions between students.

Edited May 15, 2004 by Adrian Dingle

[David Richardson]

I wonder if we could all agree on the following propositions:

1. Examinations, as such, are neither good nor bad, desirable nor undesirable - it all depends on what they're intended to achieve in an educational system.

2. Having public examinations (on the UK model) is not a necessary feature of a good educational system (the Swedish system doesn't have them, for example).

3. If you have a system of public examinations, the requirements of the exams will determine what is taught and how it is taught (i.e. the exams decide what is counted as learning - they are not mere reflections of what goes on in school independently of the exams).

4. The more exams you have in an educational system, the more difficult it is to produce and mark 'good' exams (i.e. exams which fairly test what they are supposed to be testing), since the size of the pool of exam-writing and -marking skills is finite (and most of the people who possess them are too busy teaching).

5. There are different kinds of knowledge, skills and abilities required for a 'successful' pupil.

6. Some of these are fairly easy to examine in 'mass' systems and others are almost impossible to examine without a lot of one-to-one or one-to-small group contact, which is usually too expensive and too dependent on large numbers of highly qualified and experienced examiners (who are in short supply).

7. The higher up you go in an educational system, the less important the 'easily-examined' types of knowledge are, and the more important the 'less easily-examined' types of knowledge are.

8. Being 'successful' at the higher levels requires years of practice with the right kind of exams, so if your educational system has spent its time in the early years of secondary school on the 'easily-examined' types of knowledge, the pupils will find the transition to the higher levels very difficult indeed.

Any problems with any of these? If these propositions are correct, what are the implications for educational systems?