MTAS - THE THIRD EXPERIMENT - THE BIG ONE!”
In early March this year, following the appearance on a website of the training manuel “Recruitment to Specialist Training”, The Witch Doctor wrote elsewhere:
“Consider these two documents:
DOCUMENT A
A new selection system to recruit general practice registrars: preliminary findings from a validation study. Fiona Patterson, Eamonn Ferguson, Tim Norfolk, Pat Lane, BMJ 2005;330:711-714 (26 March), doi:10.1136/bmj.330.7493.711
DOCUMENT B
“Recruitment to Specialist Training: a resource guide for selectors”
1. The Witch Doctor has noticed that in Document B, Professor Fiona Patterson is recorded as a workshop facilitator.
2. The only reference firmly quoted in Document B to back it up, is a reference to Document A ie a reference to work of same Professor Fiona Patterson, first author. Professor Patterson is Director, Organisational Psychology Programme, Department of Psychology, City University, London. Professor Patterson was also a co-founder of The Work Psychology Partnership established in 2002, the consultancy involved in writing the evanescent training manual “Recruitment to Specialist Training.” This is the consultancy charged with nationwide up-skilling of the selectors.
3. Some quotations from Document A
“Assessment centres are one of the most reliable and valid selection methods but have not been applied in the medical domain”
“Discussion
A competency based selection system for recruiting general practice registrars on the basis of an application form, referee’s report, and an assessment centre improved the reliability and validity of selection compared with traditional methods. The significant positive correlations between the competencies assessed through application, reference, and the assessment centre showed good evidence of reliability.
The validity of the assessment centre was shown by a significant positive association between performance at the centre and job performance three months into practice, and by recruited trainees being rated more highly by trainers than a matched group in another region. The reliability and validity (predictive and case comparison) findings are encouraging but preliminary, given the small sample size. Furthermore, as the comparison and matched group were demographically not equivalent to the initial intervention sample (354 applicants), caution is needed in generalisability of the results. Our findings do, however, indicate that these evidence based methods are worthy of further investigation.
The assessment centre method is also worth pursuing in other medical specialties and undergraduate selection, where the design is based on a thorough role analysis.10 Such analysis is essential if good reliability and validity is to be achieved and for a legally defensible system.19 The current assessment centre system for general practice registrars operates on an assessor to candidate ratio of 1:3 whereas traditional methods have a ratio of 1:1.5. This is an obvious saving in resources and time.”
“Ethical approval: Research ethics committee of Sheffield University.”
There has been another study in 2006 involving the selection of doctors for post graduate training in paediatrics using a competency based centre. Professor Fiona Patterson was also involved in this study which assessed 27 doctors.
“We present data which were drawn from only one area of the UK and our findings need to be validated on a wider sample.”
ie two small experiments.
Two small experiments by the same group of researchers. This does not constitute an evidence base for selection within the medical domain. They say so themselves.
There is no therefore no validation of the method. It has been rolled out prematurely as an experiment at great expense and consternation.
So, there you have it - you are an experiment. You are the further investigation. You are the wider sample. The first experiment was an interesting preamble. No more. No less. The first experiment involved small numbers and the outcome was measured after only three months! The second experiment was even smaller.
You are the real big experiment!!! You are all taking part in a psychological research study. You will eventually be published in the scientific literature. Be honoured!!
I hope the nature of this exciting experiment was explained to each and every one of you. I hope each and every one of you gave your written consent before participating.
The first preamble experiment obtained ethical approval from Sheffield University.
The second smaller experiment obtained ethical approval from The City University Psychology Department, London.
Who gave ethical approval for this third “real” experiment? Does each and every Up-Skilled Selector know this is an experiment? Did they all check that there was ethical approval before becoming Selectors? I hope so.
Or did the wobbling Up-Skilled Selectors just obey?
You need to be aware of all of this.”
The Witch Doctor | 05.03.07 - 12:18 am |
Yesterday the Ferret Fancier drew attention to the experimental nature of MTAS. He/she has explored it further. Rightly so. This needs to be brought out into the open. Never should this happen again.
This is one of the main reasons why The Witch Doctor became so outraged with the Department of Health, the Great and the Good of the medical profession and the “Up-Skilled Selectors” (consultants up and down the country who allowed themselves to be “trained” in order to participate in this flawed unproven selection process.) The Witch Doctor was astounded that these senior medics could have allowed themselves to be used in this way and apparently remain oblivious to the effects it may have, not only on the next generation of doctors, but also to the future of the NHS.
No, it should never happen again.
Senior medics need to be more vigilant, more questioning.
And when appropriate, say no.
Loudly.
The Witch Doctor - Link to a random page
_______________________________________________________________
