Dr Jonathan Fox, Consultant Nephrologist, Glasgow. Chairman, SCE Nephrology Examining Board
I am very pleased to have been asked to contribute this article to the SpR Club web site as I am keen to make sure that information about the SCE in Nephrology is disseminated widely, and, in particular, to potential candidates. I hope to answer some of the common questions about the examination.
There have now been two sittings (called ‘diets’ by the Royal Colleges of Physicians) of the SCE in Nephrology, and the process of setting the exam is well established. Key facts about the examination are given in the accompanying document, (SCE Nephrology Update for SpR Club July 2010). I also recommend reading the article ‘How to Pass the Specialty Certificate Examination’ [www.rcpe.ac.uk/exams/downloads/specialist-newsletter-april-2010.pdf] by John Mucklow (Associate Medical Director for SCEs)
Development of the SCE and the exam-setting process
About 3 years ago, around 30 UK consultant nephrologists responded to an advertisement for question writers for the SCE, and subsequently underwent training to write questions in the ‘Best of Five’ format used in the MRCP(UK) and SCEs. Thereafter, to maintain membership of the Question Writing Group (QWG), every member has to write 15 questions for each two-day QWG meeting (initially twice yearly, now once a year). Every question is then discussed by groups of about 8 consultants. Many changes are made at this stage and some questions are rejected entirely. About 300 questions have emerged from each of these meetings. Then, according to a blueprint based on the ST Curriculum, I select 280 questions to be considered at the Nephrology SCE Board, which consists of 10 consultant nephrologists from a wide range of backgrounds. Again, some questions are modified or rejected. After that, about 230 questions are chosen to submit to the Standard Setting Group (7 consultants), which sets the pass mark for the examination by the modified Angoff method (for further information, see http://www.gmc-uk.org/Assessment_good_practice_v0207.pdf_31385949.pdf). Finally, 200 questions are selected for the examination. At several points in the above process the questions are also edited by technical medical editors to ensure consistency of format.
As there is only limited overlap between the various groups that consider the questions, and only two people are involved all the way through (Jeremy Levy, SCE Board Secretary, and me), each question is seen by at least 19 different consultant nephrologists. If even one of these regards any question as unacceptable, then it does not proceed through the process.
Reliability, validity, feedback, exam preparation
The Cronbach alpha coefficient, a measure of reliability or internal consistency, is >0.8 for both diets of the Nephrology SCE so far. Despite the limitations of this statistic, particularly with relatively small numbers of candidates, it does suggest that the examination has acceptable reliability.
The SCE has been criticized for including questions based on opinion rather than solely on evidence, and I agree that this is the case. Unfortunately, if we were to stick only to strong evidence, it would be impossible to test large parts of the Curriculum. All I can say is that the opinion is not that of a single question-setter, but the balance of opinion of at least 19 established UK nephrologists, from a wide range of backgrounds.
SCE questions have also been criticized for ambiguity. However, the ‘Best of Five’ method is based on choosing the best option of those given, not the only correct answer among four obviously incorrect options. Again, it is the combined judgement of all those who have taken part in the question setting process that one answer is better than the others given.
After the exam but before the results are finalized, all questions where a majority of candidates have given the wrong answer are checked to ensure that they are valid. In addition, the ‘point biserial’ is calculated for each question. This statistic correlates the performance of the question with the paper as a whole. If it shows a poor correlation, the question is checked. Finally, a ‘cross tabulation’ is provided for each question’ which shows the percentage of candidates in each of five overall performance groups (Clear Fail, Fail, Just Pass, Pass, Clear Pass), choosing each option. This gives information about the difficulty of the question and the performance of the distractors (incorrect options). If a question fails any of these checks, it can be removed from scoring.
In response to demand from candidates, 10 additional sample questions, drawn from the first diet of the Nephrology SCE, have been added to the MRCP(UK) web site (http://www.mrcpuk.org/SiteCollectionDocuments/SCENephrologySampleQs.pdf). These illustrate the type of questions that might appear but are not intended to be a revision aid. Given the resources required to produce good questions, and the fact that some questions from the bank will be used again in future diets, we can’t publish large numbers of practice questions.
The Nephrology SCE is, of course, only a small part of the overall assessment required for gaining a CCT, and does not seek to test those areas of the curriculum best suited to other forms of assessment. So, one of the first steps for a prospective SCE candidate should be to have good look at the curriculum (the 2011 SCE is based on the 2007 curriculum http://www.gmc-uk.org/Renal_medicine_3_Jul_07_v.Curr_0028.pdf_30535449.pdf, subsequent SCEs will be based on the 2010 curriculum http://www.jrcptb.org.uk/specialties/ST3-SpR/Documents/2010%20Renal%20Medicine%20Curriculum.pdf). Candidates should be aware that the knowledge required to pass the SCE will not be acquired simply from routine clinical practice and that it is expected that a significant amount of private study will be required.
Future developments
I hoped that candidates for the Nephrology SCE in 2010 would be provided with feedback about their individual performance according to curriculum topics, so that they could identify their areas of strength and weakness, but the MRCP(UK) Central Office could not get the IT modifications made in time. This is planned for 2011.
The Nephrology Standard Setting Group is to be expanded to 8 members by the inclusion of a Nephrology Specialty Registrar who has passed the SCE. This should help ensure that an appropriate standard is set for the examination
Summary
I have been very much encouraged by the commitment of nephrologists to the development and running of the Nephrology SCE: it has been a tremendous effort for a relatively small specialty. I have also been encouraged by most of the feedback from those who have taken the SCE so far. In general, it seems to have been regarded as a useful stimulus to gaining additional important knowledge, an aspect of training that may have been a bit neglected in the past. I would, of course, be very happy to respond to any additional comments or questions from prospective candidates.
