Background The review of human participant research by Research Ethics Committees or Institutional Review Boards is a complex multi-faceted process that cannot be reduced to an algorithm. However, this does not give RECs/ IRBs permission to be inconsistent in their specific requirements to researchers or in their final opinions. In England the Health Research Authority coordinates 67 committees, and has adopted a consistency improvement plan including a process called “Shared Ethical Debate” where multiple committees review the same project. Committee reviews (...) are compared for consistency by analysing the resulting minutes. Methods We present a description of the ShED process. We report an analysis of minutes created by research ethics committees participating in two ShED exercises, and compare them to minutes produced in a published “mystery shopper” exercise. We propose a consistency score by defining top themes for each exercise, and calculating the ratio between top themes and total themes identified by each committee for each ShED exercise. Results Our analysis highlights qualitative differences between the ShED 19, ShED 20 and “mystery shopper” exercises. The quantitative measure of consistency showed only one committee across the three exercises with more than half its total themes as top themes. The average consistency scores for the three exercises were 0.23, 0.35 and 0.32. There is a statistically significant difference between the ShED 19 exercise, and the ShED 20 and mystery shopper exercises. Conclusions ShED exercises are effective in identifying inconsistency between ethics committees and we describe a scoring method that could be used to quantify this. However, whilst a level of inconsistency is probably inevitable in research ethics committee reviews, studies must move beyond the ShED methodology to understand why inconsistency occurs, and what an acceptable level of inconsistency might be. (shrink)

The review of human participant research by Research Ethics Committees or Institutional Review Boards is a complex multi-faceted process that cannot be reduced to an algorithm. However, this does not give RECs/ IRBs permission to be inconsistent in their specific requirements to researchers or in their final opinions. In England the Health Research Authority coordinates 67 committees, and has adopted a consistency improvement plan including a process called “Shared Ethical Debate” where multiple committees review the same project. Committee reviews are (...) compared for consistency by analysing the resulting minutes. We present a description of the ShED process. We report an analysis of minutes created by research ethics committees participating in two ShED exercises, and compare them to minutes produced in a published “mystery shopper” exercise. We propose a consistency score by defining top themes for each exercise, and calculating the ratio between top themes and total themes identified by each committee for each ShED exercise. Our analysis highlights qualitative differences between the ShED 19, ShED 20 and “mystery shopper” exercises. The quantitative measure of consistency showed only one committee across the three exercises with more than half its total themes as top themes. The average consistency scores for the three exercises were 0.23, 0.35 and 0.32. There is a statistically significant difference between the ShED 19 exercise, and the ShED 20 and mystery shopper exercises. ShED exercises are effective in identifying inconsistency between ethics committees and we describe a scoring method that could be used to quantify this. However, whilst a level of inconsistency is probably inevitable in research ethics committee reviews, studies must move beyond the ShED methodology to understand why inconsistency occurs, and what an acceptable level of inconsistency might be. (shrink)

We have followed with interest the commentaries arising from Moore and Donnellys1 argument that authorities in charge of research ethics committees should focus primarily on establishing code-consistent reviews.1 We broadly agree with Savulescu’s2 argument that ethics committees should become more expert, but in a different way and for a different reason. We have recently been working with the UK Health Research Authority analysing the outcomes of their ‘Shared Ethical Debate’ exercises.3 Each ShED exercise involves the circulation of a single research (...) project to a number of RECs. The resulting minutes from each REC are compared along with the final decisions made by each REC on the project under consideration. This process was originally an administrative exercise designed to promote greater consistency among the 60 or so ethics committees that were brought under the HRA ) umbrella within the UK National Health Service. Over 20 ShEDs have been run, and although the process has significant weaknesses and has not always been run consistently, the results of the exercises are fascinating. Broadly speaking, NHS RECs have been getting more consistent over time in terms of their decision-making, but the reasons for the final decisions as described in the committee minutes continue to vary widely. Qualitative research now needs to be done to understand why different committees can have such different discussions in relation to exactly the same research project and yet come to essentially the same conclusion. As part of our analysis of the ShED data, we looked at …. (shrink)