Assessing excellence in translational cancer research: a consensus based framework
Journal of Translational Medicine volume 11, Article number: 274 (2013)
It takes several years on average to translate basic research findings into clinical research and eventually deliver patient benefits. An expert-based excellence assessment can help improve this process by: identifying high performing Comprehensive Cancer Centres; best practices in translational cancer research; improving the quality and efficiency of the translational cancer research process. This can help build networks of excellent Centres by aiding focused partnerships. In this paper we report on a consensus building exercise that was undertaken to construct an excellence assessment framework for translational cancer research in Europe.
We used mixed methods to reach consensus: a systematic review of existing translational research models critically appraised for suitability in performance assessment of Cancer Centres; a survey among European stakeholders (researchers, clinicians, patient representatives and managers) to score a list of potential excellence criteria, a focus group with selected representatives of survey participants to review and rescore the excellence criteria; an expert group meeting to refine the list; an open validation round with stakeholders and a critical review of the emerging framework by an independent body: a committee formed by the European Academy of Cancer Sciences.
The resulting excellence assessment framework has 18 criteria categorized in 6 themes. Each criterion has a number of questions/sub-criteria. Stakeholders favoured using qualitative excellence criteria to evaluate the translational research “process” rather than quantitative criteria or judging only the outputs. Examples of criteria include checking if the Centre has mechanisms that can be rated as excellent for: involvement of basic researchers and clinicians in translational research (quality of supervision and incentives provided to clinicians to do a PhD in translational research) and well designed clinical trials based on ground-breaking concepts (innovative patient stratification, substantial fraction of phase I/II trials, investigator-initiated trials). Critically, the framework supports reduced bureaucracy by building on existing European evaluation systems.
The excellence framework is the product of an intense stakeholder consensus building exercise. It will be piloted during an expert peer review/site visit of at least three European Comprehensive Cancer Centres. The findings regarding content, governance and implementation can have relevance for other clinical and research fields.
Translational research can be defined as a complex process of transforming scientific discoveries, arising from laboratory, early clinical, or population studies, into clinical applications to reduce incidence, morbidity, and mortality . On average, it takes over a decade to deliver patient benefits [2, 3]. After the 2007/09 financial crisis healthcare providers find it harder to justify funding for translational research in cancer but also other fields. The Stockholm Declaration recognises that creating a strong case for funding translational research in Europe, needs proof of excellent performance by Cancer Centres that are engaged in it . Assessing excellence can stimulate continuous improvement in the way an organization perceives, plans, and performs translational research for the benefit of patients. It can also help to develop a network of excellent Centres that can focus their collaboration and share best practices through regular benchmarking.
“In the last decade, we have begun discussing the idea of translational research in every field of medicine without ever clarifying exactly what this entails and how it should be assessed.” (Researcher)
To date, no assessment framework has specifically focused on excellence along the entire continuum of translational research. Previous frameworks have focused on criteria/questions mainly for the self-guided assessment of the success of translational research organizations and/or projects [5, 6]. But there are a number of limitations to those frameworks: (i) not every success may necessarily be a sign of excellence; (ii) self-assessment is not sufficient to benchmark the performance of different Centres. Although success in translational research may be assessed by the organization itself, for the sake of credibility, excellence assessment requires an expert judgement (preferably at an international level) that is completely independent to the organization. Currently, no formal framework exists that supports peer-reviewers to judge excellence in translational research; (iii) previous frameworks were informed by a few experts but did not engage key stakeholders in setting criteria. Achieving excellence in translational research relies on people from different disciplines and functions working together to improve overall performance. Consensus building helps achieve common understanding, commitment and collaboration and is recommended for criteria development .
This study produced an excellence assessment framework that was developed using a consensus building exercise with key stakeholder groups and experts from the European cancer community. The framework will be used to identify and assess excellent translational research in a number of European Comprehensive Cancer Centres (CCC’s) (combining basic, translational and clinical research and patient care activities). The framework will be thoroughly piloted with a number of CCC’s in 2013-2014.
Consensus building took 18 months using several methods to fully engage stakeholder groups (See Figure 1). These included clinicians, researchers, senior managers and patient representatives, representing around 70 European organizations including: CCC’s, Cancer Research Centres, Clinical Cancer Centres, Cancer Units, Patient Organizations and Cancer networks. Recruiting a small group of seven acknowledged experts to provide informed review and reflection at key points complemented this process. Development of the excellence assessment framework can be summarized as follows:
Literature reviews – The European accreditation standards for CCC’s from the Organization of European Cancer Institutes (OECI) and a report from the National Cancer Institute (USA) Translational Research Working Group (TRWG) on improving translational research performance  were taken as a starting point. A systematic literature review followed to identify and critically appraise translational research models most suitable for performance assessment of CCC’s . The result was a list of 59 excellence criteria covering inputs, processes and outputs of basic, translational and clinical research & clinical care.
Stakeholder survey and focus group discussion – This initial list of 59 criteria was scored by stakeholders (N = 78) in an online survey. Evaluation of criteria by participants identified criteria as critical (if selected by more than 60% of the participants as important), optional (if selected by 40-60% of the participants as important) or not relevant (if selected by less than 40% of the participants as important). 12 of the 59 criteria scored critical (e.g. early stage clinical trials; effective transfer of innovations from basic research to clinical practice); 36 optional (e.g. improved Quality of Life from innovations implemented; innovative prevention services) and 11 not relevant (e.g. number of surgical/paediatric/radiotherapeutical subspecialities; number of radiotherapy units). Quantitative criteria generally scored lower than qualitative criteria. Next, a focus group was assembled with a representative sample of survey participants (N = 30). Paricipants clarified that the criteria that scored critical and optional should be considered when developing excellence criteria and the criteria that scored not relevant should be discarded with the exception of Health Technology Assessment.
Expert Group meeting – The updated list was sent to an expert group (2 basic, 2 translational, 2 clinical experts and 1 senior management expert). Their selection reflected more than 30 years of experience in basic, translational and/or clinical cancer research, prestigious awards and memberships in Oncology, significant current roles in European Comprehensive Cancer Centres and willingness to contribute to excellence framework development. The experts suggested a comparative review of the adapted list against recent external peer-reviewed evaluation reports from two CCC’s (see acknowledgements) to check if the criteria were reflected in these reports and to what extent additional criteria can be identified. The revised list was sent back to the experts who scrutinized each criterion, added specific points, and placed them under 10 categories that shaped an excellence assessment framework. The expert group stressed the importance of using expert peer-review when assessing the quality of translational research in combination with the assessment framework being developed.
Final validation by stakeholders – The revised version of the assessment framework was sent to the same stakeholders who had previously participated in the consensus building exercise. The participants (N = 34) made suggestions to improve criteria clarity in terms of being to the point, having short sentences, avoiding connectors such as “and”/”or” to make them less risky for misinterpretation. Ultimately, stakeholder feedback helped filter and refine the list down to 20 excellence criteria to assess excellence in translational research in CCC’s during a peer-review process and 6 additional criteria to be considered according to the preference of the CCC’s.
Critical review of excellence criteria by an external committee within an independent body, the European Academy of Cancer Sciences (EACS) - The excellence criteria that evolved from the stakeholder consensus building were critically reviewed by a committee (see endnote for the compositiona) that has been formed in the EACS to give external input and govern the excellence assessment. The committee suggested minor re-structuring to the excellence criteria. They reduced the criteria to 18 core excellence criteria, placed them in 6 themes and merged all additional criteria with the core criteria (see Table 1).
The consensus building exercise clearly identifed a need to assess excellence in translational research based on qualitative rather than quantitative criteria. Stakeholders and experts felt that whilst for instance state-of-the-art infrastructure is important to perform excellent translational research, the assessment of excellence itself should focus more on how efficiently they are being used by the organization and the quality of their outputs.
“The numbers of services/units or treatments required to be assessed as “Excellent” appear to me to be somewhat arbitrary. Quality rather than quantity should prevail.” (Researcher)
There was unanimous support for the need to minimize the bureaucracy of the excellence assessment and to have an external expert-based governance system that is independent to the organization being assessed in order to maximize transparency. The experts suggested that reports regularly prepared by the CCC’s for existing national and/or European level assessment programmes should be first assessed against the excellence criteria (Table 1). These reports contain sufficient qualitative and especially quantitative data and only data missing in such reports needs collecting. A European excellence assessment framework can only be established if the procedure exceeds the current national accreditation efforts within the European Union (EU) member states and carries sufficient credibility.
“CCC’s in the European Union member states already go through different expert based peer-reviewed national evaluations, how are we going to justify the need for and reduce the bureaucracy of a European excellence assessment?” (Experts)
It was emphasised that assessing the translational research “process” along the entire continuum from basic research upto clinical practice and back is as critical as assessing inputs and especially outputs. However, it was decided that the assessment does not have to encompass population based outcomes because that stage goes beyond the scope of most European CCC’s and is the responsibility of different authorities within the EU member states. Furthermore, it has a very long lagtime and is influenced by several non-institutional factors which makes it difficult to access data on time. A focus on organizational level assessment was preferred. However, for certain issues, such as those related to prevention and early detection, translational research should extend its scope beyond current organizational boundaries to optimize its relevance on population level. Stakeholders can be made aware of this through relevant contacts with Europe-wide organizations such as the European Public Health Alliance (EPHA)  who act as gatekeepers to the national and regional authorities across the EU-28 member states.
“It is important to establish an excellence assessment, which helps assess the highest levels of translational research quality through the innovation, productivity and efficiency of an organization.” (Manager)
Another example of careful qualitative assessment instead of quantitative data collection is the use of the scientific infrastructure that CCC’s offer to other institutions. Such criteria cannot be imposed on all CCC’s as the nature of demand depends on specific regional/national contexts.
“Of course, an excellent Centre should be willing to make these facilities available if asked for on a collaborative basis. But at my Centre, while we make some of these facilities available across the rest of city, we have not in the past five years had any requests to make these facilities available to other Centres in the nation because these Centres have their own access to these facilities.” (Manager)
Finally, excellence criteria on patient-related aspects were perceived to be specifically important.
“We must not only look at efficacy and safety in clinical study setting but even more to the value in the everyday practice of products/technologies/therapies.” (Patient representative)
By undertaking a consensus-building exercise to develop an excellence assessment framework it was clear that while consensus can exist at a general level, some disagreement is unavoidable due to the different backgrounds, experiences and interests of the stakeholder groups. For instance, clinicians and patient representatives felt that excellence assessment does not need to involve basic research and should focus more on clinical care, but basic researchers and managers felt that unless basic research is included, the entire continuum of translational research cannot be fully assessed. Furthermore, the question whether the right stakeholders and experts are involved was carefully addressed at each stage of the criteria development process. For each excellence criterion, we considered having consensus when 60% or more of the stakeholders that participated agreed and we did not consider single votes by an individual stakeholder as sufficient. Inadvertently, some perspectives may have been left out that might still show to be critical. Excellence assessment requires a degree of flexibility, which is possible to implement in a transparent manner by using an independent peer review panel. While a common format is desirable, rigid formats may be unsuitable for organizations operating in different health systems and can introduce significant bureaucracy.
Qualitative excellence criteria increase the challenge for objective rating. A flexible and meaningful rating system is therefore needed. For the final decision, strengths and weaknesses across all criteria as well as individual criterion should be considered with an agreed minimum score on each criterion. However, to what extent should there be flexibility in accommodating the limitations imposed on individual CCC’s because they have to operate in different European health systems? Regarding health care, the EU operates on a subsidiarity principle. It means that all clinical and some research fields operate within nationally set frameworks. However, transnational cooperation is valued to share knowledge and improve performance and this brings some specific rules into play. Most EU member states have competitive national assessments but none would be easily accepted for use in another member state. Instead, considering best practices from across all member states in identifying and assessing excellence in translational research is far more transparent and can give wider acceptance of such assessments across the EU. Essentially, it is about an individual Centre making its own case of why it deserves to be designated as excellent given its own operating enviornment. And then the peer-reviewers can check whether the case made by the Centre is valid during a site visit. This will be done with the help of the excellence criteria (Table 1) to some extent but experts should also be prepared to come across areas of excellence that the Centre may have forgotten to mention during its application for excellence designation. A related issue raised by some experts was about discarding or “killing” insufficiently promising translational research projects, because of insufficient innovation, a low chance of clinical implementation or probably very unfavorable cost effectiveness. Of course identifying those is a risky matter, as it is often very difficult to predict the actual clinical potential in early stages of research. Nevertheless and increasingly, early stage Health Technology Assessment techniques are being developed and applied to aid decision makers to decide about further research investments or researchers to set the specifics/demands that the research and development process should meet . One could consider looking into the availability of mechanisms to assist early stage decision-making on adequate translational research progress. It seems advisable to develop specific knowledge and development of a norm or reference material on this topic.
“Research is also a matter of intuition, intellectual flexibility and aptitude to identify opportunities. Excellence assessment must go beyond putting a tick against criteria and needs to consider the local context in which institutions operate.” (Manager)
For example, for high impact of publications it was hard to establish a minimum level because it varies greatly between the different disciplines within Oncology and among CCC’s across the EU member states. However, the average scores for each discipline can still be considered to make this criterion inclusive. Further, a range of bibliometric index other than just impact factors should be considerd i.e. citation factor, cumulative impact factor and the quality and impact of individual publications to be rigorously judged by an expert peer review team.
“One publication that shows the 100% cure of a cancer is enough. Ten publications of one-month prolongation of survival mean nothing.” (Clinician)
In Europe, the current excellence assessment primarily intends to evaluate team science due to the multifaceted nature of translational research where collaboration of different disciplines is critical to its success. The criteria that have emerged through the consensus building exercise, support this statement for example by focusing on multidisciplinary team collaboration, communication and joint publication efforts, participation of different department staff in various research projects and the outputs. A specific product of work may involve biologists, medical chemists, pharmacologists, imaging physicists and clinicians. Thus it can be difficult to identify the exact contributions made by each single member of the team and this could raise issues when individuals are evaluated for tenure or promotions etc. However, the consensus building exercise revealed that also monitoring individual efforts to some extent is needed for excellence. This could help promote a competitive attitude among researchers within and outside a Centre and help identify and reward excellent contributions of specific researchers to science that might otherwise go undetected. The individual efforts will be evaluated taking into account a range of factors: the quality of scientific outputs by clinicians pursuing a PhD in translational research; the quality of the research programs of PI’s and if they are regularly evaluated; prestigious awards, discoveries and memberships attributed to specific individuals from the Centre; investigator-initiated trials and their success rate etc. Currently, national assessment programs within some EU member states conduct these evaluations but there is no formally agreed assessment on European level. Finally, this consensus building exercise revealed a need to get some obvious basics right that might otherwise be ignored. For example, to ensure that the criteria are both meaningful and easily understood by organizations, words such as “high”, “well”, “minimum”, “significant”, “cutting-edge”, “state-of-the-art”, “substantial” etc. unless carefully explained can be easily misinterpreted by stakeholders. Stakeholders accepted to start developing an excellence framework with these words/definitions but suggested refining them based on pilots.
Piloting of the excellence framework
A committee consisting of internationally respected and renowned experts in Oncology will govern the excellence assessment process. In the EU, members of the European Academy of Cancer Sciences satisfy such requirements, and international experts (from the National Cancer Institute, USA and Accreditation Canada) will be invited to join the committee and the official peer review team. A Centre that applies to be assessed as excellent should provide documentation to the committee. This will include recent external peer reviewed evaluation reports in English that the Centre has produced for national and/or international evaluations in the past 3 years, covering basic, translational and clinical areas with specific achievements in translating innovations from bench to bedside and/or vice versa. Further documentation may be requested if the initial material is found insufficient. After an initial screening of the documents against excellence criteria (Table 1) the committee will decide if the Centre qualifies for site visit/peer review in which again the excellence criteria will be used to evaluate Centres. A minimum of three European CCC’s will pilot the excellence framework.
Assessing excellence requires a mix of quantitative and qualitative criteria retrievable through different data sources. But we need to recognise that the consensus building exercise showed strong support for qualitative criteria. This is because it will build on existing evaluation systems across the EU and other international systems (e.g. US, Canada) that already provide the necessary breadth of quantitative data. The assessment framework that we have developed will need to be thoroughly tested with European CCC’s to prove that it can help identify excellence in translational research. Although, the framework was primarily developed for Oncology, it can probably be translated to other research and/or clinicial fields after rigorous validation. Allocating governance to an external entity that has credibility and is independent of the organizations being assessed is a key ingredient. Finally, the success of the assessments will depend on minimized bureaucracy and maximized transparency and accountability during the evaluation process.
aThe composition of the committee formed in the European Academy of Cancer Sciences is: Prof. Dr. Anton Berns (Senior Group Leader Molecular Genetics, Netherlands Cancer Institute), Prof. Dr. David M. Livingston (Chairman, Executive Committee for Research, Dana-Farber Cancer Institute and Emil Frei Professor of Genetics and Medicine, Harvard Medical School), Prof. Dr. Daniel Louvard (Director of Research Centre at Institut Curie France), and Prof. Sir Bruce Ponder (Head of Department, Oncology at University of Cambridge, UK).
AR is a PhD Candidate at the Netherlands Cancer Institute, Amsterdam/University of Twente on a European project Eurocan Platform that is funded by the European Commission. Her thesis focuses on identifying and assessing excellent performance in translational research at the Comprehensive Cancer Centres in the EU. Prior to this, she holds a Masters degree in European Regional Health systems including expertise in health policy, patient advocacy, social science and Health Technology Assessment. CC is a Professor at the Cambridge Research Institute, UK who leads his own research group on Breast Cancer Functional Genomics using genomics tools such as sequencing, molecular cytogenetics, array-CGH, mRNA/miRNA profiling etc. MS is a Medical Oncologist at Breast Cancer Unit of the Institut Gustave Roussy, Villejuif, France where she is also responsible for European and international affairs. She is the chairwoman of the Accreditation and Designation programme of the OECI. HvL is a post-doctoral researcher in basic cancer research. He is currently also the manager of research operations at the Netherlands Cancer Institute, Amsterdam. WvH is a member of the board of Directors of the Netherlands Cancer Institute, Amsterdam. He is an Associate Professor at the Department of Health Technology and Services Research at the University of Twente, The Netherlands. He is also the president of the OECI.
Translational Research Working Group: National Cancer Institute. Definition of Translational Research (Adapted). http://www.cancer.gov/researchandfunding/trwg/TRWG-definition-and-TR-continuum,
Westfall JM, Mold J, Fagnan L: Practice-based research—“blue highways” on the NIH roadmap. JAMA. 2007, 297: 403-406. 10.1001/jama.297.4.403.
Balas EA, Boren SA: Yearbook of medical informatics: managing clinical knowledge for health care improvement. 2000, Stuttgart, Germany: (Schattauer Verlagsgesellschaft mbH
Ringborg U: The Stockholm declaration. Mol Oncol. 2008, 2: 10-11. 10.1016/j.molonc.2008.03.004.
Pozen R, Kline H: Defining success for translational research organizations. Sci Transl Med. 2011, 3 (94): 1-5.
Coller BS, Califf RM: Traversing the valley of death: a guide to assessing prospects for translational success. Sci Transl Med. 2009, 1 (10): 10-
International Organization for Standardization (ISO): Guidance for ISO National Standards Bodies- Engaging stakeholders and building consensus. 2010, 1, chemin de la Voie-Cruese, Case postale 56, CH-1211, Geneva 20, Switzerland: International Organization for Standardization (ISO), ISBN 978-92-67-10540-6 2010
National Cancer Institute at the National Institutes of Health: Translational Research Working Group of the National Cancer Advisory Board, US Department of Health and Human Services, National Institutes of Health. Transforming translation-Harnessing discovery for patient and public benefit. 2007, The USA: U.S. Department of Health and Human Services
Rajan A, Sullivan R, Bakker S, Harten WV: Critical appraisal of translational research models for suitability in performance assessment of cancer centers. Oncologist. 2012, 17: e48-57. 10.1634/theoncologist.2012-0216.
European Public Health Alliance. http://www.epha.org,
Ret’el VP, Bueno-de-Mesquita JM, Hummel MJM, van de Vijver MJ, Douma KFL, Kim K, van Dam FSAM, van Krimpen C, Bellot FE, Roumen RMH, Linn SC, van Harten WH: Constructive Technology Assessment (CTA) as a tool in Coverage with Evidence Development: The case of the 70-gene prognosis signature for breast cancer diagnostics. Int J Technol Assess Health Care. 2009, 25 (1): 73-83. 10.1017/S0266462309090102.
This research was funded by the European Commission FP7 programme as part of the Eurocan Platform project. Contributions by the following are valued: (i) all stakeholders and experts from the Eurocan Platform, the OECI and the (ECPC) (ii) the Netherlands Cancer Institute and the Oslo University Hospital, Norway for sharing external peer-reviewed national evaluation reports by The Dutch Ministry of Health, Welfare and Sport-Dutch Cancer Society (2004-2008) and the Research Council of Norway, Norwegian Ministry of Education & Research (2010-2011), respectively (iii) the OECI Accreditation Standards.
The authors declare that they have no competing interests.
AR and WvH conceptualized and designed the consensus building exercise. AR collected and assembled the data including inviting staleholders and experts to contribute to the development of the excellence framework. AR, HvL, CC, MS and WvH all gave input to analysing and interpreting the data. AR and WvH wrote the draft manuscript which was significantly improved due to the suggestions and comments made by all other authors HvL, CC and MS. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Rajan, A., Caldas, C., Luenen, H.v. et al. Assessing excellence in translational cancer research: a consensus based framework. J Transl Med 11, 274 (2013). https://doi.org/10.1186/1479-5876-11-274