Research Quality Plus
Research Quality Plus (RQ+) is an approach to evaluating research quality that moves beyond traditional measures of research excellence, such as scholarly peer review and bibliometrics. Originally developed to bring a more appropriate, holistic approach to the assessment of research for development, it aims to avoid some of the pitfalls of research impact evaluation by focusing on how research is positioned to achieve its aims, rather than on downstream impacts that are beyond the researchers’ sphere of control. While elements of RQ+ can be adapted for different settings and purposes, the core approach is guided by three tenets: (i) quality is multidimensional; (ii) context matters; and (iii) judgment should be based on systematic review of empirical evidence.
RQ+ was named a top ten resource for responsible research assessment by the Global Research Council in 2020,[1] and has been endorsed as good practice by the San Francisco Declaration on Research Assessment.[2]
Origins and rationale
The RQ+ approach emerged from testing and reflection by the International Development Research Centre (IDRC) and its research community from 2012 to 2015.[3] Developed for use in external reviews of research programming supported by IDRC,[4] the design and results of the first application of RQ+ were published in 2016 by Zenda Ofir, Tom Schwandt, Colleen Duggan and Robert McLean.[5] Following further applications and validation at IDRC, and work with other research institutions, RQ+ was revised and re-positioned by Robert McLean, Zenda Ofir, Amy Etherington, Manuel Acevedo and Osvaldo Feinstein in 2022.[6]
The RQ+ approach is informed by global debates on the need to reform methods of research evaluation.[7][8] It responds to the ten principles of the Leiden Manifesto[9] and other criticism of research evaluation methods that overly emphasize citations and other research publication metrics.[10][11] It also aims to address the issue of bias in peer review and other standard measures of research excellence, including the criticism that these measures often fail to capture the strengths and contributions of southern-led research.[12] Some critics have argued that publication-focused measures of research excellence are a barrier to the sharing and uptake of southern-led research[13][14] or perpetuate gender inequalities in research and academia.[15]
The RQ+ approach
RQ+ offers a practical evaluation approach that facilitates independent, systematic, and values-driven expert review. It is inspired by systems thinking, and draws on both quantitative and qualitative evidence.[3]
Rather than focusing on the downstream impacts of research, RQ+ considers how well research is positioned to be used in its context, acknowledging that many factors and stakeholders beyond researchers’ control determine how research is done and ultimately used. In this respect, it addresses current concerns[16][17][18] with research impact evaluation, which gauges research success in terms of its longer-term social impacts, such as improvements in health, economic efficiencies, or social justice.[6]
RQ+ is underpinned by three tenets:[19]
- Accept a multidimensional view of quality in research. Along with scientific rigour, dimensions of quality should include other values and objectives that matter to the research effort. The definitions of these multiple dimensions may differ according to the purposes and users of the approach.
- Take into account the context in which research happens. It is valuable to consider how the research process or results may be shaped by internal and external factors, including institutional, political or resource settings or the availability of data, in the specific research context.
- Base judgement on empirical evidence. As with the conduct of research, assessments of quality must be underpinned by empirical evidence and not just opinion. Using systematic rubrics, RQ+ asks evaluators to draw evidence-based conclusions, using multiple data sources. These include expert review of research outputs, but also the inputs of key stakeholders such the researchers themselves, other researchers in the field, and the intended users and beneficiaries of research.
Peer review plays an important role when applying the approach. But rather than relying on expert judgement alone, peer reviewers gather empirical data to better characterize the research context and draw out a multidimensional view of quality.
While IDRC has suggested that using these three tenets will ensure the core benefits of RQ+ are maintained in new applications, how they are applied will depend on context and should reflect the values and vision of the RQ+ user.[6]
The IDRC RQ+ framework
To apply the RQ+ approach, the co-designers under auspices of IDRC developed a tailored RQ+ framework that reflects values and objectives related to IDRC's research for development mission. Reflecting the three tenets of the approach, the framework defines specific dimensions and subdimensions of research quality, identifies contextual factors, and provides rubrics that guide evaluators in conducting a systematic assessment. Although it was developed to represent the values and mission of IDRC, the framework aims to inform a conceptual understanding of quality that bridges scientific efforts and social, economic and environmental impacts.[20] IDRC continues to refine details of the framework with each application, and other organizations may adapt it to their own uses.
The main dimensions of research quality identified are rigour, legitimacy, importance, and positioning for use.[20]
- Scientific rigour considers the technical quality, appropriateness, and rigour of the design and execution of the research. Subdimensions address structural design and sound use of methodologies.
- Research legitimacy considers the extent to which research addresses the concerns and insights of relevant stakeholders, how it responds to social and environmental contexts, and is seen as procedurally fair. It includes subdimensions such as gender, inclusiveness, engagement with local knowledge, and how research addresses negative consequences.
- Research importance considers how valuable the knowledge and understanding generated by the research is to its intended users, and how it contributes to research theory and practice. Subdimensions include originality and relevance.
- Positioning for use considers the extent to which the research process and outputs are managed to increase the likelihood that they will be used, and ultimately achieve influence and impact. Subdimensions address how users are engaged and the openness and actionability of the work.
IDRC's RQ+ framework identifies five main contextual factors to be considered in evaluation.
External influences include:
- Maturity of the research field—the extent to which relevant frameworks and hypotheses have been developed and subjected to testing, and how much previous research has been done in the given field.
- Risk in the political environment—such as electoral uncertainties, conflict, or humanitarian crises, that could affect the conduct of the research or its positioning for use.
- Risk in the data environment—the availability of essential data and the extent to which instruments and measures for data collection and analysis are widely agreed upon and available.
Internal influences include:
- Research capacity strengthening—how much the initiative devotes funding or technical support to ensure teams have the skills and knowledge essential to achieve their aims.
- Risk in the organizational environment—the extent to which the research effort is supported in its organizational context, such as by institutional priorities, incentives, and infrastructure.
To guide the conduct of RQ+ assessment, reviewers use standardized evaluative rubrics that set out judgement criteria for reviewers, clarifying how performance should be measured for each dimension and subdimension of research quality and each contextual factor. These rubrics facilitate a systematic and transparent approach that blends qualitative and quantitative evidence into a single evaluative assessment.[20]
Applications
Within IDRC, RQ+ has primarily been used in external summative evaluation of research projects and portfolios that it funds and supports. Its use has enabled meta-analysis across categories of interest to IDRC.[3] It may have potential for use by other research funders and managers or by universities, journals, or think tanks. Potential areas of application include in research priority-setting, planning and design, monitoring and strengthening implementation, learning and accountability, and communicating research impact and findings.[21]
Insights from IDRC’s use of RQ+
In 2015, RQ+ was used in an independent expert review of 170 studies from seven areas of research supported by IDRC in the previous five years. From the results of this retrospective analysis, IDRC drew a number of insights related to its development research mandate, which validate the strengths of southern-led research, such as:[3][22]
- Scientifically excellent research is useful research. Rather than finding a trade-off between scientific rigor and concern with positioning research for use, the results suggest these dimensions of quality are positively correlated.
- Capacity strengthening does not imply low scientific rigour. New techniques can emerge from people new to a field, and mentorship and training can have a positive influence on integrity and rigour.
- Local researchers — those embedded in the research context — are best placed to find innovative solutions.
- Quality begets quality. The findings showed that scientific rigour, legitimacy, importance and positioning for use were mutually reinforcing.
- Southern-led research demonstrated quality across all measures. It proved the most rigorous, important, useful and legitimate research for addressing development challenges in the Global South.
Research Quality Plus for Co-Production (RQ+ 4 Co-Pro)
Research co-production specialists have used the RQ+ approach to develop a tailored framework for evaluating the quality of research co-production. This iteration of the framework aims to assess the multiple and often hard to measure benefits of researchers working with research users to produce knowledge together. Following the three tenets of the RQ+ approach, the tailored RQ+ 4 Co-Pro framework identifies specific contextual factors and quality dimensions of importance to co-production research assessment.
External Links
References
- ↑ "The changing role of funders in responsible research assessment: progress, obstacles and the way ahead". DORA. Retrieved 2022-06-23.
- ↑ "Resource Library". DORA. Retrieved 2022-06-23.
- ↑ 3.0 3.1 3.2 3.3 McLean, Robert K D; Sen, Kunal (2019-04-01). "Making a difference in the real world? A meta-analysis of the quality of use-oriented research using the Research Quality Plus approach". Research Evaluation. 28 (2): 123–135. doi:10.1093/reseval/rvy026. ISSN 0958-2029.
- ↑ IDRC (2018). "In brief : Research Quality Plus".
- ↑ Ofir, Zenda; Schwandt, Thomas; Duggan, Colleen; McLean, Robert (2016). "RQ+ A Holistic Approach to Evaluating Research" (PDF). IDRC.
- ↑ 6.0 6.1 6.2 McLean, Robert; Ofir, Zenda; Etherington, Amy; Acevedo, Manuel; Feinstein, Osvaldo (2022). "Research Quality Plus (RQ+) : evaluating research differently". IDRC.
- ↑ Méndez, Ethel (July 2012). "What's in Good?" (PDF). IDRC.
- ↑ "Transforming Research Excellence: New Ideas from the Global South – African Minds". Retrieved 2022-06-27.
- ↑ Hicks, Diana; Wouters, Paul; Waltman, Ludo; de Rijcke, Sarah; Rafols, Ismael (April 2015). "Bibliometrics: The Leiden Manifesto for research metrics". Nature. 520 (7548): 429–431. Bibcode:2015Natur.520..429H. doi:10.1038/520429a. ISSN 1476-4687. PMID 25903611. Unknown parameter
|s2cid=ignored (help) - ↑ Wilsdon, James (2015). The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. 1 Oliver's Yard, 55 City Road, London EC1Y 1SP: SAGE Publications Ltd. doi:10.4135/9781473978782. ISBN 978-1-4739-7306-0. Search this book on
- ↑ National Research Council (US) Committee on Assessing Behavioral and Social Science Research on Aging (2007). Feller, Irwin; Stern, Paul C., eds. A Strategy for Assessing Science: Behavioral and Social Research on Aging. The National Academies Collection: Reports funded by National Institutes of Health. Washington (DC): National Academies Press (US). ISBN 978-0-309-10397-8. PMID 20669474. Search this book on
- ↑ Singh, Suneeta; Dubey, Priyanka; Rastogi, Apurva; Vail, Daniel (2013). "Excellence in the context of use-inspired research: Perspectives of the global South (Executive Summary)" (PDF). IDRC.
- ↑ Amano, Tatsuya; González-Varo, Juan P.; Sutherland, William J. (2016-12-29). "Languages Are Still a Major Barrier to Global Science". PLOS Biology. 14 (12): e2000933. doi:10.1371/journal.pbio.2000933. ISSN 1545-7885. PMC 5199034. PMID 28033326.
- ↑ Neylon, Cameron (2020-01-17), "Research excellence is a neo-colonial agenda (and what might be done about it)", Transforming Research Excellence: New Ideas from the Global South, Cape Town: African Minds, doi:10.5281/zenodo.3603954, ISBN 978-1-928502-06-7, retrieved 2022-06-27
- ↑ Kraemer-Mbula, Erika (2020-01-17), "Gender diversity and the transformation of research excellence", Transforming Research Excellence: New Ideas from the Global South, Cape Town: African Minds, doi:10.5281/zenodo.3603960, ISBN 978-1-928502-06-7, retrieved 2022-06-27 Unknown parameter
|s2cid=ignored (help) - ↑ FIELDING, MICHAEL (2003-07-01). "The Impact of Impact". Cambridge Journal of Education. 33 (2): 289–295. doi:10.1080/03057640302044. ISSN 0305-764X. Unknown parameter
|s2cid=ignored (help) - ↑ "Ten Reasons Not to Measure Impact—and What to Do Instead (SSIR)". ssir.org. Retrieved 2022-06-27.
- ↑ Russell, Jill; Fudge, Nina; Greenhalgh, Trish (2020-10-27). "The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it?". Research Involvement and Engagement. 6 (1): 63. doi:10.1186/s40900-020-00239-w. ISSN 2056-7529. PMC 7592364 Check
|pmc=value (help). PMID 33133636 Check|pmid=value (help). - ↑ Lebel, Jean; McLean, Robert (July 2018). "A better measure of research from the global south". Nature. 559 (7712): 23–26. Bibcode:2018Natur.559...23L. doi:10.1038/d41586-018-05581-4. PMID 29973734. Unknown parameter
|s2cid=ignored (help) - ↑ 20.0 20.1 20.2 "Research Quality Plus (RQ+) Assessment Instrument" (PDF). IDRC. January 2022.
- ↑ McLean, Robert; Ofir, Zenda; Etherington, Amy; Acevedo, Manuel; Feinstein, Osvaldo (2022). "Five illustrative uses of the RQ+ approach".
- ↑ "Why Southern research? | IDRC - International Development Research Centre". www.idrc.ca. Retrieved 2022-06-27.
This article "Research Quality Plus" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Research Quality Plus. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
