People often wonder why cities like Montreal in Canada get such good ratings. Now the means by which research projects are rated has become the focus of research too. | Image: Anthony Gerace

Recently, the German life sciences journal ‘Laborjournal’ took the daring step of conducting a thought experiment: What if researchers didn’t have to apply for money from funding organisations, but the latter had to fight over which promising researchers they were allowed to fund?

They weren’t entirely serious, though they offer an indication of just how keen the world of science is to question the status quo when it comes to research funding. There is today at least a consensus that it needs to be reformed – even among the funding organisations themselves. But how? Hopes rest on research itself: more precisely, on research on research, which could well point the way to a better future.

Metrics, the replication crisis and a flood of publications

To sum up the problem, the current unease in the academy is focused on three areas. The most important is metrics. The quality of researchers is measured primarily in quantitative terms – how often someone is cited, and the journals in which they are published. Even the journals themselves are given a number stating how often their articles are cited on average – this is the infamous ‘impact factor’.

The second is the replication crisis, which refers to those instances in the natural and social sciences when experiments by one group can’t be replicated by another. But reproducibility is essential, and a lack of it undermines the credibility of the sciences overall.

Ultimately, researchers publish as many papers as possible because this increases their chances of getting funding. And those who have already been given funding once are most likely to get it again. As it says in the Gospel according to St Matthew: To him who has, more shall be given. But this ought not to be the case either. If applications are evaluated by experts from the same research field, this ought to ensure that only the best applications are funded. But today’s procedures often mean that women and people from ethnic minorities are assessed less favourably, as they are underrepresented in the evaluating panels.

The metrics that have become common in our scholarly system have led to some bizarre effects. Ruth Müller is an associate professor of science and technology policy at the Technical University in Munich. She has recently been involved in studies proving that postdocs in the life sciences overall don’t choose their research questions for purposes of knowledge gain, but depending on what topics are more likely to guarantee publication in a high-impact-factor journal. But this stands in contradiction to the notion of scientific and scholarly relevance.

Research on research is intended to help reveal these mechanisms and eliminate abuses of the system – all in aid of better research funding and hence also of better research. A dedicated Research on Research Institute (RoRI) was even founded in 2019, run by the Universities of Leiden in the Netherlands and Sheffield in the UK, and supported both by a private foundation and by a company active in digital business. RoRI has been linking up with existing reforms such as the DORA Declaration of 2012, which committed funding organisations to abandoning the impact factor, thus focusing on research content instead of a journal’s reputation. RoRI director James Wilsdon, who teaches research policy at University College London, says: “We’re testing new tools to see if and how research assessment can be streamlined and made more efficient”. These include the use of artificial intelligence and chance elements. “RoRI is part of a global effort to establish a responsible culture of research assessment”.

Winning the tiebreak

The Swiss National Science Foundation (SNSF) is already integrating such findings in its assessment practices. For example, it has introduced partial randomisation, which means selecting projects by lot when there are too many funding applications of a similarly high level. “Studies have shown that reviewers find it difficult to decide between applications that are situated in a mid-level grey zone”, says Katrin Milzow, the head of the Strategy Division at the SNSF. She is part of RoRI’s management committee.

In a pilot call for proposals, the SNSF is paying special attention to applications that receive both very good ratings and very bad ratings at the same time. These different assessments often point to interesting potential, says Milzow. The SNSF is also engaged in a RoRI project of its own: the Career Tracker Cohorts. Over the next ten years, in a collaboration with the University of Bern, it will conduct a survey of postdocs who’ve received funding, and postdocs who haven’t. This long-term study is intended to provide information both on career paths and also on the motives and assessments of young researchers themselves – in other words, their own, subjective view of the academic system and the steps they take in the course of their career. It is hoped that these findings will help the SNSF to tailor its research funding more precisely to the needs of young researchers.

The SNSF has now also begun requiring narrative CVs. These offer applicants an opportunity to comment on their research results and link them to their biography. “This reduces the outsized influence of journal indicators on the review process, and makes the evaluation system better integrated”, says Milzow. She emphasises that RoRI findings are creating an evidence base for the further development of research funding.

The German Research Foundation (DFG) is also reforming its funding practices, although it isn’t going as far as the SNSF. In future, it’s going to curb its “narrow focus on quantitative metrics” when evaluating proposals. Other funding organisations, however, are expressing caution about research on research. The Austrian Research Promotion Agency (FFG) believes that the best way to improve research is simply to have the right instruments for each target group. The Centre National de la Recherche Scientifique in France, however, doesn’t want to comment on the topic. It’s not caught on everywhere yet.

A wild vision called metascience

Michael Nielsen is a quantum physicist and programmer at the Recurse Center in New York, and his colleague Kanjun Qui is a software developer. They are forging ahead with research on research, and have even written a kind of manifesto: ‘A Vision of Metascience’. These two bold pioneers of research on research want nothing less than to construct a machine for the rapid improvement of “the social process of science”.

By ‘social process’, Nielsen and Qui mean “the institutional practices, incentives and norms” that are prevalent in scholarship. “When we talk about changes in social processes, we mean changes in peer review, or in how to cope with risk when dealing with funding institutions”. For example, if the failure rate of funded projects is below fifty per cent, this means that the funding organisation in question has taken too few risks. In such a case, its programme manager should be fired, even if it remains unclear how ‘failure’ is actually defined. Nielsen and Qui offer another example: funding organisations ought to set up a ‘Hall of Shame’ in which all the successful researchers should be honoured who failed to get funding because the organisation in question didn’t recognise their potential.

Ruth Müller welcomes both the growing awareness of the problem on the part of the funding organisations, and their willingness to undergo reform. She hopes that analysing funding allocation – which is one aspect of research on research – will become a perfectly normal research topic, just as others study the practices of researchers active in the lab or out in the field. “Scientists take on all kinds of tasks, including evaluating research. We have to analyse this as a social practice so that we can improve our evaluation processes”.

However, we’re not that far yet, as Müller admits. She is critical of the fact that some of those active in research on research are unaware of the state of knowledge in the social sciences and want to reinvent the wheel. When Müller speaks about the social system of science, she means more than Nielsen and Qui. To her, researchers are part of society. They both shape society and are at the same time steeped in its power relations and norms. The ‘social system of science’ to her means more than just the process of peer review in which a collective is involved.

Research on research on research

‘Research on research’ has been a topic of science and technology studies since the 1960s. Even back then, the sociologist Thomas Kuhn remarked that scientific facts are not objective, natural facts, but a product of the social conditioning that shapes researchers’ investigations. Nor are evaluations objective. They reflect the prejudices of those participating in the process, for example. “A good researcher isn’t always a good peer reviewer – evaluating people is a practice that needs to be learned, just like research”, says Müller.

To Müller, the independence and transparency of research on research is important, although in this regard things are not currently what they should be: “Peer review is often examined in the context of research projects that have been commissioned and only result in a limited number of publications. The results are available to funding organisations and haven’t become part of any collective body of knowledge”. But research has to be accessible to all researchers if they are to build up a new body of knowledge. Müller is in favour of open calls for research-on-research projects with third-party funding. She cites the German Volkswagen Foundation’s ‘Researching Research’ projects as a positive example.

The aim of research on research was from the start to improve research funding, and thereby improve research itself. Thanks to its international networking through RoRI, it has already led to reforms in many funding organisations. This has a positive impact on research. Now, perhaps, it ought to apply itself more often to its own field of study – in other words, conducting research on research on research.