27 mars 2019
Peter van den Besselaar, Ulf Sandström Bibliometric indicators are increasingly used to evaluate individual scientists–as is exemplified
by the popularity of the many other publication and citation-based indicators used in evaluation. These indicators, however, cover at best some of the quality dimensions relevant for assessing a researcher: productivity and impact. At the same time, research quality has more dimensions than productivity and impact alone. As current bibliometric indicators are not covering various important quality dimensions, we here contribute to developing better indicators for those quality dimensions not yet addressed. One of the quality dimensions lacking valid indicators is an individual researcher’s independence. We propose indicators to measure different aspects of independence: two assessing whether a researcher has
developed an own collaboration network and two others assessing the level of thematic independence. Taken together they form an independence indicator. We illustrate how these indicators distinguish between researchers that are equally productive and have a considerable impact. The independence indicator is a step forward in evaluating individual scholarly quality.
13 juli 2018
Peter van den Besselaar, Ulf Sandström Peer and panel review are the dominant forms of grant decision-making, despite its serious
weaknesses as shown by many studies. This paper contributes to the understanding of the
grant selection process through a linguistic analysis of the review reports. We reconstruct
in that way several aspects of the evaluation and selection process: what dimensions of the
proposal are discussed during the process and how, and what distinguishes between the
successful and non-successful applications? We combine the linguistic findings with
interviews with panel members and with bibliometric performance scores of applicants.
The former gives the context, and the latter helps to interpret the linguistic findings. The
analysis shows that the performance of the applicant and the content of the proposed study
are assessed with the same categories, suggesting that the panelists actually do not make a
difference between past performance and promising new research ideas. The analysis also
suggests that the panels focus on rejecting the applications by searching for weak points,
and not on finding the high-risk/high-gain groundbreaking ideas that may be in the proposal.
This may easily result in sub-optimal selections, in low predictive validity, and in
Keywords Peer review Panel review Research grants Decision-making Linguistics
LIWC European Research Council (ERC)
4 juli 2018
Ulf Sandström Många tar för givet att forskning med svagt genomslag är koncentrerad till vissa småskaliga universitet högskolor. Detta motiverar närmare undersökning eftersom det förhållandet att svensk forskning i väsentlig grad skulle förbättras om verksamheten flyttades från de regionala högskolorna till universiteten behöver i så fall beläggas med fakta. Om det är så att de stora universiteten dragit ifrån och gör bättre resultat än vad som framgick av en tidigare undersökning (Sandström 2015) borde detta kunna förklaras av att forskningsresurserna kanaliserats till dessa lärosäten. Men frågan är hur det egentligen ser ut? Har de stora dragit ifrån och har de små förlorat i samma mån?
20 juni 2018
Ulf Sandström This bibliometric evaluation of wildlife research, funded by the Wildlife
Management Fund through the Swedish Environmental Protection Agency
(SEPA) during 2003–2014, highlights how the international publications
have developed for the funded research leaders and co-applicants during the
period 2006 until 2014. The following questions have guided the evaluation:
1) Has the SEPA programme for wildlife research payed off in relation to
input of resources?
2) Has SEPA and its Wildlife Research Committee chosen the best available
researchers for the projects?
3) Does SEPA’s funded wildlife research represent a reasonable project portfolio
in an international perspective?
4) Does SEPA have a gender-wise equal distribution of research funds?
Nearly 95% of all resources have gone to sub-programmes devoted to large
carnivores, general biology and social science/humanities. Those areas that
have received most of the resources can therefore have dedicated researchers,
where most of their publications have focused on the game programme, the
other and the smaller areas more or less fall outside. Within the aforementioned
areas, game research has yielded good results.
The bibliometric evaluation suggests that the SEPA has a good exchange
of resources in terms of number of articles and expected citation response
from the larger research community. Particularly the programme for large
carnivores has proved to be an investment with good productivity and substantial
recognition from the international research community.
During the programme period, citation strength increases significantly,
from 40% to 60% of researchers have strong achievements, i.e. they are
included in the top 20% of Swedish researchers.
7 juni 2018
Koen Jonkers, Ulf Sandström, Peter van den Besselaar The Marie Sklodowska Curie Action (MSCA) fellowship scheme aims, as a part of the European
framework programmes, to promote scientific excellence, mobility and research collaboration in the
European Research Area. As most elements on the EU Framework Programmes, it also aims to
widen capacity development throughout the EU in Member States with different levels of scientific
development. This report analyses the mobility, publication and international co-publication
behaviour of a group of European researchers that have taken part in the Marie Sklodowska Curie
Action (MSCA) Fellowship schemes. It compares researchers that received their PhD from
organisations in two groups of countries before and after being granted the fellowship.
The first group of countries (from North-Western Europe: FPIC receives a relatively large share of
their research funding budget from the European Framework Programmes and a relatively low
share from the European Structural and Investment Funds. The second group of countries (from
South and Eastern European: ESIFIC) presents a lower Framework Programme funding intensity
but the Funding intensity of the European Structural and Investment Funds is higher. The funding
intensity levels associated with these broad programmes are taken as an indication of the level of
scientific development. It strongly correlates with the average impact of the publications made by
researchers in these countries. Also relevant to this analysis is that the first group of countries tend
to host more MSCA fellows than they send whereas the reverse holds for the second group group.
The analysis measures performance as the sum of the citation impact of a researchers publications.
Before the grant one observes a difference between the performance of applicants from South and
Eastern Europe (ESIFIC) on the one hand and those from North Western Europe (FPIC) on the
other. Over time the median performance gap disappears: there is convergence in the median
performance of researchers from the two country groups. However due to a larger number of
outliers (top performers) in North Western European countries there remains a difference in the
When comparing MSCA applicants with other grant schemes, one finds that the MSCA applicants
perform well before and after the grant - though as expected below the performance of researchers
funded by the highly selective ERC junior grant which tend to be more senior. The MSCA applicants
show a marked improvement after the grant in comparison to before. This in contrast to a similar
national individual fellowship in an EU MS.
Post grant performance is mainly correlated to pre-grant performance. One does not find a
significant correlation with the quality of the research environment (as proxied by citation impact of
the host organisation). This is surprising because the quality of the host environment is an explicit
Post grant international collaboration behaviour is mainly correlated to pre-grant international
collaboration: it appears as if the well connected remain well connected also after being funded.
What we did find was that after the grant a considerable share of the increase in co-authored high
impact papers are co-published with researchers from North Western Europe: this suggests the
MSCA mobility experience leads to productive research links.
The potential for robust evaluations, either in the form of counterfactual analyses or randomised
controlled experiments should be taken into account at the planning and implementation phase of
the Framework Programmes.
26 maj 2018
Ulf Sandström, Peter van den Besselaar Understanding the quality of science systems requires international comparative studies,
which are difficult because of the lack of comparable data especially about inputs in research.
In this study, we deploy an approach based on reasonable comparative data that focus on
change instead of on levels of inputs and outputs, as this approach to a large extent eliminates
the problem of measurement differences between countries. Using input-data related to
output data (top publications in Web of Science) we first show which national science
systems are more efficient (where performance increase is stronger than expected change in
funding) and systems which are less efficient. We then discuss our findings using popular
explanations of performance differences: differences in the level of competition, differences
in the level of university autonomy, and differences in the level of academic freedom.
Interestingly, the available data do not support the common explanations. Good functioning
systems are characterized by a well-developed ex post evaluation system combined with
considerably high institutional funding and low university autonomy (meaning a high
autonomy of professionals). On the other hand, the less efficient systems have a strong ex
ante control, either through a high level of so-called competitive project funding, or through
strong power of the university management.
24 maj 2018
Ulf Sandström, Peter van den Besselaar Policymakers and researchers have long sought measures to compare countries’ scientific performance. The
most widely used has been to divide levels of public R&D spending by numbers of publications or citations. Simply dividing funding by outputs, however, is not likely to give an accurate portrait of a research system’s efficiency. Countries differ greatly in how their research budgets are organised and administered, in how PhD studentships are financed, for example. And while in theory the data for OECD statistics are collected in the same way everywhere, in practice this is not the case. These factors and others make it unwise to use R&D spending levels when comparing the performance of national research systems. In this paper we propose an alternative.
24 april 2018
Ulf Sandström, Jörg Müller, Anne Laure Humbert, Sandra Klatt The present paper report the findings of the cross-country survey regarding gender diversity in R&D teams across Europe and its link to performance indicators carried as part of the GEDII project. The empirical evidence is based upon 1,357 complete questionnaire submissions across 159 teams in the following 17 countries: Austria, Belgium, Czech Republic, Denmark, Finland, France, Germany, Italy, Lithuania, the Netherlands, Norway, Poland, Portugal, Spain, Sweden, Switzerland and the UK.
Most teams were recruited from Spain (approximately 500 individual responses. and Sweden (approximately 300 responses. followed by Germany, the UK, the Netherlands approximating about 100 individual responses each. The fieldwork was conducted between March 2017 and January 2018. Despite concerted efforts, response from the private sector was negligible.
R&D teams reaching a sufficiently high response rate threshold were included in the analysis of the diversity-performance link.
Web of Science publications as well as patents were collected for all members of the participating groups. Bibliometric indicators including such size-dependent indicators as the Field Adjusted Performance (FAP) and Percentile Model (PModel) were calculated in order to compare performance of research groups across scientific fields. Patent indicators counted the number of patents per team.
Gendered processes within teams were captured through the Gender Diversity Index (GDI), a composite indicator developed in another part of this project. The GDI measures the representation and attrition of women and men within teams along seven dimensions of diversity, such as education, age, marital status, care responsibilities, team tenure, seniority and contract type. The GDI provides a score bound between 0 and 1, where 1 signals a more inclusive team.
Our preliminary analysis shows that more inclusive teams – that is, teams with a score close to 1 on the Gender Diversity Index – tend to perform better and generate more research output. When controlling for gender stereotypes, gender balance and the representation of women within teams, a score of 1 on the GDI is associated with an increase of 0.91 FAP. Less inclusive teams need on average an additional 0.91 senior researchers in order to perform as well as more inclusive teams.
There is no statistically significant effect on the quality rank of the published research (Percentile Model). Initial modelling also does not indicate a significant mediation effect of team processes such as team climate, power disparity, perception of leadership style or diversity climate.
4 december 2017
Ulf Sandström, Peter van den Besselaar Understanding the quality of science systems requires international comparative studies, which are difficult because of the lack of comparable data especially about inputs in research. In this study, we deploy an approach based on reasonable comparative data that focus on change instead of on levels of inputs and outputs, as this approach to a large extent eliminates the problem of measurement differences between countries. Using input-data related to output data (top publications in Web of Science) we first show which national science systems are more efficient (where performance increase is stronger than expected change in funding) and systems which are less efficient. We then discuss our findings using popular explanations of performance differences: differences in the level of competition, differences in the level of university autonomy, and differences in the level of academic freedom. Interestingly, the available data do not support the common explanations. Good functioning systems are characterized by a well-developed ex post evaluation system combined with considerably high institutional funding and low university autonomy (meaning a high autonomy of professionals). On the other hand, the less efficient systems have a strong ex ante control, either through a high level of so-called competitive project funding, or through strong power of the university management.
28 september 2017
Peter van den Besselaar, Ulf Sandström A recent paper in this journal compares the Norwegian model of using publications counts for university funding with a similar intervention in Australia in the mid-1990 s. The authors argue that the Norwegian model (taking into account the quality of publications) performs better than the Australian (which did neglect paper quality other than being peer reviewed). We argue that these conclusions are in contrast to the evidence provided in the article, and therefore should be considered incorrect.