4 april 2016
Ulf Sandstrom Myths about successful researchers are common and occasionally Nobel laureates are used to draw hasty conclusions concerning the value of bibliometrics and citation analysis. Sometimes these conclusions might even have an impact on research policy. As long as the analysis is based on large groups of laureates it is quite in order, but there are also some cases where Laureates personal life paths occur in science policy debates. This paper will deal with two cases where myths stand in stark contrast to bibliometric data.
15 december 2015
Ulf Sandstrom During 2015, all research performed from 2008 to 2014 at Örebro University, as well as research at Örebro University Hospital, was the subject of evaluation. This report – ORU2015 – presents the background, planning and implementation of the research assessment and its results. Chapter I includes the panel evaluations, and chapter II presents the bibliometric analysis.
7 december 2015
Ulf Sandstrom Forskningsrådet Formas verksamhet som forskningsstödjande statligt organ inriktar sig mot fyra huvudområden: Miljö, Areella näringar, Samhällsbyggnad samt Grundläggande och gränsöverskridande naturresurssystem. Dessa fyra områden kan i sin tur brytas ned i tjugotvå delområden för vilka det i princip är möjligt att följa den vetenskapliga publiceringsverksamheten eftersom dessa delområden beskrivs i termer av de tidskriftsklasser som används av den bibliografiska databasen Web of Science (tillhandahålls av företaget Thomson Reuters). Rapportens huvudsakliga frågeställningar är följande: Hur ser prestationer från den svenska forskningen ut inom dessa områden och hur bidrar Formas till denna forskning?
15 november 2015
Wang Qi, Ulf Sandström The aim of this article is twofold: (1) to provide a methodology for measurement of cognitive
distance between researchers and (2) to explore the role of cognitive distance on the results of
peer review processes. Cited references and the content of articles are used to represent their
respective scientific knowledge bases. Based on the two different approaches—Author-
Bibliographic Coupling analysis and Author-Topic analysis—we apply the methodology on a
recent competition for grants from the Swedish Strategic Foundation. Results indicate that cognitive
distances between applicants and reviewers might influence peer review results, but that the
impact is to some extent at the unexpected end. The main contribution of this article is the
elaboration on the relevance of the concept of cognitive distance to the issue of research evaluation
in general, and especially in relation to peer review as a model used in grant decisions.
12 november 2015
Peter van den Besselaar, Ulf Sandström We take up the issue of performance differences between male and female researchers, and investigate the change of performance differences during the early career. In a previous paper it was shown that among starting researchers gendered performance differences seem small to non-existent (Van Arensbergen et al. 2012). If the differences do not occur in the early career anymore, they may emerge in a later period, or may remain absent. In this paper we use the same sample of male and female researchers, but now compare performance levels about 10 years later. We use various performance indicators: full/fractional counted productivity, citation impact, and relative citation impact in terms of the share of papers in the top 10 % highly cited papers. After the 10 years period, productivity of male researchers has grown faster than of female researcher, but the ﬁeld normalized (relative) citation impact indicators of male and female researchers remain about equal. Furthermore, performance data do explain to a certain extent why male careers in our sample develop much faster than female researchers’ careers; but controlling for performance differences, we ﬁnd that gender is an important determinant too. Consequently, the process of hiring academic staff still remains biased.
1 augusti 2015
Ulf Sandström This study is a follow-up to Sandström & Hällsten (2008); it showed that nepotism or what is in most discussions known as conflict of interest (CoI), had a significant influence over grant decisions in the Swedish Medical Research Council (MFR). Data covered the year 2004, ten years after the Wennerås & Wold study, published in Nature 1997. The size of the effect was the same in 2004 as it was in 1994. Conflict of interest seemed to be a persisting phenomenon in the council procedures. Could it be a structural effect of the institutional set-up?
After the 2008 study, there was a discussion in the Swedish professional journal Läkartidningen in 2009. Hällsten & Sandström wrote that it was time to highlight the problems concerning the role of conflict of interest in the research policy debate. The Swedish Research Council entered into the debate, they more or less had to as they were since 2004 engaged in giving yearly reports on figures for parity in the distribution of grants between men and women. But, these reports didn’t tell the full story; they didn’t control the track record of applicants and didn’t relate the performance to the rewards. As it turned out, the council had no knowledge about the self-selection effects.
In the answer, the council representatives explained why controversial patterns were found: the members of the Council and the review panels consisted of "prominent members from prominent research environments". Since these prominent researchers also produced “family eggs” that eventually sought funding from the Research Council, it was natural that it appeared as nepotism in an analysis of the type that had been performed. Nepotism was a natural thing in the context of a council that produced excellent research (Läkartidningen nr 37/2009).
A natural response to that type of reasoning is to control whether it is the cases that panel members of the MFR are excellent high-profile researchers. This is an empirical question and it can be sorted out. Therefore, we turn to the bibliometric tool-box. All panel members that were active during the period 2008-2009 were analyzed for the period 2000 2007 (investigation was done in 2012 with citations to 2011). Using standard bibliometric indicators it was found that this group of 84 people had a total citation level that was on par with Swedish universities, even just below the nine largest Swedish universities' citation level (per paper).
Furthermore, this report applies a new size-dependent indicator model (Sandström & Sandström 2010) which rank all Swedish researchers based on production and citations. Not more than one out of three of the panel members would qualify among the top 5% of Swedish researchers, which is a fairly inclusive definition of what would count as excellent research. Foreign members, from the Nordic countries mainly, seem to perform even worse.
15 juli 2015
Peter van den Besselaar, Ulf Sandström The main rationale behind career grants is helping top talent to develop into the next generation leading scientists. Does career grant competition result in the selection of the best young talents? In this paper we investigate whether the selected applicants are indeed performing at the expected excellent level – something that is hardly investigated in the research literature. We investigate the predictive validity of grant decision-making, using a sample of 260 early career grant applications in three social science fields. We measure output and impact of the applicants about ten years after the application to find out whether the selected researchers perform ex post better than the non-successful ones. Overall, we find that predictive validity is low to moderate when comparing grantees with all non-successful applicants. Comparing grantees with the best performing non-successful applicants, predictive validity is absent. This implies that the common belief that peers in selection panels are good in recognizing outstanding talents is incorrect. We also investigate the effects of the grants on careers and show that recipients of the grants do have a better career than the non-granted applicants. This makes the observed lack of predictive validity even more problematic.
15 juni 2015
Ulf Sandstrom, Agnes Wold Women make up 30% of top researchers in Sweden. Thus, the fact that women receive less than 20% of funds for research at centres of excellence cannot be explained by a lack of top female researchers. It is more probable that notions of who are ‘excellent’ are coloured by gender prejudices. At the same time, half the recipients of excellence funding cannot be characterised as top researchers.
1 juni 2015
Ulf Sandström, Agnes Wold Women make up 30% of top researchers in Sweden. Thus, the fact that women receive less than 20% of funds for research at centres of excellence cannot be explained by a lack of top female researchers. It is more probable that notions of who are ‘excellent’ are coloured by gender prejudices. At the same time, half the recipients of excellence funding cannot be characterised as top researchers.
20 november 2014
Ulf Heyman, Ulf Sandström