Quantity and/or Quality? The Importance of Publishing Many Papers

Ulf Sandström, Peter van den Besselaar Do highly productive researchers have significantly higher probability to produce top cited papers? Or do high productive researchers mainly produce a sea of irrelevant papers—in other words do we find a diminishing marginal result from productivity? The answer on these questions is important, as it may help to answer the question of whether the increased competition and increased use of indicators for research evaluation and accountability focus has perverse effects or not. We use a Swedish author disambiguated dataset consisting of 48.000 researchers and their WoS-publications during the period of 2008–2011 with citations until 2014 to investigate the relation between productivity and production of highly cited papers. As the analysis shows, quantity does make a difference.

Towards Field Adjusted Production: estimating research productivity from a zero-truncated distribution

Timo Koski, Erik Sandström, Ulf Sandstrom Measures of research productivity (e.g. peer reviewed papers per researcher) is a fundamental part of bibliometric studies, but is often restricted by the properties of the data available. This paper addresses that fundamental issue and presents a detailed method for estimation of productivity (peer reviewed papers per researcher) based on data available in bibliographic databases (e.g. Web of Science and Scopus). The method can, for example, be used to estimate average productivity in different fields, and such field reference values can be used to produce field adjusted production values. Being able to produce such field adjusted production values could dramatically increase the relevance of bibliometric rankings and other bibliometric performance indicators. The results indicate that the estimations are reasonably stable given a sufficiently large data set. Waring distribution, Productivity, Citation Analysis, Ranking, Research Policy

What is the Required Level of Data Cleaning? A Research Evaluation Case

Peter van den Besselaar, Ulf Sandström Bibliometric methods depend heavily on the quality of data, and cleaning and disambiguating data are very timeconsuming. Therefore, quite some effort is devoted to the development of better and faster tools for disambiguating of the data (e.g., Gurney et al. 2012). Parallel to this, one may ask to what extent data cleaning is needed, given the intended use of the data. To what extent is there a trade-off between the type of questions asked and the level of cleaning and disambiguating required? When evaluating individuals, a very high level of data cleaning is required, but for other types of research questions, one may accept certain levels of error, as long as these errors do not correlate with the variables under study. In this paper, we present an earlier case study with a rather crude way of data handling as it was expected that the unavoidable error would even out. In this paper, we do a sophisticated data cleaning and disambiguation of the same dataset, and then do the same analysis as before. We compare the results and discuss conclusions about required data cleaning. Keywords: Coupling data sets, Data cleaning disambiguation, Data error.

Myter om nobelpristagare och deras tidiga uppmärksamhet (Myths about Laureates early recognition)

Ulf Sandstrom Myths about successful researchers are common and occasionally Nobel laureates are used to draw hasty conclusions concerning the value of bibliometrics and citation analysis. Sometimes these conclusions might even have an impact on research policy. As long as the analysis is based on large groups of laureates it is quite in order, but there are also some cases where Laureates personal life paths occur in science policy debates. This paper will deal with two cases where myths stand in stark contrast to bibliometric data.

Bibliometric Report to ORU 2015

Ulf Sandstrom During 2015, all research performed from 2008 to 2014 at Örebro University, as well as research at Örebro University Hospital, was the subject of evaluation. This report – ORU2015 – presents the background, planning and implementation of the research assessment and its results. Chapter I includes the panel evaluations, and chapter II presents the bibliometric analysis.

Forskarprestationer inom dedicerade Formasområden och tidskriftsklasser

Ulf Sandstrom Forskningsrådet Formas verksamhet som forskningsstödjande statligt organ inriktar sig mot fyra huvudområden: Miljö, Areella näringar, Samhällsbyggnad samt Grundläggande och gränsöverskridande naturresurssystem. Dessa fyra områden kan i sin tur brytas ned i tjugotvå delområden för vilka det i princip är möjligt att följa den vetenskapliga publiceringsverksamheten eftersom dessa delområden beskrivs i termer av de tidskriftsklasser som används av den bibliografiska databasen Web of Science (tillhandahålls av företaget Thomson Reuters). Rapportens huvudsakliga frågeställningar är följande: Hur ser prestationer från den svenska forskningen ut inom dessa områden och hur bidrar Formas till denna forskning?

Gender differences in research performance and its impact on careers: a longitudinal case study

Peter van den Besselaar, Ulf Sandström We take up the issue of performance differences between male and female researchers, and investigate the change of performance differences during the early career. In a previous paper it was shown that among starting researchers gendered performance differences seem small to non-existent (Van Arensbergen et al. 2012). If the differences do not occur in the early career anymore, they may emerge in a later period, or may remain absent. In this paper we use the same sample of male and female researchers, but now compare performance levels about 10 years later. We use various performance indicators: full/fractional counted productivity, citation impact, and relative citation impact in terms of the share of papers in the top 10 % highly cited papers. After the 10 years period, productivity of male researchers has grown faster than of female researcher, but the field normalized (relative) citation impact indicators of male and female researchers remain about equal. Furthermore, performance data do explain to a certain extent why male careers in our sample develop much faster than female researchers’ careers; but controlling for performance differences, we find that gender is an important determinant too. Consequently, the process of hiring academic staff still remains biased.

Early career grants, performance, and careers A study on predictive validity of grant decisions

Peter van den Besselaar, Ulf Sandström The main rationale behind career grants is helping top talent to develop into the next generation leading scientists. Does career grant competition result in the selection of the best young talents? In this paper we investigate whether the selected applicants are indeed performing at the expected excellent level – something that is hardly investigated in the research literature. We investigate the predictive validity of grant decision-making, using a sample of 260 early career grant applications in three social science fields. We measure output and impact of the applicants about ten years after the application to find out whether the selected researchers perform ex post better than the non-successful ones. Overall, we find that predictive validity is low to moderate when comparing grantees with all non-successful applicants. Comparing grantees with the best performing non-successful applicants, predictive validity is absent. This implies that the common belief that peers in selection panels are good in recognizing outstanding talents is incorrect. We also investigate the effects of the grants on careers and show that recipients of the grants do have a better career than the non-granted applicants. This makes the observed lack of predictive validity even more problematic.

Excellenssatsningarna - belöning för kön eller toppforskning?

Ulf Sandstrom, Agnes Wold Women make up 30% of top researchers in Sweden. Thus, the fact that women receive less than 20% of funds for research at centres of excellence cannot be explained by a lack of top female researchers. It is more probable that notions of who are ‘excellent’ are coloured by gender prejudices. At the same time, half the recipients of excellence funding cannot be characterised as top researchers.