National research assessment exercises: a measure of the distortion of performance rankings when labor input is treated as uniform

Measuring the efficiency of scientific research activity presents critical methodological aspects, many of which have not been sufficiently studied. Although many studies have assessed the relation between quality and research productivity and academic rank, not much is known about the extent of distortion in national university performance rankings when academic rank and the other labor factors are not considered as a factor of normalization. This work presents a comparative analysis that aims to quantify the sensitivity of bibliometric rankings to the choice of input, with input considered as only the number of researchers on staff, or alternatively where their cost is also considered. The field of observation consists of all 69 Italian universities active in the hard sciences. Performance measures are based on the 81,000 publications produced during the 2004-2006 triennium by all 34,000 research staff, with analysis carried out at the level of individual disciplines, 187 in total. The effect of the switch from labor to cost seems to be minimal except for a few outliers.