Research Evaluation

Impact data and citation analysis - an important part of the information retrieval service for the Max Planck Society

As public funds for research should be used as efficient as possible, objective evaluation criteria to select the most promising fields of research have become very important. Also the Max Planck Society (MPG) is confronted with the need to sell its research using publication and citation counts. Generally, the evaluation of MPG research is based on the peer review process. Citation data become increasingly important (informed peer review) in the following cases:

  1. appointments
  2. quality assurance of the institutes (consulting committee, board of trustees)
  3. presentation of MPG research in an international comparison (for the MPG headquarters)
  4. optimization of publications (selection of proper journals)
  5. self-monitoring of the impact of individual researchers

Institute's managements and the headquarters of the MPG increasingly request impact data from the information retrieval service groups. Analysis of MPG specific issues as well as interpretation of the data are connected to this service. However, there is no official opinion about the informative value of citation data and its usage

The main sources of citation data have been ISI (formerly known as: Institute for Scientific Information, now:Thomson Reuters), Chemical Abstracts (CAS), and Scopus (Elsevier). Additionally, Google Scholar and community-based products such as Cite Seer (for computer science) which are freely available on the internet.

The ISI data are available on the Web of Science (WoS) from Web of Knowledge by Thomson Reuters. Other hosts offer the same data (although for a limited time frame) with more flexible search systems. Statistical analytic tools such as offered by STN International (FIZ Karlsruhe) in combination with home-brew programs are essential for extensive citation analysis. The ISI products Essential Science Indicators (ESI), the HighlyCited database, country specific high-impact databases, and the Journal Citation Reports (JCR) with Journal Impact Factors (JIFs) are very important for special evaluation purposes. However, Journal Impact Factors are not suitable for evaluation of persons or institutions!

Also other non-normalized prarmeters (h index, Nature Index, etc.) should not be used or at least very cautiously. Such indicators depend very much of the research area, age of scientist (institute), and other factors. It is much more appropriate to use percentile ranks which are normalized to the category and publication year. For evaluation of persons and institutes, beam plots of persons and institutes have been shown to be very meaningful [1,2,3,4]. In the following, beam plots from 2014 of three different persons are presented.

Person 1 is an example for a researcher who was active for 39 years but achieved only a mediocre impact.

The black data points are inverted percentiles of individual publications. The red triangles show medians of the years. The red line indicates the median over all publications.

Person 2 is an example for a researcher who started to publish around the same time as person 1. Person 2 stayed longer active in research and has achieved a significantly higher impact than person 1.
Person 3 is an example for a younger researcher than persons 1 and 2. Person 3 has achieved an extraordinary impact within the 21 years of scientific research.

[1] J. Am. Soc. Inform. Sci. Tech., 62, 1370 (2011), DOI: 10.1002/asi.21534 .
[2] EMBO reports, 14, 226 (2013), DOI: 10.1038/embor.2013.9.
[3] Scientometrics, 98, 487 (2014), DOI: 10.1007/s11192-013-1161-y.
[4] J. Am. Soc. Inform. Sci. Tech., 65, 206 (2014), DOI: 10.1002/asi.22996.

Go to Editor View