Ludtke, O., and Robitzsch, A. (2009). Evaluation of a group agreement: a critical review of a resampling approach for random groups. organ. Res. Methods 461-487. doi: 10.1177/1094428108317406 Keywords: Interrater Agreement, rwg, multi-level methods, data aggregation, group agreement, reliability Multi-level management researchers have mainly used consensus composition or reference models for aggregation of individual data at a higher level of analysis. The composition of the consensus assumes that there is sufficient consensus within the group regarding the management structure of interest; in the absence of agreement, the entire governance structure is unsustainable. At the same time, guidelines to help leadership researchers make decisions about data aggregation have received little attention. In particular, a discussion of how data aggregation decisions can improve or conceal the theoretical contribution of a study – a central priority of this article – has not been addressed in depth. Recognizing that empirical generalization depends on the accuracy with which aggregation decisions are applied, we examine the often overlooked assumptions associated with the most common consensus statistic used to justify data aggregation – rWG and rWG (J) (James, Demaree and Wolf, 1984).
Based on a dataset published as part of a “Leadership Quarterly Special Issue” (Bliese, Halverson, Schriesheim, 2002), we highlight the potential abuse of rWG and rWG (J) as the only statistic justifying aggregation at a higher level of analysis. We conclude with prescriptive implications for promoting consistency in the way leadership research is conducted and reported at multiple levels. Dunlap, W. P., Burke, M. J., and Smith-Crowe, K. (2003). Accurate tests of statistical significance for average difference indices to interrater agree. J. Appl. Psychol. 88, 356-362.
doi: 10.1037/0021-9010.88.2.356 Allen, N. J., Stanley, D. J., Williams, H.M., and Ross, S. J. (2007). To assess the impact of non-response on the impact on the diversity of working groups. organ. Res. Methods 10, 262-286. doi: 10.1177/1094428106/294731 George, J.M. (1990).
Personality, affect and group behaviour. J. Appl. Psychol. 86, 1075-1082. doi: 10.1037/0021-9010.75.2.107 Biemann, T., Cole, M., and Voelpel, p. (2012). Within the group: on the use (and abuse) of rWG and rWG (J) in leadership research and some guidelines for good practice. Leadersh.
Q. 23, 66-80. doi: 10.1016/j.leaqua.2011.11.006 Cohen, A., Doveh, E., and Nahum-Shani, I. (2009). Test agreement for multi-item scales with rwg (j) and ADm (j) indices. organ. Res. Methods 12, 148-164. doi: 10.1177/1094428107300365 Burke, M. J., Finkelstein, L.M., and Dusig, M.
S. (1999). For average indices of variation for the estimate of the Interrater agreement. organ. Res. Methods 2, 49-68. doi: 10.1177/10944281921004 van Mierlo, H., Vermunt, J. K., and Rutte, C.
G. (2009). Compose group constructions from survey data at the individual level. organ. Res. Methods 12, 368-392. doi: 10.1177/1094428107309322 George, J.M., and James, L. R. (1993). Personality, affect and behavior in revisited groups: Comment on aggregation, analysis levels and a common application of analysis and analysis.