Accession Number : ADA136709

Title :   Estimating Within-Group Interrater Reliability with and without Response Bias.

Descriptive Note : Technical rept.,


Personal Author(s) : James,L R ; Demaree,R G ; Wolf,G

PDF Url : ADA136709

Report Date : 09 Dec 1983

Pagination or Media Count : 37

Abstract : This article presents methods for assessing agreement among the judgments made by a single group of judges on a single variable in regard to a single target. For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower-level manager, or a team. The variable on which the target is judged could be overall publishability in the case of the manuscript, managerial potential for the lower-level manager, or team cooperativeness for the team. The methods presented are based on new procedures for estimating interrater reliability. For situations such as the above, these procedures are shown to furnish more accurate and interpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability. In addition, the proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency, social desirability) on estimates of interrater reliability. (Author)

Descriptors :   *Group dynamics, *Agreements, *Judgement(Psychology), *Ratings, *Reliability, Estimates, Response, Bias, Variables, Targets, Teams(Personnel), Cooperation, Methodology, Accuracy, Psychological tests, Statistical processes

Subject Categories : Psychology

Distribution Statement : APPROVED FOR PUBLIC RELEASE