Multiple Facets of Cross-Informant Agreement

Meta-analyses of correlations between ratings of child, adolescent, and adult psychopathology by different informants have shown low to moderate levels of agreement between different informants. This means that no one informant is likely to provide precisely the same information as other informants. Instead, comprehensive assessment requires data from multiple informants, as well as other kinds of data, to take account of the fact that people’s functioning varies from one context to another and that informants shape information according to their own perspectives. Although numerous studies have replicated the modest levels of agreement between different sources of data regarding psychopathology, much remains to be learned about factors affecting agreement levels and, ultimately, how to utilize multi-source data most effectively.

Three new studies shed light on different facets of these important issues. In one study, an Erasmus University team compared cross-informant agreements for a Dutch general population sample of 1,875 individuals assessed on seven occasions spanning 24 years from when the youngest were 4 years old to when the oldest were 40 years old (Van der Ende, Verhulst, & Tiemeier, 2012). Data were analyzed for 12,059 pairs of parent, teacher, spouse/partner, and self-reports obtained with ASEBA instruments. It was found that correlations among informants’ ratings of Internalizing and Externalizing problems depended more on the particular kinds of informant pairs than on the problem type or age group. However, differences between informants’ ratings of Internalizing problems increased as the participants grew older. Furthermore, self-ratings generally yielded higher problem scores than parent, teacher, or spouse/partner ratings.

To determine whether self-ratings produced higher problem scores than ratings by others in other societies, Rescorla et al. (2012) compared YSR ratings by 27,861 11-18-year-olds with CBCL ratings by their parents in population samples from 25 societies. Mean scores on all problem scales were higher on the YSR than the CBCL in all 25 societies, significantly so in all societies except Puerto Rico. Averaged across societies, significantly more youths than parents endorsed each of the 98 problem items common to both forms. Despite the consistently higher YSR than CBCL problem scores, correlations between mean item ratings by youths and parents for the 98 items ranged from .72 (Japan) to .94 (Romania), with a mean = .85. This indicated high agreement between youths and parents with respect to the items they rated low, medium, or high in each society. Because youths and parents tended to rate the same items as low, medium, or high, the higher scale scores on the YSR than on the CBCL were not due to youths endorsing different problems than their parents did. Furthermore, bi-society Q correlations between the mean rating received by each item in each society averaged .72 across all pairs of societies for the YSR and .73 for the CBCL. These large Q correlations indicated considerable consistency in the problems that received low, medium, or high ratings across the different societies.

The Van der Ende et al. (2012) and Rescorla et al. (2012) studies showed that self-reports typically yield higher problem scores than collateral reports across broad spans of ages and societies in general population samples. To determine whether such findings apply to youths assessed in forensic contexts, Penney and Skilling (2012) compared YSR and CBCL scores for 373 Canadian 12-19-year-olds who had been arrested and were undergoing court-ordered assessments. As found in the general population samples studied by Van der Ende et al. and Rescorla et al., YSR scores were significantly higher than CBCL scores on the Withdrawn/Depressed and Somatic Complaints syndromes. However, unlike the general population samples, YSR scores were significantly lower than CBCL scores on the Aggressive Behavior syndrome and did not differ significantly on the other syndromes. Cross-informant rs of .34 for Internalizing and .47 for Externalizing were in the same general range as for most other samples. Other measures indicated that YSR-CBCL discrepancies were higher in families experiencing high levels of stress and conflict than in other families. Furthermore, youths who scored high on measures of arrogance and deceitfulness provided more discrepantly low ratings on the YSR Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior syndromes than other youths. Measures of psychopathic features and self-deceptive enhancement were also associated with more discrepantly low ratings on some YSR scales. The authors concluded that the discrepantly low YSR ratings could reflect youths’ intentional efforts to misrepresent themselves and/or impaired insight into their problems.

References: Penney, S.R., & Skilling, T.A. (2012). Moderators of informant agreement in the assessment of adolescent psychopathology: Extension to a forensic sample. Psychological Assessment, 24, 386-401.
Rescorla, L.A. et al. (2012). Cross-informant agreement between parent-reported and adolescent self-reported problems in 25 societies. Journal of Clinical Child & Adolescent Psychology, 41, in press.
Van der Ende, J., Verhulst, F., & Tiemeier, H. (2012). Agreement of informants on emotional and behavioral problems from childhood to adulthood. Psychological Assessment, 24, 293-300.