Hartmann, D. P. Reflections in the choice of Interobserver`s reliability estimates. Journal of Applied Behavior Analysis 1977,10, 103-116. Cohen, J. A coefficient of agreement for nominal scales. Educational and psychological measure 1960,20, 37-46. This technical report provides detailed information on the reasons for using a common computer computing program (Microsoft Excel®) to calculate different forms of interobserver agreement for continuous and discontinuous datasets. In addition, we offer a brief tutorial for using an Excel table to automatically calculate the traditional total number, partial match in intervals, exact tuning, trial test, interval interval, multiple interval, total duration and average duration of interobserver duration algorithms. We conclude with a discussion of how practitioners can integrate this tool into their clinical work.
The idea that practicing behavioural analysts should collect and report reliability or interobserver agreement (IOA) in behavioural assessments is demonstrated by the Behavior Analyst Certification Board`s (BACB) assertion that behavioural analysts are responsible for the use of “different methods of evaluating the results of measurement methods such as inter-observer agreement, accuracy and reliability” (BACB, 2005). In addition, Vollmer, Sloman and St. Peter Pipkin (2008) argue that the exclusion of these data significantly limits any interpretation of the effectiveness of a behavioural change procedure. Validity requirements in a behavioural assessment study should therefore be conditional on the inclusion of insurance data (Friman, 2009). In light of these considerations, it is not surprising that a recent review of articles in the Journal of Applied Behavior Analysis (JABA) from 1995 to 2005 (Mudford, Taylor, Martin, 2009) revealed that 100% of articles reporting continuously recorded dependent variables contained IOA calculations. These data, as well as previously published reports on reliability procedures in JABA (Kelly, 1977), suggest that the inclusion of IOA is in fact a trademark – if not a standard – of behavioural evaluation. 1 Note that this calculator is based on examples of IOA (Improving and Assessing the Quality of Behavi measurement) data, presented in Chapter 5; 102-124) was tested by Cooper, Heron and Heward (2007). For all algorithms, there was a 100% match between the values derived from the IOA using the calculator described in this article with those reported in Cooper et al. House, A.
E., Farber, J. and Nier, L. L. Accuracy and Speed of Reliability Calculation with different dimensions of the Interobserver agreement. Lecture at Postersession, Association for Advancement of Behavior Therapy, New York, November 1980.