Inter-Rater Agreement on the Scale Developed for Identification of Autism Spectrum Disorder in Early Childhood

Introduction

Autism spectrum disorder (ASD) is a neurodevelopment disorder which presents as delayed social communication skills and restricted repetitive pattern of behaviour.1 The symptoms of the autism generally present at an early age and are predominately seen in the second year of life. Early identification and diagnosis of ASD is highly beneficial as it facilitates parents to access the early intervention which can lead to improved child’s quality of life.² The children with ASD have co-morbid emotional or behavioural difficulties which might be the primary concern of both parents and healthcare workers, and these need to be addressed while performing the assessment of young children with any behavioural challenges.² The screening is the most important aspect of assessment while deciding any intervention, as prevalence of autism has increased to 3 per 100,³ indicating the need for early screening of children with any behavioural issues. The screening and identification of ASD at an early age will help to diagnose the condition, thereby facilitating early intervention. Various factors such as inadequate training on screening, lack of knowledge, lack of time at the outpatient department affect the screening for ASD in a clinical setting. This can be accounted for the lack of change in the average age at which the ASD can be picked up.⁴

It has been noted that around 10% of parents do not show any concern if their child has ASD and approximately 50-70% of the professionals rely on clinical presentation alone, instead of using validated tools for diagnosis of ASD. These issues pose a greater challenge in the identification of ASD.⁵

In the clinical setting, various scales are available to aid in the diagnosis of the children with ASD.^6,7 Recently published screening tool by Thomas et al. is a simple tool that blends with the socio-cultural milieu of our geography.⁸ Evidence of repeatability by various healthcare providers will enhance the confidence in this scale for general use. The present study was planned with an aim to achieve interrater agreement on a newly developed scale for identifying ASD in early childhood. The scale is comprised of 40 items which were extracted from the 120 pooled items with each component extracted based on the sensitivity and specificity for developing a new scale. The scale included items related to cognitive, behaviour and communication components.⁸

Objective

To estimate the inter-rater agreement on a newly developed scale for identification of ASD.

Materials and Methods

Study design

It was a prospective study conducted in a tertiary care hospital setting.

Subjects

Children who were already diagnosed with ASD and under treatment at different centres like JSS Sahana School, District early intervention centre, District disability centre and tertiary health care centre.

Intervention

Two independent raters working with children with special health care needs used the investigational screening tool. One of the rater was a clinical psychologist and the other was a physiotherapist.

The first rater administered the screening tool and scored the items. Forty-eight hours later, same children were reassessed by the second rater at a different location using the investigational screening tool. Both the raters were blinded regarding the diagnosis. Once the scoring was done, the data was deposited with the investigator with coded identities

Demographic and baseline clinical data were collected by the investigator.

Outcomes

The basic outcome of the study was the inter-rater agreement on the individual items of the developed scale, which contained 40 different items. The secondary outcome was the intra-rater agreement on the complete scale.

Sample size

Sample size was calculated based on the formula to use for a binary outcome for the agreement.

N = [Z_α P_o (1 - P_o )]/δ² (1- P_e )² ]

where Z_α is 1.96 for an α error of 0.05, P_o is the observed proportion (by pilot study), P_e is the expected proportion and δ is the width of 95% confidence interval (CI) of Kappa statistic κ. Based on our pilot study, we considered P_e as 0.5, P_o as 0.7, and a δ of 0.1. The sample size was estimated as 45.

Results

A total of 45 pairs of assessments of children who met eligibility criteria were included in the study. The demographic details are described in Table 1

A total of 45 children were included, among which 32 were males and 13 were females with a mean age of 62.13±1.6 months. The children were predominantly from urban domicile. The data of both raters were analysed using kappa statistics to determine the interrater agreement. Both the raters demonstrated very good inter-rater agreement with K statistic varying between 0.683 to 1 and P value was 0 for each item of the scale assessed (Table 2).

The component one of the scale had an inter-rater agreement of 0.683 indicating good inter-rater agreement and components from 2nd to 40th were between 0.818 to 1 which indicated very good inter-rater agreement with P value of 0. The overall agreement on the scale was 0.8813 which indicates good inter-rater reliability.

Discussion

The purpose of the current study was to estimate the inter-rater agreement on the scale developed to identify children with autism spectrum disorders in early childhood. The sample size was adequate to analyze the rater agreement. The evaluation was done by two raters from different disciplines working with children with special needs. The higher level of rater agreement indicates that the scale provides the same information with good sensitivity, while a lack of rater agreement suggest that the scale may not have good specificity or it may suggest that the two raters might have provided varying information.

Our study results demonstrated a very good inter-rater agreement on the scale among all the 40 items. The scale included 40 items pooled from different sections such as visual, hearing, olfactory, oral (taste), tactile, repetitive and restricted interest, persistent preoccupation with parts of objects, socio-communication, play behaviour, sensory integration, behavioural difficulties not specific to autism, major physiological functions, associated medical conditions/ disorders.

The participants in the present study were 45 children with ASD in the age range of 62.13±1.6 months (5 years ± 4 months). In this study, we assessed the interrater agreement and excellent agreement was observed between the two raters, implying that the scale can be used for further studies.

While performing the assessment, the raters faced certain issues such as conflicting statements made by mothers during the assessment for the questions. The items in the scale can help in identifying some aspects of Attention deficit hyperactivity disorder (ADHD) and intellectual disability as the item description is closely related to these conditions. However, further studies are required to confirm the items that can assist in the identification of ADHD and intellectual disability.

Maddox et al., (2021) reported that PARS – ASD (paediatric anxiety rating scale modified for autism spectrum disorder) demonstrated good to excellent interrater reliability and also showed that the values are not vulnerable to random fluctuation which is important to detect the treatment changes in the individuals with ASD.⁹

The recent research on early identification of ASD has strong focus on motor and behavioural responses. The children diagnosed late with ASD may have early deficits in the social behaviour, attention and tolerance to touch.⁵

Conclusion

In general, the study findings revealed strong consensus among the raters for all 40 items on the scale. The scale exhibited consistent information and high sensitivity. However, instances of disagreement between raters may indicate a potential lack of specificity in the scale or variations in the information provided by the two raters.

Conflicts of interest

None