First Video-Based Pragmatics Assessment Now Available Online

by Adriana Lavi, PhD, and Kristin Ferrell, PhD

The Clinical Assessment of Pragmatics (CAPs) is a norm-referenced video-based pragmatic language battery of tests for children and young adults ages 7 through 18 years. The CAPs, which offers an Online Kit that was released by the publisher WPS in May, is composed of six subtests. Each of the CAPs subtests is based on a well-defined pragmatic language construct. The CAPs is the first assessment tool of its kind to evaluate pragmatic judgment and performance using a series of video-based social scenarios. It is a reliable test that yields valid results on pragmatic judgment and the use of social language and nonverbal cues, such as facial expressions, prosody, and gestures. The CAPs test yields four types of scores: raw scores, scaled scores, percentile ranks, and composite and index scores to assist examiners in obtaining a comprehensive, pragmatic language profile.

One of the most notable benefits of the CAPs is its unique test design, which consists of videos that are true-to-life interactions. The videos are presented in relevant, life-like content, and the actors in the videos are from a wide variety of ethnic and cultural backgrounds. Verbal dialogue in the videos is easy to attend to and understand. It is presented at a rate that is controlled for speed without being unnaturally slow. The vocabulary used in the videos is appropriate to the testing age range (7 to 18 years), and the real-life situations are those which might be expected to occur in environments with which the participants could be expected to be familiar.

CAPs evaluates both examinees’ level of pragmatic judgment (meaning their ability to comprehend social situations), and their ability to express themselves in an appropriate manner within various social situations. The pragmatic performance aspect of this test is a crucial feature, which is unique because it allows the examiner an opportunity to elicit the participants’ both verbal and non-verbal responses. Beginning with ‘superficial’ layers of instrumental social situations, this test delves into every level of pragmatics, and assesses ‘intricate’ high-level skills, such as the examinees’ ability to express sadness, gratitude, frustration, support, and surprise, as well as their ability to use nonverbal language such as facial expressions and prosody.

A key area that may have been overlooked by traditional testing is the examinees’ use of higher-level pragmatic language, specifically the ability to use affective communication and paralinguistic (nonverbal) cues. For example, the Paralinguistic Decoding subtest is the most unique standardized measure that assesses the ability to use various non-verbal cues, such as facial expressions, tone of voice, inflections in prosody, gestures, and overall body language to express a variety of communicative intents. The CAPs is an effective means by which speech-language pathologists, as well as other related practitioners, can obtain a greater and comprehensive understanding of their examinees’ pragmatic language needs, such as awareness of basic social routines, the ability to read a variety of dynamic contextual cues and non-verbal language, the ability to use social routine language, and the ability to express higher level language, such as emotions and use nonverbal cues.

The CAPs uses a series of video-based social scenarios. Examinees are presented with a social situation in a video-based format and are asked two types of questions. On the pragmatic judgment (receptive pragmatic) subtests, the examinees are asked to judge the appropriateness of a variety of social situations by answering the following: “Did anything go wrong in this video?” and “What went wrong?” On the pragmatic performance (expressive pragmatic) subtests, the examinees are presented with a social situation and are asked: “Show me, what would you say and how?” The CAPs test can be administered with relative ease. The video content can be accessed via a CAPs memory stick or online. Scoring has been simplified by listing the scoring criteria and rubrics in the Examiner Record forms. A listing of the most common correct and incorrect responses is provided as well.

Researchers and practitioners have long argued for the need to develop pragmatic language assessments that target the unique social language characteristics of students with autism and pragmatic language impairment, such as higher level language expression, inferential thinking, and understanding the minds of others. Current means of assessing students who fall into this complex ‘grey area’ of higher-level pragmatic language ability have long relied on careful dynamic and informal observations and documentation. This comes with a major cost in time and labor to identify evidence to indicate that these students qualify for special services through public schools. However, even with careful dynamic observations and assessment, it is difficult to elicit skills within the suspected areas of weakness or those that the students’ caregivers and educators express concerns about. School-based observations that target social interaction and socialization are most often impossible or insufficient.

For example, it may be impossible to observe a student’s ability to express sorrow, affection, consolation, support, gratitude, etc., from school-based observations during students’ recess or lunchtime. The CAPs presents a viable testing method: a comprehensive test of pragmatic language ability that elicits responses through a set of video-based role plays of real-life situations. This method is not only able to evaluate students’ instrumental and “surface” pragmatic language skills, but it can be sensitive to the higher level pragmatic skills, such as understanding of and expression of facial expressions, body language or ability to appropriately express affective language. This test replaces the use of one-dimensional and static pictured stimuli of social situations with real-life scenarios presented in a video format. The nature of social interactions is dynamic, continuous, and fast. Pictures of social interactions are static and may not effectively elicit authentic responses, such as understanding of sarcasm, arrogance, etc. The use of real-life social situations is the closest method to elicit students’ ability to read dynamic contextual cues and nonverbal language.

In addition to assessing pragmatic judgment and pragmatic performance skills, this test differentiates social language as either instrumental or affective (non-instrumental) communication. In instrumental communication, the primary goal is to relay information effectively to the interlocutor, and where communication is used as a means to an end (i.e., communication is focused on benefitting the self). Focus is heavily emphasized on what is being said as opposed to affective or emotional functions. Because difficulty understanding others’ emotions and perspectives is a highlighted characteristic in individuals with autism and social communication disorder, instrumental communication is often used. This is critical in the assessment of such individuals. Affective communication involves higher-level communication skills, such as expressing emotions (i.e., joy or sorrow) to another person. It is a key component of non-verbal communication and also requires higher-level thought processing. ‘This metacognitive ability requires more than simply engaging in comprehension and production of pragmatically acceptable communication” (Carrow-Woolfolk, 1999).

The normative data for the CAPs test are based on the test performance of 914 examinees across 9 age groups (7 years, 0 months to 18 years, 11 months) in 15 states. The data were collected by 23 state licensed speech-language pathologists. To ensure representation of the national population, the CAPs test standardization sample was selected to match the US Census data reported in the ProQuest Statistical Abstract of the United States, 2017. The sample was stratified within each age group by the following criteria: gender, race or ethnic group and geographic region. A series of reliability and validity studies were conducted to include: internal consistency (ranging from .79 to .94), SEM analyses, interrater (ranging from .90 to .94), test-retest reliability (ranging from .80 to .95), and content, construct and criterion validity, as well as inter-correlations of the CAPs subtests. Clinical validity studies revealed sensitivity ranging from .90 to 1.0 and specificity ranging from .85 to .97. When comparatively analyzing scores of clinical groups (high functioning autism, social communication disorder) and typically developing examinees, comparisons using Mann- Whitney U test showed that there was a significant difference among all three study groups (p<0.001).

Caution should be taken when considering one’s performance on any standardized assessment. Examiners must use a variety of different assessments and strategies to obtain relevant functional and developmental information about an individual, including information provided by caregivers and teachers and information obtained from observations across a variety of contexts. Assessors must not rely on a single measure or assessment used as the sole criterion for making clinical decisions. Even though the CAPs test was designed carefully with extensive research and statistical analysis, results obtained on the CAPs test should be supplemented with a variety of other assessment standardized and informal measures such as systematic observations, contextual analyses, etc.

Adriana Lavi, PhD, is the author of the CAPs, and Kristin Ferrell, PhD, is a senior project director at WPS. The CAPs Online Kit was released in May 2020.

Related:

• 17 Assessment Terms and What They Mean
• Video: 7 Rating Scales Available Online Now
• Social-Emotional Learning: Why We Must Act Now
• Webinar: Pandemic Distress: Navigating the Course with Social-Emotional Learning
• Telepractice and COVID-19 resources
• 5 Tips for a Smooth Transition into Teletherapy and Assessment