Tests of Information Processing Speed

Cognitive impairment is common in multiple sclerosis (MS), affecting 40% to 65% of patients and contributing to functional limitations that result in poor health-related quality of life (HRQOL).^{1 2} The predictive value of cognitive dysfunction in HRQOL remains even after controlling for subjective distress variables such as depression and fatigue.³ The most commonly identified cognitive difficulties associated with MS include problems with working memory, as well as learning and memory for new information.^{4 5} Information processing speed has also been identified as a primary cognitive deficit in MS.^6–8 Cognitive difficulties can lead to problems for people with MS in their daily lives, such as their ability to maintain employment.^{1 9} Such deficits can also negatively affect their relationships with intimate partners, family members, caregivers, coworkers, and their health-care team.¹⁰

A challenge with any type of cognitive assessment is selecting measures that possess good psychometric properties (ie, reliability, validity, sensitivity to impairment, specificity to the types of impairment encountered in MS). While it is important for clinicians to select tests that do not cause stress for the individual completing the test, it is also important to ensure that even subtle cognitive deficits are detected. Thus, some measures that are designed to be quite challenging to complete must also be included.

One such task that has been reported anecdotally to be unpleasant, both for the patients undergoing it and for the clinicians administering it, is the Paced Auditory Serial Addition Test (PASAT).^{11 12} In this test, patients listen to a time-spaced series of single-digit numbers (from 1 to 9) and add each number to the previous number presented. The speed with which the patient must process information during this task can be manipulated by varying interstimulus intervals (ISIs). The PASAT has consistently been shown to be sensitive to the deficits in information processing speed (IPS) and working memory often observed in individuals with MS.^{13 14} Given its efficacy in identifying cognitive impairment as well as its ease of administration and cost-effectiveness, the 3-second ISI version has been included as the only cognitive measure in the widely used Multiple Sclerosis Functional Composite (MSFC), a now frequently used outcome measure in clinical trials.¹⁵ Both clinical and research staff in MS clinics are becoming more familiar with this instrument, and clinic nurses are often called upon to administer it to their patients or research participants.

Despite its efficacy in identifying impairment, the PASAT has been found to have a number of problems. Performance is confounded by prior exposure, age, education, and mathematical ability, and the test has been documented to induce anxiety, sadness, and hostility in those who undergo it.^{16 17} This effect may correlate with lower performance,¹⁸ so that the score achieved may not solely reflect underlying cognitive impairment.

Several authors have proposed using the Symbol Digit Modalities Test (SDMT)^{12 19} as an alternative measure to the PASAT. The SDMT measures IPS in the visual modality. Patients are presented with a key that includes nine numbers, each paired with a different symbol. Below this key is an array of these same symbols in pseudorandom order paired with empty spaces. Patients must then provide the correct numbers that accompany the symbols as indicated in the key. Both written and oral response versions are used, but the oral version is primarily recommended for use with people with MS so as to minimize the influence of motor deficits. The SDMT takes only a few minutes to administer (including task instructions), and thus some have argued that it is especially suitable for use as a brief screening tool.²⁰ The SDMT has been demonstrated to be a viable tool to replace the PASAT in the MSFC given that it possesses slightly better predictive validity and is easier to administer.²¹ This may render the PASAT unnecessary.

A more recently developed tool to evaluate IPS deficits in MS is the Computerized Test of Information Processing (CTIP).²² This is composed of three reaction time (RT) measures of increasing complexity. The first task is a simple reaction time (SRT) task that involves the patient pressing a key in response to seeing an “X” on the computer screen. The SRT measurement represents motor speed and can be separated out from an individual's score in order to remove this motor component as a confounding variable from the other RT measures. The second is choice reaction time (CRT). The patient is presented with one of two words and must press one of two keys in response. The additional decision component of the CRT task increases the cognitive load. Finally, the third task, semantic search RT (SSRT), further increases the cognitive load by requiring more complex processing, with the patient having to consider the meaning of the stimulus in order to make an appropriate decision as to whether a stimulus is a member of a presented semantic category (eg, UTENSIL: chair, correct response is NO versus UTENSIL: spoon, correct response is YES). Each of the three tasks (SRT, CRT, SSRT) contains three blocks of ten trials each, for a total of 30 trials in each task. The CTIP is presented by computer, and total administration time is approximately 15 minutes. Unlike the PASAT, the CTIP is not influenced by an individual's mathematical ability, is less frustrating, and is free from practice effects.^{23 24}

In the current study, each of these three measures of IPS was administered in the context of a larger neuropsychological battery. In addition to the neuropsychological measures, subjects completed a questionnaire designed to assess their subjective opinion of the tests they underwent. Subjects rated the three identified measures using a 6-point Likert scale in three ways: likeability, difficulty, and appropriateness for measuring perceived MS-related cognitive deficits. The goal of the current study was to determine how people with MS and healthy controls rated the PASAT, CTIP, and SDMT on these three dimensions. It was expected that the PASAT ratings would differ significantly from those of the SDMT and CTIP, consistent with the accumulated evidence of negative perceptions of the PASAT. We also investigated the relationship between subjective ratings and objective performance on these measures to determine whether a person's performance on a test influences his or her subjective experience of the task.

Methods

Subjects

Sixty-nine individuals with a confirmed diagnosis of relapsing-remitting MS were recruited from the MS clinic of the Ottawa Hospital (MS group). These individuals all had a mild level of physical disability (ie, an Expanded Disability Status Scale [EDSS] score of under 6.0) and a disease duration of less than 10 years (mean [SD], 4.41 [3.03] years). Sixty-eight age-, education-, and IQ-matched healthy controls (control group) were recruited by word of mouth from the community (family [no first-degree relatives], friends, etc.), as well as newspaper and website advertisements. All participants were fluent in English, were between the ages of 18 and 65 years, and had between 12 and 21 years of education (Table 1). All participants were free from previous neurologic, medical, or psychiatric illnesses (besides MS) that may have impaired their cognitive performance.

Demographic variables

Procedure

The study was approved by the Ottawa Hospital Research Ethics Board. After informed consent was obtained, subjects completed a demographic questionnaire. The North American Adult Reading Test (NAART) was administered as an estimate of premorbid intellectual ability, with participants required to achieve an estimated IQ score of at least 90 to be eligible, to ensure that any cognitive deficits were not due to premorbid intellectual limitations. Subjects completed the PASAT (3-second and 2-second), the SDMT, and the CTIP in the context of a larger battery of neuropsychological tests. After undergoing all cognitive measures, subjects completed a newly developed self-report questionnaire (Test Acceptability Questionnaire) that records subject perceptions of tests using a Likert scale in each of three categories, with identification of the “best” and “worst” test in each category. The three categories were based on likeability, perceived difficulty, and appropriateness for capturing cognitive deficits of MS. An abbreviated form of the questionnaire is shown in Figure 1. The full form of this questionnaire recorded subjective opinions of all tests in the neuropsychological battery. For the purposes of this article, we focused only on the measures of IPS (PASAT, SDMT, and CTIP).

Test Acceptability Questionnaire

Data Analysis

A mixed-factor, two-way analysis of variance (ANOVA) was performed, comparing the MS and control groups on the ratings (likeability, difficulty, appropriateness) across all three tests (PASAT, SDMT, CTIP). This was followed by analyses of simple effects. Finally, Pearson correlations were calculated between subjective ratings (on each of the three domains) and objective performance on each of the three tests (PASAT, SDMT, CTIP). Given the multiple comparisons, a Bonferroni correction was applied.

Results

A significant three-way interaction (ie, group vs. rating vs. test) was found. Thus, the way in which the two groups (MS vs. control) rated the tests according to likeability, difficulty, and appropriateness differed between the three tests (PASAT, SDMT, CTIP). In order to facilitate interpretation of this interaction, simple effects analyses were performed. Performance data are shown in Table 2, and subjective ratings in Table 3.

Test performance

Subjective ratings

Group Differences

The MS group and the control group generally did not differ in their ratings of the three tests on the three different dimensions. The sole exception was with regard to the appropriateness dimension. The MS group rated the PASAT as more appropriate for measuring MS-related cognitive deficits than did the control group (F _1,134 = 6.57; P = .011). The mean (SD) rating of the MS group was 4.82 (1.29) (ie, closer to “moderately well” at measuring MS-related cognitive deficits), whereas the mean rating of the control group was 4.18 (1.61) (closer to “somewhat well”).

Likeability

Both the MS and control groups rated the SDMT higher on likeability than the PASAT (MS: t = 7.14, P < .0001; control: t = 5.95, P < .0001; Figure 2). Both groups also rated the CTIP as more likeable than the PASAT (MS: t = 6.36, P < .0001; control: t = 4.05, P < .0001). Only the control group rated the SDMT as more likeable than the CTIP (t = −2.33; P = .023), with the MS group rating these two measures as equivalent in likeability.

Mean ratings of likeability (1 = strongly dislike; 6 = strongly like)

Difficulty

Both the MS and control groups rated the PASAT as more difficult than both the SDMT (MS: t = 13.79, P < .0001; control: t = 15.34, P < .0001) and the CTIP (MS: t = 18.69, P < .0001; control: t = 22.47, P < .0001; Figure 3). Both groups also rated the SDMT as more difficult than the CTIP (MS: t = 4.50, P < .0001; control: t = 6.36, P < .0001).

Mean ratings of difficulty (1 = very difficult; 6 = very easy)

Appropriateness

The control group did not report any test as being more appropriate for measuring MS-related cognitive deficits than any other test (Figure 4). All tests were subjectively considered to measure these deficits “somewhat well” to “moderately well.” In contrast, the MS group found both the PASAT (MS: t = −2.73, P = .008) and the SDMT (MS: t = −2.80, P = .007) to be more appropriate for measuring MS-related deficits than the CTIP. However, the PASAT and SDMT were rated as equivalent in appropriateness.

Mean ratings of appropriateness for measuring MS-related cognitive deficits (1 = very poorly; 6 = very well)

Subjective and Objective Performance

Table 4 shows the significant correlations found between objective performance on the three tasks and subjective ratings of these same tasks on the three dimensions measured. Better performance on the PASAT was associated with higher ratings of likeability for the control group but not for the MS group. In addition, subjects in both groups who performed better on the PASAT were more likely to rate it as easier. The only significant correlation for the SDMT was with regard to control group ratings of difficulty, with better performance on the task associated with lower perceived difficulty.

Correlations between subjective ratings and objective performance

Discussion

The primary goal of this study was to determine how people with MS and healthy controls rated the PASAT, CTIP, and SDMT in terms of their perceived likeability, difficulty, and appropriateness for measuring perceived MS-related deficits. It was hypothesized that the PASAT ratings would differ significantly from those of the SDMT and CTIP, consistent with the accumulated evidence of negative perceptions of the PASAT. The hypothesis was confirmed. Both the MS and control groups rated the PASAT as less likeable and more difficult than both the SDMT and the CTIP. Thus, a negative perception of the PASAT seems to be unrelated to the presence of disease. Given that sensitivity to cognitive deficits in MS has already been demonstrated for both the SDMT and CTIP,^{21 23 24} subjecting individuals to the PASAT may be unnecessary.

The results with respect to appropriateness revealed that only people in the MS group found the PASAT (2-second only) to be more appropriate if they performed well on that measure. The clinical significance of this finding is unclear. The appropriateness dimension essentially measures the face validity of the tests (ie, how subjects rate the test based on what they think occurs in MS). The expectation that subjects' perceptions will be accurate presumes that they have a clear understanding of what MS-related deficits are. However, this is not necessarily the case. It is widely believed that people with MS who are cognitively impaired are in fact less able to estimate their own cognitive abilities at any given time point than are those who care for them.^25–28 Therefore, the ratings of appropriateness obtained in the current study should not be assumed to accurately represent how appropriate a task is for measuring MS-related deficits (all three tests have been demonstrated to be sensitive to MS-related impairment).^{13 14 21 23 24} Rather, the ratings reflect the individual perceptions of those undergoing the tasks.

Turning to the findings in relation to the SDMT and the CTIP, both groups found these measures to be more likeable and less difficult than the PASAT. Both groups found the CTIP to be easier than the SDMT, although the control group found the SDMT to be more likeable. As noted above, both measures have been demonstrated to be sensitive to MS-related cognitive deficits. However, the two measures have not yet been studied together to determine whether one is more sensitive to impairment (or change in cognition over time) than the other. Our research group is currently working on this issue.

There were almost no differences in ratings between the MS and control groups with regard to their subjective perceptions. Having MS did not generally have an impact on how the tests were perceived. The one exception was that the MS group rated the PASAT as more appropriate for measuring MS-related cognitive deficits than the control group. However, when one examines the mean (SD) ratings for each group (MS: 4.82 [1.29]; control: 4.18 [1.61]), the statistically significant difference may not be of sufficient magnitude to be clinically significant. The MS group's ratings of the PASAT were closer to “moderately well” in capturing MS-related deficits, whereas the control group's ratings were closer to “somewhat well.” The difference of less than 1 point on a 6-point Likert scale does not suggest that opinions between the two groups differ to any considerable degree.

The relationship between subjective ratings and objective performance on these measures was examined to determine whether how well a person performs influences his or her subjective experience of the task. No consistent relationship was demonstrated between subjective ratings and objective performance across tasks, particularly with regard to the domains of likeability and difficulty. Of the three tests administered, the PASAT was most likely to yield positive correlations between difficulty and performance. Those who performed well were more likely to rate the test as easier. Perhaps these individuals were less susceptible to the negative influence of factors that have been identified in the literature as potentially affecting performance (eg, math anxiety). The fact that the SDMT and the CTIP were less likely to show a relationship to performance suggests that subjects completing these tasks are less aware of how they are performing. This makes intuitive sense. One often knows when one is struggling with a math calculation; however, one may be less aware of how many symbol matches are typically achieved or the typical response time to stimuli on a screen. With the SDMT and CTIP, poor performance is less obvious to the person completing the task. Presumably, then, these tasks are less likely than the PASAT to induce negative feelings.

The Test Acceptability Questionnaire used in the current study to evaluate subjects' perceptions of the tasks completed is a qualitative measure developed by our research team and has not been validated elsewhere in the literature. Our research team is pursuing further work with this instrument in this ongoing longitudinal study, and data on test-retest reliability and other psychometric properties will be available in the future.

Conclusion

The current findings clearly document that when assessing IPS in a neuropsychological battery, the SDMT and CTIP are preferred over the PASAT. The SDMT and CTIP are both rated as more likeable and less difficult than the PASAT, and these ratings have little to do with actual performance on these tasks. It seems unnecessary to subject people to an unpleasant test such as the PASAT given the existence of other measures that have been demonstrated to be just as sensitive in detecting IPS. Although the PASAT was established as part of the MSFC, the SDMT in particular has been shown to be a viable alternative. Substitution of the SDMT for the PASAT in the MSFC not only is feasible but would be welcomed by the individuals being assessed.

PracticePoints

Information processing speed (IPS) deficits are common in individuals with MS.

The Paced Auditory Serial Addition Test (PASAT) is a widely used measure of IPS (and working memory) in MS, but anecdotal reports suggest that patients are averse to it, and there is evidence of confounding factors that may affect performance.

Of the three measures of IPS administered here—the PASAT, the Symbol Digit Modalities Test (SDMT), and the Computerized Test of Information Processing (CTIP)—the PASAT was reported as the least likeable and most difficult.

Given that the SDMT and CTIP are similarly sensitive to cognitive impairment in MS, these measures may provide a more palatable alternative to the PASAT.