Screening Instruments for the Early Detection of Cognitive Impairment in Patients with Multiple Sclerosis

Multiple sclerosis (MS) is a demyelinating disease of the central nervous system that affects cognitive and physical functioning.^1–4 Cognitive impairments affect 45% to 65% of patients with MS.³ Over time, changes in a patient's cognition can have greater effects on the patient's life. Early screening to detect impaired cognition, followed by a comprehensive neuropsychological evaluation, could allow for earlier treatment,⁵ better education, and counseling for individuals with MS and their families.

Screening for cognitive impairments has relied on rating scales completed by patients and their informants, along with brief standardized tests. Currently available brief standardized tests include the Neuropsychological Screening Battery for Multiple Sclerosis and the Screening Examination for Cognitive Impairment.⁶ The Brief International Cognitive Assessment for Multiple Sclerosis has recently been recommended as a less time-consuming option.⁷ Although relatively brief in administration, these batteries still take 15 to 30 minutes to administer, require training,⁶ and do not include self-report measures, which can be important predictors of functional outcome.⁸ In contrast, the Symbol Digit Modalities Test (SDMT), which typically measures processing speed,⁹ takes only 5 minutes to administer. Because processing speed is a primary deficit in individuals with MS, the SDMT may be a particularly good candidate for cognitive screening purposes; previous studies have found it to be effective.¹⁰ Similar to the batteries mentioned previously herein, however, the SDMT does not provide information on real-life outcomes.

The Multiple Sclerosis Neuropsychological Screening Questionnaire (MSNQ) was developed to screen patients quickly for cognitive impairment in everyday activities. The results of the self-report version, however, were more related to measures of depression.¹¹ In addition to depression, other factors can influence subjective cognitive concerns.¹² Levels of anxiety, fatigue, and self-efficacy¹² have been found to shape perceived cognitive abilities, which suggests that attitudes, beliefs, and expectations regarding cognitive functioning are important variables that need to be addressed in treatment.⁸ Particularly, perceptions of cognition and depression tend to overlap, as depression can influence one's perception of cognitive difficulties.² In contrast to the self-report version of the MSNQ, the informant version of the MSNQ had a significant relationship to objective testing. Not all patients, however, will have an informant available to accompany them to their clinic visit, and some may not have an informant at all. Because questionnaires mailed to informants cause delays, we evaluated two alternative self-reports of cognition. The objective of this study was to identify the most accurate and efficient means to screen for cognitive impairment in patients with MS by comparing various self-reports, both with each other and with the SDMT. Raw scores for the SDMT were used so that clinic personnel could readily administer the test.

Methods

One hundred fifteen consenting participants with a definite diagnosis of MS were recruited from an MS clinic in New Jersey. The study was approved by the institutional review board of Albert Einstein College of Medicine (Bronx, NY). Neuropsychological testing was conducted by a postdoctoral fellow in the clinic. Seven patients failed the effort test (Test of Memory Malingering¹³ or the Forced Choice of the California Verbal Learning Test, Second Edition¹⁴) and were excluded. Eight additional participants were excluded because they were unable to complete all the study questionnaires in the allotted time. Descriptive statistics of the variables of interest were similar for completers versus noncompleters. Table 1 presents the demographic data of the final sample. All the participants completed an evaluation consisting of a clinical interview, self-reports, informant reports, and neuropsychological testing.

Characteristics of the 100 study participants

Measures

The Cognitive Index: Composite Z Score Index Derived from the Minimal Assessment of Cognitive Function in Multiple Sclerosis

All the patients were given the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS), a 90-minute battery of tests validated for individuals with MS.¹⁵ The MACFIMS includes the following: Controlled Oral Word Association Test¹⁶; Judgment of Line Orientation Test¹⁶; California Verbal Learning Test, Second Edition¹⁴; Brief Visuospatial Memory Test–Revised¹⁷; Paced Auditory Serial Addition Test¹⁸; SDMT⁹; and Delis-Kaplan Executive Function System Sorting Test.¹⁹ Scores from the MACFIMS were converted to z scores; 12 of 13 measures of the MACFIMS were averaged to derive a single metric. The composite z score was termed the Cognitive Index (CI) and represented patients' overall cognitive function. (Because the Brief Visuospatial Memory Test–Revised Recognition Trial has a low ceiling—the highest score that can be attained is greater than the 16th percentile—the score for this measure was not included in the CI.) Table 2 presents the patients' performance and frequency of impairment on the MACFIMS. Impairment was defined by the CI as 2 or more test failures, using cutoff values of less than or equal to −1.5 and −2.0 SD below the mean. A dichotomous measure of cognitive impairment, based on the CI (pass/fail), was defined by a criterion set at either −2.0 or −1.5 SD below the mean. This CI was also used in the analysis as a continuous variable.

Performance on the MACFIMS: frequency of failures (N = 100)

The SDMT is part of the CI; therefore, an inflated correlation with the CI is expected if the SDMT is not removed when calculating the CI. Because the SDMT is a measure of processing speed (a primary deficit in individuals with MS²⁰), it was not removed from the CI in analyses that involved only the self-reports as predictors of impairment. However, all analyses that included the SDMT as a predictor were performed with the SDMT excluded from the CI.

Neuropsychological Test

One neuropsychological test, the SDMT, was used as a screening tool for comparative evaluation. It typically measures processing speed⁹ and is one of seven tests that compose the MACFIMS. Because this test takes only 5 minutes to administer and is effective for screening purposes,¹⁰ it was a good candidate for determining whether this single test could accurately correspond to the derived CI. Raw scores were used to make it easier for nurses to administer and interpret results. Standard scores, adjusted for age and education,²¹ were analyzed separately to confirm that any clinically significant findings would not be affected by their use.

Neuropsychological Self-reports

Patients completed self-report measures related to self-appraisals of neuropsychological functioning.

The MSNQ–Patient Version. The MSNQ–Patient Version (MSNQ-P) is a 15-item questionnaire, validated for the MS population,²² of cognitive and neuropsychiatric dysfunction. Patients rate themselves from 0 (never; does not occur) to 4 (very often; very disrupted) on specific cognitive and behavioral problems that may arise in daily life. For example, they may be asked, “Do you lose your thoughts while listening to somebody speak? Do you have difficulty controlling your impulses?”

Behavior Rating Inventory of Executive Function–Adult Version, Self-Report. The Behavior Rating Inventory of Executive Function–Adult Version (BRIEF-A) captures individuals' views of their own executive functioning (see the study by Roth et al.²³ for psychometric properties). It contains 70 items and yields an overall score (Global Executive Composite) that combines two indices: the Metacognitive Index (MI) and the Behavioral Regulation Index (BRI). The MI (regulation of cognition, eg, I lie around the house a lot; I forget what I am doing in the middle of things) and the BRI (regulation of behavior, eg, I have problems waiting my turn; I have emotional outbursts for little reason) were recently validated for individuals with MS.²⁴ Because one aim of the study was to identify a screening tool that clinic personnel could administer and interpret, raw scores were used, and only the MI and BRI summary indices are reported.

Problem-Solving Inventory. Consisting of 32 items rated using a 6-point Likert format (see the manual by Heppner²⁵ for the psychometric properties), the Problem-Solving Inventory (PSI) is a self-appraisal measurement of how well people believe they can handle problems. The PSI probes individuals' global beliefs about everyday problems and their perception of their problem-solving ability.²⁶ In this regard, the PSI has been interpreted as an indicator of confidence in cognitive abilities and self-efficacy in managing real-life problems.^{8 27} A total score is derived from three subscales: problem-solving confidence (PSC; self-assurance, belief and trust in problem-solving skills), approach-avoidance style (tendency to avoid or withdraw from problems), and personal control (belief that one is in control of emotions and behavior when solving problems). Higher scores reflect a negative self-appraisal. The PSI was added to the study because Rath et al.²⁷ demonstrated that the way in which an individual self-appraised his or her capabilities, using the PSI, accounts for a significant proportion of the variance in community functioning after traumatic brain injury. The PSI was added late in the present study, and, thus, only 41 participants completed this measure; descriptive statistics, however, summarized for completers versus noncompleters of the PSI, and a multivariate analysis of variance of the two groups indicate that the two groups were not significantly different. In this study, the adolescent version of the PSI was used.

Self-reported Mood: the Beck Depression Inventory–II. The Beck Depression Inventory–II (BDI-II) is a 21-item multiple-choice questionnaire that measures the presence and severity of depressive symptoms. Each item is scored from 0 to 3 in terms of intensity, with total scores ranging from 0 to 63.²⁸ The BDI-II is widely used in MS research and shows high internal consistency in psychiatric clinic outpatients (Cronbach α = 0.92)²⁸ and in the MS population (Cronbach α = 0.86).²⁹ The BDI-II was added as a covariate in this study to investigate the role of depression in objective cognitive performance and self-reports of cognition for individuals with MS.

Statistical Analysis

Statistical analyses were performed using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp, Armonk, NY). Descriptive statistics were used to determine whether the data met the necessary assumptions for parametric statistical analysis. Collinearity diagnostic tests were performed. Receiver operating characteristic (ROC) curve analyses were performed to test each of the self-report measures and the SDMT for classification accuracy in terms of identifying cognitive impairment based on the criterion-determined dichotomous outcome measure of the CI. Hierarchical linear regression analyses were conducted to determine predictors that best explain impairment as expressed by the continuous measure of the CI. The PSI was excluded from this set of analyses because of the small subset of the sample that completed this measure. A separate, secondary analysis was performed to explore the linear relation between the PSI and the CI.

Results

A visual inspection of the frequency histogram and the P-P plot for the outcome measure (the CI) confirms that the data were normally distributed. Diagnostic statistics to assess collinearity (variance inflation factor and tolerance) demonstrated that multicollinearity was not a problem with the given set of predictors, and application of the Durbin-Watson statistic showed independence of serial residuals. Table 3 presents the descriptive statistics of the outcome measure, five predictor/classification variables, and a covariate (BDI-II); Table 4 presents the Pearson correlations among these variables. Also examined were the linear relations between depression and self-reports, which were moderately strong (BDI-II/BRIEF-A BRI: R ² = 0.32; BDI-II/BRIEF-A MI: R ² = 0.39; BDI-II/MSNQ: R ² = 0.43). The more depressed the patients were, the more negatively they appraised their cognition. The CI was not significantly related to depression (R ² = 0.03). There was also a strong positive linear relation among all the self-reports (BRI/MSNQ-P: R ² = 0.48; BRI/MI: R ² = 0.52; MI/MSNQ-P: R ² = 0.58). The PSI was excluded from this intercorrelation analysis because it was not included in the hierarchical regression analysis on grounds of the small number of 41 patients associated with this measure. When a separate correlation analysis with the PSI and the other measures was performed, however, the results showed that the PSI scales were mostly significantly correlated with the other measures with moderate coefficients (Table 5).

Descriptive statistics of variables used in the analyses

Intercorrelations among predictor variables, depression, and the CI

Intercorrelations among PSI scales, predictor variables, and depression (SDMT included in the CI)

Impairment Classification and the Accuracy of Self-reports

According to ROC curve analyses with the CI criterion set at −2.0 SD below the mean, none of the self-report measures were significant predictors of the CI (MSNQ-P: AUC = 0.607, P = .231; BRIEF-A BRI: AUC = 0.629, P = .149; BRIEF-A MI: AUC = 0.627, P = .125). With the CI criterion set at −1.5 SD below the mean, the MSNQ-P was significant, whereas the two summary indices of the BRIEF-A were just shy of significance. The ROC curve analyses of the SDMT (excluding the SDMT from the CI) were significant given both criteria. Classification accuracy, as measured by area under the ROC curve (AUC), was high for the SDMT when the criterion was −2.0 SD and was moderately high when the criterion was −1.5 SD. Specificity was high when the SD criterion was set at −2.0 and was moderately high when it was set at −1.5 SD. Moderate sensitivity was obtained for both criteria, with a somewhat lower cutoff score given the −2.0 SD criterion. Minimal change was observed regardless of whether the SDMT was included in the CI; excluding the SDMT from the CI resulted in a higher SDMT cutoff score (≤44 vs. ≤39) when the criterion was set at −1.5 SD. There was no significant difference when SDMT-transformed scores, as opposed to raw scores, were used in the analyses. Classification accuracy, sensitivity, and specificity were not high for any of the other measures. The PSI was not statistically significant for classifying impairment according to either criterion. Table 6 presents the diagnostic accuracy of the SDMT, MSNQ-P, and BRIEF-A.

Diagnostic accuracy of the SDMT, MSNQ-P, and BRIEF-A

Models to Predict the CI

Scatterplots of each predictor versus the outcome variable were reviewed. The CI versus the SDMT illustrates a moderately strong positive linear relation (R ² = 0.50, P < .001). Weak negative linear relations exist between the CI and the self-reports (CI/MSNQ-P: R ² = 0.05, P = .026; CI/BRI: R ² = 0.07, P = .008; and CI/MI: R ² = 0.03, P = .073).

Hierarchical linear regression analyses were performed to identify the best model for predicting impairment, with the SDMT being excluded from the CI. In this linear regression and the ones that follow, multicollinearity and independence of serial residuals were not a problem, as is evidenced by tolerance and the Durbin-Watson statistic. Theoretical considerations determined the order of entry. The literature underscores that behavioral and emotional control are fundamental to effective cognitive functioning.^{23 30}

The BDI-II was entered at the first step to statistically adjust for depression as a covariate (model 1). The SDMT was entered at the second step (model 2), followed by the self-reports (BRIEF-A BRI, MSNQ-P, and BRIEF-A MI). The SDMT alone emerged as a significant predictor of impairment: the BDI-II in model 1 yielded an adjusted R ² of 0.02 (P = .088). Adding the SDMT in model 2 yielded an adjusted R ² of 0.50; this change in R ² of 0.48 was significant (P < .001). The self-reports did not provide significant explanatory power (ΔR ² of 0.01 to <0.01 in each subsequent model) (Table 7).

Hierarchical linear regression analyses for prediction of the Cognitive Index (SDMT excluded)

Hierarchical regression was also performed using standardized z scores of the SDMT, with similar results. The analysis conducted with raw scores provided somewhat greater explanatory power (adjusted R ² = 0.50 for raw scores vs. 0.44 for z scores in the models with the SDMT). When the SDMT was retained in the composite index, the explanatory power was only slightly greater than when the SDMT was excluded from the CI: adjusted R ² = 0.56 for raw scores versus 0.53 for z scores. For both conditions, the SDMT alone was found to best explain variance in the CI. One standard deviation change in the SDMT leads to a 0.70 SD change in the CI, as opposed to 0.74 when the SDMT is retained in the composite CI.

Secondary Analysis

Self-reports (MSNQ-P and BRIEF-A) as Predictors of the CI

Secondary hierarchical linear regression analyses were performed with only self-report measures as predictors. In the first analysis, the BDI-II was entered along with the constant in the first step (model 1), the BRIEF-A BRI was added in a second step (model 2), followed by the MSNQ-P and the BRIEF-A MI at steps 3 and 4 (models 3 and 4). Analyses of self-report measures demonstrated that the BRI was significantly, albeit weakly, related to the CI. The BDI-II in model 1 yielded an adjusted R ² of 0.02 (P = .095). Adding the BRI in model 2 yielded a ΔR ² of 0.042 (P = .04), and the coefficient on the BRI variable was itself significant (B = −0.019, SE = 0.009, P = .04). The remaining self-reports entered in subsequent steps did not provide additional explanatory power for the CI. In fact, the adjusted R ² decreased with each new entry (Table 8).

Hierarchical linear regression analyses for self-reports' ability to predict the Cognitive Index (SDMT included)

CI versus PSI

An additional secondary exploratory analysis was conducted with the CI and PSI scales. Multivariate analysis of variance was conducted to assess group differences between completers (n = 41) versus noncompleters (n = 59) of the PSI for the variables of age, education, and CI. The results demonstrated that there were no differences between the two groups (Wilks Λ = 0.98, F _3,96 = 0.61, P = .609). A scatterplot of the PSI confidence scale/PSC versus the CI revealed a moderate linear relation (r ² = 0.21, P = .003).

Hierarchical linear regression was performed with the CI entered as a criterion variable. The BDI-II was entered at the first step to statistically adjust for depression, followed by the PSI scales. The PSC, approach-avoidance style, and personal control subscales were then added individually to determine whether model fit improved at each step. As Supplementary Table 1, which is published in the online version of this article at ijmsc.org, shows, only the PSC subscale contributes to a significant increase in R ².

Discussion

This study investigated the relative contributions of subjective reports and a brief objective cognitive test for accurate screening of cognitive impairment in patients with MS. Various subjective instruments were compared with the SDMT. Hierarchical regression analyses indicated that the SDMT, which takes approximately 5 minutes to administer and can be administered without extensive training, was a better screening tool for identifying patients with cognitive impairment than were the investigated self-report instruments. The SDMT is part of the composite CI, and, thus, the SDMT was omitted from the CI in all analyses that had the SDMT as a predictor. Each time, the SDMT consistently emerged as the strongest screening instrument for cognitive impairment. The finding that the SDMT has a moderately strong relation to the CI provides further support for its use as a cognitive screen.

Consistent with previous studies, the present findings show the SDMT to be a reliable, accurate predictor of cognitive impairment in patients with MS. Previous studies, however, concluded this by defining impairment as a failure in performance, based on z scores,^{31 32} on two or more individual neuropsychological tests for each component of a neuropsychological test battery, specifically the Neuropsychological Screening Battery for Multiple Sclerosis or the Brief Repeatable Battery of Neuropsychological Tests. The present study used a novel index of cognitive function by computing a composite z score from the MACFIMS battery, which was designed specifically for the MS population.

Different definitions of impairment exist.^33–35 Cognitive impairment has significant negative ramifications at multiple levels. Finding a definition of impairment that could most accurately identify patients for a more comprehensive neuropsychological evaluation has significant importance. The CI is an outgrowth of Halstead's Impairment Index, in which seven tests were combined into a single index.³⁶ For Halstead, the brain was an organ of adaptation. As such, it was not considered accurate to separate out skills as measured by specific tests needed to perform a task; rather, function should be evaluated on all tests taken together. Effective behavior was viewed as ultimately tapping multiple facets of cognition.^{36 37}

Using the CI, a criterion of −1.5 or −2.0 SD below the mean was used to define cognitive impairment. Individuals were classified as having cognitive impairment based on a score of 44 or less on the SDMT (given a criterion on the average z composite of the MACFIMS of −1.5 SD below the mean) or a score of 36 or less (given a criterion of −2.0 SD). The SDMT performed better at identifying cognitive impairment given the more stringent criterion of −2.0 SD. The classification accuracy was 89% with sensitivity of 69%, indicating that the SDMT may serve as a valuable screening tool. It had specificity of 90.8%, indicating that it is better at ruling out those who do not have cognitive dysfunction versus merely detecting those who do have cognitive dysfunction.

Parmenter et al.'s research on the SDMT as a screening tool,¹⁰ which defines cognitive impairment as failure on at least two tests in the MACFIMS battery (−1.5 SD criterion) with the SDMT removed from the battery, found that an SDMT score of 55 or less resulted in classification accuracy of 72% (N = 100), with sensitivity of 82% and specificity of 60%. However, their findings had a false-positive rate of 40%. In contrast, the cutoff values for −1.5 and −2.0 SD below the mean of the CI yielded false-positive rates of only 16.7% and 9.2%, respectively. It is not clinically prudent to overidentify patients as having neuropsychological deficits, particularly given the expense of a full neuropsychological work-up and the anxiety and stress associated with the lengthy wait period before the work-up is administered.

For comparison, we applied Parmenter et al.'s method and likewise defined impairment as failing two or more MACFIMS tests at −1.5 SD or less. The results indicated that, given this approach, the optimal cutoff value for this sample is 48, not 55, indicating some difference between the two samples. Two other studies used the criterion of failing two or more tests at −1.5 SD or less in a neuropsychological battery to classify impairment, albeit different batteries than the MACFIMS. They examined the SDMT as a screening device and found criterion scores of 40³¹ and 49,³⁸ placing our criterion of 44 midway between the other criteria.

The intention of the SDMT as a screening tool is not to replace other available screening tools, which assess more than just the speed of processing and may, therefore, be better at detecting cognitive impairments. Rather, the intention is that the SDMT may serve as a useful screening tool when more comprehensive screening tools or staff resources are not available. For example, it can be administered by nurses rather than by neuropsychologists/neurologists.

The secondary analyses of this study, in which self-reports were examined in hierarchical regressions with the SDMT in the CI, showed that these various measures were not strong predictors of cognitive impairment. Of this set of predictors, the BRIEF-A BRI had statistically significant and greater explanatory power than did the BRIEF-A MI or the MSNQ-P, which were not statistically significant. A subsequent analysis of the PSI on a subsample of 41 individuals showed that the PSC subscale of this self-report instrument yields greater explanatory power than does the BRIEF-A BRI but was still a considerably weaker predictor than the SDMT.

With one notable exception,³⁹ research has consistently found that patient self-reports of cognitive abilities are confounded by their affect (depression, anxiety) and fatigue.^40–42 The present work, however, adjusted for depression by including the BDI-II as a covariate in the regression analyses and found that the BRIEF-A BRI and the PSC subscale of the PSI remained significant predictors. Thus, self-reports of cognitive dysfunction were not explained simply by depression.

Although subjective reports were less accurate screening instruments than objective cognitive measures, recent literature on subjective reports indicates that they may provide unique, complementary information⁴⁰ that informs the rehabilitation process.⁸ Patients' self-reports, specifically confidence in their ability to resolve everyday functional problems²⁷ or perceived self-efficacy for managing symptoms,^{43 44} can guide rehabilitation efforts, and “targeting objective deficits in cognitive remediation is necessary, but not sufficient.”^{8 (p326)} It may be useful to look at what other factors better account for the discrepancy between objective performance and self-reports (eg, mood, attitudes, beliefs, and expectations about cognitive functioning). In this regard, the results of this study and previously published work suggest that cognitive impairment screening may require a multidimensional approach that pairs objective neuropsychological tests with subjective self-reports of cognitive status and related factors.⁸

Although the SDMT was selected here as a screening tool candidate due to its brevity and ease of administration, other tests from the MACFIMS should be examined as well in future studies. Future research could further examine the value of the composite CI as a basis for classifying cognitive impairment compared with other definitions of impairment.

Conclusion

This study shows that a brief neuropsychological test, the SDMT, administered during a routine clinical visit, may serve as a valuable screening tool for cognitive impairment. In general, self-report measures were not found to be strong predictors of cognitive impairment. It is noteworthy, however, that those measures that self-appraised regulation of behavior and expectations about cognitive functioning were statistically significant predictors of objective measures of cognitive status. Such subjective reports may provide complementary information concerning functional status that is not adequately captured by objective neuropsychological tests. This possibility remains to be explored in future studies.

Two limitations are noteworthy. First, only 21% of patients were impaired on the SDMT, which is relatively low compared with other studies.^{10 45} Second, neither the duration of MS symptoms nor the participants' level of physical disability was reported, which, therefore, could limit the ability to compare the findings with those of other studies or to interpret them in a clinical context. Future research should use very large samples (thousands of patients), fully characterized, for better representativeness of the study sample and generalizability of results instead of the relatively low number of participants currently reported in most studies.^{2 10 31 38}

PracticePoints

The Symbol Digit Modalities Test, a 5-minute, easy-to-administer neuropsychological test, is shown to be an accurate screening tool for cognitive impairment (as defined by a composite score from a neuropsychological battery of tests) in patients with MS compared with self-reports of cognitive function.

Self-reports of cognition vary in their association with objective impairment; the Problem-Solving Inventory, which is designed to probe patients' beliefs and expectations (attitudes and affective reactions) about their own abilities to manage problems, attained the strongest association with objective cognitive status.

Objective neuropsychological tests and self-report measures of cognitive function provide different information concerning the individual's cognitive and behavioral status. The results of this study suggest that screening requires a multidimensional approach that should not rely on neuropsychological test scores alone.