Improving Detection of Change in Motor Functioning in Multiple Sclerosis Using Video-Assisted Composite Measures

Clinical assessment of patients with multiple sclerosis (MS) is notoriously difficult largely due to the heterogeneous clinical symptoms.¹ With the availability of increasing numbers of effective disease-modifying therapies and expanding treatment targets, selecting appropriate outcome measures is increasingly important.² For example, in a large phase 3 trial of patients with secondary progressive MS, the effect of natalizumab on progression was assessed using various measures. Although no treatment effect was found on Expanded Disability Status Scale (EDSS) score progression, a reduction in progression of upper extremity function (UEF) impairment was detected.³ The accurate assessment of disability is also relevant in the evaluation of symptomatic treatment options such as fampridine (also called dalfampridine).⁴ Fampridine improves walking speed and UEF in a subset of patients with MS, albeit patient-perceived improvement does not fully correspond with improvement measured using clinical instruments.^5–7

The EDSS is the most widely used outcome measure in MS trials, although it has several limitations (eg, high interrater and intrarater variability, disproportional impact of ambulation on the total score).^8,9 To improve the assessment of commonly affected functional domains in MS, the Multiple Sclerosis Functional Composite is used to assess ambulation (Timed 25-Foot Walk test [T25FW]), UEF (Nine-Hole Peg Test [NHPT]), and cognition.^10,11 Despite generally good psychometric properties, the individual components have several shortcomings.^12,13 The T25FW and the NHPT measure only certain aspects of ambulation and UEF (walking speed and manual dexterity, respectively), which do not fully capture the broader aspects of functioning. Also, the T25FW may not be sensitive enough to detect abnormalities in patients with mild ambulatory impairment.¹⁴ Because of these shortcomings, clinically relevant treatment effects might be missed.

Video-assisted assessment of motor functioning has the potential to improve disability evaluation. Recording patients’ movements on video may improve certain psychometric properties, such as intrarater and interrater variability, and promotes data quality by allowing a visual check on compliance relative to the standardization of performance.^15,16 Also, standardized activities of daily living (ADLs) can be added to the assessment and are valuable in measuring patients’ functioning.¹⁷

We investigated the additional value of video-assisted composite measures, consisting of classic neurologic tests and ADLs, in relation to conventional measures and aimed to enhance the detection of changes in mobility and UEF in patients with MS.

METHODS

Written informed consent was obtained from all patients before study participation, and the study was approved by the respective ethics committees. Data used in this study were part of a multicenter project to develop the ASSESS MS system, performed in 4 large European MS centers located in Amsterdam, the Netherlands, and in Basel, Bern, and Lucerne, Switzerland.^18,19 Patients with MS were recorded using a three-dimensional depth-sensing and color camera (Kinect, Microsoft Corp) with the aim of training machine learning algorithms to automatically quantify motor functioning.

Participant Recruitment

Patients were recruited at the Amsterdam University Medical Center, Vrije Universiteit Medical Center between August 1, 2017, and April 30, 2018. Patients initiating treatment with fampridine were eligible for study participation because fampridine can have positive treatment effects on mobility and UEF.^5–7 Inclusion criteria conformed to the official treatment label of fampridine, a diagnosis of MS according to the revised 2010 McDonald criteria,²⁰ age older than 18 years, ambulatory impairment defined as an estimated EDSS score between 4.0 and 7.0, and no additional diseases that contributed to disability. Contraindications as stated in the fampridine product label were handled as exclusion criteria.²¹ Patients were assessed at baseline before fampridine treatment and at follow-up after a minimum of 2 weeks of fampridine treatment. Patient and disease characteristics were collected during the baseline visit: age, sex, EDSS score,⁸ disease type, and disease duration.

Conventional Measures

The T25FW was used to measure mobility, and the NHPT was used to measure UEF. Each test was performed twice during each visit, as implemented in the Multiple Sclerosis Functional Composite.¹⁰ For each visit, 2 trials were averaged for each patient. A decrease of at least 20% in baseline walking time was used to indicate significant improvement in mobility function.^13,22,23 For the NHPT, trial results of the dominant and nondominant hands were averaged. Similar to the T25FW, a decrease in completion time of at least 20% between visits was used as the determinant of significant UEF improvement.²²

Video-Assisted Composite Measures

In addition to the conventional measures, the mobility composite (MOB-COM) consisted of 3 traditional neurologic tests and 4 ADLs (FIGURE 1A). The UEF composite (UEF-COM) consisted of 3 classic neurologic tests and 2 ADLs. The MOB-COM and UEF-COM movements were recorded using the Microsoft Kinect camera (FIGURE 1B). Each color video was rated by 2 neurologists (of C.E.P.vM., M.D.S., S.S.), who were blinded to the visit type (ie, before or during fampridine treatment) and to the patients’ performance on the conventional measures. The classic neurologic tests were quantified according to predetermined rating scales based on the Neurostatus EDSS functional system scoring definitions.²⁴ A 5-point Likert scale was created ranging from 0 (normal) to 4 (unable to perform due to disability) for the ADLs. Videos were rated using reference videos.¹⁶ A reference video displays a person performing a test according to the predefined scale, the gold standard. A consensus score was calculated based on the video ratings of the 2 neurologists using an algorithm similar to that described by Sarkar et al,²⁵ which takes into account individual rater bias. These scores were used in the statistical analyses. Videos of insufficient quality or inaccurately performed movements were excluded from the analysis. A more comprehensive description of the standardized movements, video rating, and score calculation can be found elsewhere.^16,17 During follow-up visits, significant improvement on the MOB-COM and the UEFCOM was defined as improvement on 2 or more tests (ie, ≥1-point decrease) and a decline on no more than 1 test (ie, ≥1-point increase) compared with baseline.

Video-Assisted Composite Measures.

Clinically Relevant Improvement

Clinically relevant improvement was determined with the Global Rating of Change (GRC) scales.²⁶ At the follow-up visit, patients rated their perceived change in mobility and in UEF. Patients with positive GRC scales scores were classified as having clinically relevant improvement, whereas negative GRC scales scores or scores of 0 were classified as no clinically relevant improvement.

Statistical Analysis

All statistical analyses were performed using IBM SPSS Statistics for Windows, version 24.0 (IBM Corp). Categorical variables are summarized as frequencies with percentages. The mean and SD are used to summarize continuous variables that were normally distributed; the median and interquartile range are used to summarize irregular distributions. The proportional agreement between the video-assisted composite and conventional measures was determined using 2×2 contingency tables for mobility (T25FW and MOB-COM) and UEF (NHPT and UEFCOM). The added value of the composite measures was determined and visualized using Venn diagrams showing the overlap in patients who improved on the various outcomes for mobility and UEF.

RESULTS

A total of 43 patients with MS initiating fampridine treatment were included and completed baseline measurements. Two patients discontinued treatment prematurely due to adverse effects, and 2 patients refused follow-up assessment because of time constraints. The baseline characteristics of the 39 patients who completed the study are summarized in TABLE S1, which is published in the online version of this article at IJMSC.org. No adverse events occurred while performing the tests.

FIGURE S1 shows the proportions of patients with significant mobility and UEF improvement, as assessed using the GRC scales, conventional measures, and video-assisted composite measures. The proportions of patients who improved on the classic neurologic tests and ADLs were similar to patients with improved MOB-COM (both 50.0%) and UEF-COM (54.5% and 45.5%, respectively).

Comparison of Conventional and Video-Assisted Composite Measures

TABLE S2 shows the distribution of improved mobility on the conventional and video-assisted composite measures. Three patients (7.7%) improved on the MOB-COM without improvement on the T25FW. Conversely, 5 patients (12.8%) improved on the T25FW without improvement on the MOB-COM. For the remaining patients (31 [79.5%]), the MOB-COM and T25FW outcomes were congruent.

Results of the UEF-COM and the NHPT, shown in TABLE S3, are congruent for 32 patients (82.1%). Four patients (10.3%) improved on the UEF-COM with no improvement on the NHPT. Conversely, 3 patients (7.7%) did not improve on the UEF-COM but improved on the NHPT.

Detection of Clinically Relevant Improvement

The overlap between improvement on the GRC scale and the conventional and video-assisted composite measures is depicted in FIGURE 2. In total, 26 of 39 patients (66.7%) reported improvement in mobility on the GRC scales. Of these patients, the number also showing improvement on the T25FW and the MOB-COM was similar (10 and 8 patients, respectively) and largely overlapped (6 of 12 patients). Concerning UEF, 13 patients (33.3%) reported improvement on the GRC scales. Although no overlap could be detected between these patients and those showing improvement on the NHPT (n = 3), an improved UEF-COM largely concurred with GRC scales improvement (n = 3 of 4).

Venn Diagrams Depicting the Overlap Between Patient-Perceived Improvement and Detected Improvement.

DISCUSSION

In this study we investigated the value of adding 2 video-assisted composite measures to conventional measures to improve detection of change in motor functioning. We found that the composite measures detected additional clinical improvement, as perceived relevant by the patient. The additional clinical value was most evident for UEF assessment.

The present findings suggest the importance of multidimensional assessment to detect changes in motor functioning. The composite measures used herein consisted of several classic neurologic tests and ADLs. In contrast, the conventional T25FW primarily tests walking speed which results in both ceiling and floor effects in patients who are more or less severely disabled, respectively.^13,14 Similarly, a floor effect has been reported for the NHPT.²⁷ In the present cohort, a ceiling effect is unlikely to have occurred because patients had relatively mild disabilities (highest EDSS score of 6.5).

The potential value of composite measures was most clearly present for UEF, as some overlap existed between the UEF-COM and the GRC scales. The inclusion of several ADLs in the composite measures might have contributed to this finding. In the present cohort, approximately 50% of all tests that contributed to improvement of the composite measure were ADLs. A previous study with similar methods also found that ADLs may be more valuable than classic neurologic tests.¹⁷ The NHPT and the T25FW also correlate to the ability to perform ADLs.^12,28 However, these correlations are based on patient-reported outcome measures (ie, questionnaires), which differ from the performance tests used in the present study.

Video-assisted assessment of motor functioning has the potential to improve disability evaluation in several ways. By using reference videos,²⁹ it may improve psychometric properties of outcome measures, such as intraobserver and interobserver variability. Digital video data could be used to build machine learning algorithms that automatically quantify motor performance. Video-assisted assessment could also shorten administration times and associated administrative burdens relative to other assessments, particularly the EDSS, which takes up to 30 minutes to obtain, including administration time. The tasks that made up the composite scores were shown on a screen integrated into the system and took 10 to 20 minutes to perform, depending on the level of disability. Ultimately, once determined safe for the patient, automated, video-assisted disability assessment has the potential to be performed by patients themselves at their preferred location and time instead of in a hospital or research clinic. To achieve this, machine learning algorithms of sufficient quality need to be developed.

This study has several limitations. The first is the small sample size. Another is, for the assessment of clinically relevant change, that potential placebo effects are not accounted for because no control group was available. Nevertheless, the goal of this study was not to assess the efficacy of fampridine but to detect change in performance. The present findings also provide limited insight into patients in the lower and upper ranges of EDSS scores because disability in this cohort of patients ranged from 3.5 to 6.5.

Another limitation is that the clinometric properties of the GRC scales scores as an assessment tool for clinically significant change are less established than those of the T25FW and the NPHT. Therefore, despite its use to determine relevant change from the clinicians’ or patients’ perspective, it is unclear whether the improvement of mobility on the GRC scale and the UEF truly reflects (subjective) improvement at the group level. Nonetheless, we believe that this 1-item patient-reported outcome stands more closely to the perceived treatment effect on the individual patient level.

Last, potential learning effects in performing the video-assisted composite at follow-up were not studied, although we assume that this is not a major bias because the composite measures were assessed once each visit with an interval of at least 2 weeks. Further improvement of the composite measures to detect changes in motor functioning might be achieved by adding ADLs with various difficulty levels to decrease floor effects or increase ceiling effects.

CONCLUSIONS

In this study, we demonstrated that the use of video-assisted composite measures may enhance the detection of motor functioning changes in patients with MS. This was most evident for the assessment of UEF. Using video-assisted composite measures as outcome measures in MS clinical practice and trials may enhance the detection of treatment effects, but further exploration is needed in a larger cohort of patients.

PRACTICE POINTS

» The clinical disease course of multiple sclerosis is heterogeneous, and, therefore, the assessment of treatment and rehabilitation effects is challenging. Adequate assessment of motor functioning is a crucial facet to monitor disease course.

» Video-assisted composite measures detected additional change in motor functioning. This is more evident for the assessment of upper extremity functioning than for mobility. Video-assisted composite measures may enhance detection of treatment and rehabilitation effects.