Practice Points
- A novel battery of self-administered, smartphone application–based neuroperformance tests would allow more frequent, remote assessments of motor and cognitive function in people with multiple sclerosis (MS).
- Outcomes derived from the smartphone app correlate with walking speed, manual dexterity, and processing speed tests from the previously validated Multiple Sclerosis Performance Test.
- Open-eye balance tests using the Cleveland Clinic Balance Assessment App demonstrate sensitivity to disease status and test-retest reliability in people with MS, meriting further investigation as a novel performance measure.
The development of valid, reliable outcome measures for multiple sclerosis (MS)–related disability is key to both the measurement of treatment effect in clinical trials and the management of chronic disease in individual patients. Standard outcome measures currently in use for MS, such as the Expanded Disability Status Scale, are not sensitive to changes in disability among groups with the most and least amount of disability and focus primarily on ambulation.1,2 To address some of these concerns, the MS Functional Composite (MSFC) was created; later, the Multiple Sclerosis Performance Test (MSPT) was developed as an iPad adaptation of the MSFC.3
However, the MSPT, along with traditional outcome measures, must be performed during an in-office visit, which presents challenges for people with MS. First, they may have access barriers, including living a significant distance from the MS clinic, work and family obligations, financial strain, and physical limitations due to the disease.4 Second, changes in neurological function that occur between visits may go unmeasured, such that it is difficult to appreciate a true level of disability because of infrequent assessment. The availability of in-home, self-administered tests of neurological function, used in conjunction with teleneurology visits, would reduce the number of visits necessary as well as allow more clinical course data to be captured. Finally, although there have been attempts to better capture disability in MS, such as the MSFC, measures that are more sensitive to change across the disability spectrum are still needed. In this study, a smartphone application–based battery of tests for cognitive and motor function was created by adapting several component tests of the MSPT and developing additional novel assessments (Figure S1; all supplemental materials are available in a PDF at the end of the online article). Convergent validity with the analogous MSPT tests, effect size, and test-retest reliability were analyzed by performing a cross-sectional study of a group of people with MS and a control group without MS.
Methods
Participants
A convenience sample of 23 people with MS and 23 control participants completed both the MSPT and the newly developed smartphone batteries. All tests were performed during a single visit at the Cleveland Clinic Mellen Center for Multiple Sclerosis Treatment and Research. Both groups had the same study inclusion criteria: 18 to 75 years old, able to understand the purpose of the study and provide informed consent, and able to comprehend the written and/or audio instructions for the software tool, which were provided in English. Exclusion criteria included a serious eye or vision problem (eg, uncorrected low vision as visual acuity of less than 20/60 of the better eye, cataracts, glaucoma) that would affect their ability to use technology. An additional inclusion criterion for participants with MS was a clinician-confirmed diagnosis of MS. Additional inclusion criteria for the control group were no cognitive impairment as defined by a score of 26 or greater on the Montreal Cognitive Assessment (MoCA), a Patient Health Questionnaire-9 score of less than 15, and no diagnoses of neurological, developmental, or medical illnesses. We did not impose these same additional criteria upon the MS group, as doing so might select for healthy people with MS and limit generalizability.
Participants with MS were recruited from the Mellen Center, and participants without MS were recruited via an institutional review board–approved advertisement that was posted on the Mellen Center website and displayed in flyer form around the Cleveland Clinic campus.
Data Collection
Procedure
For this pilot stage of smartphone app testing, the component tests were self-administered by participants in the clinic while in the presence of a research coordinator. Validation of tests in a controlled environment was desired so that test instructions could be modified, if necessary, before testing in a remote home environment. Each MSPT iPad test was performed once, and each smartphone-based test was performed twice. There was a 30-minute rest period between each of the 3 trials to ensure participants were not fatigued. The order of smartphone and MSPT administration was counterbalanced between groups to mitigate any learning effects. Auditory and written instructions for each of the component tests performed on the iPad and smartphone were delivered via the application.
MSPT Battery
The iPad-based MSPT battery development and tests have been described in prior validation publications.3,5-7 In brief, the battery includes a manual dexterity test (MDT), walking speed test (WST), and processing speed test (PST). The MDT is an adaptation of the 9-Hole Peg Test (9HPT), the WST is an adaptation of the Timed 25-Foot Walk, and the PST is an adaptation of the Symbol Digit Modalities Test (SDMT).
Smartphone Battery
The components of the smartphone battery were developed as adaptations of the MSPT along with several novel tests. The same smartphone device (iPhone 8) was used by all participants in the study; a research coordinator was present to answer questions for any participants unfamiliar with the device.
In the finger tapping test (FTT), participants are instructed to use a single finger and tap back and forth between 2 dots on the screen for 30 seconds. The test is conducted twice with both hands. The number of valid taps per second is recorded.
A digital version of the Timed Up and Go test (TUG) was developed (ie, remote TUG [rTUG]). To set up, the coordinator marked a line on the floor 3 m from a chair. The same chair, with armrests, was used for all trials. The smartphone was carried for all trials, from beginning to end, either in a pants pocket or in the participant’s hand. Participants started in the seated position and pressed a timer on the smartphone. The participant then walked o the line, turned around, walked back to the chair, and immediately stopped the timer when they sat down. A modified version (rMTUG) was also conducted, in which the participant performed the same procedures as previously described, but instead of walking to the prespecified 3-meter line, they were instructed to walk 3 normal-sized steps and turn around. This was done to create a version of the TUG test with less required setup to facilitate the ease of remote assessments in the future. For the rTUG and rMTUG, participants were instructed to walk at a comfortable and safe pace. To further assess lower extremity function and balance, a digital version of the 5 Times Sit-to-Stand test (5STS) was developed (ie, remote 5STS [rSTS]). Participants began sitting in a chair and pressed a timer on the smartphone. The participant was then instructed to stand up as quickly as they could 5 times. Participants held the phone while performing the r5STS and stopped the timer upon reaching the standing position for a fifth time; a coordinator observed to ensure that exactly 5 repetitions of the sitting-to-standing movement were completed. For each of these 3 tests (rTUG, rMTUG, r5STS), time to complete the task, in seconds, was recorded.
Finally, 2 balance tests were created using the Cleveland Clinic Balance Assessment App (CC-BApp), which has been previously validated for the assessment of postural stability in people with Parkinson disease.8 Trunk rotation as well as trunk movements in the medial-lateral and anterior-posterior directions were collected. The participant started a timer on the phone (either in their pocket or in their hand) and stood with their feet together and eyes open for 30 seconds. They were instructed to remain as still as possible while visually fixating on a marked target on the wall. The test was repeated for another 30 seconds with their eyes closed; a research coordinator stood close by to steady the participant if needed. Using the recorded movements of the center of mass, we calculated an ellipsoid volume that, with 95% probability, contained the center of the points of sway.
To assess cognitive function, a smartphone version of the tablet-based PST was created, which is an adaptation of the SDMT, as described previously. Number of items answered correctly was recorded for the PST.
Usability Questionnaire
To assess the usability of the smartphone application, all participants completed a standardized questionnaire of 9 items, with set responses ranging from 1 (strongly disagree) to 5 (strongly agree) (Table S1).
Statistical Analysis
Participants’ demographic characteristics were summarized as counts and percentages for categorical variables and median and IQR for continuous variables. Comparison of the characteristics between groups was determined using Fisher exact test or Pearson χ2 test for categorical variables and the Wilcoxon rank sum test for continuous variables. Test results of the first
smartphone trial were used for convergent/divergent validity and sensitivity analyses.
Spearman rank (ρ) correlation coefficients were calculated to evaluate relationships between pairs of analogous neuroperformance measures from smartphone and MSPT (iPad-based) data. Additionally, to assess divergent validity, Spearman coefficients were calculated for each of the component smartphone tests when paired with an MSPT test of a different functional domain (ie, lower extremity, fine motor, or cognitive function). Correlation magnitudes were interpreted as negligible (< 0.10), weak (0.10-0.39), moderate (0.40-0.69), strong (0.70-0.89), or very strong (≥ 0.90).9
Analysis of covariance (ANCOVA), with age as a covariate, was used to compare smartphone measure performance between groups. Effect sizes were calculated using partial η2 and interpreted as small (0.01-0.059), medium (0.06-0.13), or large (≥ 0.14).10
Participants who had a complete set of smartphone measurements (both trials) were included in these analyses. Test-retest reliability of the smartphone battery was examined separately for both groups through the intraclass correlation coefficient (ICC) using a single-measurement, 2-way mixed-effects model with absolute agreement (ICC (2,1)). ICC values were interpreted as indicative of poor (< 0.50), moderate (0.50-0.75), good (0.75-0.89), or excellent (≥ 0.90) reliability. SEM and smallest detectable change (SDC) at a 95% CI were calculated to quantify absolute reliability: SEM = SD × √((1 - ICC)); SDC = SEM × 1.96 × √2 .11
All analyses were conducted using R version 4.3.1 (R Foundation for Statistical Computing). All procedures were performed in compliance with relevant laws and institutional guidelines and have been approved by the institutional review board of the Cleveland Clinic (#19-719). Written informed consent was obtained from all participants.
Results
Participant Characteristics
A total of 23 people with MS and 23 control participants completed the assessments. As shown in Table 1, the people with MS were older (median age, 58.48 [IQR, 42.33-62.99] years) compared with control participants (29.40 [IQR, 24.45-54.05] years) (P = .003), but there were no significant differences for sex, ethnicity, or race. Participants with MS predominantly had a diagnosis of relapsing-remitting MS (61%), a median disease duration of 12.18 years, and a range of disability as measured by the Patient-Determined Disease Steps (PDDS),12 with 22% reporting no disability (PDDS score, 1) and 52% reporting some degree of gait impairment (PDDS score, 4-7).
Statistical Comparisons
Table 2 reports the correlation coefficients between the smartphone tests and corresponding MSPT tests, separated by group. For people with MS, correlations were significant for all comparisons except the balance tests vs WST and FTT (left) vs MDT (left). Strong correlations (ρ > 0.70) were observed when comparing rTUG vs rMTUG (ρ = 0.96), rTUG vs WST (ρ = 0.80), rMTUG vs WST (ρ = 0.80), and number correct on PST iPad vs number correct on PST smartphone (ρ = 0.78); this suggests a strong linear correlation and agreement between measures. Moderate correlations were observed for FTT (right) vs MDT (right) (ρ = –0.46) and r5STS vs WST (ρ = 0.55).
Divergent validity analyses (Table 3) showed negligible to moderate correlations (ρ ≤ 0.69) between smartphone mobility measures (rTUG, rMTUG, r5STS) and nonmobility iPad measures (PST, MDT).
After accounting for age as a covariate, a large effect size was observed in FTT (η²p = 0.27 and 0.20 for right [FTT(R)] and left [FTT(L)], respectively) between people with MS and control participants (Table 4). Medium effect sizes were observed for rTUG (η²p = 0.07), ellipse area with open eyes (EA[O]) (η²p = 0.07), and ellipse volume with open eyes (EV[O]) (η²p = 0.08). The balance measures with closed eyes, EA(C) and EV(C), had small effect sizes (η²p = 0.04 and 0.05, respectively), as did the rMTUG (η²p = 0.05) and r5STS (η²p = 0.02) durations. Finally, the total correct (TC) on PST demonstrated a negligible effect size (η²p = 0.00).
In people with MS, excellent reliability was observed for rTUG (ICC = 0.99), rMTUG (ICC = 0.90), r5STS (ICC = 0.99), EV(C) (ICC = 0.90), FTT(L) (ICC = 0.99), and PST(TC) (ICC = 0.93) (Table 5). Good reliability was observed for FTT(R) (ICC = 0.86), EA(C) (ICC = 0.89), EA(O) (ICC = 0.77), and EV(O) (ICC = 0.82).
The majority of both people with MS and control participants strongly agreed they had previous experience with a computer (82% and 91%, respectively). They also had similar responses upon completion of the smartphone battery, except fewer people with MS reported being “not tired” (median [IQR] of 5 [4-5] in people with MS vs 5 [5] in control participants; P = .035) (Table 6). Most people with MS agreed or strongly agreed that they had no problems seeing information displayed on the smartphone screen or concentrating on the tests (5 [5] and 5 [4-5], respectively). Likewise, most found that they were given adequate time to complete the tasks (5 [5]) and did not foresee any problems should they take the test again (5 [5]).
Sample size was originally calculated for 28 per group to detect correlations of r equal to or greater than 0.50 with 80% power at α = 0.05. However, the COVID-19 pandemic limited recruitment. The achieved sample (n = 23 per group, 46 total) provided adequate power (> 80%) to detect correlations of r equal to or greater than 0.60. For test-retest reliability, approximately 20 participants were required to estimate an ICC equal to or greater than 0.60 with 80% power at α = 0.05 and a 10% dropout rate.
Discussion
We created and pilot-tested smartphone-based neuroperformance tests to assess the level of cognitive, fine motor, and lower extremity function disability in people with MS. The data show that PST, rTUG, and rMTUG tests had strong correlations with their respective MSPT tests, and FTT (right and left) had large effect sizes for distinguishing participants with MS from control participants. Our study also included the first smartphone-based 5STS test used in MS, showing moderate correlation with the WST from the MSPT. Further, we demonstrated that open-eye balance measures originally developed for people with Parkinson disease (ie, CC-BApp) could differentiate people with MS from control participants. These results suggest that FTT, rTUG/rMTUG, and r5STS may provide useful data for measuring function in people with MS; biometric balance tests, particularly open-eye tests, merit further testing as performance-based measures in MS.
The MSFC was developed to provide a multidimensional objective assessment of neurological function but is limited by the need for administration by trained personnel. The MSPT improved upon the MSFC, as it is an in-office, validated iPad adaptation that can be self-administered and automatically uploaded into the electronic medical record. We aimed to further advance characterization in function by creating tests that could be self-administered and performed more frequently outside the clinic setting. The neuroperformance testing administered via smartphone application was not merely a recreation of the MSPT component tests in a different format but a set of related but novel tests accessible via smartphone. The tests are not only self-administered but can also be performed at a range of disability levels with no equipment other than a personal phone. Current outcome measures sometimes have additional barriers to completion, such as the Timed 25-Foot Walk, which requires patients to measure a specific distance. In this study, rMTUG showed promise as a possible substitute for rTUG, increasing ease of home administration; rMTUG demonstrated convergence with rTUG (ρ = 0.96 [IQR, 0.89-0.98]), and excellent test-retest reliability (ICC = 0.90). This is in agreement with other studies that have validated the traditional nondigital version of TUG as a valid measure of lower extremity function in MS.13
Prior research has shown that the 5STS test is valid in assessing function in MS.14 Our study results similarly found moderate r5STS correlation with WST (ρ = 0.55) and agreement between repeat tests (ICC = 0.99), but r5STS was not as sensitive to disease status (η²p = 0.02; P = .34). r5STS may only moderately correlate with WST due to the tests measuring different domains of lower extremity and physical function.
A novel aspect of the study was the application of biometric balance testing, previously validated as an outcome measure for individuals with Parkinson disease, to people with MS.8 In this study, EA and EV did not demonstrate convergent validity with the WST, which likely reflects that the WST is not a measure of balance. Additionally, the test-retest reliability was good for people with MS but only moderate or poor for control participants, indicating potential concerns with the current version of the smartphone test. Further, sensitivity to disease status (control participants or people with MS) was medium for the eyes-open tests but small for eyes-closed tests. Overall, balance is an important clinical concept, but the smartphone balance tests likely would need further refinement before implementation.
This study builds on other attempts to develop smartphone-based applications for assessing neurological function in MS. Several examples of other developed applications are Floodlight MS, elevateMS, Konectom, and dreaMS.15-18 Our application is similar to these other suites of tests in that it includes the 9HPT/FTT, PST, and a test of balance. However, the inclusion of 5STS and TUG/MTUG as measures of lower extremity function is unique. Another strength of the present study is the range of disability levels represented in our cohort; although our cohort was small compared with cohorts of other studies, ours included a greater proportion of people using canes or bilateral supports. Our study also highlights the utility of the FTT as an alternative manual dexterity test to the 9HPT. A test similar to the FTT has shown concordance with 9HPT and the ability to distinguish between individuals with MS and control participants.19 This is consistent with our findings that FTT moderately correlates with MDT (ρ = –0.46 and –0.41 for the right and left hands, respectively); moreover, we found good test-retest reliability of the FTT (ICC = 0.86 and 0.90 for the right and left, respectively). FTT also has the added benefit of requiring less equipment than the 9HPT.
This study is limited by its small sample size and the fact that it is a convenience sample, which resulted in a significant difference in age and a (nonstatistically significant) difference in sex distribution between groups. Although we adjusted for age through an ANCOVA, it is possible that age-related cognitive and physical changes could have contributed to the differences observed. Thus, future studies should ensure matching between people with MS and control participants based on age, sex, and MoCA score. Also of note, the MS group had an overrepresentation of men (48%) compared with the overall MS population (~25%). Another potential limitation is that we did not utilize traditional MSFC tests as a comparison with smartphone test outcomes. We chose to compare smartphone tests with the MSPT instead, because we use it daily in clinical practice and it was previously validated against the MSFC.
There are additional limitations to our administration of smartphone tests. Variability in location of the phone (ie, pocket vs hand) could have altered test outcomes. Further, although all the tests can be performed at home, this study was completed in a controlled clinic environment; further testing outside the clinic and longitudinal testing are needed. When administered at home, the tests should be performed in the presence of another person for the prevention of falls or injuries (ie, during the closed-eye balance test). In future studies, administration of rTUG/rMTUG and r5STS could be improved to minimize any lag time; efforts will be made to optimize voice command or sensor-based data to eliminate the need to manually start/stop trials. In addition, as with all previously developed smartphone applications, a significant limitation is the integration into clinical workflow. The application developed and tested in the study was built to integrate with the electronic medical record through a single sign-on process utilized for our current patient portals. This integration was not utilized in this study and would need to be formally tested in future studies to demonstrate usability and feasibility for patients. Finally, as with any new health technology, it will be important to ensure this application is distributed in an equitable manner. Consideration should be given to ways in which people with low computer literacy or no access to a smartphone can still benefit, and the test instructions should be translated into additional languages for wider generalization.
Conclusions
Our study data demonstrate that a self-administered, smartphone-based application can be a valid, reliable, and sensitive means of administering neuroperformance testing to people with MS. Future work is needed to validate these measures longitudinally and develop implementation strategies for routine clinical workflows.
