Interrater Reliability of Four Sensory Measures in People with Multiple Sclerosis

Sensory impairment is a significant problem for people with multiple sclerosis (MS). Although complete loss of sensation is rare, up to 80% of patients present with some sensory impairments,^1–3 and this is often the first sign of MS. Sensory loss can be a significant cause of disability,^{4 5} and several studies have found that somatosensory impairments have a significant effect on gait^{6 7} and balance^{8 9} and lead to decreased mobility, independence, and quality of life^{9 10} in people with MS.

The assessment of sensory impairment is not common in clinical practice, but it is clear that sensation plays an important role in motor control^{11 12} and is related to gait and balance function in people with MS.¹³ However, there is little research into how exercise and rehabilitation affect the sensory system; one possible reason for this is the lack of information on valid and reliable assessments.

Several subjective and objective sensory tools (visual analogue scales, verbal analogue scales [VASs], the Nottingham Sensory Assessment [NSA],¹⁴ the Erasmus MC modifications to the revised NSA [EmNSA],¹⁵ Semmes-Weinstein monofilaments [SWMs],⁸ and vibration sensitivity testers⁵) have been used in studies. However, their psychometric properties have not been investigated for people with MS. Furthermore, a recent systematic review of measures of sensation in neurologic conditions stated that gold standard sensory measures are lacking and that very limited advice exists regarding which measurement tools should be used for different domains and patient populations.¹⁶ Interrater reliability is a particularly important property for outcome measures for use with people with MS because they have interactions with many different health-care practitioners during their disease.

This study aimed to determine the interrater reliability of the VAS for sensation, the EmNSA, SWMs, and the neurothesiometer (NT) in the MS population. These measures were chosen because they are used in clinical practice, are reported in the literature, and assess both patient-reported and objective measures of sensation.

Methods

Participants

One hundred twenty information leaflets were posted by the MS office coordinator to randomly selected participants in the MS Society's databases. Interested participants were asked to contact the lead investigator. Individuals who were able to walk with or without an aid were included in the study (score of 0–4 on the mobility section of Guy's Neurological Disability Scale)¹⁷; this was screened by telephone when participants expressed interest to the lead investigator.

Individuals were excluded if they had a relapse in the previous 3 months, corticosteroid treatment within 28 days of the start of this study, a recent fracture or surgical procedure that might restrict their mobility or function, or a history of peripheral neuropathy. Thirty-seven of 40 interested participants met the eligibility criteria and were recruited to this study. On the assessment day, each participant gave informed consent. The study was approved by the Education and Health Sciences Research Ethics Committee, University of Limerick, Limerick, Ireland. Demographic data, including age, sex, time since diagnosis, and mobility level, were collected during the initial assessment by the interviewer.

Measures

The measures were chosen because they are used in clinical practice, are reported in the literature, and assess patient-reported and objective measures of sensation.

Neurothesiometer

The NEU 1500 NT (Scientific Laboratory Supplies Ltd, Nottingham, England) was used to determine the vibration threshold. This instrument delivers vibrations of increasing strength (measured in volts or micrometers) and determines the minimum threshold at which vibration is perceived.¹⁸ In this study, the method of limits as published by Bril and Perkins¹⁹ in 2002 was used. First, the investigator demonstrated the expected vibration sensation (buzzing sensation) on the patient's hand. Each participant was asked which side of the body was more affected regarding sensation. Then the participant was asked to close their eyes or turn their head away, and the NT was then placed over selected areas of the participant's more affected side (self-reported). The voltage was gradually increased until the individual reported feeling vibration. The minimum reading at which “buzzing” was perceived was recorded. Three readings were taken from the same place, and the average was calculated. Four sites on the plantar aspect of the foot were tested: the distal phalanx of the great toe, directly over the joint line of the first and fifth metatarsophalangeal joints, and the center of the plantar surface of the heel. The NT was found to have good interrater reliability in people with diabetic polyneuropathy (interclass correlation coefficient [ICC(2,1)] = 0.9441, P < .001),²⁰ and a recent study suggested good test-retest reliability for people with MS (ICC[2,1] = 0.92–0.81 for different testing areas, significant at the 0.1% level).²¹

The VAS

The VAS was used to determine patients' subjective feeling of sensation in the upper and lower extremities. These scales have been used widely to assess variations in intensity of musculoskeletal pain and have moderate-to-good reliability (Spearman's rank correlation values varied from 0.60 to 0.77).²² In the present study, we included a variety of sensory stimuli (for details see Supplementary Table 1, which is published in the online version of this article at ijmsc.org). These represent the domains often reported by patients in clinical practice.

The EmNSA

The EmNSA¹⁵ is a screening tool for sensory impairments of the upper and lower extremities and consists of the following items: light touch, pressure, pinprick, sharp-blunt discrimination, and proprioception. This is a modified version of the original NSA scale for patients with stroke developed by Lincoln et al.¹⁴ It was reported to have good-to-excellent interrater reliability in neurologic and neurosurgical inpatients with intracranial disorders.¹⁵ In this study, the testing protocol as published by Stolk-Hornsveld et al.¹⁵ was used. When the participant correctly identified the test sensation on three trials, it was considered “normal”; when the participant identified the test sensation but not on all three trials in each region of the body, it was considered “impaired”; and, finally, when the participant did not identify the test sensation on any of the three trials, it was considered “absent.” Each testing sense (light touch, pressure, pinprick, sharp and blunt, and proprioception) for each region of the body (upper limb: fingers, hand, forearm, and arm; lower limb: toes, foot, leg, and thigh) was analyzed separately.

Semmes-Weinstein Monofilaments

The Rolyan SWM set (Homecraft Rolyan, a Patterson Medical Company, Ashfield, Nottinghamshire, England) is a standard set of monofilaments of different strengths and was used to assess the threshold for light touch. The Rolyan SWM set consists of handheld testing filaments that are precisely calibrated and of equal length (38 mm). In this study, a six-piece foot kit (filaments 2.83, 3.61, 4.31, 4.56, 5.07, and 6.65) and a testing protocol as published by Mawdsley et al.²³ were used. Participants were asked to close their eyes during the testing procedure and to respond with a “yes” if they could feel the test stimulus. For each trial, the monofilament was applied to the skin surface, and once fully bent it was held on the skin for approximately 1.5 seconds and then removed. The lightest monofilament (2.83) was used first, followed by a thicker monofilament if the previous one was not detected. The SWM was found to have good interrater reliability in people with peripheral nerve injury (ICC[2,1] = 0.965).^{24 25} In this study, SWMs were used on four sites each on the upper and lower limbs. For statistical analysis, to allow comparison with previous studies,^26–28 we converted the filament size into the ordinal scale to be able to calculate ICC(2,1): 1 represented normal sensation; 2, diminished light touch; 3, diminished protective sensation; 4, loss of protective sensation; and 5, untestable or deep pressure sensation only.

Testing Area for EmNSA and SWMs

Upper limb: fingers: palmar aspect: first, third, and fifth distal phalanges; hand: palmar aspect: head of second metacarpal bone, head of fifth metacarpal bone, and center of the thenar eminence; forearm: anterior aspect of the ulnar styloid, center of the forearm, anterolateral aspect, 2 cm distal to the elbow joint line; arm: anteromedial aspect, 2 cm proximal to the elbow joint, center of the anterior aspect of the humerus and the lateral aspect, 2 cm distal to the acromion.

Lower limb: toes: plantar aspect: first, third, and fifth distal phalanges; foot: dorsal aspect: base of the fifth metatarsal bone, second metatarsal bone, and center of the midtarsal line; leg: medial malleolus, center of the anterior border of the tibia, lateral aspect of the fibular head; thigh: medial femoral epicondyle, anterior aspect, center of the line of the femur, and greater trochanter.

Procedure

One senior (MU) and one basic-level physiotherapist collected data. Both raters received the same training and an information manual to ensure standardization of approach. No communication was allowed between raters. Neither rater had access to the other rater's testing scores. Participants were tested twice on the same day, with a 1.5-hour interval between tests. Background information was collected by the first rater, and then assessment with the NT, VAS, EmNSA, and SWMs took place in random order. The testing order was identical for both raters. Participants were randomly assigned to either rater 1 or rater 2 for the first assessment.

Statistical Analysis

Statistical analyses were undertaken using IBM SPSS Statistics for Windows, version 20 (IBM Corp, Armonk, NY). For the EmNSA test, we initially used percentage agreement and Cohen's kappa.²⁹ However, the distribution of responses across the categories was found to be highly skewed toward the “normal” response category. Therefore, we computed the prevalence-adjusted bias-adjusted kappa³⁰ for the dichotomized response categories “normal” and “impaired/absent,” with associated prevalence and bias indices. In addition to the reliability statistics, we calculated the percentage of normal sensory responses (Ps%) for each test, where a normal sensory response was defined as a “normal” test result recorded by both raters. The Ps% provided an important estimate of the percentage of people with MS who would be classified as having normal sensation by the test.

Continuous data from the VAS and NT tests were analyzed using the two-way random single measure with absolute agreement ICC(2,1) with 95% confidence intervals (95% CIs) and Bland and Altman analysis.³¹ Standard error of measurement (SEM) was also calculated as SEM = SDdiff/√2 (under the assumption of similar variances where SDdiff = the standard deviation of the differences between test and retest observations).^{32 33} For the SWM test, Cohen's kappa and percentage agreement were computed, and to allow comparison with previous studies, ICC(2,1) values were computed and expressed as the quadratic weighted kappa, as outlined by Fleiss and Cohen.³⁴

Values for ICC(2,1) measures were interpreted using the guidelines of Portney and Watkins,³⁵ where less than 0.50 represents poor reliability, 0.50 to 0.75 represents moderate reliability, and greater than 0.75 suggests good reliability. The classification of reliability for categorical data proposed by Landis and Koch³⁶ was used for kappa and prevalence-adjusted bias-adjusted kappa: less than 0, poor; 0.00 to 0.20, slight; 0.21 to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and 0.81 to 1.00, almost perfect.

Sample Size

Sample size calculations were based on optimal and minimal levels of reliability. Assuming that the proportion of positive ratings is 0.5, a large-sample, 0.05-level, two-sided test of the null hypothesis that kappa is 0.4 will have 80% power to detect an alternative kappa of 0.90 when the sample size is 27.³⁷ Also, a sample size of 33 is sufficient to detect a true reliability of 0.7 from a null value of 0.4, with 80% power where ICC(2,1) is used to measure reliability.³⁸

Results

Of the 34 participants tested, 26 were women (76.5%). The mean (SD) age was 49.5 (11.13) years. The mean (SD) time since diagnosis was 12 years and 10 months (5.84 years). The greatest percentage of participants (41.2%) scored 0 on Guy's Neurological Disability Scale, which indicated that they reported that gait was not affected. The greatest percentage of participants (61.8%) were diagnosed as having relapsing-remitting MS. Twenty-nine participants (85.3%) reported a sensory problem on VAS testing. Descriptive study statistics are presented in Table 1. Detailed results of the interrater reliability testing for the NT, VAS, EmNSA, and SWMs are given in Tables 2, 3, 4, and 5.

Descriptive study statistics

Interrater reliability results for the neurothesiometer

Interrater reliability results for the verbal analogue scale for sensation

Interrater reliability results for the Erasmus MC modification to the revised Nottingham Sensory Assessment

Interrater reliability results of Semmes-Weinstein monofilaments

The ICC(2,1) values for NT can be considered generally good, with the highest ICC(2,1) for the plantar first metatarsophalangeal joint (ICC[2,1] = 0.84, P < .001). The lowest ICC(2,1) was found for the plantar aspect of the heel (ICC[2,1] = 0.73, P < .001) (Table 2). High ICC(2,1) values were supported by low SEM values and low mean differences, with associated 95% CIs that include zero. Bland and Altman plots for NT (Figure 1) show excellent agreement for low vibration thresholds and good agreement for higher vibration thresholds, and almost all the differences are within the limits of agreement (±1.96 SDdiff).

Interrater variability of the vibration threshold for four locations on the foot measured by the neurothesiometer expressed as Bland and Altman plots

Good ICC(2,1) values, greater than 0.75 (P < .001), for the VAS were found for the hands and feet for question 1, relating to having numbness, and question 3, relating to having pins and needles, tingling, or burning sensation (Table 3). These results are supported by low SEM values and low mean difference values. For questions 2, 4, 5, and 6, the ICC(2,1) values ranged from 0.60 (P < .001) for question 5, relating to having problems feeling pain in the hands, to 0.85 (P < .001) for question 4, relating to having any pain in the hands. However, for those questions, the Ps% was high, reflecting a low number of reported sensory problems.

Good interrater reliability values were found for EmNSA for all sections of the test except for the sharp and blunt test (Table 4). However, further analysis showed a high Ps% for most of the test sections, indicating that nearly all the participants had normal sensory responses. Sharp and blunt was the only section of the test to identify sensory problems, and for this test the interrater reliability was poor.

Cohen's kappa and quadratic weighted kappa showed generally poor values for SWMs for the upper and lower limbs (Table 5). Percentage agreement between the two raters suggested moderate agreement. Cumulatively, these results suggest poor interrater reliability of SWMs.

Discussion

Despite the high incidence of sensory problems in people with MS and the importance of sensory feedback for motor performance, there is little information about the psychometric properties of sensory tests for people with MS, and their use in clinical practice and research is limited. Therefore, this study aimed to establish the interrater reliability of the VAS, SWMs, EmNSA, and NT in people with MS.

The reliability of the VAS and the NT was generally considered good. The interrater agreement for EmNSA was found to be high (except for sharp and blunt sense). However, the very high Ps% may indicate a poor ability of the test to detect sensory problems. The results of interrater reliability for SWMs were generally poor, with moderate percentage agreement between raters.

The NT was found to have good reliability, with high ICC(2,1) values, generally greater than 0.75. The SEM values were approximately 5 to 6 points on a scale from 0 to 51, which was generally less than 10% of the range of the scale. This study confirms previously reported results in a diabetic neuropathy population, where Jurado et al.²⁰ reported good interrater reliability, with an ICC(2,1) of 0.94. To our knowledge, only one study so far, by Newsome et al.,⁵ has investigated the interrater reliability of the vibration threshold in people with MS, using the Vibratron II (Physitemp Instruments Inc, Clifton, NJ), and also found good reliability. In a previous study,¹³ we reported the relationship between foot vibration threshold and clinical measures of walking and balance in people with MS and the predictive validity of foot vibration threshold. Additionally, we found good test-retest reliability²¹ of the NT for ambulatory people with MS. Cumulatively, these studies provide initial evidence that NT has good psychometric properties and, therefore, may be considered for measuring sensation in people with MS.

Data analysis for the VAS revealed high ICC(2,1) values (significant at the 0.1% level) for the hand and the foot, suggesting good interrater reliability according to the guidelines of Portney and Watkins.³⁵ However, further analysis showed that some questions had a very high Ps%, where both raters selected either 10 or 0 on the VAS, reflecting a low prevalence of sensory problems. For these questions, the reduced variability of the responses across the range of possible VAS scores had the effect of inflating the ICC(2,1) scores for the sample. Therefore, reliability estimates of these questions should be interpreted with caution. When searching the available literature, we did not find any studies using the VAS to test sensation in people with MS. However, the VAS is commonly used in clinical practice,^{22 39–41} has the advantage of being easy to administer, and considers the patient's report of sensory problems. Patient-reported outcome measures are increasingly important in evaluating interventions in health-related quality of life in people with MS,⁴² and the VAS is a potentially useful tool in evaluating patient-reported sensation in people with MS. Based on the present data, we suggest that question 1, relating to feeling numbness in the hands and feet, and question 3, relating to having pins and needles, tingling, or burning sensation in the hands and feet, may be useful, but this requires confirmation in further studies. Additionally, the optimal wording of VAS questions for other sensory domains requires further investigation.

The EmNSA is a tool that was designed to detect sensory impairments in the upper and lower limbs, testing senses such as light touch, pinprick, and proprioception. We found that many EmNSA tests had zero variability in some response categories and very high Ps% values, indicating low test sensitivity in detecting sensory problems. Data analysis suggested generally good reliability across all tests, except for the sharp and blunt test. However, owing to high Ps% values for many sections of EmNSA, these results require verification in larger studies. Similar to ours, high weighted kappa values were found by Stolk-Hornsveld et al.,¹⁵ where predominantly good-to-excellent interrater reliability was reported. The present analysis found that the interrater reliability can be artificially inflated by low response variability and where there are high Ps% values on the tests. Therefore, these results suggest using the EmNSA scale with caution, and these findings require verification in larger studies. Future studies should report the percentage of maximum responses on the scale to distinguish between these two factors.

The SWM was the only test that reported relatively low Ps% values, especially in the lower limb, suggesting that monofilaments were sensitive in detecting sensory problems. However, all three statistical methods used to establish reliability reported generally low values, suggesting poor interrater reliability. The present results are contrary to previously reported findings by Novak et al.,^{24 25} who reported good interrater reliability of SWMs for nerve injury, with an ICC(2,1) of 0.97, and Schreuders et al.,²⁶ who found a high ICC(2,1) of 0.86 in Charcot-Marie-Tooth disease. The present results corresponded with findings by Collins et al.,⁴³ where poor-to-moderate interrater reliability was reported in healthy participants, and by Rozental et al.,⁴⁴ where fair-to-slight reliability was reported in asymptomatic individuals with peripheral nerve injuries and compression syndromes. The testing technique, level of training, force used to bend a monofilament when testing, and location of the sensory receptors on the testing area may be important factors that lead to variability of the results. The SWMs are a portable and easy-to-administer testing tool that may be more useful as a diagnostic tool than as a measure of outcome.

The lack of experience by the testers in using SWMs, NT, and EmNSA and the differences in the amount of clinical experience are sources of potential measurement error. It is difficult to conclude whether the reliability estimates were due to the raters or the measures in this study. However, both raters received standard training and practice opportunities before the trial commenced and are representative of the variety of clinicians who would assess people with MS during their disease.

We did not collect data for participants who were unable to walk, so these results may not be applicable to that population. It is also possible that the wording of some questions for the VAS was not specific enough to identify sensory problems in the group and that that led to many “normal” answers, which artificially inflated the ICC(2,1) values. Further clarification of the wording of the VAS questions to optimize reliability is needed. To date, the NT has been used in sensory testing of the foot, and a limitation of this study is the lack of available data of this measure for the hand.

The Ps% by both raters was high for some sensory tests. This may suggest that the participants did not have sensory problems, or it could be indicative of the inability of a test to detect sensory problems. More than 85% of our participants reported sensory problems; therefore, further evaluation of the responsiveness of these tools is required. Lack of variability in response data and high Ps% values can artificially inflate the test reliability measures. We recommend reporting Ps% values in conjunction with reliability results for this reason.

Conclusion

The evidence from this interrater reliability study suggests that two questions (feeling numbness in the hands and feet and having pins and needles, tingling, or burning sensation in the hands and feet) on the VAS for sensation and the NT are reliable sensory measurement tools in people with MS. This is reflected by the high ICC(2,1) values, significant at the 0.1% level, with narrow 95% CIs and low SEM values, generally less than 10% of the range on the scale. On the other hand, results for SWMs and EmNSA showed either poor interrater reliability or questionable results, suggesting that these two scales need to be further investigated before they can be applied to practice. It is also possible that these scales may not be suitable when the scores of two different raters are used.

PracticePoints

We suggest using the neurothesiometer for objective sensory measurement of vibration threshold in people with MS.

The verbal analogue scale is a self-reported, patient-friendly, and reliable scale that is easy to use in clinical practice.

Erasmus MC modifications to the revised Nottingham Sensory Assessment and Semmes-Weinstein monofilament tests should be used with caution because the results of interrater reliability are not satisfactory.