Development and Validation of the Actionable Bladder Symptom Screening Tool for Multiple Sclerosis Patients

Multiple sclerosis (MS) is a progressive neurologic disorder that results in demyelination within the central nervous system.^{1 2} Initially, it can produce a wide range of neurologic symptoms; as the disease progresses, it often leads to reductions in mobility and cognitive function.² The disease affects women three times as often as men and is the most prevalent neurologic disorder in individuals aged 20 to 45 years.³ Many people with MS eventually develop some form of lower urinary tract dysfunction.^4–6 This dysfunction has been attributed to a lack of coordination between centers in the brainstem and the sacral part of the spinal cord.⁵ Neurogenic detrusor overactivity (NDO), a bladder disorder characterized by spontaneous involuntary contractions of the detrusor muscle, affects up to 75% of MS patients.⁷ Resulting symptoms include urinary urgency, frequency, and incontinence.^{1 8} These symptoms may have psychosocial, physical, and sexual effects that reduce patients' overall health-related quality of life (HRQOL).⁹ The most bothersome symptom has been reported to be urgency incontinence, which may have substantial and multidimensional impacts including major social, occupational, and domestic problems.

Despite the large percentage of MS patients with some form of urinary dysfunction, urologic evaluation and treatment are underutilized in this population.⁸ When NDO is identified, treatment usually begins with oral anticholinergic medication. In some patients, however, it may have dose-limiting side effects as well as insufficient efficacy.^10–14 In these cases, intermittent catheterization can be used, but most patients find this inconvenient and unpleasant, and it may also be ineffective.¹⁵

Successful management of bladder problems in MS patients depends on clear communication of the nature and extent of the bladder symptoms between patient and physician. This may be facilitated by the use of a short, simple, patient-friendly screening tool to help assess patients during clinical visits. This study was conducted to develop and validate a new screening tool, the Actionable Bladder Symptom Screening Tool (ABSST). This tool is designed to help clinicians identify MS patients with urinary incontinence who may benefit from the various treatment options now available.

Methods

Study Objective

Although the objective of this study was to develop a clinically relevant and statistically sound screening tool, currently there is no regulatory guidance (from either the US Food and Drug Administration [FDA] or other government agencies) on how to develop a screening tool. It was decided to follow the FDA Guidance for Industry: Patient-Reported Outcome Measures ¹⁶ for the content validation and measurement property assessment involved in developing this tool. Classic psychometric methods were used to identify optimally performing items and to validate the instrument.

Qualitative Study Methodology

The study was conducted sequentially, beginning with a literature review of all published patient-reported outcome (PRO) instruments specific to bladder health and urinary incontinence. It was determined that an existing instrument would not fulfill the stated criteria for the screening tool; therefore, development of a new instrument was initiated, with patient input obtained through accepted qualitative methods. Following analysis and interpretation of the qualitative data, a multisite observational study was conducted to assess the measurement properties of the new instrument (Figure 1).

Steps in developing the Actionable Bladder Symptom Screening Tool

Literature Review

Existing PRO instruments were reviewed in three phases. A literature review was conducted to identify screening tools and questionnaires assessing bladder symptoms and their impact on patient quality of life (QOL). Of the 17 instruments identified in the literature, 11 met the criterion of specifically addressing incontinence. The second round of evaluation focused on these 11 instruments (listed below), which were reviewed by the study committee co-chairs, a urologist and a neurologist:

Incontinence Impact Questionnaire-Short Form-7 (IIQ-7)

Urinary Incontinence Handicap Inventory (UIHI)

Urinary Symptom Profile (USP)

Incontinence Quality of Life (I-QOL)

Overactive Bladder Questionnaire (OAB-q)

Incontinence Stress Questionnaire-Patient (ISQ-P)

Kings Health Questionnaire (KHQ)

Male Urinary Symptom Impact Questionnaire (MUSIQ)

Nocturia Quality of Life Questionnaire

International Consultation on Incontinence Modular Questionnaire-Male Lower Urinary Tract Symptoms (ICIQ-MLUTS)

International Consultation on Incontinence Modular Questionnaire-Female Lower Urinary Tract Symptoms (ICIQ-FLUTS)

Evaluation of these 11 instruments was based on several criteria, including their perceived sensitivity, MS specificity, coverage of multiple dimensions or concepts, ease of scoring, scoring that would support an action (eg, referral), and feasibility of use in a clinical setting. No single instrument met all the criteria. Four of the 11 instruments (IIQ-7, USP, I-QOL, ISQ-P) were selected by the co-chairs to be reviewed by a larger steering committee consisting of key urology and neurology opinion leaders (including the co-chairs). The committee applied the above criteria to assess the feasibility of adapting these four instruments (or items from these instruments) for use as a screening tool for NDO in MS patients. The consensus of the committee was that a new instrument should be developed.

Concept Elicitation, Item Generation, and Cognitive Debriefing

The new instrument (the ABSST) was developed through a qualitative process. Specifically, face-to-face concept elicitation (CE) interviews with MS patients with NDO were conducted in order to identify relevant concepts. Expert clinician input from urologists, neurologists, psychiatrists, and MS nurse specialists in Europe (n = 5) and the United States (n = 6) was also gathered. Using this information, an item-generation meeting was held to review the CE interviews, achieve agreement on a list of items for a new, MS-specific questionnaire, and reach a consensus on the preliminary structure and format of the questionnaire. Finally, the face and content validity of the new questionnaire were tested via face-to-face cognitive debriefing interviews with MS patients with NDO. The goals of the cognitive debriefing interviews were to obtain detailed information on how patients interpreted the questions and response choices, whether the wording was appropriate, and whether instructions and formats were understood. Written informed consent was required before a patient could participate in any part of the study. All research involving patient interviews and patient completion of questions were in compliance with the Helsinki Declaration and had institutional review board approval from the Copernicus Group (protocol #MAP1-11-049).

Multisite Observational Study Methodology

The observational study was a US-based, nonrandomized, multicenter, stand-alone study that assessed the psychometric properties of the ABSST among male and female MS patients in the United States with and without NDO. The patients were recruited through neurology practices to complete the PRO assessments (the ABSST and the OAB-q Short Form [SF]) as well as a demographic and health information form. De-identified data from the completed ABSST were shared with the referring clinician, who then indicated whether or not referral to a urologist was recommended based on the patient's responses. The presence of NDO was not a requirement for inclusion in the observational study.

Concurrent Instruments

Concurrent instruments were those instruments used in the observational study alongside the newly developed ABSST. These instruments are typically used for comparison with the instrument of interest in terms of statistical performance.

Urology Assessment Form: Clinician Completed. This two-item questionnaire was completed by the clinician after the patient filled out the ABSST. The clinician was asked whether he or she would refer the patient to a urologist based on the patient's responses to the ABSST. The clinician was also asked to list the specific items that most influenced this decision.

Overactive Bladder Questionnaire Short Form. The OAB-q SF consists of a 6-item symptom-bother scale and a 13-item HRQOL scale. The HRQOL scale is divided into three subscales: coping (5 items), sleep (3 items), and emotional social (5 items). The OAB-q SF has well-documented reliability and validity across domains. The OAB-q SF was scored according to the published algorithm. Items were transformed and summarized into two domain scores: Symptom Severity and Total HRQOL. The Symptom Severity score (average of items 1 through 6) ranged from 0 to 100, with higher scores reflecting greater symptom severity or bother. The Total HRQOL score (average of items 1 through 13) ranged from 0 to 100, with higher scores reflecting better HRQOL.^{17 18}

Scoring of the ABSST

The interim scoring algorithm for the 17-item version of the ABSST was developed as three domains (Bladder Symptoms, Coping Strategies, and Impact of Bladder Symptoms) plus a total score. Each item on the ABSST was scored from 0 (no symptoms or impact) to 3 (extreme symptoms or impact), and scores were calculated as the sum of the items on the instrument (total score) or the sum of the items within an individual domain.

Analytical Methodology

The classic psychometric properties of the ABSST were assessed through the analyses listed below.

Item Completion

Excessive numbers of missing responses and utilization of response categories were analyzed.

Item and Scale Distribution

Items from the ABSST were assessed for floor and ceiling effects and missing data. “Floor effect” refers to a high percentage of patients obtaining the lowest scores possible, while “ceiling effect” refers to a high percentage of patients achieving the highest scores possible.

Reliability (Internal Consistency Reliability)

Internal consistency reliability is determined by measuring the homogeneity of items belonging to the same scale or domain. This criterion was investigated by calculating the Cronbach α for each domain and was considered to be met if the Cronbach α was 0.70 or greater.^19–21 The impact of item removal on internal consistency reliability was also examined. The Cronbach α was calculated with each item removed from its corresponding domain to assess the impact. For example, if the removal of an item causes the α to increase, then that item may not be fitting well into its domain. The ABSST domains and ABSST total score were tested for internal consistency reliability.

Concurrent Validity

Concurrent validity was measured by examining the correlations of the ABSST domains and ABSST total score with the OAB-q SF. A logical pattern of correlations should exist among the instruments. Domains with similar content should correlate more highly than domains with less similar content. If data were normally distributed, correlations were calculated using the Pearson product moment correlation. If the data failed normality tests, the correlations were calculated using the Spearman rank correlation. It was expected that the Bladder Symptoms domain on the ABSST would have a high positive correlation and the Impact of Bladder Symptoms domain on the ABSST would have a moderate positive correlation with the Symptom Severity domain on the OAB-q SF. It was expected that the Coping Strategies domain on the ABSST and the ABSST total score would have moderate-to-high positive correlations with the Total HRQOL score on the OAB-q SF.

Inter-item Correlations

The ABSST was examined for inter-item correlations. Items that correlated highly with one another (α > 0.70) were reviewed for inclusion in the final instrument, as items with such a high correlation may be so similar in content that the instrument may be capturing redundant information.

Predictive Validity, Sensitivity, and Specificity

Logistic regression was used to determine the predictive validity of the ABSST total score to identify patients who would receive a recommendation to see a urologist. The predictive value was based on a blinded clinician response of “yes” or “no” to the question of whether the clinician would recommend that a patient see a urologist based on the patient's responses to the ABSST. This “gold standard” classification of patients who should or should not be referred to a urologist was then compared to the classification based on the ABSST total score.

Results from logistic regression models testing different cut-points predicting the recommendation were summarized to assess the performance of the screening instrument at a specific total score. The different cut-points were used to categorize the patient's need for referral to a urologist. The results were summarized to assess the predictive ability of the screening tool as well as its odds ratio, sensitivity and specificity, positive predictive value (PPV), negative predictive value (NPV), percentage of patients correctly classified, and area under the receiver operating characteristic (ROC) curve.

Specifically, the odds ratio was defined as the odds of an MS patient being referred to a urologist to the odds of the patient not being referred to a urologist. Values greater than 1 indicate increased likelihood of being referred to a urologist; for example, an odds ratio of 1.7 indicates that a patient is 1.7 times more likely to be referred to a urologist. Sensitivity refers to those results that are truly positive results (ie, would refer to a urologist), while specificity refers to those results that are truly negative results (ie, would not refer to a urologist). Minimum criteria for sensitivity and specificity were 75% and 80%, respectively.

Positive predictive value refers to the proportion of positive test results that are true positives (ie, the proportion of patients with positive test results who would actually be referred to a urologist), while the NPV refers to the proportion of negative test results that are true negatives (ie, the proportion of patients with negative test results who would not actually be referred to a urologist). Values closer to 100% indicate a higher proportion of true positives or true negatives for PPV and NPV, respectively. The percentage of correctly classified patients is the percentage of patients correctly classified as either being referred to a urologist or not being referred to a urologist. The area under the ROC curve refers to the ability to correctly classify those who would or would not be referred to a urologist.²² The C statistic is the area under the ROC curve. Values closer to 1 approximate a perfect model.

Results

Qualitative Study (Concept Elicitation, Item Generation, and Cognitive Debriefing)

A total of 20 patients with MS participated in the CE interviews; their ages ranged from 32 to 62 years, with mean of 50 years. Most of the patients were female (80%) and white (95%). Only 10% of the patients were employed full time, and 90% had some college education. Sixty-five percent of patients rated the severity of their MS symptoms over the past 6 months as moderate to severe. Three domains were identified during CE: Bladder Symptoms, Coping Strategies, and Impact of Bladder Symptoms.

Item generation was performed during a 1-day meeting, during which the results of the CE interviews were reviewed. This meeting involved the screening tool development team and two clinicians specializing in the treatment of MS patients (a neurologist and a urologist). A total of 28 items were forwarded on to the cognitive debriefing stage.

A total of 15 patients participated in cognitive debriefing. Most of the patients were female (n = 13, 87%). Their ages ranged from 40 to 79 years, with a mean of 52 years (SD = 10.1).

The tool was shortened to 16 items based on feedback from patients and assessment of clinical utility by the clinicians. One item was added asking patients if they would like to receive help for their bladder problems, for a grand total of 17 items. The questionnaire maintained the domains elicited from the CE interviews: Bladder Symptoms (5 items), Coping Strategies (4 items), and Impact of Bladder Symptoms (7 items). The domains and associated item content are shown in Table 1.

Domains and item content of the ABSST

Multisite Observational Study

A total of 151 patients, all MS patients with and without NDO, were recruited by 28 clinicians in various US geographic locations to participate in the study. Patients had a mean age of 48.2 (SD 12.11) years, with a range of 27 to 66 years. Patients had been diagnosed with MS an average of 9.1 (SD 7.24) years previously. A total of 69.5% of patients described the severity of their MS symptoms over the past 6 months as mild, 23.8% described their symptoms as moderate, 1.3% described them as severe, and 4.6% indicated none or not applicable. Consistent with the stratified recruitment, 41.1% of patients reported having a history of or currently having urinary incontinence and/or urinary urgency. Clinicians reported that they would refer 32.5% of patients to a urologist based on patient responses to the ABSST.

Item Completion and Floor and Ceiling Effects

No items on the ABSST had an excessive number of missing responses, and all response categories were used. Patients had an average ABSST total score of 9.1 (SD 9.86) on a scale from 0 to 48, with higher scores representing greater effect of NDO on symptoms, coping strategies, and impact. Domain and total scores are shown in Table 2. The majority of items had a greater-than-expected number of responses at the low end of the response scale. Both the Coping Strategies and Impact of Bladder Symptoms domains showed a scale-level floor effect, with over 20% of responses yielding the lowest possible scale score. The Coping Strategies domain had 48.6% and the Impact of Bladder Symptoms domain had 50.0% of responses in the lowest response category (Table 3). This is not surprising given that over 50% of patients reported not having a history of or currently having urinary incontinence and/or urinary urgency.

Summary of psychometric results for the ABSST: descriptive analysis

Summary of psychometric results for the ABSST: domain floor/ceiling effect

Internal Consistency Reliability and Inter-item Correlations

All ABSST domains and the ABSST total score met the threshold for good internal consistency (α ≥ 0.70). The Cronbach α values for the Bladder Symptoms, Coping Strategies, and Impact of Bladder Symptoms domains were 0.89, 0.85, and 0.90, respectively. The ABSST total score demonstrated strong internal consistency (Cronbach α = 0.95).

Inter-item correlations were determined using the Spearman rank correlation. A few items in the ABSST were highly correlated. Item 4 (use of urine-leakage protection items such as pads, underpads, shields, protective underwear, and protective bedding during the day) and item 5 (use of urine-leakage protection items such as pads, underpads, shields, protective underwear, and protective bedding during the night) had a correlation of 0.803. Item 1 (urinate right away) and item 7 (need to urinate right away) had a correlation of 0.805. It must be noted here that correlation between items of greater than 0.70 may indicate that the items are capturing similar content, which is an argument for not including one of the items in the final instrument.

Concurrent Validity

The OAB-q SF contains two domains: Symptom Severity (with possible scores of 0–100) and Total HRQOL (with possible scores of 0–100), with higher scores indicating worse symptom severity or worse HRQOL, respectively. Patients had a mean Symptom Severity score of 20.2 (SD 22.91) and a mean Total HRQOL score of 14.9 (SD 20.81), which is in the moderate range of severity for each domain.

The ABSST domains and ABSST total score were both highly correlated with the OAB-q SF Symptom Severity and Total HRQOL scores (Table 4). Every domain had a Spearman correlation coefficient of at least 0.782 (0.70 is the minimum value demonstrating acceptable concurrent validity). The ABSST domains and ABSST total score were both highly correlated with the history of urinary incontinence item (obtained from the inclusion/exclusion criteria) (point biserial correlation [|rpb|] ≥ 0.707) as well as with ABSST item 17, request for help with bladder problem (|rpb| ≥ 0.708) (Table 5).

Summary of concurrent validity results for the ABSST: ABSST and OAB-q SF

Summary of concurrent validity results for the ABSST: ABSST and patient report

Predictive Validity

Logistic regression was used to determine the predictive validity of the ABSST total score in identifying patients who might receive a recommendation to see a urologist (Table 6). The predictive value (dependent variable) in the regression was the blinded clinician rating of referral to a urologist based on review of the patient's responses to the 16-item ABSST. Results from logistic regression models testing different total scores (from 0, indicating no bladder health symptoms or impacts, to 48, indicating severe bladder health symptoms or impacts) were summarized with odds ratio, sensitivity and specificity, PPV, NPV, percentage correctly classified, and area under the ROC curve. Given the criteria stated previously, a total score or cut-point of at least 8 is appropriate for whether a patient should be referred to a urologist. Specifically, 87.4% of patients scoring 8 or higher are correctly classified as actually having a bladder problem.

Predictive validity: summary of the performance of the 17-item ABSST total score at various cut-points in predicting clinician referral to a urologist (N = 151)a

The overall ABSST total score C statistic was equal to 0.928 (where a C statistic equal to 1.00 indicates perfect fit). The corresponding Hosmer-Lemeshow goodness-of-fit test was not significant (P = .5180), indicating the appropriateness of the logistic regression model (Figure 2). In this range, which may be considered mild to moderate, sensitivity and specificity are quite high (85.7% and 88.2%, respectively).

Receiver operating characteristic curve for the performance of the 17-item Actionable Bladder Symptom Screening Tool total score at various cut-points in predicting clinician referral to a urologist

Discussion

The need for a well-developed and validated screening tool for bladder problems in people with MS is well established based on patient-reported symptomatology and impact. Screening tools can be developed through several research avenues, including use of key opinion leader input and/or a literature review of previously validated PRO measures.²³ Our approach to creating a new screening tool was multifaceted and iterative, with both qualitative and quantitative methods used in its development. At each stage of the process, rigorous steering committee feedback (including an informed clinical perspective) was also captured and considered.

Given that the ABSST is a screening tool to be completed by the patient, it was important to capture the patient's “voice” within the instrument and ensure that the correct content was being captured. Our qualitative methods accomplished this, making the screening comprehensible (content validity) from the patient perspective. Classic quantitative test theory was used to further confirm the validity and establish the reliability of the screening tool. Although there is no specific FDA guidance for developing screening tools, the methodologies noted above and described throughout this article adhere closely to the FDA final PRO guidance.¹⁶

The primary objective of this study was to develop and validate a patient-completed screening tool for clinicians—including urologists and neurologists—to use to screen for NDO incontinence in MS patients amenable to treatment. The urinary symptoms most commonly reported by MS patients interviewed were urgency, nocturia, leakage, frequency, and incontinence. The impact of bladder problems included issues that were emotional (feeling embarrassed, frustrated, worried, annoyed, depressed, angry), social (problems going out with friends), and sexual (limited sex life) and involved relationships with the patient's partner, spouse, and/or friends. Having to change soiled clothes, effects of the symptoms on walking, and impact on sleep were also concerns for MS patients.

Several ABSST items and domains showed floor effects. It is important to interpret these results in the context of the characteristics of the patients completing the questionnaire and the intended use of the instrument. A total of 58.9% of patients reported no history of or current urinary incontinence or urgency, so a large number of responses at the bottom of the scales was not unexpected, especially given that the ABSST is intended to be used as a patient-generated screening tool for clinicians. The high floor effects may be indicative of a less variable patient population, affecting correlation instruments. Specifically, we would expect to see similar floor effects in our concurrent measures used in this study.

The scores for the proposed domains of the ABSST and its total score correlated highly with the OAB-q SF scales. In addition, this new instrument meets the current content validity and psychometric testing thresholds established by the FDA. Importantly, the ABSST is a screening tool specific to the MS population that is designed to alert clinicians to potential problems with overactive bladder, while use of the OAB-q SF presupposes that the patient already has overactive bladder. Therefore, the ABSST can be used as a first step to identify those MS patients with potential overactive bladder issues. The decision regarding the desired balance between sensitivity and specificity should be based on disease severity and the potential risks of patient misclassification. These risks may range from nontreatment of symptoms (undersensitive) to the performance of unnecessary tests with their associated costs (underspecified).

Development of a screening tool for clinical use commonly does not go beyond the realm of qualified clinical input. Specifically, these screening tools are generally developed without direct qualitative patient input and feedback. Other avenues of development include the methodologies used by Ibrahim et al.²⁴ 2009 and Jinks et al.²³ 2001. Specifically, Ibrahim et al. used quantitative methodologies with an already established instrument, while Jinks et al. pooled several well-established PRO measures (not intended for screening use) for screening tool development. While both Ibrahim et al. and Jinks et al. used established and respected methods for developing screening tools for psoriatic arthritis and knee pain, respectively, it is the additional patient input gathered from the initiation of the development process that makes the ABSST unique.

Although the ABSST has been developed using a unique methodology, the instrument meets standard thresholds for screening tool validation. The quantitative results of this study showed the ABSST to be a sensitive and specific screening instrument. Specifically, clinicians in the study would have recommended to almost one-third of patients that they consider treatment and possibly see a urologist based on responses to this instrument. This is indicative of the population quotas recruited (ie, 41.1% of the patients reported having a history of or currently having urinary incontinence and/or urinary urgency). The tool also showed good reliability in that both the individual domain scores and the total score met the threshold for good internal consistency using the Cronbach α.^19–21

Established criteria for evaluation of psychometric properties of a screening tool depend on the prevalence of the condition. Concern is generally focused on specificity, as it has a greater impact on predictive values; therefore, a reliable and valid screening tool will also have sufficient ability to differentiate patients who do not have the condition (specificity) from those who do and may need to be referred (sensitivity).²⁵ For the ABSST, with an assumed prevalence of NDO of 75%, criteria for sensitivity and specificity are recommended to be 90% and 80%, respectively.²⁵ At this level, 87.4% of patients will be correctly classified with scores of 8 or greater.

A limitation of this study involves the recruitment of the patient population in the quantitative study. Specifically, as mentioned previously, the average ABSST total score was 9.1 (SD 9.86) on a scale from 0 to 48. The quantitative study was designed to assess use of the ABSST in patients with and without urinary incontinence. More than two-thirds of the patients described their MS as mild within the 6 months preceding the study, and over half reported not having any urinary incontinence or urinary urgency. However, the distribution of mild, moderate, and severe disease in patients in this study is similar to that in the general MS population and suggests that the screening tool is generalizable to this population.

The current version of the ABSST consists of items covering symptoms, coping strategies, and impact of bladder symptoms in MS patients. This tool is designed to provide screening information useful to clinicians in identifying patients who need treatment for MS-related urinary symptoms and who may need to be referred for urologic evaluation.

The ABSST offers benefits to both urologists and neurologists. Urologists sometimes perceive a gap in communication on the part of neurologists regarding MS patients' needs. These patients often are referred for urologic evaluation only after suffering from urologic problems for many years, when the problems may be more difficult to treat. On the other hand, referral of all MS patients for urologic evaluation would be under-specific and not cost effective. Urologists would find an MS-specific bladder health screening tool to be very helpful in the early identification of patients who may need urologic evaluation and treatment. Neurologists may find that this tool allows for quick, simple, and reliable screening, especially for patients who need more comprehensive management of bladder symptoms. If the ABSST is to be adapted to assess the effect of NDO on urinary symptoms in a clinical trial or for a labeling claim, additional validation in a larger population of patients with varying levels of disease severity would be recommended.

PracticePoints

Although bladder symptoms are common in MS patients and can have a major impact on their day-to-day lives, they are often overlooked in the clinical management of the disease.

Both neurologists and urologists would benefit from the availability of a bladder health screening tool specifically for use in MS patients.

The Actionable Bladder Symptom Screening Tool is a reliable, sensitive, and specific screening tool for people with bladder problems related to MS. It may be especially useful in identifying MS patients who could benefit from further urologic evaluation and treatment.