Lower urinary tract symptoms (LUTS)/benign prostatic hyperplasia (BPH) is not a life-threatening malignant disease. Treatment practices are generally not standardized within a center because decision-making may vary according to patients’ preferences and the subjective judgment of the surgeon [1,2]. Bladder outlet obstruction (BOO) is one of the most important components to assess in patients with LUTS/BPH. However, it is difficult to implement urodynamic studies to diagnose BOO routinely in all patients [3].

In our previous study, we developed a nomogram to predict having BOO and requiring prostatic surgery based on urodynamically determined BOO using our urodynamics database that we have developed over the last 13 years (since 2004) [4]. These prediction models are expected to help support physicians in decision-making and in informing patients of their risk of requiring prostatic surgery. As a next step, it is necessary to confirm the transportability of the model into different but related populations. The successful implementation of prediction models in clinical practice generally requires validation of their performance [5]. The successful estimation of model performance is needed to make judgments regarding model reproducibility and model transportability [6].

Clinical prediction models are commonly developed to facilitate diagnostic or prognostic probability estimations in daily medical practice [6]. A user-friendly modality is necessary for the use of such formulas and nomograms to become a widespread part of routine practice. However, any such app or software should be validated through usability testing among potential end-users. In this study, we aimed to externally validate the prediction model for both having BOO and requiring prostatic surgery using 2 independent data sets from tertiary referral centers and to validate a mobile app for using this model through usability testing.

In this study, we externally validated the overall accuracy of previously proposed prediction models for BOO and prostatic surgery. These prediction models showed an excellent discriminant ability in the independent patient cohort. In addition, the models were well calibrated and had high net benefits. They could be used to support physicians in decision-making and in informing patients of their risk of undergoing prostatic surgery in routine clinical use among the general population.

In this external validation study, the prediction model for requiring prostatic surgery showed high values for accuracy, with a sensitivity of 80.6% and a specificity of 73.2%. The validity of the test is measured by sensitivity and specificity [8]. This is because the PPV and NPV depend directly on the prevalence of the disease in the population. Assuming all other factors remain constant, NPVs will increase as prevalence decreases [8]. In the present study, the NPVs were very high (92.0%). This means that the rate of patients requiring BPH surgery in the validation group was lower than in the original group among patients who underwent a urodynamic study.

The calibration plot for requiring prostatic surgery showed very high agreement. A decision curve analysis also showed high net benefits at all probability thresholds. Our predictive models were analyzed to determine whether they made accurate predictions of whether prostatic surgery was required, even without TPV or BOOI. These models are expected to be of great help for surgical decision-making in real clinical practice. However, in the external validation group, there were not many patients with a high probability, i.e., 70%–80% or more, in case of absence of BOOI or prostate volume, we should pay attention to the interpretation of these results. It is necessary to add more cases and revise the nomogram in the near future. The external validity of the prediction model of BOO was high, with an AUC of 79.5%. However, it decreased to 64.8% when TPV was absent. The specificity of the prediction model of BOO was 97.8%, and that of BOO without TPV also had a specificity of 96.7%. In contrast, the sensitivity of the prediction model for BOO was relatively low, with or without TPV, with values of only 24.0% and 11.5%. Several studies have explored the correlation between BOO and TPV. Although TPV can be an important factor in causing BOO, no strict relationship has been found between these parameters [9]. It has been reported that not only prostate volume, but also morphological and/or functional changes, such as intravesical protrusion, the urethral prostatic angle, and the ratio of the transitional zone volume, affect BOO [10-12]. The combination of high specificity and low sensitivity for the prediction of BOO in this study indicates that age, Qmax, PVR, and TPV are the minimum necessary factors that cause BOO, while other factors may act indirectly.

The calibration plot of the prediction model of BOO showed that the overall accuracy was acceptable, although it displayed a tendency towards underestimation at low predicted probabilities and overestimation at high predicted probabilities. This means that conservative judgments are required when using these models to predict BOO in high-probability patients with a very large prostate or a very low Qmax. This was also confirmed by the decision curve analysis. The net benefit changed to a negative value at a probability of approximately 80%. In the BOO prediction model without TPV, although the calibration plot did not deviate much from the ideal line, the decision curve analysis showed that the net benefit was not high for all thresholds. Although the prostate volume is not strictly related to BOO, it can be inferred that predicting BOO without TPV is not accurate. In the case of the absence of a PFS for various reasons, obtaining information on the TPV may have clinical benefits for predicting BOO.

Many algorithms or formulas have been developed to support medical professionals in the diagnosis or management of certain diseases or conditions [13]. However, directly applying such models in daily practice is not an easy task. A mobile or PC app can be used to support physicians in decision-making about patients with specific conditions. Algorithms or formulas can support medical professionals in the diagnosis or management of certain diseases or conditions. We first developed formulas for calculating the probabilities of both BOO and requiring prostate surgery, with the goal of creating a smartphone-based mobile health app and PC-based software to facilitate the widespread use of the formulas. Using such technologies is advantageous compared to conventional paper-based methods. First, data entry and computation are very efficient and fast. Second, such methods are easier, simpler, and more user-friendly. Therefore, the number of health care mobile apps predicting risk factors or prognoses is rapidly increasing. However, an app should be tested and validated among potential end-users before it comes into use in clinical practice [14,15]. If the user interface of the software is not well constructed, user errors can occur, which may lead to potentially serious consequences in the ultimate decision-making process. Therefore, usability testing is as important as developing the formulas.

Our study has a few limitations. First, differences were found in the baseline characteristics of patients between the original development cohort and the external validation cohort. However, no significant differences were found in the clinical parameters of patients who required BPH surgery. It is suggested that this reflects the environmental variation across referral centers, which differ in terms of the patient population, disease severity, socioeconomic status, and/or proximity to primary or secondary referral centers. Patients with more severe symptoms requiring surgery may be referred to an institution similar to the institution where the development cohort was created. It was also observed that the percentages of patients who received PFS and required surgery were 71.8% and 24.7%, respectively. However, no differences were found in the major parameters of patients who required surgery, such as BOOI, TPV, total IPSS, and the IPSS voiding subscale score. This means that patients who were referred to the 3 centers differed in their clinical characteristics, but that the decision about prostatic surgery was made on the basis of similar criteria in these 3 independent institutions. Therefore, it seems that there was no problem regarding the criteria for surgical decision-making in the development cohort.

Second, in this study, a usability test was performed among a relatively small number of participants at the end of the user-interface development. Additionally, the PC version of the software was not tested. There are 2 different types of usability tests in medical device development: formative and summative usability tests. A formative test is conducted during the development of the medical device to identify product strengths and shortcomings. The present test was performed as a summative test, where there should be at least 25 users to reveal subtle interaction issues that might not be identified in a test involving fewer participants [16]. However, during the development of the apps, the authors circulated and reviewed development prototype versions more than 10 times before the usability test. Therefore, we believe that most user errors and interface errors were corrected before the test.

In conclusion, external validation of the newly developed probability models demonstrated a moderate level of discrimination, adequate calibration, and high net benefits for predicting both having BOO and requiring surgery. Moreover, usability testing showed that the smartphone app was user-friendly and that no major human errors occurred in its use.