Article Text

Modelling the cost-effectiveness of introducing rapid syphilis tests into an antenatal syphilis screening programme in Mwanza, Tanzania
1. P Vickerman1,
2. R W Peeling2,
3. F Terris-Prestholt1,
4. J Changalucha1,
5. D Mabey1,
6. D Watson-Jones1,
7. C Watts1
1. 1London School of Hygiene & Tropical Medicine, London, UK
2. 2Sexually Transmitted Disease Diagnostic Initiative, World Health Organization WHO/TDR, Geneva, Switzerland
1. Correspondence to:  Dr Peter Vickerman  London School of Hygiene and Tropical Medicine, Keppel Street, London, WC1E 7HT, UK; peter.vickerman{at}lshtm.ac.uk

Abstract

Objectives: A study found screening (with rapid plasma reagin (RPR)) pregnant women for maternal syphilis was cost-effective in Mwanza, Tanzania. Recently, four rapid point-of-care (POC) syphilis tests were evaluated in Mwanza, and found to have reasonable sensitivity/specificity. This analysis estimates the relative cost-effectiveness of using these POC tests in the Mwanza syphilis screening intervention.

Methods: Empirical cost and epidemiological data were used to model the potential benefit of using POC tests instead of RPR. Reductions in costs relating to training, supplies, and equipment were estimated, and any changes in impact due to test sensitivity were included. Additional modelling explored how the results vary with prevalence of past infection, misclassified RPR results, and if not all women return for treatment.

Table 1

Model parameters for estimating the cost-effectiveness of the POC test. Cost estimates are in 2005 US$COSTr = ST+(EcδE)+TδT+(Δdrtro)+(PlδP+Po)+RδR Equation 5 where most parameters are defined as before, although Es and Ef are not needed because POC tests do not require a shaker or fridge, and Δrt and Δrd are the test and drug costs when using POC tests: Δrt = CrN(1+δW) Equation 6 Δrd = Cdrug[NRPRθSr+(1−Spr)(NNRPRθ)] Equation 7 where Cdrug is the penicillin cost per woman treated, Cr is the POC test cost, and δW is the relative wastage of POC tests. Model parameters The model was mainly parameterised using data from three studies undertaken in Mwanza, with uncertainty ranges being assigned to most model inputs (table 1). Cost and intervention output data were obtained for 1998–1999 from Terris-Prestholt et al.4 Data on the accuracy of the clinic RPR test compared to the RPR test undertaken at a reference laboratory (table 1), and the number of screened women that would have tested positive with a laboratory based RPR or TPHA test, were obtained from a prospective cohort study undertaken in 1997–1999.3,21 Lastly, sensitivity/specificity estimates for the four syphilis POC tests when compared to the TPPA test undertaken in a laboratory setting were obtained from an evaluation of these tests undertaken in Mwanza in 200419 (table 1 and appendix 1: to view appendices 1-6 visit the STI website—http://www.stijournal.com/supplemental). The tests were evaluated on blood in an antenatal clinic (ANC) setting and on blood and serum in a laboratory setting. The ANC results were used to estimate the impact of using POC tests on blood, and because of a lack of other data the laboratory results were used to estimate the impact of using the tests on serum. The sensitivity of the tests for HTAS was also evaluated, but only on 13–20 HTAS samples. Table 1 also includes the current WHO procurement cost for the POC tests. The POC tests were also assumed to need less training and supply deliveries, and no centrifuge was needed if the tests were used on whole blood. Cost-effectiveness of intervention with POC test Using equations 3 and 5, the cost-effectiveness of using a specific POC test (CEr) can be estimated by dividing the estimated cost of the intervention while using that POC test by projections of the DALYs saved from averting adverse birth outcomes: Because of uncertainty in many model inputs, an uncertainty analysis was undertaken on the projected cost (COSTr) and impact (NrHi) of using each POC test. Uniform distributions were associated with the uncertainty intervals for most model parameters except the sensitivity and specificity of the POC tests which were assumed to be normally distributed. For each test, random sampling of the parameter uncertainty ranges was used to obtain 500 parameter sets. These produced paired estimates for the impact and cost of the intervention. The distribution obtained was used to obtain uncertainty bounds for the intervention’s cost-effectiveness, which was compared to the cost-effectiveness of the RPR test.4 A multi-linear regression analysis was then undertaken to determine which parameters contributed most to the uncertainty in the cost-effectiveness projections. Threshold cost of POC test The threshold cost (Cr*) at which a POC test becomes as cost-effective as the RPR test can be obtained from equation 8 to give (see appendix 3 for the derivation): For each POC test, the cost-threshold was calculated for different HTAS sensitivities while other parameters were varied to obtain the lowest cost-threshold at that sensitivity. Sensitivity analysis to generalise to other settings To generalise the results of this analysis to other settings, the effect of variations in key parameters on the cost-effectiveness of using the RPR or Bioline test (on blood) was explored. Firstly, many women screened with the RPR test do not receive treatment at their initial visit6,11,12,15,22 or do not wait long enough for treatment, and so the cost-effectiveness of the RPR test was estimated for different return rates (60–80%).6,10,11,22 Secondly, because of the difficulty in interpreting RPR test results in clinic settings11,13,14 and the frequent occurrence of RPR biological false positives,13,17,18,23 the cost-effectiveness of the RPR test was estimated for different sensitivities to HTAS (75–100%),11,14 and for different prevalences of biological false positives (NRPR was varied between 563–130113,17). In contrast, the cost-effectiveness of the POC test was estimated for different prevalences of old infection by varying the ratio θ between 104–188%.17,24,25 Lastly, the cost-effectiveness of both tests was estimated for a wide range of syphilis prevalences (5–21%). The combined effect of these sensitivity analyses on the cost-threshold for each POC test was analysed by incorporating the return rate for the RPR test into equation 9. RESULTS Cost-effectiveness of POC tests The estimated cost-effectiveness of using the four syphilis POC tests on whole blood or serum are shown in table 2. The results show that all POC tests, except the Determine test, will on average result in more DALYs saved from averting adverse birth outcomes than the RPR test. This is due to their higher sensitivity for HTAS. However, because of the high cost of each POC test, they all result in higher costs, about 6% greater with Bioline test, 21–23% with Visitect and Syphcheck tests, and 36–39% with Determine test. This results in all POC tests, except Bioline, being less cost-effective than the RPR test (ranging from 15–41% higher cost per DALY saved than RPR). In contrast, although our projections suggest the Bioline test is on average more cost-effective than RPR, 49% of the simulations from the uncertainty analysis suggest otherwise (when used on blood). This is highlighted in appendix 4, which shows the incremental (additional) cost and effectiveness of each POC test relative to the RPR test from the uncertainty analysis. The plots show that there is substantial uncertainty in the incremental impact of the POC tests, but little variation in the incremental cost. Table 2 Cost-effectiveness per DALY saved (2005 US$) of each POC test using Tanzanian life expectancy. Uncertainty bounds are in brackets

When a multi-linear regression analysis was undertaken on the input and output from each uncertainty analysis, the variability in the cost-effectiveness of each POC test was found to be mainly dependent on the uncertainty in the sensitivity of the POC tests for HTAS. This is illustrated in fig 1, and shows that for each 5% increase in the sensitivity of the POC tests there is an approximate $1 improvement in their cost-effectiveness ratio. Figure 1 Projected cost-effectiveness (per disability-adjusted life year (DALY) saved in 2005 US$) of each syphilis point-of-care (POC) test for different assumptions about their sensitivity for high titre active syphilis (HTAS). The projections assume that the tests are used on blood in an antenatal clinic setting and the bounds refer to the variability due to the uncertainty in the other parameters. CE, cost-effectiveness; RPR, rapid plasma reagin.

Cost-effectiveness of RPR and POC tests in other settings

A sensitivity analysis was undertaken to explore how the cost-effectiveness of the RPR or POC test changes between settings. The results showed that the prevalence of syphilis, past infections, or RPR false positives has little effect on the cost-effectiveness projections. In contrast, the cost-effectiveness of the RPR test worsens by 15% if its sensitivity for HTAS had been 75% instead of 86%, and by 25–65% if 20–40% of women had not returned for treatment. In these two scenarios, the Bioline POC test is highly likely to be more cost-effective than the RPR test, whereas the other POC tests will only be more cost-effective if the return rate for treatment is towards the lower end of the range considered or their sensitivity for HTAS is towards the high end of the estimated confidence intervals. In contrast, if the RPR test’s sensitivity for HTAS had been 100% then the cost-effectiveness of the RPR test would have improved by 14%. In this situation, the POC tests will only be as cost-effective as the RPR test if they are much cheaper than their current cost, or not all women return for treatment with the RPR test. Appendix 6 shows how the cost-effectiveness of using the RPR or Bioline test changes for these specific scenarios.

Cost thresholds for POC tests

The results in fig 1 show that at current prices only the cheapest POC test (Bioline) could be as cost-effective as the RPR test in Mwanza. Indeed, if they are 100% sensitive for HTAS, the cost of the POC tests has to be less than $0.63 for their cost-effectiveness to be similar to the RPR test. However, if the sensitivity of the POC tests is only 85%, then even the Bioline test would have to be at least$0.13 cheaper than its current cost ($0.47). Appendix 5 shows the threshold costs for each POC test, and shows that for a specific HTAS sensitivity, the cost-thresholds only vary by$0.04 between tests and whether they are used on blood or serum.

For different POC and RPR test sensitivities and return rates, fig 2 shows the lower-bound cost-thresholds for the POC tests. The dashed line signifies the projections for Mwanza (RPR test sensitivity for HTAS = 86%). The figure shows that for every 5% decrease in the return rate or sensitivity of the RPR test for HTAS, the cost of a POC test can be about $0.10 greater. In contrast, for every 5% decrease in the sensitivity of the POC test for HTAS, its cost has to decrease by about$0.12.

Figure 2

Threshold costs (in 2005 US$) for POC test to be as cost-effective as RPR test for different sensitivities of POC and RPR test for HTAS and different return rates for treatment with the RPR test. Dashed line signifies the cost thresholds for Mwanza. HTAS, high titre active syphilis; POC, point-of-care; RPR, rapid plasma reagin. DISCUSSION This study compared the likely cost-effectiveness of four syphilis POC tests against the cost-effectiveness of the RPR test in an antenatal syphilis screening intervention in Mwanza, Tanzania. Uncertainty in the POC test sensitivity for HTAS resulted in uncertainty in their likely impact and cost-effectiveness. Despite this uncertainty, the model projections suggest that all the POC tests, except Determine, are likely to result in marginally greater impact than the RPR test due to their higher HTAS sensitivity in this setting. However, at current prices, only the Bioline test may be as cost-effective, and will at best result in an 11% improvement in the cost-effectiveness ratio (at 100% HTAS sensitivity). In contrast, the other POC tests need to be 33–50% cheaper to be as cost-effectiveness as the RPR test. For example, if the POC tests are 95% sensitive for HTAS, then they would need to cost ∼$0.54 per test (if used on blood), whereas for each 5% reduction in sensitivity the price of the POC tests need to drop by about $0.10. These results illustrate that only Bioline can be considered as a replacement for RPR in Mwanza, and highlights the importance of HTAS sensitivity on threshold prices for POC tests. The antenatal clinic in Mwanza was well equipped with a centrifuge, electric shaker and refrigerator. This meant the RPR test could be done relatively quickly on-site, and all women were tested and treated during the same visit. In other less well equipped clinics, or in settings without electricity, blood samples are sometimes sent to central laboratories for testing, or are left to stand to separate the serum needed for RPR testing. These delays can result in up to 20–40% of screened women not receiving treatment.7,10–12 In these settings the model projects the cost-effectiveness of using the RPR test could worsen by 25% and 65% for an 80% and 60% return rate, respectively. Even at the cheaper of these cost-effectiveness levels, it is highly likely that the Bioline test will be more cost-effective than the RPR test, whereas the Syphcheck and Visitect tests will be more cost-effective if their sensitivity to HTAS is greater than 80%. This illustrates the usefulness of POC tests in resource poor settings, because they can be used on whole blood, and do not require electrical equipment. The sensitivity of the RPR test used at the Mwanza clinic was quite low for HTAS (86.2%). Had the intervention achieved 100% sensitivity, the intervention’s cost-effectiveness would have improved by 14%, and none of the POC tests would have been more cost-effective at current prices. This illustrates that the RPR test is likely to be the best choice in settings with good facilities and well trained staff. However, the conditions for undertaking the RPR test will be far from perfect in many clinics in developing countries.10,13 If the RPR test had been 75% sensitive for HTAS,14 then the Bioline test would be more cost-effective if its HTAS sensitivity is above 80%, whereas the Visitect and Syphcheck tests would need an HTAS sensitivity above 95%. All the POC tests would be more cost-effective if the RPR test had a HTAS sensitivity below 65%. This again illustrates the benefit of using POC tests in settings that are not ideal for the RPR test. Another important question is whether our results would vary in other epidemiological settings. However, although this analysis showed that the cost-effectiveness of each test is highly dependent on syphilis prevalence, the relative cost-effectiveness of the POC tests compared to RPR is not. This implies that syphilis prevalence should not be used as a decision criterion for using either type of test. One weakness of current POC tests is that individuals remain positive after resolution of infection. This means their use could result in substantial over-treatment, especially if syphilis treatment has been occurring.17,18,24,25 Similarly biological false positive RPR results can also result in substantial over-treatment.17,18,23 However, apart from the possible negative social implications of wrongly diagnosing women for syphilis,26 this analysis showed that the effect of over-treatment on the cost-effectiveness of either type of test is small because treatment costs are relatively small. For example, if the prevalence of resolved infections was fourfold greater than in Mwanza, then the cost-effectiveness of using POC tests only worsens by 3%. This illustrates that the prevalence of biological false positives or resolved infections is unlikely to be important in deciding whether to use POC or RPR tests in a particular setting, except if there are major negative implications associated with wrongly diagnosing women. However, because of the substantial over-treatment that can occur with either test, care should always be taken in emphasising that the test is being used for screening and would normally require confirmatory testing, but given the potential threat for the unborn child treatment is given even when this cannot be done. This is especially important when treatment strategies also invite partners for treatment. Key messages • More studies need to be done evaluating the sensitivity of point-of-care (POC) tests for high titre active syphilis (HTAS). However, if the limited data suggesting POC tests have a high sensitivity for HTAS are correct, POC tests could result in similar numbers of “at risk” women being treated for maternal syphilis. • Unless the cost of available POC tests reduces to below US$0.63, the rapid plasma reagin (RPR) test is likely to be a more cost-effective option for maternal syphilis screening interventions in well equipped settings with well trained staff such as Mwanza.

• In settings where not all screened women are treated because of delays between testing and treatment, or the RPR test has a lower sensitivity for high titre active syphilis (HTAS) than in Mwanza, then POC tests could improve the cost-effectiveness of maternal syphilis screening interventions. This is likely to include many rural or resource-poor settings.

• Because POC tests are not dependent on expensive equipment or electricity, they could enable the scaling up of maternal syphilis interventions to settings that would otherwise be hard to reach due to operational difficulties in using the RPR test.

CONCLUSION

More studies need to be done evaluating the sensitivity of POC tests for HTAS. However, if the limited data suggesting POC tests have a high sensitivity for HTAS are correct, POC tests should be considered as an alternative to the RPR test in many settings. In Mwanza, the Bioline, Visitect and Syphcheck tests are all likely to have resulted in more “at risk” women being treated, and the Bioline test would have cost a similar amount per DALY saved. Indeed, the Bioline test is likely to be more cost-effective than the RPR test in settings where either not all screened women are treated, or the RPR test has a lower sensitivity for HTAS than in Mwanza. This is likely to include many settings where maternal syphilis screening currently takes place,5,6,10–13,15,22 and specifically many rural or resource-poor settings. Alternatively, the RPR test is likely to be a better option in well equipped settings with well trained staff. For the other POC tests to be as cost-effective as the RPR test and competitive with the Bioline test, they need to be at least as cheap as the Bioline test or highly sensitive for HTAS. Lastly, because POC tests are not dependent on expensive equipment or electricity, they could enable the scaling up of maternal syphilis interventions to settings that would otherwise be hard to reach due to operational difficulties in using the RPR test. This is an important advantage of POC tests, and their use in low coverage areas should be seen as an important priority for health providers.

Acknowledgments

This research was funded by WHO/TDR and the Wellcome Trust. David Mabey, Charlotte Watts, Deborah Watson-Jones and Peter Vickerman also receive funding from the DFID funded AIDS Knowledge Programme. The views expressed are those of the authors and cannot be taken to reflect the official opinion of the London School of Hygiene and Tropical Medicine.

AUTHOR CONTRIBUTIONS  PV participated in the planning of the analysis, collated already published and unpublished data from Mwanza, developed and adapted the impact and cost model for estimating the cost-effectiveness of the POC tests, performed all simulations and sensitivity analysis, interpreted results and wrote the manuscript. FTP, RWP, DM, JC, DWJ and CW all helped in interpreting the model results, and contributed to the manuscript. In addition, FTP, RWP, DM, and CW all helped in planning the analysis, FTP developed the initial cost-effectiveness spreadsheet model, RWP and DM had the initial idea for the study and gave continual expert advice, and JC and DWJ provided published and unpublished data and expertise.

Supplementary materials

• Files in this Data Supplement:

Footnotes

• Competing interests: none declared

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.