Objective To compare the performance of the Focus HerpeSelect-2 enzyme immunoassay (EIA) with the gold standard herpes simplex virus (HSV) type 2 western blot, among HIV-1-uninfected men and women in east and southern Africa.
Methods 3399 HIV-1-uninfected women and men from seven countries in east and southern Africa were tested for HSV-2 antibody using the Focus HerpeSelect-2 EIA. The performance of the HerpesSelect-2 EIA was compared with the gold standard HSV-2-specific western blot.
Results Two-thirds (2294/3399) of participants were male and two-thirds (2242/3399) were from east Africa. By western blot testing, HSV-2 prevalence was 68%; 59% in men and 85% in women. At the manufacturer's recommended cut-off value of greater than 1.1, the HerpeSelect-2 EIA had a sensitivity of 98.3% and specificity of 80.3%. Receiver operating characteristic plot analysis indicated that the optimum cut-off was 2.1 or greater, with sensitivity 93.9% and specificity 90.5%. Diagnostic accuracy was modestly higher for southern Africa (area under the curve (AUC) 0.979, 95% CI 0.970 to 0.988) compared with east Africa (AUC 0.954, 95% CI 0.942 to 0.965; p<0.001 for southern vs east Africa).
Conclusions The Focus HerpeSelect-2 EIA has acceptable diagnostic accuracy for the determination of HSV-2 serostatus in African HIV-1-uninfected adults. An assay cut-off value of 2.1 or greater results in approximately 90% sensitivity and specificity, against a gold standard HSV-2 western blot. Diagnostic accuracy differed slightly by geographical region.
- Focus HerpeSelect-2 EIA
- western blot
Statistics from Altmetric.com
Herpes simplex virus (HSV) type 2 is the most frequent cause of genital ulcer disease worldwide and is an important risk factor for HIV-1 acquisition.1 The HerpeSelect-2 enzyme immunoassay (EIA) (Focus Technologies, Cypress, California, USA) is a commercially available, type-specific serological test for the detection of antibodies to HSV-2 glycoprotein G (gG) that is frequently used in epidemiological research studies of HSV-2. However, the HerpeSelect-2 EIA has been reported to have poor specificity for serological diagnosis of HSV-2 among some African populations, particularly for samples with index values (ie, the ratio of the optical density of the sample to the optical density of a standard calibrator) in the low positive range (values between 1.1 and 3.4).2–4 Studies comparing the HerpeSelect-2 EIA with gold standard assays, such as HSV-2-specific western blot, have proposed various cut-offs (index values 3.1–3.5) to improve specificity,2 3 5 6 but these were generally done with relatively small populations, only among men or women, or in single geographical areas. We compared the performance of the Focus HerpeSelect-2 EIA with the gold standard for HSV-2 serological diagnosis, HSV-2 western blot, among nearly 3400 HIV-1-uninfected men and women from seven countries in east and southern Africa.
Population and procedures
Between November 2004 and April 2007, 3408 HSV-2/HIV-1-co-infected women and men and their HIV-1-uninfected heterosexual partners were enrolled in the Partners in Prevention HSV/HIV Transmission Study, a randomised clinical trial of acyclovir HSV-2 suppressive therapy to reduce HIV-1 transmission (http://ClinicalTrials.gov/ number NCT00194519). All participants were 18 years of age or older, and HIV-1-uninfected partners could be either HSV-2 seropositive or seronegative. Couples were from 14 sites in seven African countries in east (Kenya, Rwanda, Tanzania and Uganda) and southern Africa (Botswana, South Africa and Zambia). As previously reported, HSV-2 suppression provided to the HIV-1-infected partners did not reduce HIV-1 transmission risk to their initially HIV-1-uninfected partners.7 For the present study, we assessed the HSV-2 serological status of the HIV-1-uninfected partners from a blood sample collected at the enrolment visit.
Institutional review boards at the University of Washington and at all collaborating site organisations approved study procedures. All participants provided written informed consent.
At study enrolment, HIV-1-uninfected partners provided a serum sample for HSV-2 serological testing using the HerpeSelect-2 EIA; 12 laboratories performed the testing for the 14 study sites. The manufacturer's instructions for this assay define a negative result as an index value less than 0.9, an indeterminate result as an index value between 0.9 and 1.1, and a positive result as an index value greater than 1.1. Archived serum aliquots from the enrolment visit were also batch tested at the end of the study at the University of Washington using a HSV type-specific western blot.8 Western blot readers were blinded to the results of the HerpeSelect-2 EIA. All sites participated in an external quality assurance programme using a HSV-2 proficiency panel developed at the University of Washington.9
The sensitivity and specificity of Focus HerpeSelect-2 EIA results, compared with western blot, were calculated, and receiver operating characteristic (ROC) curves were constructed to describe test performance. Optimal EIA index result cut-offs were identified, aiming to achieve both test specificity and sensitivity of approximately 90%. Individuals with indeterminate western blot results (ie, HSV-2 gG band indistinct or not apparent after pre-absorption against HSV-1 antigens)10 were excluded from sensitivity/specificity and ROC analyses. To assess the effect of between-laboratory variation on our estimates of test performance, we calculated adjusted ROC analyses that controlled for clustering by laboratory.11 Statistical calculations were performed using PASW Statistics 17.0 and Stata 10.1.
Of 3408 HIV-1-uninfected participants enrolled in the study, HerpeSelect-2 EIA and western blot results were available for 3399 (99.7%). Of these, 2294 (67%) were male and 2242 (66%) were from east Africa. The median age was 34 years (IQR 28–41). By western blot testing, HSV-2 seroprevalence was 68% (2304/3399), including 85% (944/1105) in women, 59% in men (1360/2294), 68% (1532/2242) in east Africa and 67% (772/1157) in southern Africa. One hundred and nine participants (3%) had indeterminate (n=109) western blot results.
There was generally strong agreement between HerpeSelect-2 EIA and western blot results (table 1). Using the manufacturer's recommended cut-off (index value >1.1) and compared with western blot, the HerpeSelect-2 EIA had a sensitivity of 98.3% (2264/2304) and specificity of 80.3% (792/986) compared with western blot for determination of HSV-2 infection (table 2). At an index value cut-off of 3.5 or greater, sensitivity was 82.9% (1888/2304) and specificity improved to 95.1% (938/986). ROC plot analysis comparing a range of HerpeSelect-2 cut-offs versus gold standard western blot indicated strong agreement between the two tests: (area under the curve (AUC) 0.962, 95% CI 0.954 to 0.971). An optimum cut-off of 2.1 or greater was identified, giving a sensitivity of 93.9% and a specificity of 90.5%.
We assessed whether demographic factors resulted in differences in the diagnostic accuracy of the HerpeSelect-2 EIA. By ROC testing, overall test performance was modestly better in men (AUC 0.960, 95% CI 0.951 to 0.969) than in women (AUC 0.944, 95% CI 0.914 to 0.973; figure 1A), although this was not statistically significant (p=0.3). Diagnostic accuracy was modestly greater for samples from southern Africa (AUC 0.979, 95% CI 0.970 to 0.988) compared with east Africa (AUC 0.954, 95% CI 0.942 to 0.965; figure 1B) and this difference was statistically significant (p<0.001). The optimal index value cut-off was 2.7 in east Africa (sensitivity 89.8%, specificity 90.3%) and 1.3 in southern Africa (sensitivity 96.8%, specificity 90.4%). There were small differences in test performance between countries in the same geographical region (table 3). Among samples from Uganda, specificity was 81% at an index value of 2.1 or greater, and the optimal index value was 3.4 (sensitivity 90.4%, specificity 90.1%). Test performance did not differ for those with or without a history of genital ulcer disease in the 3 months before enrolment (AUC 0.903, 95% CI 0.732 to 1.000 vs AUC 0.954, 95% CI 0.945 to 0.962; p=0.50). In addition, age had no effect on test performance: age 25 years or less (AUC 0.968, 95% CI 0.951 to 0.985) vs age greater than 25 years (AUC 0.961, 95% CI 0.951 to 0.970; p=0.38).
We investigated whether between-laboratory variation accounted for differences in diagnostic accuracy. We recalculated the AUC, 95% CI and p values for the above comparisons with adjustment for clustering by laboratory, and obtained results that were essentially identical to the unadjusted analyses. In the adjusted models, there were no statistically significant differences in adjusted AUC estimates for comparisons of gender (p=0.43), genital ulcer disease history (p=0.51), or age group (p=0.47). A statistically significant effect of region remained present in the analysis adjusting for site laboratory (adjusted AUC 0.978, 95% CI 0.961 to 0.995 for southern Africa vs 0.953, 95% CI 0.939 to 0.966 for east Africa, p=0.02).
We considered the proportion of individuals with positive and indeterminate western blot results at various low positive HerpeSelect-2 EIA values. For those with EIA results in the manufacturer's indeterminate range (ie, 0.9–1.1, n=62), 29% were positive and 18% indeterminate by western blot. For those with EIA results between 1.2 and 2.4, 55% were positive and 13% indeterminate by western blot. Finally, for those with EIA results between 2.5 and 3.4, 80% were positive and 6% indeterminate by western blot. Overall, of the 109 persons with indeterminate western blots, 75 (69%) had HerpeSelect-2 EIA values greater than 1.1 and 37 (34%) greater than 2.4 (table 1). There were no significant differences in the frequency of indeterminate HSV western blots by gender (p=0.12) or region (p=0.10).
This multinational study of nearly 3400 HIV-1-uninfected adults represents the largest investigation of the performance of the Focus HerpeSelect-2 EIA to date in sub-Saharan Africa. In this population with high HSV-2 seroprevalence, a Focus HerpeSelect-2 EIA index value cut-off of 2.1 or greater resulted in approximately 90% sensitivity and specificity. There was some variation in optimal cut-off values by subgroups defined by gender and region. Of the approximately 15% of sera that had index values 1.2–3.4, two-thirds were confirmed as HSV-2 seropositive by western blot.
Previous studies have demonstrated reduced specificity of the HerpeSelect-2 EIA at an index value cut-off greater than 1.1 for samples from Africa4 6 potentially due to cross-reactivity with HSV-112 or HSV-2 sequence diversity.13 In our study population from east and southern Africa, an index cut-off value of 2.1 or greater improved test specificity to approximately 90%, similar to a meta-analysis that suggested index values between 2.2 and 3.5 increased specificity14; in our population, a cut-off of 2.1 or greater also provided a sensitivity of approximately 94%. Studies from Rakai, Uganda, have suggested higher index value cut-offs improve specificity, with an optimal trade-off between sensitivity and specificity identified at index values of 3.2–3.4.3 15 Our results for east Africa in general and Uganda in particular, are consistent with those findings. For samples from southern Africa, we found an index cut-off value of 1.3 provided an acceptable trade-off between sensitivity and specificity, lower than the optimal index value of 3.3 from one previous study from that region.5 Regional differences in test specificity could be a result of cross-reacting antibodies due to possible HSV-2 antigenic differences.3 Importantly, between-laboratory variations in test performance did not appear to account for differences in diagnostic accuracy. We also found gender differences in test performance, although this was not statistically significant; at each cut-off value, test sensitivity was better in women while specificity was higher in men. The potential for differential test performance by gender may merit further investigation.
Although improving test specificity is important in epidemiological studies and for clinical diagnosis, shipping sera to a reference laboratory for western blot confirmatory testing is neither feasible nor cost-effective. Studies have suggested that sequential serological testing with the Focus HerpeSelect-2 assay plus other commercially available type-specific assays, such as Kalon HSV-2 gG2 ELISA (Kalon Biological Ltd, Surrey, UK) or Biokit HSV-2 rapid test (Biokit USA Inc, Lexington, Massachusetts, USA) could be feasible alternatives to western blot for confirmatory testing of sera that are initially positive on the HerpeSelect-2 EIA,16 although other studies have not demonstrated that serial algorithms improve diagnostic accuracy.6 17 The approximately 90% sensitivity and specificity values we calculated with modified cut-offs may not be sufficiently precise for the use of this assay for serological diagnosis of HSV-2 in all clinical care settings.
In summary, the Focus HerpeSelect-2 EIA had an acceptable diagnostic accuracy in HIV-1-uninfected women and men from east and southern Africa, compared with gold standard HSV-2 western blot, particularly when using an index cut-off of 2.1 or greater.
The Focus HerpeSelect-2 EIA has good diagnostic accuracy in east and southern Africa.
Using an index value of 2.1 or greater to define a positive result, compared with gold standard HSV-2 western blot, resulted in approximately 90% sensitivity and specificity.
Diagnostic accuracy was modestly better for samples from southern Africa compared with those from east Africa.
The authors are especially thankful to Anne Cent from the University of Washington Clinical Virology Laboratory for performing HSV western blot testing and to Katherine Thomas for assistance with statistical analyses. The authors are grateful to the study participants for their participation and dedication. They wish to thank the study team members at the research sites and at the University of Washington for their contributions to study implementation and data collection.
Partners in prevention HSV/HIV transmission study team
University of Washington coordinating centre and central laboratories, Seattle, USA
Connie Celum (principal investigator), Anna Wald (protocol co-chair), Jairam Lingappa (medical director), Jared M Baeten, Mary Campbell, Lawrence Corey, Robert W Coombs, James P Hughes, Amalia Magaret, M Juliana McElrath, Rhoda Morrow, James I Mullins
Study sites and site principal investigators:
Cape Town, South Africa (University of Cape Town): David Coetzee; Eldoret, Kenya (Moi University, Indiana University): Kenneth Fife, Edwin Were; Gaborone, Botswana (Botswana Harvard Partnership): Max Essex, Joseph Makhema; Kampala, Uganda (Infectious Disease Institute, Makerere University): Elly Katabira, Allan Ronald; Kigali, Rwanda (Rwanda Zambia HIV Research Group and Emory University): Susan Allen, Kayitesi Kayitenkore, Etienne Karita; Kisumu, Kenya (Kenya Medical Research Institute, University of California San Francisco): Elizabeth Bukusi, Craig Cohen; Kitwe, Zambia (Rwanda Zambia HIV Research Group and Emory University): Susan Allen, William Kanweka; Lusaka, Zambia (Rwanda Zambia HIV Research Group and Emory University): Susan Allen, Bellington Vwalika; Moshi, Tanzania (Kilimanjaro Christian Medical College, Harvard University): Saidi Kapiga, Rachel Manongi; Nairobi, Kenya (University of Nairobi, University of Washington): Carey Farquhar, Grace John-Stewart, James Kiarie; Ndola, Zambia (Rwanda Zambia HIV Research Group and Emory University): Susan Allen, Mubiana Inambao; Orange Farm, South Africa (Reproductive Health Research Unit, University of the Witwatersrand): Sinead Delany-Moretlwe, Helen Rees; Soweto, South Africa (Perinatal HIV Research Unit, University of the Witwatersrand): Guy de Bruyn, Glenda Gray, James McIntyre; Thika, Kenya (University of Nairobi, University of Washington): Nelly Rwamba Mugo.
Data management was provided by DF/Net Research, Inc (Seattle, USA) and site laboratory oversight was provided by Contract Lab Services (University of the Witwatersrand, Johannesburg, South Africa).
Funding This study was supported through research grants from the Bill & Melinda Gates Foundation (grant ID 26469) and the US National Institutes of Health (National Institute of Allergy and Infectious Diseases grant P01 AI030731). The funders played no role in design, data collection, analysis, interpretation, or writing of the report.
Competing interests None.
Patient consent Obtained.
Ethics approval This study was conducted with the approval of the University of Washington, Human Subjects Division and all collaborating site institutions.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.