Infection with herpes simplex virus type 2 (HSV-2) is a key risk factor for HIV in sub-Saharan Africa.1 Serological testing for HSV-2 may be clinically useful in this setting and elsewhere as an indicator of an individual’s risk of HIV infection, for accurate calculation of per-contact risk of transmission of HIV, for differential diagnosis of genital ulcers and, potentially, for interventions that focus on HSV-2 status to prevent acquisition or transmission of HIV.1 One of the most commonly used type-specific HSV-2 ELISAs in sub-Saharan Africa, the HerpeSelect ELISA test (Focus Diagnostics, Cypress, California, USA), has been shown to have relatively poor specificity2 3 when compared with the monoclonal antibody (MAb) ELISA and the University of Washington Western blot gold standard in specific African populations.

A study in five African countries found that the HSV-2 HerpeSelect yielded similar results to the Western blot in samples from South Africa and Zimbabwe.4 However, samples from Kenya and Uganda4 were found to have higher HSV-2 positivity detected than the Western blot suggesting the possibility of false-positive HerpeSelect HSV-2 results. Another possible explanation is lower sensitivity of the Western blot assay. As there is potential for under-detection of HSV-2 antibodies among recent seroconverters using the Western blot assay,5 additional performance research has incorporated HSV-2 recombinant gG ELISA inhibition testing as an alternative reference gold standard to the Western blot.4 This recombinant inhibition test measures antibody binding to multiple epitopes of HSV-2 glycoprotein G (gG2) and uses the differential absorption of type-specific antibodies to identify potential false-positive results.4

To evaluate the performance of HSV-2 serological testing among young men in Kisumu, Kenya, who were HIV-seronegative, we conducted a study within the framework of a randomised controlled trial (RCT) of male circumcision for the prevention of HIV infection.6 We present here the performance of the type-specific HSV-2 HerpeSelect ELISA and Kalon HSV-2 (Kalon Biological Ltd, Aldershot, UK) ELISAs compared with two confirmatory assays of the Western blot and recombinant gG ELISA inhibition testing.

METHODS

Study population and enrolment

Uncircumcised men aged 18–24 years of age in Kisumu, Kenya, were invited to participate in the RCT of male circumcision. The primary aim of this RCT was to determine the effectiveness of male circumcision in reducing HIV incidence.6 Study participants were recruited from sexually transmitted infection (STI) clinics, workplaces and community organisations. This study includes men who were initially screened to participate in the RCT who consented to serological testing. Study inclusion criteria included being uncircumcised, HIV seronegative, sexually active (defined as reporting sex within the last 12 months) and having haemoglobin 9.0 g/100 mL, as previously described.6 Data analysis inclusion required that samples be available for shipment to the University of Manitoba, Canada, by early February 2002. A total of 120 men completed the initial screening.

Type-specific HSV-2 HerpeSelect and Kalon ELISA serological testing was conducted at the University of Manitoba Department Of Medical Microbiology Laboratory in Winnipeg, Canada, per manufacturer’s instructions.2 7 According to manufacturer’s instructions for both assays, index values of <0.9 were classified as negative, those >1.1 as positive and others as equivocal. Due to budget constraints, a total of 101 samples were randomly chosen from the initial 120 collected for Western blot analyses conducted at the University of Washington. Of the 19 samples without Western blot results, a total of 13 (68%) were HerpeSelect seropositive and 7 (37%) were Kalon seropositive. HSV inhibition testing, as previously described,4 was conducted on a random sample of 100 samples at Focus Diagnostics. Briefly, for the latter, sera were absorbed with HSV-1 and HSV-2 antigen and samples exhibiting ⩾60% inhibition with the HSV-2 antigen were considered positive for the HSV-2 antibody.

The sensitivity and specificity were calculated for (i) the Kalon ELISA, (ii) the HerpeSelect and (iii) the HerpeSelect followed by confirmation with the Kalon ELISA of HerpeSelect positive samples. Summary receiver operating characteristic (ROC) curves were also estimated.8 The ROC curves were plotted as sensitivity on the ordinate and 1-specificity (false positivity rate) on the abscissa. Explanatory power of predictive models was quantified using the c-statistic, which is equivalent to the area under the ROC curve in this context and is a commonly used measure of model predictive power.9

RESULTS

HSV-2 seroprevalence differed substantially by HSV-2 detection method among men who were HIV seronegative (aged 18–24 years): HerpeSelect and Kalon testing yielded 69.8% (83/119) and 39.0% (46/118) seropositivity, after excluding 1 and 2 equivocal samples, respectively. Of 101 Western blot testing results, HSV-2 Western blot seropositivity was 24.8% (25/101), of which two samples were classified as having atypical profiles to HSV-2 proteins. Of these 101 sera, 70 were positive by HerpeSelect (30 negative, 1 equivocal) and 39 were positive by Kalon (61 were negative, 1 equivocal) (using package insert standards for positivity). Of the 19 samples without Western blot results, HSV-2 results were similar to those with Western blot results: a total of 13 (68%) were HerpeSelect seropositive and 7 (37%) were Kalon seropositive. Inhibition testing among 100 samples yielded 45.2% seroprevalence (42/93) after excluding 7 samples with equivocal results.

As shown in table 1, the HerpeSelect ELISA test had 100% sensitivity (25/25) but 40% specificity (30/76) using Western blot as the reference standard. Kalon yielded a slightly lower sensitivity (92%, 23/25) but a substantially higher specificity (79%, 60/76) versus the Western blot, with the same manufacturer-specified cut-offs. In this setting, with a prevalence of approximately 25% using Western blot as the gold standard, the HerpeSelect ELISA had a positive predictive value (PPV) of 0.36 and negative predictive value (NPV) of 1.0, and the Kalon ELISA had a PPV of 0.59 and NPV of 0.97.

When Western blot results were negative, HerpeSelect and Kalon yielded concordant results in 46 of 74 cases (table 2); however, in the remaining 28 cases, the Kalon test was negative while the HerpeSelect test was positive. Among these Western blot−/HerpeSelect+/Kalon− discordant samples, the median index value was 1.95 (range 1.15–7.43) for HerpeSelect (mean 2.41, SD 1.47) and 0.44 (range 0.19–0.89) for Kalon (mean 0.47, SD 0.23). Of the 46 concordant results, 30 were dually HerpeSelect and Kalon seronegative and 16 were dually HerpeSelect and Kalon seropositive. Among these Western blot−/HerpeSelect+/Kalon+ samples, the median index value was 5.70 (range 2.22–7.43) for HerpeSelect (mean 5.17, (SD 1.98) and 2.37 (range 1.17–4.26) for Kalon (mean 2.38, SD 0.93). Among the 25 Western blot positive sera, HerpeSelect and Kalon ELISAs were highly concordant. The only two sera with Western blot+/HerpeSelect+/Kalon− results had index values of 1.24 and 4.83 for HerpeSelect and 0.12 and 0.29 for Kalon.

Sensitivity and specificity were calculated for a variety of additional, dichotomous cut-offs for both ELISA tests (table 1). Using the Western blot as the reference gold standard, the sensitivity of the HerpeSelect ELISA was generally higher than that of Kalon up to a cut-off of 2.0, although relative differences in sensitivity decreased with higher chosen diagnostic cut-offs for both tests. The Kalon test had consistently higher specificity than the HerpeSelect ELISA, with the exception of a chosen cut-off of 3.4 where the HerpeSelect ELISA performed at specificity of 80% but with a lower sensitivity of 80%. Specificity for HSV-2 detection was generally under 80% for both serological assays over a wide range of investigated cut-offs. The Kalon optimised trade-offs at a cut-off value of 1.2 (sensitivity 92%, specificity 80%), while the HerpeSelect ELISA optimised trade-offs between sensitivity and specificity at a dichotomous cut-off of 3.5 (sensitivity 80%, specificity 80%). The combination of HerpeSelect ELISA and Kalon did not perform markedly better than use of either test separately (table 1).

Using the inhibition test as the reference standard, and considering the manufacturer’s suggested cut-offs, the HerpeSelect ELISA test had 100% sensitivity but low specificity (54%), while the Kalon test had sensitivity 80% and specificity 82%. The HerpeSelect optimised sensitivity/specificity trade-offs at the cut-off of 2.5 (sensitivity 74%, specificity 73%); Kalon optimised that trade-off at a cut-off of 1.0 (81% and 82%). Again, the combination of HerpeSelect ELISA and Kalon did not improve sensitivity and specificity.

Among sera with both Western blot and inhibition test results (n = 76), the concordance between the two tests was 74% (56/76.). Against the inhibition test, the Western blot had relatively lower sensitivity (48.6%) with high specificity (95.1%). In contrast, the sensitivity and specificity of the inhibition test against Western blot were 89.5% and 68%, respectively (data not shown).

ROC curve analysis using Western blot as the gold standard gave 0.833 for HerpeSelect, 0.869 for Kalon and 0.880 for both tests together. ROC curves for all three of these results using Western blot as gold standard are shown in fig 1. Using inhibition as the gold standard yielded 0.806 for HerpeSelect ELISA, 0.866 for Kalon and 0.861 for both tests together.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Receiver operating characteristic (ROC) curve analysis using Western blot as the reference standard.

DISCUSSION

To our knowledge, this is the first study to compare the performance of both the HerpeSelect and Kalon HSV-2 ELISA in African sera with Western blot and inhibition testing confirmatory assays. Among young men in Kisumu, HSV-2 seroprevalence varied notably by the diagnostic method choice, ranging from 25% with the Western blot to 67% with the HerpeSelect. Using the Western blot as the reference gold standard, the HerpeSelect and Kalon type-specific ELISAs (employed per manufacturer’s instructions) had high sensitivities of over 90% (100% for HerpeSelect, 92% for Kalon). However, the specificity of the Kalon ELISA (79%) was superior to that of the Herpes-Select ELISA (40%). When inhibition testing was employed as an alternative reference standard, HSV-2 performance results were largely comparable with those based on the Western blot reference gold standard, with the exception of a slightly lower sensitivity for Kalon.

These data complement previous studies of HSV-2 ELISA performance comparisons, in particular by incorporating HSV-2 inhibition testing results. All assays consistently showed low specificity of the HerpeSelect in Kisumu male sera when using the manufacturer’s recommended cut-off. Among 330 samples from four African countries, Van Dycke et al used the Western blot and monoclonal antibody (MAb) ELISA to test samples found to be initially discordant between the HerpeSelect ELISA and a monoclonal antibody test to identify potential HSV-2 false-positive results.2 A specificity of 57% was found for the HerpeSelect among Kenyan samples. Hogrefe et al examined 781 samples from five African countries using the Western blot and HerpeSelect ELISA. Among Western blot/HerpeSelect discordant sera, the inhibition assay was used and the specificity of sera from Kenya was found to be 93% versus the Western blot and 97% versus the inhibition assay.4 The differences in specificity results between our data and Hogrefe et al4 may be related to populations studied as no other explanatory factor has been identified. It is worth noting that although the specificity of the HerpeSelect ELISA has been found to be relatively low among populations with a low HSV-2 seroprevalence (<5%) in the United States,10 or by geographical location (for example, Vietnam11), the specificity seen here in Kisumu is notably lower and may be specific to certain geographical areas of Africa.

The present study indicated that the use of a higher HerpeSelect ELISA cut-off of 3.5, rather than 1.1, would maximise overall HerpeSelect ELISA performance, but would sacrifice sensitivity, which would be reduced to 80%. Our results are consistent with results from Rakai, Uganda, that found a cut-off of 3.4 for the HerpeSelect ELISA resulted in sensitivity and specificity figures both of ∼85% when compared with Western blot.

Unlike a previous investigation in a population of men in Brazil at high risk of both HSV-2 and HIV,12 we did not find any advantage to retesting with the (more specific) Kalon ELISA after a positive HerpeSelect. Whether the Western blot or the inhibition test was chosen as the reference standard in this Kenyan population, the combination of HerpeSelect followed by Kalon retesting (according to manufacturer’s instructions) resulted in HSV-2 testing results very similar to those found using a testing strategy of the Kalon ELISA alone, with about 80% test specificity. When an algorithm was chosen with a high cut-off of 3.4 for HerpeSelect ELISA, followed by retesting of positive results with the Kalon ELISA at a cut-off of 1.3, the sensitivity and specificity combination was still not higher than Kalon ELISA testing alone.

Interpretation of HSV-2 serological assays is dependent upon the choice of the reference “gold standard”. PCR is considered the true gold standard for HSV ascertainment but does not ascertain past exposure to HSV-2.13 For HSV serological performance studies, the well-validated Western blot is considered the reference standard of choice but requires expertise for appropriate interpretation.14 The concordance between Western blot and inhibition test results of 74% was notably lower than the 95% concordance rate found in Hogrefe et al’s study in African sera.15 Further understanding, however, is needed for the interpretation of samples in our Kenyan study that were (i) Western blot negative and both HerpeSelect and Kalon ELISA seropositive (n = 16), and (ii) Western blot negative, and HerpeSelect positive and Kalon ELISA seronegative (n = 28), in order to determine whether the gold standard Western blot was falsely negative. It is possible that a proportion of the “false-positive” samples by either ELISA may have actually been relatively recent HSV-2 seroconverters not detected by the Western blot. The median time of HSV-2 seroconversion following documented primary HSV-2 infection has been shown to be the shortest for the HerpeSelect test (21 days) when compared with the Western blot (40 days) or the Kalon ELISA (120 days).5

In these sera from young men in Kisumu, Kenya, the sensitivity of the Western blot may have been suboptimal. Another possible explanation for the lower specificity observed with HerpeSelect is that it may have reacted to a variant strain of HSV-2 that was not detected by either the Western blot or Kalon assays. Of note, HSV-2 sequences in patients with HSV-2 from sub-Saharan Africa have not yet been well-studied and are not characterised in the GenBank database.16 Potential differences in HSV-2 strain or variants resulting in abnormal Western blot staining patterns could result in false-negative Western blot results. Alternatively, the HerpeSelect ELISA may be cross-reactive with an unidentified non-HSV-2 virus to a greater extent than the Kalon ELISA, although cross-reactive antibody may not necessarily have been due to another infectious agent. Both HerpeSelect and Kalon ELISAs are based on gG2 antigens using a similar expression system, although sites of truncation of recombinant antigens used in the two ELISAs are different.

Results presented here have several key advantages: the study included men at high-risk of HSV-2 seroconversion and laboratory testing was performed using four different type-specific serological assays. A limitation of this study is that it is based on a small sample size of young men from Kisumu, Kenya. Given that PPV and NPV values depend strongly upon prevalence, these values cannot be generalised outside of this study. Further, due to cost constraints, a subset of sera was not tested with the Western blot, although HSV-2 ELISA seroprevalence rates were similar among sera with and without Western blot testing results.

In this study of young men from western Kenya, the Kalon ELISA alone performed better than the HerpeSelect ELISA and there was no indication that any chosen algorithm for combined type-specific ELISA testing resulted in preferential test results. Further improvements in type-specific serological assays are needed for African sub-populations where currently available tests have not shown optimal sensitivity and specificity.