As in much of Africa, young people in South Africa are particularly at risk for HIV infection. In 2003, a nationally representative household survey of close to 12 000 people aged 15–24 years found that 15.5% of South African young women were infected with HIV compared with 4.8% of young men.¹

Despite the fact that 50% of all new global HIV infections occur among young people, to date our knowledge of the impact of adolescent HIV prevention interventions in developing country settings is limited. The few rigorous studies from developed and developing countries have shown that, while it is possible to improve knowledge, attitudes and to some extent sexual behaviours, it has been more difficult to show an effect on pregnancy, sexually transmitted infection (STI) or HIV rates.^2–9 In order to have the greatest impact on stemming the epidemic and to ensure efficient use of resources, it is critical to provide evidence on the effect of adolescent focused prevention programmes.

There is a need to bring prevention interventions to scale, and with speed, to have a substantial impact on the HIV epidemic among youth.^10,11 Nevertheless, national programmes do not lend themselves to being evaluated using methods such as the randomised controlled trial (RCT). Although the RCT is viewed as the gold standard for proving the efficacy of interventions, there has been debate in the literature about the appropriateness of RCTs in evaluating public health programmes, particularly large programmes that have multiple components and complex causal pathways. Over the years several authors have argued for alternative approaches and methodologies to determine the effect of public health interventions in achieving desired public health effects.^12–14 This paper aims to describe the challenges faced when trying to evaluate a national programme, specifically loveLife, and the types of evidence that could be used in the absence of an RCT to better understand the effect of the programme.

THE LOVELIFE PROGRAMME

In the context of high HIV prevalence levels and associated risk behaviours among adolescents, a national HIV prevention intervention for youth, called loveLife, was launched in South Africa during 1999. loveLife is a national initiative combining high-powered multimedia with comprehensive youth-friendly reproductive health services in public clinics and countrywide community outreach and support programmes. The intervention aims to reduce HIV, other STIs and unwanted pregnancies among South African youth. Specifically, its goal is to trigger and sustain changes in sexual behaviour and related social norms to halve the rate of new HIV infections among young people. The target age range for loveLife is 10–17 years, although many programmes reach individuals in older age groups, and older youth are not excluded from participating in programmes.

loveLife’s activities are broad ranging in scope, content and level of engagement, and they operate at multiple levels: the individual, peer group, family and community, and nationally at a societal/cultural level. Media programmes, including billboards, television, radio and printed materials, promote HIV risk reduction and the concept of a positive lifestyle to South African youth by providing limited factual information, challenging social norms and stimulating public debate around issues relevant to HIV risk, such as condom use, multiple partners and gender norms.

To provide youth, parents, organisations and communities with more substantive and face-to-face experiences that aim to reduce HIV risk, loveLife also offers comprehensive, interactive educational programmes. These programmes extend into communities and are implemented by young people themselves (loveLife’s community-based peer educators are called groundBREAKERS and Mpintshis). loveLife’s outreach programmes provide service delivery, institutional support and capacity building, and deliver what are called “loveLifestyle” experiences for young people. The programmes aim to inspire and motivate young people to take control of their lives, set goals, make healthy choices, and navigate and reduce their HIV risk. An important element of the community outreach and clinical service programmes are loveLife Y-Centers (youth centres), franchises (youth-serving community organisations affiliated with loveLife) and National Adolescent-Friendly Clinic Initiative (NAFCI) clinics. These organisations act as hubs for community outreach from which loveLife’s educational programmes are implemented directly in schools and other community venues through engaging young people in a range of recreational activities.

The theoretical framework for loveLife’s behaviour change process draws on a number of behavioural theories including diffusion of innovations, ecological theory and the theory of reasoned action.^15–17 Diffusion is a process whereby members of a social system communicate about an innovation over time.¹⁵ This model focuses on social networks, opinion leaders and change agents. loveLife’s programmes operate by using change agents (the groundBREAKERS and Mpintshis) and opinion leaders (eg well-known South Africans such as Nelson Mandela) to communicate about new ideas and previously unacceptable behaviours, such as talking to children about sex. Further, by working in schools and with national media organisations loveLife capitalises on existing social networks to communicate the programme’s messages about behaviour change and to challenge social norms that may act as barriers to HIV prevention.

Ecological theory acknowledges that behaviour change is not only an individual-level construct, but that influences also operate at the dyad/small group, organisational, community and cultural/societal levels.^17,18 A central tenet of ecological theory is to tailor HIV intervention programmes to target the level at which risk is manifest, rather than focusing exclusively on the individual. loveLife’s programmes aim to address HIV risk by working to change norms and attitudes around issues that affect HIV risk, such as condom use, HIV testing and disclosure, and gender power relations, at multiple levels.

Lastly, loveLife operates within the theory of reasoned action, in which individuals take into account the implications of their behaviour within a particular context before they decide to change their behaviour.¹⁶ The theory places importance on the role of normative beliefs and locates young people’s behaviour in the framework of both their attitudes and subjective norms or social influences. loveLife’s programmes aim to change societal norms and attitudes that place young people at risk of HIV, particularly through promoting dialogue and debate about youth sexuality issues.

Specifically, loveLife’s programmes use the following approaches:

• social networks and young people as change agents to implement and diffuse programmes;

• addressing the various levels at which HIV risk is manifest (individual, small group, community, organisational and societal);

• working to change norms and attitudes that influence HIV risk behaviour;

• providing comprehensive, factual and personalised sexual and reproductive health (SRH) and HIV/AIDS information;

• providing positive lifestyle experiences for youth, which give them skills and motivation to reduce their HIV risk (motivation, art, music, debate, creative problem solving, health and fitness programmes, HIV/STI/pregnancy risk reduction skills);

• providing comprehensive, quality SRH services for adolescents.

CHALLENGES IN EVALUATING LOVELIFE

loveLife is an excellent example of a programme that cannot be readily evaluated using an RCT. Nevertheless, given its national scope, comprehensive nature and the investment made in it, understanding its effect is critical. Given that loveLife is a national intervention, all South African adolescents are theoretically exposed. There is thus, no obvious control group, and because the programme is comprehensive and involves a number of national programmatic elements, randomisation of the campaign is not possible. Although staged programmatic role-out with intervention and control areas may have been a possibility for some of the regional outreach and service components, the programmes were specifically targeted at previously disadvantaged communities identified by partners in national and local government. Sites chosen for programmes were selected based on need and a desire for maximum impact, rather than the requirements of an evaluation. Further, the intervention implementers went to scale rapidly—a priority given the speed at which the epidemic was growing in South Africa. The programme, therefore, rolled out before a baseline evaluation was conducted.

A further challenge in evaluating a complex programme such as loveLife was determining how to define programme exposure. In 2003 there were over 16 different programmes designed to work synergistically. In the 2003 national survey, 65% of youth reported awareness of four or more programmes.¹ Thus, looking at one programme in isolation may not capture a true representation of how the programmes work together. In addition, programme exposure data are based on self-report and thus subject to recall and social desirability bias. All these issues contribute to the challenge of evaluating a complex intervention of this scope with a predetermined set of factors that provide a less than ideal evaluation situation, as is so often the case in intervention implementation outside of the context of study settings.

PROPOSED SOLUTIONS TO THE CHALLENGES

Given the above challenges and loveLife’s national and community scope, two different study designs were originally planned for intervention evaluation. First, a series of nationally representative household surveys collecting both behavioural and biological markers of HIV prevention were planned and the first implemented in 2003.^1,19 The aims of this survey were to determine:

• the prevalence of HIV infection, associated risk behaviours and exposure to the loveLife programme among young people age 15–24 years in South Africa

• whether there is a 50% decrease in HIV prevalence nationally among young people age 15–24 years over a five-year period (to be able to measure this within age groups and by sex);

• whether young people exposed to loveLife have a lower prevalence of HIV and related risk behaviours compared with those young people who have not been exposed.

The focus on 15–24-year-olds was guided by the low HIV prevalence in youth under 15; thus detecting changes in HIV over time would require a very large sample size. Further, given that loveLife started in 1999, it would allow us to determine the intervention’s reach in older populations and measure its effect in youth who may have been part of the target age group before the survey but no longer were. The survey used a three-stage disproportionate, stratified design using the 2001 census enumeration areas as the primary sampling unit to identify a nationally representative household sample of young people, 15–24 years of age, living in the nine provinces of South Africa. Households within sampled areas were enumerated and one eligible young person in each household was randomly selected to take part in the interview, irrespective of whether they were sexually active or not. Study participants were asked to complete a comprehensive interviewer-administered questionnaire on HIV risk, protective behaviours and exposure to loveLife. They were also asked to provide an anonymous oral fluid specimen to test for HIV infection. It was originally planned that this survey would be repeated over time to measure changes in the desired outcomes, but challenges in accessing funding for intervention-specific surveillance type data collection means that the survey will probably not be repeated as planned.

Second, a quasi-experimental, repeated, cross-sectional, community-based study was designed.²⁰ This design was chosen as it was hypothesised that youth living in communities with more intensive loveLife activities, specifically communities with Y-Centers or NAFCI—the “hubs” of loveLife outreach and programmatic implementation—would be at lower risk for HIV, STIs and related risk behaviours compared with youth living in communities without these intensive activities. The strength of this study included allowing for a control arm and the measurement of other biological markers, specifically gonorrhoea and chlamydia, which was not logistically or financially possible in the national survey.

The study has three arms: 11 communities with Y-Centers, 11 with NAFCI clinics and 11 control communities (which all had standard public sectors clinics) selected from matching health districts. Households falling within 2 km of a study centre or control clinic were enumerated and one young person aged between 15 and 24 years from each household was randomly sampled (again irrespective of sexual activity). Like the national survey, participants took completed an interviewer-administered questionnaire and provided anonymous samples to test for HIV, and gonorrhoea and chlamydia. The baseline surveys were conducted from August 2002 to January 2003. As with the national survey, insufficient funds are available for the follow-up survey to thus far be conducted.

In the light of loveLife’s evaluation plan, Habicht et al provide a useful framework for assessing different types of evidence to develop a more comprehensive picture of intervention impact, particularly in the case where an RCT has not been conducted.¹³ This framework provides a continuum of evidence for an intervention’s effect moving from adequacy to plausibility to probability. Adequacy designs ask whether the expected changes occurred. They can evaluate performance, such as how well programme activities met expected objectives (these are often called process indicators: how many centres opened, how many condoms were distributed, how many individuals in the population used the programme), and can also look at impact by assessing whether health or behavioural indicators have improved among programme recipients or the target population as a whole.¹³ Although this design cannot causally link the observed health or behavioural outcomes to the specific intervention itself, it can supply information on whether the services were provided, if services were used and if the target population was reached. If these objectives were not met (ie no youth accessed the programme) then one would not expect the programme to have an effect on the outcome of interest.

In plausibility designs the evaluations attempt to document impact and rule out alternative explanations. This is often done by including a comparison group and by dealing with confounding variables. Plausibility assessments encompass a continuum from weak to strong; with stronger findings the other possible explanations for the observed associations can be discarded. Probability designs may provide the strongest evidence that any observed effect was due to the intervention and aim to ensure that there is only a small probability that observed differences between intervention and control sites were due to confounding or chance. The RCT falls into this category.

EVALUATION OF LOVELIFE: INDICATORS OF ADEQUACY AND PLAUSIBILITY

Providing evidence for the adequacy of loveLife’s impact on desired outcomes is possible (table 1). There are good monitoring systems in place to determine whether services are available, accessible, of good quality and are being used by the target population, and whether the target population is being reached. Data monitoring from the fourth quarter of 2004 indicated that there were over 235 NAFCI clinics, 16 Y-Centers, 532 schools implementing loveLife programmes, 908 groundBREAKERS and that 153 543 young people participated in “loveLifestyle” programmes nationally.²¹ In the 2003 survey, 85% of all youth reported awareness of loveLife, ranging from 65% in rural farming areas of South Africa to 93% in urban formal areas, and over a third reported participation in loveLife’s programmes.¹⁹ Outreach activities aim to target rural and disadvantaged youth who have been shown to be less likely to receive prevention interventions. In addition, programmes seem to be reaching their target audience as they are designed to be youth friendly and accessible to youth by being open on weekends and afternoons.

Determination of the plausibility of loveLife’s programmes having had an effect on observed changes in sexual behaviour or HIV infection becomes more complicated. The first challenge is determining the best way to classify exposure to the programme. When loveLife started, programme exposure was crudely defined as awareness of the programme (Have you heard of or seen loveLife?); yet by 2003, 85% of youth were aware of the programme and in the community survey baseline these numbers were even higher.^1,20 Given that we hypothesise that participation in loveLife’s programmes will probably have a greater impact on HIV risk behaviour than awareness of the programmes alone, we place more emphasis on examining participation in programmes than awareness. To deal with the multitude of possible programme exposures we have grouped exposure based on type (eg outreach v media) and examined programme effect in this manner rather than trying to look at over 16 different programmes individually. Although this definition does not get at the complexity inherent in each programme, looking at programmes in isolation, when they were designed to work synergistically, may not be accurate either. Ideally, detailed information on the breadth (ie number of programmes youth participate in) and depth (ie amount of exposure time), the type of involvement and level of engagement with the programmes should also be measured when evaluating such programmes. Another important difficulty in trying to isolate the effect of one programme operating at a national level is that a multitude of other HIV prevention programmes are also being implemented both at the community and national level, thus any observed effect on behaviour will most probably be the combined effect of all prevention programmes that youth are exposed to. It is doubtful that the different effects of a multitude of programmes will ever be fully untangled.

One means to address sole reliance on self-reported exposure data in the surveys involves using ecological measures of exposure to loveLife programmes by analysing the impact of—for example—living with a 2 km radius of a NAFCI clinic. The community survey allows for the opportunity to examine the effect of living in a community with a high intensity of loveLife programmes as opposed to individual-level exposure to the programme.²⁰ Methods to improve self-reported behaviors, particularly those that are subject to social desirability bias, are also an important element to consider for future surveys.

Once programme exposure is defined, the challenge of examining associations between loveLife and selected outcomes, and determining whether such associations are valid, remains. Although we cannot draw causal conclusions from these current studies given their cross-sectional nature, steps can be taken to strengthen the plausibility that loveLife may have had an impact on observed changes. These steps include first, measuring potential confounders and controlling for them using multivariable statistical methods. In multivariable models using data from the 2003 national survey, both sexually experienced males and females who reported participation in loveLife had significantly less chances of being infected with HIV (males: odds ratio (OR) 0.60 (95% CI 0.40 to 0.89); females: 0.61 (0.43 to 0.85)) after controlling for socioeconomic factors (education, rural/urban residence, having electricity in the home, marital status, age); ever testing for HIV; personally knowing someone who had died of AIDS; participation in youth groups; and awareness of two other national prevention campaigns (SoulCity and the Red Ribbon campaign).¹⁹ Second, assessing whether results are consistent when examining associations between loveLife and different outcomes. In the national survey, youth who participated in loveLife programmes, for example, were also more likely to report using a condom at last sex (males: OR 2.2 (1.7 to 2.9); females: OR 2.1 (1.3 to 3.2)); were more likely to have talked to their parents about HIV (males: OR 1.9 (1.6 to 2.1); females: OR 2.1 (1.5 to 2.9)); to report having changed their behaviour due to HIV (males: OR 1.9 (1.4 to 2.7); females: OR 1.6 (1.3 to 2.0)); and to report having a stronger sense of future optimism (males: OR 1.8 (1.4 to 2.4); females: OR 1.5 (1.3 to 1.9)).²² Third we aim to determine if dose–response associations are present in the expected direction. Again, from the national survey we found that youth who participated in two or more programmes compared to no programme were less likely to be infected with HIV (1 programme v 0: OR 0.68 (0.54 to 0.85); ⩾2 v 0: OR 0.44 (0.33 to 0.58)) and more likely to use condoms consistently (1 programme v 0: OR 1.47 (1.18 to 0.85); ⩾2 v 0: OR 2.28 (1.71 to 3.05)) compared with youth who participated in only one programme compared with no programme participation.

Perhaps one of the biggest challenges with evaluating national programmes stems from the cost of evaluation and ensuring appropriate commitment to and financial planning for rigorous evaluations. Evaluations such as the one described are expensive, although considerably less expensive than an RCT. Since the start of the loveLife evaluation, funding constraints have resulted in many of the planned evaluation activities being no longer feasible, in particular the two large surveys. Other methods that could be used to strengthen the evidence of loveLife’s effect, include the following.

• Epidemic modelling to estimate reductions in HIV incidence that would be expected with reductions in risk behaviour and HIV prevalence associated with prevention exposure in South African youth. This could involve developing a model incorporating potential and observed relationships of prevention exposure to levels of sex-specific and age-specific risk behaviours and HIV prevalence; conducting epidemic simulations to compare HIV infections expected in youth nationally with and without prevention programmes, to estimate infections averted; and exploring the potential gains in HIV infections averted associated with expansion and/or intensification of various prevention components.

• Conducting economic costings of the implementation of loveLife, overall and for programme components, by geographic area and over the phases of programme implementation for five years.

• Spatial mapping to examine the variation in loveLife’s spread geographically, and also to determine whether the greatest change in HIV prevalence is observed in the areas where there is the greatest exposure to loveLife.

Despite reductions in large-scale evaluation activities, loveLife continues to collect programme monitoring and process evaluation information to deal with the issues of programme access, availability, coverage and utilisation. In addition, smaller-scale surveys have been conducted to measure programme exposure and knowledge of current programmes.

CONCLUSION

The RCT remains an important tool for providing evidence on the effect of interventions. Nevertheless, in the context of an urgent need to prevent new HIV infections, with the aim of meeting the UN Millennium Development Goals for youth and HIV, there is a need to role out interventions at scale and with speed.¹⁰ Organisations such as the WHO have advocated for a “Steady, Ready, Go!” approach for rolling out interventions on a large scale based on a number of criteria.¹⁰ Given this, there is a need to expand and implement rigorous methods that will provide information on the effect of large-scale, complex behavioural interventions as alternatives to RCTs. Given the need to identify successful prevention interventions, we must look at multiple sources of data to understand the full picture in relation to the effect of prevention programmes. While a programme like loveLife may never be able to prove causation, putting together pieces of the puzzle to understand what did happen will provide valuable information, not only for loveLife, but for other prevention programmes globally.

AUTHOR CONTRIBUTIONS

AEP developed and implemented the evaluation activities and wrote the paper; CM analysed the data related to the evaluation and conceptualised and edited the paper; SB helped to conceptualise the ideas in the paper and edit the paper; and HVR was the principal investigator of the evaluations and was involved in conceptualising and editing the paper.