METHODS

Questionnaire development

Focus group discussions or panels are often used for exploring a specific set of issues. The group is “focused” around a collective activity, varying from watching and discussing a movie to talking about a particular set of questions on one specific topic.¹⁴ Such a topic could include patients’ views on quality of care. Focus group discussions with experienced patients can result in a broad range of potential quality of care indicators from the patients’ point of view. Frequent users of healthcare services like the chronically ill become experts by experience¹⁵; people infected with HIV may be considered as one such group.

To make sure that we were dealing with subjects who were representative of those infected with HIV, participants were members of and were recruited through the Dutch HIV Patient Society at the end of 1998. Letters were sent to their members in which patients were asked to participate in the focus group discussions. Two focus groups were used to determine subjects’ opinions about what constitutes good quality of care from their point of view and the bottlenecks encountered in the current healthcare system. A flow chart detailing the development of the questionnaire is shown in fig 1.

The first session was an open group discussion of the definition of good quality of care and the problems to be resolved. These sessions were tape recorded and translated into formal statements of care aspects mentioned by either one or both groups. The first focus group session resulted in 56 statements of care aspects. In the second session the statements of care aspects derived from the first session were grouped and labelled by the participants and prioritised into five levels of importance ranging from 1 (relatively unimportant) to 5 (extremely important). This was done according to the concept mapping process to quantify the relative value of quality of care aspects derived from the focus groups. Concept mapping is a type of structured conceptualisation which can be used by groups to develop a conceptual framework to guide evaluation of planning.¹⁶

The QUOTE questionnaire contains 10 generic aspects. These aspects were included in the questionnaire regardless of their importance rating. Further selection of the aspects by the researchers was based on the sorting and rating of the aspects derived by the participants in the second focus group session. These were classified by the researchers into the following sub-dimensions of quality of care: continuity of care, treatment, accessibility, autonomy, costs, information, competence, efficiency, privacy, and accommodation. The most important aspect within each sub-dimension was selected for the questionnaire. The list was completed using all aspects with a relatively high importance score (>3.5) as derived from the concept mapping procedure and regardless of their sub-dimension.

Empirical testing

Based on the patients’ ratings, 17 items were selected and used in the questionnaire together with the 10 permanent generic QUOTE items (see table 2 for complete list). Quality aspects varied from items generally used in patient satisfaction scales to items tailored to the wishes of this specific group of patients. The final questionnaire contained 13 generic items, including the 10 generic items of the already existing QUOTE questionnaires, and 14 HIV specific items.

In order to pilot the questionnaire, two general practitioners who were not eligible for participation in the prospective study sent it to the patients with HIV in their practices. No reminder was sent. To verify whether the focus groups were successful in determining all relevant quality of care items, an open question asking about additional aspects was included in the questionnaire.

Measures

In the final questionnaire quality of care aspects were formulated as importance and performance statements. These statements refer to three healthcare providers: general practitioner (GP), medical specialist (SP), and AIDS nursing consultant (AC). Respondents were asked to rate all 27 aspects with respect to the three categories of healthcare providers. For the AIDS nursing consultant four items were not relevant and were left out (see table 4). Importance and performance were measured using a 4-point Likert scale. For the importance aspects scores were calculated for the following categories (1=“not important”, 2=“fairly important”, 3=“important”, and 4=“extremely important”) by linear transformation of standardised values (Z-scores) to values between 0 and 10.⁵ For the performance aspects the response categories (1=“no”, 2=“not really”, 3=“on the whole yes”, and 4=“yes”) were dichotomised into percentages “yes” and “no”. The performance score (P) represents the proportion of respondents who were not satisfied with the care received.

The concept “quality of health care from the patients’ perspective” was operationalised as the product of importance and (perceived) performance according to the formula: Q_ij = I_ij × P_ij. The quality improvement score (Q) on a health service (j) by an individual patient (i) is equal to the importance score (I) multiplied by the (perceived) performance score (P). This formula is based on the PES model suggested by Zastowny et al.¹⁷ The conceptual framework of the formula is further explained by Sixma et al.⁵ A high quality improvement score means that better care is recommended.

Statistical analyses

Data analyses included item and inter-item analyses (non-response, skewness, correlations) to test internal consistency, explorative factor analysis (principal component analysis with varimax rotation and Kaiser normalisation), and reliability analysis. Factor analysis was performed separately on the 13 generic and 14 HIV specific aspects. Reliability scores refer to the total scale as well as to the different subscales within the generic and HIV specific items. Reliability and validity of the scales were evaluated primarily by looking at the importance scores. These importance ratings are assumed to be more stable and less situation dependent than performance scores. Feasibility of the new QUOTE-HIV patients’ instrument was established by a comparison of the quality improvements scores within and between the different healthcare services evaluated. The analyses were performed using SPSS 10.0.7 for Windows (SPSS, Chicago, IL, USA).

RESULTS

Patients

The two focus panels met at the end of 1998 and comprised three women and 12 men infected with HIV of mean age 49 years (range 30–62) and a mean duration since diagnosis of HIV infection of 9 years (range 1–15). Eleven were using anti-retroviral therapy. Six were diagnosed with AIDS and three others had symptomatic HIV infection.

The questionnaire was sent to 80 people infected with HIV; 44 questionnaires were returned (55%). Responders were predominantly homosexual men educated to higher level with a variable time since HIV diagnosis (table 1), which is a good representation of the HIV infected population in Amsterdam.¹⁸

Test characteristics of the questionnaire

With respect to the validity of the conceptual framework of the QUOTE instruments, separate analyses of the generic and HIV specific items indicated that each loaded onto a single factor (table 2).

For the total questionnaire the internal consistency as represented by Cronbach’s alpha (≥0.80) was good (table 3). The generic and HIV specific parts were found to have sufficiently reliable scales (≥0.70) except for the HIV specific part of the GP importance scale (0.60). Inter-item correlations indicating the discrimination values of the items are also shown in table 3.

To test the feasibility of the QUOTE-HIV instrument the 27 items were applied to the quality of care given by GPs, HIV specialists, and AIDS nursing consultants. Table 4 shows three scores for each care provider: the importance score (I), the performance score (P), and the quality improvement score (Q).

The mean importance score shows what is important to patients with each care provider together with their expectations. For GPs, patients rated aspect 3 (“to be taken seriously”) of the highest importance (I=8.5), and for specialists and AIDS nursing consultants they rated aspect 16 (“having special knowledge about HIV”) of the highest importance (I-SP=10; I-AC=8.6). Importance scores ranged from 8.5 to 2.7 for GPs, from 10.0 to 3.7 for specialists, and from 8.6 to 4.2 for AIDS nursing consultants on a scale from 0 (“unimportant”) to 10 (“extremely important”).

The performance score shows the percentage of patients who were not satisfied with the delivered care—for example, the performance score of the specialist for aspect 7 (“never keeps me waiting longer than 15 minutes”) is 0.27, which means that 27% of the respondents reported having to wait longer than 15 minutes in the specialist’s waiting room.

The quality improvement score shows the relative score of priorities and performances on quality. The importance of aspect 6 (“can be easily reached by phone”) for the specialist is valued at 6.5 with 31% of patients reporting that they could not reach their specialist by telephone. The quality improvement score of 2.0 is rather high as a result of the product of a relatively high performance score (P=0.31) and a medium importance score (I=6.5). Thus, according to patients, quality improvement is needed on this aspect.

The quality improvement score can also be used to compare within and between professions and institutions on both individual aspects and combinations of aspects. The quality improvement score of one aspect over several types of healthcare services can be considered. The quality improvement score for aspect 4 (“being aware of my situation at home and at work”) differed for the GP (Q=1.1), specialist (Q=2.0), and AIDS nursing consultant (Q=2.7). This indicates that, from the perspective of patients with HIV, quality improvement concerning this aspect is needed for AIDS nursing consultants and (to a somewhat lesser extent) medical specialists.

DISCUSSION

Focus group discussions resulted in 27 items (13 generic and 14 HIV specific) being used in the QUOTE-HIV questionnaire. Factor analysis showed that the questionnaire could be divided into two parts: a generic part and an HIV specific part. The internal consistency of the total questionnaire was good (Cronbach’s alpha ≥0.80). Feasibility of the questionnaire was shown by the diversity of importance and performance scores for GPs, HIV specialists, and AIDS nursing consultants. The QUOTE-HIV therefore seems to be a useful instrument for measuring the quality of care from the perspective of patients infected with HIV.

Working with focus groups has its limitations. Firstly, the result of the discussions may be biased by the fact that only a small number of patients were involved, all of whom were members of the Dutch HIV Patient Society. To find out whether the focus groups were successful in determining the relevant quality of care items, an open question was included in the questionnaire. Some patients used this question to comment on the list of items but no new items were mentioned, so we can assume that the questionnaire contains the most relevant items of quality of care for people infected with HIV. Secondly, working with focus group panels includes a subjective element by the moderator and observer(s) who are part of the group dynamics. This influence is minimised by using more than one panel and by recording the meeting on audio or videotape.

We did not subdivide each sub-dimension (continuity of care, autonomy, etc) into different scales because the questionnaire has to be easy to handle and can therefore contain only a limited number of aspects.

With respect to the quality of care aspects reported in table 3, in those such as “take enough time to talk with me”, “treated by the same person”, and “is organised in such way I cannot hear conversations at the desk or in the consulting room” it is hard to distinguish parts of the items from those used in many other satisfaction with care studies such as those reported by Hall and Dornan¹⁹ and Wensing.²⁰ However, this is only true for the part of the items which looks at their content and at the way they were formulated. Most items are more informative and more practical than those in existing patient satisfaction scales and better reflect the needs of specific categories of patients.⁸ For instance, items concerning medication can be expected to be important to everyone in general, but for HIV infected persons who have to take a lot of drugs in complicated dosing schedules such items are especially important. The patients in our study mentioned three medication aspects (“give information about possible side effects of drugs”, “inform me about the pros and cons of a treatment”, and “give information about the use of my HIV medication”) among the five most important aspects of specialist care. Other instruments developed from a patient’s viewpoint to measure patients’ satisfaction such as the QPP (Quality from Patients’ Perspective),²¹ the PSQ (Patient Satisfaction Questionnaire),²² and the PPS (Patient Perspective Survey)²³ have been developed for patients in general. These questionnaires do not focus on patients with a specific disease, and this is an important difference from the principles of each QUOTE questionnaire.

The QUOTE-HIV questionnaire differs from traditional patient satisfaction questionnaires in several respects. Firstly, unlike most satisfaction instruments, QUOTE-HIV offers an importance as well as a performance component so the importance component can be added as a weight factor in the judgement of quality of care. For an aspect with a low performance score that is not rated as highly important one can determine whether improvement is needed, unlike those in which an interaction between performance and importance scores gives a high quality improvement score. The QUOTE-HIV questionnaire is therefore exceedingly useful for identifying aspects that really need improvement. Secondly, patients played a crucial role in the development of this instrument, which ensures that the items important to them are included. As a result, an equal number of items encountered specifically by HIV infected patients as well as known patient satisfaction items were generated.

In table 4 a quality improvement score of 3.5 for GPs is shown for the aspect “give information about my HIV medication”. The QUOTE-HIV questionnaire therefore indicates the need for an education course for GPs in which more attention is paid to the use of HIV medication.

To limit the number of items in the questionnaire it could be restricted to a single time rating of the importance of an item in general. We wanted to compare importance scores as well as performance scores for GPs and HIV specialists and therefore asked for an importance rating for each healthcare provider separately.

Further research is needed for specific patient groups. We did not look at different disease categories or medication use; future studies should validate the instrument for specific categories.

We consider that this instrument can be used widely although some adaptation to the local situation in other parts of Europe might be warranted. Healthcare systems in Europe differ between countries and sometimes between regions in the same country, so the instrument has been provided with a method to adapt it to specific characteristics of health care. For such cross-cultural validation, additional focus group discussions with patients will play an important role. The scope of the instrument is not limited to providers but can be used for functions of care such as the provision of care for non-institutionalised patients with HIV.