The Scorecard has been developed through a consultative process involving globally acknowledged experts from civil society, UN agencies, and research and public health institutions. It follows on from AAI’s previous Scorecards, the AIDS Accountability Country Scorecard and the AIDS Accountability Scorecard on Women.

AAI Scorecard Grades

Based on the limitations of the data (discussed in detail below), AAI places countries in five broad ‘grades’, from A to E. The grade is based on the percentage reported by the country according to the following formula: A (81-100%); B (61-80%); C (41-60%); D (21-40%); E (0-20%) – from A (very good) to E (very poor).

Data limitations
 

The UNGASS database represents the largest global data set on various aspects of country responses to HIV and AIDS. UNAIDS performs a verification and validation process thereby removing significant inaccuracies where possible. However, although data may not always fulfill reporting criteria, it remains the most useful global data available with standardized indicators for global and regional M&E comparisons. Yet, further below, this scorecard report will discuss reasons why this data does not allow for straight-forward comparison between countries or other more rigorous and detailed forms of analyses. 

The first reason for skepticism surrounding data submitted by some countries is that governments submit the data directly to UNAIDS and may or may not include civil society stakeholders in the process. UNAIDS strongly encourages the meaningful participation of civil society in the reporting process and in many countries this has been the reality and thus contributed not only to better reporting, better data quality but positively impacted on relations between stakeholders leading to more effective and integrated partnerships beyond the reporting process.      

However, in some countries civil society has found it necessary to submit shadow reports (see section above) as a means to contend government data submission. Furthermore questions around data have been raised even by civil society stakeholders that have not, for a variety of reasons, submitted shadow reports. These reservations need to be noted when working with the data and carefully considered when using it for analysis.

The second limitation of the data refers to data quality. A variety of issues exist, including but not limited to:

•    Sample size: Sample sizes of respondents vary and in some cases cannot be said to be robust.
•    Geographic coverage: Some data reflects only urban statistics and not rural circumstances, for example.
•    Income and class differences: Data collection methods may lead to data being skewed in terms of class and income if concerted efforts are not made to reflect the full spectrum.
•    Data collection methods: Data that is collected in face to face interviews in government surveys may not reflect reality. For example, countries where same sex activities are illegal may not completely capture truly accurate data. Similarly online surveys cannot guarantee that individuals with homophobic ideals do not log on and attempt to influence data outcomes.  
 
A third and important component of data reporting is that of whether countries consider it necessary to report on all indicators and how often they choose to do so. Bio-behavioral surveys are expensive and time-consuming and may not be the most appropriate use of limited government budgets in countries that are trying to roll out other parts of HIV prevention, treatment, care and support campaigns. For this reason AAI is aware that as we call for governments to improve reporting to UNGASS, we acknowledge that at country level leaders may not deem it a top priority. In providing national and global advocacy groups and other stakeholders with the scorecard analysis they are able to determine as experts in their field whether government decisions were well-based or indeed a sign of a lack of accountability.

A further limitation of the data is that the AAI Scorecard methodology reflects a measure of government performance based on one indicator in one reporting process. For this reason a country may receive a top score (A) on a particular indicator in the LGBT Scorecard yet at local level circumstances for LGBT people may be dire. This limitation of the work should be kept in mind when reading the element reports. The Scorecard methodology is a limited snap-shot assessment of a deep and complex issue. Thus it remains just one way to measure performance and that other issues, such as policy, criminalization etc in countries may differ either slightly or significantly from the grades viewed in each element and reported by countries.

For these and other reasons, some of the LGBT Scorecard Element Reports will focus on a particular limitation of the data. This is done with a view to creating and feeding into a dialogue around the UNGASS Reporting process. 

Important methodological notes

Sometimes 100% coverage may indicate a methodological issue in the collection of the data. For example 100 percent coverage on testing: very often these high figures represent the percentage of men who attend a Voluntary Counseling and Testing clinic and who agree to be tested and receive their results rather than the real denominator which is supposed to be Number of most-at-risk population included in the sample of a Bio-Behavioral surveillance survey. 

It is worth noting that some countries conflate MSM and MSW as well as MSW and FSW thus making it difficult to distinguish if male data presented is that applicable to MSM or MSW, note Algeria for example. In the Peru Country Report the authors discuss the inability of the data to separate out MSWs and MSM, hence the same figure was provided for both sub-groups. This is problematic for many reasons; however, AAI does not have resources to investigate data that has already been verified and validated by an agency with the resources such as UNAIDS has and thus decided to include this data.

In the 2010 round of reporting UNAIDS moved Mexico from the Latin American region to join Canada and the United States of America (USA) to be part of North America. The remaining Latin American countries were renamed central and Southern America. For our purposes we have not moved Mexico as it is more useful to include them in Latin America for comparative purposes, considering the low reporting from the North American countries on the indicators studied by this research.

For the purposes of our research female sex workers and IDUs have been excluded. The rationale for this has been that although we acknowledge that some female sex workers may be having sex with women, and that some male sex workers have female clients, we have assumed that the majority of male sex workers have male clients and that the majority of female sex workers also have sex with male clients, thus have included only the male sex workers as sexually diverse.

We also acknowledge that in some research the correlation between injecting drug users and sex workers has been recognized as being fundamental to the epidemiology of HIV for LGBT individuals yet we have excluded it as a result of being unable to ascertain sexual behavior from drug behavior and that country to country variations are too large and complex to factor in.

The AAI Collaborative Research Process

All AAI research and Scorecards are created in a consultative process with experts from various applicable fields. AAI is determined to conduct needs based research that produces effective and useful advocacy tools and in doing so consults with a wide variety of experts in order to achieve this goal. In the development of our ratings and communication initiatives, AAI relies on the active participation of a broad group of stakeholders with extensive experience and different types of expertise. In assembling our Panels and Forum we make sure that the knowledge and experience of people in the global South, and especially of women, is given prominent representation.

This process takes place with three groups of experts each with specific roles:
1.    AIDS Accountability Expert Panel: a multi-stakeholder group of about 100 invited experts with whom we regularly consult in our research process to review AAI’s ratings. The Expert Panel continues to play an important role in both the research and output processes.
2.    Development Teams: rating-specific teams of some 6-10 people who develop Scorecards.
3.    Group of 9: a small group of experts that reviews ratings at a late stage in the process to make recommendations to AAI regarding the launch of the tool.

The Development Team for the AIDS Accountability LGBT Scorecard: