How does TTCSP rank Think Tanks?

Each year, our best efforts have gone into generating a rigorous, inclusive, and objective process. However, we recognize the impossibility of entirely ridding the Index from bias. Inevitably, personal, ideological, disciplinary, and regional biases of those consulted throughout the process may have influenced the rankings. While some have suggested that we move to a small group or panel of experts and journalists to make the selections, we are unwavering in our commitment to an open and democratic process. Given the safeguards in place– the transparent process, the provisions set by the detailed nomination and selection criteria, and the annually increasing participation of think tanks and experts from every region of the world– we are confident in the quality and integrity of the Index. Additionally, with the growing involvement of the expert panelists, the nomination and ranking process has also been dramatically improved. Together, we believe these measures insulate the nomination and selection process from any significant charges of bias and misrepresentation. We would like to underscore that the GGTTI is but one measure of a think tank’s performance and impact, and has been designed for use in conjunction with other metrics to help identify and evaluate public policy research organizations around the world. An organization’s inclusion within the Index does not indicate a seal of approval or endorsement for the institution, its publications, or its programs by the TTCSP. Conversely, an organization’s exclusion from the rankings does not necessarily indicate poor quality, effectiveness, or performance. There are 8000+ think tanks around the world completing exceptional work to help bridge the gap between knowledge and policy. This report is no more than one effort to highlight some of the world’s leading think tanks.

The Think Tank Index Process

Before even beginning the nomination and selection process, the team conducted extensive research in order to update and verify the Think Tank and Civil Societies Program’s (TTCSP) Global Think Tank Database. More than 60 TTCSP staff spend more than a month collecting and verifying information on all known think tanks so they could be included in the process. All think tanks are eligible to participate in the peer and expert and nomination and raking process. There are no fees or barriers to participation. A month in advance of the nomination and selection process launch, a letter announcing the commencement of the Global Go To Think Tank Index (GGTTI) is sent to the more than 50,000 individuals and organizations in our database. Think tanks are then asked to make recommendations for improving the nomination and selection process, in addition to potential Expert Panel nominees. A letter requesting evaluations of the efficacy and validity of the previous year’s rankings criteria, and nomination and selection process, was also sent to expert panelists from previous years.

• Nominations: Expert Panel Member – Typically occurs between April and August.
  In preparation for the Global Go To Think Tank Indexing process, a call for nominations is issued for qualified individuals to serve on the Regional, Functional, and Special Areas of Distinction Panels. In addition, asked for comments, suggestions and recommendations for how we might improve the Index.

• Round I: Nominations – August to September
  A call for Nominations was sent to over 7,500 think tanks and approximately 8,500 plus journalists, public and private donors, and policy-makers from around the world. These nominations are tabulated and institutes with 15 or more nominations are included in the next step of the Think Tank Indexing process. All of the top ranked think tanks from the previous year are automatically included in the ranking ballot. Peers and experts are also encouraged to send the names and contact information for newly established think tanks (last 24 months) and to report any issues or corrections to the previous year’s Global Go To Think Index Report.

• Round II: Peer/Expert Rankings – October
  Think tanks with 15 or more nominations are placed in an electronic ranking survey. A letter announcing the second round was emailed to all the think tanks, journalists, public and private donors, and policy maker groups who agreed to participate in the process. The rankings are tabulated and the list of finalists was generated for the Expert Panel to review and make final selections. This year, Regional and Functional Expert Panels are created for every category. These specialists are then consulted to help assure the quality and accuracy of the nominations before they are placed on the final rankings survey.

• Round III: Expert Panel Selects Go To Think Tanks – November to January
  The members of the Expert Panel received information packets by email in order to facilitate the final selection process. Individuals who served on the previous year’s expert panel as well as those who are nominated this year are invited to serve on the current years Expert Panel. Experts from every region and functional area are represented on the Expert Panel. Panelists typically submit their rankings and recommendations by mid November.

• Launch: Global Go-To Think Tank Rankings are Announced (Typically 3rd week of January)
  The Global Go-To Think Tank Rankings is announced in New York, Washington D.C., and at host organizations in every region of the world.

THINK TANK TYPOLOGY/THINK TANK AFFILIATIONS
Autonomous and Independent: Significant independence from any one interest group or donor and autonomous in its operation and funding from government.
Quasi Independent: Autonomous from government but an interest group (i.e. unions, religious groups, etc.), donor or contracting agency provides a majority of the funding and has significant influence over operations of the think tank.
University Affiliated: A policy research center at a university.
Political Party Affiliated: Formally affiliated with a political party.
Government Affiliated: A part of the structure of government.
Quasi Governmental: Funded exclusively by government grants and contracts but not a part of the formal structure of government.
For Profit: Public policy research unit located within a corporation or operating as a free standing for-profit think tank.

NOMINATIONS AND CRITERIA
It is essential that institutions consider a variety of criteria in making their decisions. These may include, but are not limited to:

● The quality and commitment of the think tank’s leadership (chief executive and governing body). This involves effectively managing the mission and programs of the think tank, mobilizing the financial and human resources necessary to fulfill the mission and monitoring the quality, independence and impact of the think tank;
● The quality and reputation of the think tank’s staff. Ability to assemble a critical mass of highly skilled, experienced and productive scholars and analysts who are recognized as either emerging or established experts in their respective area of research;
●The quality and reputation of the research and analysis produced. The ability to produce high quality, rigorous, policy oriented research that is accessible to policymakers, media and the public;
● Ability to recruit and retain elite scholars and analysts;
● Academic performance and reputation. This involves the academic rigor associated with the research conducted. This includes formal accreditation of a think tank’s scholars and analysts, the number and type of scholarly publications produced such as: books, journals and conference papers and the number of presentations delivered at scholarly and other professional meeting and the number and type of citations of the think tanks scholars’ research in scholarly publications produced by other scholars;
● The quality, number, and reach of its publications;
● The impact of a think tanks research and programs on policymakers and other policy actors. Policy recommendations considered or actually adopted by policymakers, civil society or policy actors;
● Reputation with policymakers (name recognition associated with specific issues or programs, number of briefings and official appointments, number of policy briefs and white papers produced, legislative testimony delivered);
● A demonstrated commitment to producing independent research and analysis. This involves standards and policies for producing rigorous evidence based research and analysis that are posted and monitored by the organization, research teams and individual researchers.  This includes disclosure of conflict of interest (financial, institutional or personal) and a commitment to nonpartisanship and established professional standards for research in the social sciences;
● Access to key institutions. The ability to reach and connect with key audiences and personnel such as government officials (elected and appointed), civil society, traditional and new media, and academia;
● Ability to convene key policy actors and to develop effective networks and partnerships with other think tanks and policy actors;
● Overall output of the organization (policy proposals, web visits, briefings, publications, interviews, conferences, staff nominated to official posts);
● Utilization of research, policy proposal and other products. The effective transmission and utilization of policy briefs, reports, policy recommendations and other products by policymakers and the policy community, number of current and former staff serving in advisory roles to policymakers, advisory commissions, etc., awards given to scholars for scholarly achievement or public service;
● Usefulness of organization’s information in public engagement, advocacy work, preparing legislation or testimony, preparing academic papers or presentations, conducting research or teaching;
● Ability to use electronic, print and the new media to communicate research and reach key audiences;
● Media reputation (number of media appearances, interviews and citations);
● Ability to use the Internet including social media tools, to engage with policymakers, journalists and the public;
● Web Site and Digital presence. The quality, accessibility, effective maintenance of the organization’s web presence, as well as, the quality and level Digital traffic and engagement. (quality, accessibility and navigability of web site, number of website visitors, page views, time spent on pages, “likes” or followers);
● Level, diversity and stability of funding.The ability of an organization to mobilize the necessary financial resources to support and sustain the think tank over time (endowment, membership fees, annual donations, government and private contracts, earned income);
● Effective management and allocation of financial and human resources. The ability of a think tank to effectively manage its money and people so that they produce high quality outputs that achieve maximum impact;
● Ability of the organization to effectively fulfill the terms of the gifts, grants and contracts from government(s), individuals, corporations and foundations who have provided the financial support to the think tank (financial stewardship);
● The organization’s ability to produce new knowledge, innovative policy proposals or alternative ideas on policy;
● Ability to bridge the gap between the academic and policymaking communities;
● Ability to bridge the gap between policymakers and the public;
● Ability to include new voices in the policymaking process;
● Ability of organization to be inscribed within issue and policy networks;
● Success in challenging the traditional wisdom of policymakers and in generating innovative policy ideas and programs.
● The impact on society. Direct relationship between the organization’s efforts in a particular area to a positive change in societal values such as significant changes in the quality of life within respective country (amounts of goods and services available to citizens, state of physical and mental health, quality of environment, quality of political rights, access to institutions)

THINK TANK ASSESSMENT TOOL
Clearly, assessing the impact of think tanks is not an easy endeavor to undertake given the various and conflicting actors, events, and politics involved in the policy making process. Despite the significant challenges in establishing a causal relationship between knowledge and policy, it is necessary for think tanks to understand and effectively respond to the growing chorus of questions being raised by donors, journalists, and the public about the role and influence of think tanks in civil societies and governments around the world.

Think tanks can employ a variety of metrics to assess the impact including such measures as an increase in research and analysis they produce as well as to account for their contributions to the policymaking environment and civil society. McGann’s recent (2008) research has focused on developing a comprehensive assessment tool for evaluating a think tank’s impact. The impetus for this research, in part, was the apparent confusion that exists about the differences between outputs and impacts. In various studies and surveys that McGann has conducted over the years, researchers and think tanks responded curiously when asked about impact on public policy and how they measure it. The overwhelming response was to provide a list of research outputs (number of books published, conference held, web hits, media appearances, etc.). Outputs, however, are not the only way to measure impact.

The metric provided below is designed to serve as a catalyst for a discussion on how to effectively measure the impact of think tanks. It is provided here as background for the think tank ranking process in the hopes that it will help clarify the distinction between outputs and impacts and provide a useful tool as you prepare your rankings. We ask that you consider the following indicators when contemplating the impact of think tanks:
• RESOURCE INDICATORS: Ability to recruit and retain leading scholars and analysts; the level, quality, and stability of financial support; proximity and access to decision-makers and other policy elites; a staff with the ability to conduct rigorous research and produce timely and incisive analysis; institutional currency; quality and reliability of networks; and key contacts in the policy academic communities, and the media
• UTILIZATION INDICATORS: Reputation as a “go-to” organization by media and policy elites in the country; quantity and quality of media appearances and citations, web hits, testimony before legislative and executive bodies; briefings, official appointments, consultation by officials or departments/agencies; books sold; reports distributed; references made to research and analysis in scholarly and popular publications and attendees at conferences and seminars organized
• OUTPUT INDICATORS: Number and quality of: policy proposals and ideas generated; publications produced (books, journal articles, policy briefs, etc.); news interviews conducted; briefings, conferences, and seminars organized; and staff who are nominated to advisory and government posts
• IMPACT INDICATORS: Recommendations considered or adopted by policymakers and civil society organizations; issue network centrality; advisory role to political parties, candidates, transition teams; awards granted; publication in or citation of publications in academic journals, public testimony and the media that influences the policy debate and decision-making; listserv and web site dominance; and success in challenging the conventional wisdom and standard operating procedures of bureaucrats and elected officials in the country

Beyond this quantitative assessment, an effective evaluation of impact should also involve NGOs, as well as members of the government and policymakers, to ascertain the degree to which they have utilized the grantee’s research output. This participation can be obtained through interviews, surveys, questionnaires, and focus group meetings, utilizing the Outcome Mapping which “moves away from assessing the products of an activity or a program to focus on changes in behaviors and relationships (outcomes) which can lead to changes.” Impact can be viewed as positive if it “changes the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly.”

Although this qualitative assessment is essential because it recognizes that policy impact can be successfully achieved even if policy prescriptions are not directly translated into actual policy, we recommend that this assessment should be translated into numerical rankings, thereby allowing comparisons with baseline data for effective monitoring and evaluation in the future.