whythawk ratings: measuring effective development

NOTE: To use the advanced features of this site you need javascript turned on.

home
Ratings Methodology

The rating of individual organisations is based on responses Whythawk Ratings receives from information gathered during an extensive on-site interview as well as from three separate questionnaires designed around observable elements.

In the first instance, a generic questionnaire focuses on the organisation's structure, capacity to deliver and financial management. Thereafter a second questionnaire assesses the human resource capacity, focusing on the level of education, skills and relevant experience the organisation has at hand. Finally, a random sampling of the organisation's clients is interviewed, with the aim of determining the overall level of client satisfaction and their successes post-intervention.

Information gathered during these stages are divided into three categories: administrative for internal purposes, sector specific for wider feedback, and responses used for the development of ratings criteria.

Scores for each criterion are converted according to a scale and related criteria are grouped together to create a scorecard. Each of the criterion are given appropriate weightings within the scorecard and the scorecards themselves rolled up into an overall rating using similar scaling, are weighted according to importance.

Our scaling technique is a dynamic distribution that allows Whythawk Ratings to enhance the value of comparative evaluations as the base of rated organisations expands.

Whythawk Ratings publishes the results from this process in an organisation specific report that: provides the overall rating, details the rating for each scorecard, and gives a one to two page summary of the organisation's operations, service offering and character.

Whythawk Ratings’ process for rating development and poverty relief organisations includes a site visit to the organisation and a meeting with the management team. This overview outlines the steps of the rating process and what management at a development organisation should expect during a site visit by Whythawk’s analysts. Once a rating relationship is initiated, the maintenance of the rating is not onerous, if there is ongoing communication.

Rating Process

Getting Started

Whythawk develops metrics and methodology for an entire development sub-sector.  As part of that process the majority of industry actors are identified and approached to be part of the rating process.  Organisations may also approach Whythawk on their own with a view to being rated. The rating process generally takes one to two weeks to complete. However, if a rating is needed within a shorter time frame, Whythawk will work to accommodate the request.

Whythawk prefers to receive a rating package in advance of the meeting date.  This allows the ratings team to adequately prepare and begin identifying key factors and trends. If information is received in advance, the analysts can provide management with feedback for the meeting agenda. Whythawk will also require access to the organisation’s complete database of clients / beneficiaries so that these may be interviewed to further inform the rating.

Site Visit
In virtually all cases, Whythawk requests a site visit. The visit allows Whythawk analysts to meet various members of the management team, board, staff, and other decision makers. A site visit can also enhance Whythawk’s understanding of the organisation’s operations and service area characteristics. The site visit will include a rating interview that generally lasts from one to two hours. In addition, a tour of the facility and the service area are helpful for Whythawk’s analysis.
Follow Up
After the site visit, there usually is a need for clarification or expansion of various rating issues. If applicable, Whythawk will send a list of additional data needs and follow up with a call to the designated contact. At this point, Whythawk should have received all the information necessary to complete the rating analysis. To convey transparency in the rating process, Whythawk analysts will provide the rated organisation with an indication of the rating level they will recommend to the rating oversight committee.
Oversight Committee

Oversight committees are composed of individuals selected for their impartiality and experience in the sector concerned. The committee is able to evaluate the consistency and objectivity of the process as well as indicating whether individual ratings should be reviewed.

There is an appeal process if the rated organisation believes that certain quantitative or qualitative factors were not incorporated in the rating. Although rare, the appeal process allows the rated organisation to present new information to support the factors that it believes may not have been given appropriate consideration.

Following the appeal process, the committee will reconvene to establish a final rating, and this process is resolved expeditiously. An appeal of an existing rating should be resolved within two business days.

Ratings Report
After the rating has been finalised, the analytical team produces a draft of the overview report for review. The review of the draft is geared toward evaluating the accuracy of the information rather than editing Whythawk’s opinion. In addition, this process helps prevent the release of any confidential information. Whythawk’s full rating analysis reports are then disseminated to subscribers.

What to Expect During a Site Visit

The site visit is an integral component of the rating process and assists Whythawk analysts in understanding the organisation’s operational, financial, and strategic goals.  Whythawk views management practices as an important factor in a rating and believes the opportunity to spend time with management is essential.

Discussion with management is informal but structured around a specific questionnaire which allows Whythawk to objectively compare different organisations active in the industry. The goal of the meeting with members of the management team is to develop an understanding of its management philosophy, mission, and strategic goals. 

A tour of the organisation’s office is helpful to gain a better understanding of the level of investment in the infrastructure. In addition, a tour sometimes is valuable to allow Whythawk analysts to become familiar with the economic status of the area and the distances for clients.

Rating Maintenance and Ongoing Disclosure

The monitoring of Whythawk’s ratings and timely update of changes in service quality is a key service that Whythawk provides to subscribers. Once a rating has been assigned, the surveillance of the rating is not time consuming, provided that there is ongoing communication with the organisation.

Whythawk expects each rated organisation will supply information on a timely basis.  In addition, rated organisations should notify Whythawk of any material changes in any information previously provided and disclose potential material events.  Whythawk requires continuing disclosure from the organisation to maintain the rating.

Whythawk views continuing disclosure practices as a significant indicator of management quality and comments on each organisation’s disclosure policy in its written reports. The disclosure provided is assessed in terms of timeliness, completeness, accuracy, and format.

Whythawk believes those organisations that have adopted excellent disclosure practices will be rewarded with greater support from the industry.

Internal reviews of Whythawk’s portfolio are done on an ongoing basis. Lower rated and troubled organisations are reviewed more frequently. A rating update review is usually less intensive than the initial rating review, since Whythawk has established a relationship and is knowledgeable of the organisation’s historical performance, service area, and governance structure.