Overview of Methods to Collect Information
The following table provides an overview of the major methods used for collecting data during evaluations. Advantages -can complete anonymously -inexpensive to administer when need to quickly -easy to compare and questionnaires, and/or easily get lots of analyze surveys, information from -administer to many checklists people in a non people threatening way -can get lots of data -many sample questionnaires already exist when want to fully -get full range and depth understand someone's of information impressions or -develops relationship interviews experiences, or learn with client more about their -can be flexible with answers to client questionnaires -get comprehensive and when want impression historical information of how program -doesn't interrupt operates without program or client's documentation interrupting the routine in program review program; is from -information already review of applications, exists finances, memos, -few biases about minutes, etc. information to gather accurate information about how a program actually operates, particularly about processes -view operations of a program as they are actually occurring -can adapt to events as they occur Method Overall Purpose Challenges
-might not get careful feedback -wording can bias client's responses -are impersonal -in surveys, may need sampling expert - doesn't get full story
-can take much time -can be hard to analyze and compare -can be costly -interviewer can bias client's responses -often takes much time -info may be incomplete -need to be quite clear about what looking for -not flexible means to get data; data restricted to what already exists -can be difficult to interpret seen behaviors -can be complex to categorize observations -can influence behaviors of program participants -can be expensive
observation
focus groups
case studies
explore a topic in depth -quickly and reliably get through group common impressions discussion, e.g., about -can be hard to analyze -can be efficient way to reactions to an responses get much range and depth experience or -need good facilitator for of information in short suggestion, safety and closure time understanding common -difficult to schedule 6-8 - can convey key complaints, etc.; useful people together information about in evaluation and programs marketing to fully understand or depict client's -fully depicts client's -usually quite time experiences in a experience in program consuming to collect, program, and conduct input, process and results organize and describe comprehensive -powerful means to -represents depth of examination through portray program to information, rather than cross comparison of outsiders breadth cases
Ethics: Informed Consent from Program Participants
Note that if you plan to include in your evaluation, the focus and reporting on personal information about customers or clients participating in the evaluation, then you should first gain their consent to do so. They should understand what you're doing with them in the evaluation and how any information associated with them will be reported. You should clearly convey terms of confidentiality regarding access to evaluation results. They should have the right to participate or not. Have participants review and sign an informed consent form. See the sample informed-consent form.
How to Apply Certain Methods
Purposes and Formats of Questions Developing Questionnaires Conducting Interviews Conducting Focus Groups Developing Case Studies
Selecting Which Methods to Use
Overall Goal in Selecting Methods:
The overall goal in selecting evaluation method(s) is to get the most useful information to key decision makers in the most cost-effective and realistic fashion. Consider the following questions: 1. What information is needed to make current decisions about a product or program?
2. Of this information, how much can be collected and analyzed in a low-cost and practical manner, e.g., using questionnaires, surveys and checklists? 3. How accurate will the information be (reference the above table for disadvantages of methods)? 4. Will the methods get all of the needed information? 5. What additional methods should and could be used if additional information is needed? 6. Will the information appear as credible to decision makers, e.g., to funders or top management? 7. Will the nature of the audience conform to the methods, e.g., will they fill out questionnaires carefully, engage in interviews or focus groups, let you examine their documentations, etc.? 8. Who can administer the methods now or is training required? 9. How can the information be analyzed? Note that, ideally, the evaluator uses a combination of methods, for example, a questionnaire to quickly collect a great deal of information from a lot of people, and then interviews to get more in-depth information from certain respondents to the questionnaires. Perhaps case studies could then be used for more in-depth analysis of unique and notable cases, e.g., those who benefited or not from the program, those who quit the program, etc.
Four Levels of Evaluation:
There are four levels of evaluation information that can be gathered from clients, including getting their: 1. reactions and feelings (feelings are often poor indicators that your service made lasting impact) 2. learning (enhanced attitudes, perceptions or knowledge) 3. changes in skills (applied the learning to enhance behaviors) 4. effectiveness (improved performance because of enhanced behaviors) Usually, the farther your evaluation information gets down the list, the more useful is your evaluation. Unfortunately, it is quite difficult to reliably get information about effectiveness. Still, information about learning and skills is quite useful.
