Download to read offline
Bm515 h8
- 1. USABILITY EVALUATION USER TESTS 1
- 2. TERM PAPER SCHEDULE December 19th Set your topic/research question Make a short lit. rev. and write the draft version (2-3 pages) Write methodology Turn-in your draft: 10 pts Late submission: -2pt per day January 09th Paper Submission Late submission: -5 pt per day 2
- 3. TERM PAPER Survey One approach is to choose an area of research, read all the relevant studies, and organize them in a meaningful way. An example of an organizing theme is a conflict or controversy in the area, where you might first discuss the studies that support one side, then discuss the studies that support the other side. Another approach is to choose an organizing theme or a point that you want to make, then select your studies accordingly. Empirical Gather data on an HCI topic 3
- 4. LITERATURE REVIEWS VERSUS RESEARCH ARTICLES Literature reviews survey research done in a particular area. Although they also evaluate methods and results, their main emphasis is on knitting together theories and results from a number of studies to describe the "big picture" of a field of research. Research articles, on the other hand, are empirical articles, specifically describing one or a few related studies. Research articles tend to focus on methods and results to document how a particular hypothesis was tested. The introduction of a research article is like a condensed literature review that gives the rationale for the study that has 4 been conducted.
- 5. EMPIRICAL PAPER -STRUCTURE Title Abstract Introduction (Lit. Rev. & Res.Q) Method Results Discussion Bibliography 5
- 6. ABSTRACT A one paragraph summary A statement on objective/purpose of the investigation Description of participants Brief description of what participants did Summary of findings 6
- 7. INTRODUCTION Lit review Background & rationale (previous research, what they found, what they identify as possible issues/questions) Use EBSCO, ScienceDirect, SCI, SSCI, etc. Statement of purpose “The current study was conducted to evaluate the effect of X on Y” .... or ..... “to find out what are the factors that lead to Z” or “to determine the relationship between A and B” 7
- 8. METHOD Enough detail for a reader to replicate Who participated (number, characteristics, volunteer or randomly selected) What materials were employed (systems, questionnaires - design, validity and reliability) What data was collected (dependent variables i.e. scores, ratings, responses) What were participants required to do (where, who, 8 sequence of events - include instructions & tasks)
- 9. RESULTS How have the data been treated? Text and graphs Statistics - descriptive/inferential Summarize the results 9
- 10. INDEPENDENT AND DEPENDENTVARIABLES Independent - what the experimenter does to the subject e.g. exposed to an interface, training, mental model, or selected by age or gender levels & controls Dependent - any behavior/ performance/ attitude of the subject which is measured as the ‘outcome’ e.g. scores on a test, type of knowledge 10
- 11. DISCUSSION Interpretation- what do the results mean in terms of your original question why do you think they turned out like this Critique your study (limitations) and recommend improvements Suggestions for further research 11
- 12. CREDIBILITY OF THESTUDY Definition of the construct being measured Congruence between method & question Measurement Bias: instruction & instrument (wording), administering Reliability (stable/decision consistency) the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects Validity strength of our conclusions, inferences or propositions Were we right? 12
- 13. PILOT YOUR METHOD…. Try your method before capturing data for real….. Ask friends to answer your survey, take your test, perform your experiment etc. Look for issues that confuse them (or you!) - modify accordingly 13
- 14. STATISTICS (IF YOU CHOOSEQUANTITATIVE APPROACH) Descriptive/summary stats are sufficient Mean (median, mode) Range Standard deviation (if you know how) Run tests if you are comfortable Provide tabulated raw data if possible, (put in appendices if large) 14
- 15. COMMON PITFALLS Rambling, unfocused style Keep a question in mind as you write All claims and opinions, no evidence Cite literature that supports your argument Misses relevant topics from class Try to see how the readings and lectures fit 15
- 16. DETERMINANTS OF USABILITYRATING Effectiveness Design Process Usability Efficiency Rating Satisfaction 16
- 17. PLAN A USABILITYEVALUATION Describe: What data will you collect? What will these data tell you? What data collection methods will you employ? How long will it take to produce the test results? What form of feedback will you provide? List advantages/disadvantages of this plan 17
- 18. ? THOUGHT ACTIVITY VESTEL is designing an interface for their new VCD product (includes remote control) for Turkish market which aims to be the ‘most usable’ on the market - how would you test this claim? ASELSAN wishes to test its new weapon control system . You are charged with designing the test. Educational software company has developed an educational game for use in elementary school - they want to know if it is usable/enjoyable. Municipality of Ankara is producing an information booth for use in visitors centers, ASTI, etc. offering info and advice to visitors. They ask for a usability test. (Zero learning time) 18
- 19. SETTING USABILITY CRITERIA “Product X is usable to the extent that 70% of users, with no additional training, can perform all tasks with 95% accuracy, 25% faster than existing application use, and report at least equivalent satisfaction” 19
- 20. OR..... “Product X is usable to the extent that 80% of users, with 2 days training, can perform 90% of routine tasks with >90% accuracy, as efficiently as with the existing application, and report increases in satisfaction” 20
- 21. INSTEAD OF.... “Product X is usable” (meaningless statement for HCI) “This new application is more usable than the old application” (begs the question...”More usable in what sense? And for whom? And where?” ) 21
- 22. WHO SETS THEUSABILITY CRITERIA? Purchasers can be basis for contract Designers basis for design targets Evaluators provide context/limits of generalization for evaluation Users Key stakeholders with privileged knowledge 22
- 23. HOW ARE CRITERIADERIVED? User analysis Task analysis Situation analysis 23
- 24. OUTPUT Scenarios of use “Stories” of interaction in which users, tasks and contexts are described Scenarios form basis of decisions on Effectiveness Efficiency Satisfaction 24
- 25. FRAMEWORK FOR USABILITYEVALUATION Approach and Type Approach refers to source of data User, Expert, or Model Type refers to purpose of evaluation Diagnostic (Formative) or Metrication (Summative) Any evaluation method is a combination of approach and type 25
- 26. EVALUATION APPROACH The approach defines the source of the data i.e., where does the evaluator gain the data about usability? from real users? (User-based) from usability experts or self evaluation? (Expert-based) from the application of a formal theory or model? (Model-based) 26
- 27. CONDUCTING YOUR TEST:THINGS TO CONSIDER… How many users? Length of test session? Where to conduct the session? Role of facilitator: put participant(s) at ease (testing the material, not them) observe and take notes not to intervene or assist Role,placement and responsibilities of other observers Verbal protocol (“think-aloud”) Token reward for participation (if appropriate) 27
- 28. DATA COLLECTION Quantitative data Number of errors made using the system Time required for activity(s) Qualitative data Ease of use – are materials convenient, easy to locate, to use? Learners’ reactions to materials, activities, evaluation 28
- 29. ANALYZING & REPORTING YOURUSABILITY RESULTS Quantitative data descriptive data (number of users, time spent, errors) be sure and discuss any data tables (what do they mean?) Qualitative data consolidate your observations (negatives and positives!) extract common themes identify critical themes (e.g. length of time required) perform member checking if possible determine solutions for addressing the problems 29 summarize and present your findings and solutions
- 30. ANALYZING & REPORTING YOURUSABILITY RESULTS Observations Interpretation Recommendation 30
- 31. PROTOCOL Introduction Thank you...for agreeing to participate in this session. Product Description...A CD-ROM "book" on the topic of visual design for instructional multimedia. Purpose of session...is to make this product better. This product does have problems. Any problems you have or find with the product is with the product, not your fault. Instructions... I'll be asking you do certain things with the program and watching and writing notes as you do them. That's just to help me remember how things went later on. To help me do this, I'd like you to "think out loud" as you use the program and make your decisions to do certain things. I'd like you to try and perform the given tasks on your own as best you can. If you’re really stuck, I may be able to help, but I’d really like you to try it without my help. At any time, you can quit a particular task and move on or you may choose to quit the entire session. 31
- 32. Efficiency OBSERVATION SHEET Start time: Finish time: page/link name Notes +/- name of starting page Efficiency Effectiveness 32
- 33. AND NOW.....THE USER How should we think of users? User as a psychological being Parameters of human information processing Emergence of skilled behavior 33
- 34. NIELSEN (1993) -THE RANGE OF USER TYPES: Knowledgeable about Domain Minimal Computer Experience Extensive Computer Experience 34 Ignorant about Domain
- 35. FOUR LEVELS OFTHE USER IN HCI Psychophysiological HCI issue: brain-computer interaction (BCI) Perceptual HCI issue: Screen layout, readability Cognitive HCI Issue: Task Structure, Human Learning Social HCI issue: CSCW, Organizational impact 35
- 36. BASIC PROPERTIES OFALL USERS Changes with experience Actively learns Limited attention Makes mistakes Models the system in their mind Remains unique Goal oriented 36
- 37. Some Resources http://www.hcibib.org/ http://www.acm.org/sigchi/ Usability.gov (research based evidences) http://www.microsoft.com/playtest/publications.htm comp.human-factors (USENET news group) Some Journals ACM transactions on computer-human interaction Human-computer interaction ACM Interactions International journal of human-computer interaction User modeling and user-adapted interaction Computers in Human Behavior Human factors in computing systems http://www.sciencedirect.com/science/journals 37