Iterative usability evaluation of DSLs
Download as PPS, PDF0 likes240 views
The document discusses domain-specific languages (DSLs) and methods for evaluating the usability of DSLs. It notes that DSLs aim to raise the level of abstraction by focusing on domain concepts rather than computation concepts. This can provide benefits like productivity gains. However, the document states that software language engineers often do not evaluate how their languages impact the software development process. It then presents Barišić's work on introducing DSL usability evaluation into the DSL development lifecycle. This includes designing effective experiments to provide qualitative and quantitative feedback to DSL developers. The goal is to produce user-centered DSL design and foresee quality while the language evolves. The document outlines one case study where two
![?
SE
U
RE
[Mernik, 2005]
5](https://image.slidesharecdn.com/quasar2013-131028071707-phpapp02/85/Iterative-usability-evaluation-of-DSLs-5-320.jpg)
![[Barisic, 2011c]
11](https://image.slidesharecdn.com/quasar2013-131028071707-phpapp02/85/Iterative-usability-evaluation-of-DSLs-11-320.jpg)
![Our evaluation technique was tested with two individuals
(two physics experts) in order to verify it and to test the
teaching materials and questionnaires
As time constrants and equipment turn out to be adequat
there was no need to change prepared materials
[Howatt 1995]
23](https://image.slidesharecdn.com/quasar2013-131028071707-phpapp02/85/Iterative-usability-evaluation-of-DSLs-23-320.jpg)
![Literature
[Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to
develop domain-specific languages, 2005, ACM Computing Surveys
[Gabriel2010] Gabriel, P., Goulão, M. & Amaral, V. (2010). Do Software
Languages Engineers Evaluate their Languages? in XIII Congreso
Iberoamericano en "Software Engineering" (CIbSE'2010)
[Barisic2011a] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in
Use of DSLs: Current Evaluation Methods’. Proc. 3rd INForum - Simpósio de
Informática (INForum2011), Coimbra, Portugal, September 2011
[Barisic2011b] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in
Use of Domain Specific Languages: a Case Study’. Proc. Evaluation and Usability
of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011
[Barisic2011c] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘How to
reach a usable DSL? Moving toward a Systematic Evaluation’, Electronic
Communications of the EASST, 2011
[Barisic2012] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Evaluating
the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and
Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI
Global, 2012)
35](https://image.slidesharecdn.com/quasar2013-131028071707-phpapp02/85/Iterative-usability-evaluation-of-DSLs-35-320.jpg)
Iterative usability evaluation of DSLs
- 1. Ankica Barišić, PhD student Supervisors: Vasco Amaral, Miguel Goualo
- 2. A language is a means of communication The user interface is a realization of a language A language is a model that describes the allowed terms and how to compose them into valid sentences 2
- 3. General Purpose (programming) Languages (GPLs) User has to master programming concepts User has to master domain concepts Domain Specific (modeling) Languages (DSLs) Meant to close gap between PROBLEM DOMAIN and SOLUTION DOMAIN Reduce the use of computation domain concepts Focus on the domain concepts 3
- 4. Increasingly popular Raise the abstraction level (closer to the domain) Narrow the design space Several benefits claimed, in well-defined domains Productivity gains Better time to market Avoid error-prone mappings between domain and software development concepts Leverage the expertise of domain experts 4
- 6. a model of the User Different knowledge sets Characteristics chosen are dependent on application domain the hardware-software set of computing, sensing, communication, and interaction resources e.g. operating systems, memory size, network bandwidth, input and output interaction devices the social and physical Platform Environment Where the interaction is actually taking place Different languages may have different contexts of use Their users are likely to have different knowledge sets A minimum set of ontological concepts is required to use the language 6
- 7. The user's view of the Quality of a product Measured in terms of the result of using the product, rather than its properties 7
- 8. In general, software language engineers DO NOT evaluate their languages with respect to their impact in the software development process in which the DSLs will be integrated Or, if they do, they are extremely shy about it 8
- 9. 9
- 12. • Introduce DSLs’ Usability evaluation during DSLs’ life-cycle • • • • iterations Design an effective experimental evaluation of DSLs that will provide qualitative and quantitative feedback for DSLs developers Produce user-centered design of DSL Foresee the Quality of a DSL while in an iterative evolution step Merge the Software Language development process with the Usability Engineering process 12
- 13. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 13
- 14. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 14
- 15. Barišić, Monteiro, Amaral, Goulão, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages“, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012 15
- 17. 17
- 18. per group 18
- 19. Language goals: Support users in creating and manipulating Evaluation: Compare two different DSL solutions within same domain sharing the same goal For the purpose of having means to reason about what is more usable: Abstract syntax definition Transformation Interaction 19
- 20. Development environment and methodology: Epsilon Model-Driven development Domain: Management and organization Transformation output: MS Excel template Place: Summer school on MDD of DSLs, Faculty of Organizational Science's 20
- 21. Domain Experts: subjects Usability Experts: experiment designers Language Developers: lecturers, subjects End Users: subjects Lecturers Experiment designers Subjects 2 2 39 21
- 23. Our evaluation technique was tested with two individuals (two physics experts) in order to verify it and to test the teaching materials and questionnaires As time constrants and equipment turn out to be adequat there was no need to change prepared materials [Howatt 1995] 23
- 24. Instrument: background questionnaire RQ1:Is querying with Pheasant more effective than with C+ +/BEE? RQ2:Is querying with Pheasant more efficient than with C+ +/BEE? RQ3:Are participants querying with Pheasant more confident on their performance than with C++/BEE? Our goal is to: analyze the performance of Pheasant programmers plug-ins for the purpose of comparing it with a baseline alternative (C+ +/BEE) with respect to the efficiency, effectiveness and confidence of defying queries in Pheasant from the point of view of a researcher trying to assess the Pheasant DSL, in the context of a case study on selected queries 24
- 25. H1null Using Pheasant vs. C++/BEE has no impact on the effectiveness of querying the analysis framework H1alt Using Pheasant vs. C++/BEE has a significant impact on the effectiveness of querying the analysis framework H2null Using Pheasant vs. C++/BEE has no impact on the efficiency of querying the analysis framework H2alt Using Pheasant vs. C++/BEE has a significant impact on the efficiency of querying the analysis framework H3null Using Pheasant vs. C++/BEE has no impact on the confidence of querying the analysis framework H3alt Using Pheasant vs. C++/BEE has a significant impact on the confidence of querying the analysis framework 25
- 26. We focus on presenting six examples, each focusing in some of the features we chose to evaluate Participants are asked to give themselves a mark for feeling of correctness of their trial Session take the time needed for each group to understand the examples 26
- 27. Every participant has four queries, specified in English, to be rewritten in previously learned language Subjects makes selfassessment of his replay rating his feeling of correctness Example: Build the decay of a D0 particle to a Kaon Pion 27
- 28. Results obtained with Pheasant were clearly better then those with C++/BEE Pheasant allowed non-programmers to correctly define their queries. The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using Pheasant The feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative. 28
- 29. The participants were asked to judge the intuitiveness, suitability and effectiveness of the query language. The goal was to evaluate: Overall reactions Query language constructs Affect to query language was rated by: Query language constructs Participants’ comments 29
- 30. Foster higher productivity of DSL end users Increased quality in use and quality of experience while using a DSL Replication of experiments Reusing data and recommendations Reduce complexity Traceability of usability recommendations and design changes Reduced cost of evaluation, development and support Increased reliability of usability evaluation results 30
- 31. 31
- 32. 32
- 33. Foster higher productivity of DSL end users Increased quality in use and quality of experience while using a DSL Replication of experiments Reusing data and recommendations Reduce complexity Traceability of usability recommendations and design changes Reduced cost of evaluation, development and support Increased reliability of usability evaluation results 33
- 34. 34
- 35. Literature [Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to develop domain-specific languages, 2005, ACM Computing Surveys [Gabriel2010] Gabriel, P., Goulão, M. & Amaral, V. (2010). Do Software Languages Engineers Evaluate their Languages? in XIII Congreso Iberoamericano en "Software Engineering" (CIbSE'2010) [Barisic2011a] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of DSLs: Current Evaluation Methods’. Proc. 3rd INForum - Simpósio de Informática (INForum2011), Coimbra, Portugal, September 2011 [Barisic2011b] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Quality in Use of Domain Specific Languages: a Case Study’. Proc. Evaluation and Usability of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011 [Barisic2011c] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘How to reach a usable DSL? Moving toward a Systematic Evaluation’, Electronic Communications of the EASST, 2011 [Barisic2012] Barišić, A., Amaral, V., Goulão, M., and Barroca, B.: ‘Evaluating the Usability of Domain-Specific Languages’, in Mernik, M. (Ed.): ‘Formal and Practical Aspects of Domain-Specific Languages: Recent Developments’ (IGI Global, 2012) 35
Editor's Notes
- #2: {}