Pilot implementation of Active Learning at an Engineering Department

Pilot implementation of Active
Learning at an Engineering
Department for the development of
generic skills of students

The pilot course
Decision Support Systems (DSS)

Technical University of Crete, Department
of Production Engineering & Management,
Evangelia (Lia) Krassadaki, PhD candidate

Indicatively presentation of practices applied in universities for
the improvement of generic skills of students

• A’ trend: as part of the curricular
– Through courses (without activities for skills assessment) of the curriculum. A certification-
transcript is given at the end of studies to students (the case of Luton University)
– Autonomous courses for generic skills improvement. The grade at the course certifies the level
of the skills possessed by students. For example, the case of General Education program at
college level-USA.
– Writing/speaking intensive courses. The case of USA universities.
– E-Portfolios (University of Denver, Virginia Tech, Queensland University of Technology, etc)
• B’ trend: as extracurricular activities
– Activities of Career Offices sometimes by giving a certificate and sometimes without it. For
example, the transcript ‘Nottingham Advantage Award’ from Nottingham University.
– Surveys for skills improvement for graduates or students. Characteristic case the yearly
graduate survey in Australia ‘Graduate Course Experience’-GSA (118 thousand graduates
participated the year 2009).
– Specially designed tests for the evaluation of generic skills, like the national level test in
Australia ‘Graduate Skills Assessment’-GSA for assessing critical thinking, problem solving,
interpersonal understanding and written communication.


The educational context of the pilot
implementation at Technical University of Crete
Goals:
Performance improvement at the content of
the course DSS (content)
Generic skills improvement: Communication
(written-oral) & Team-working
Active participation of students
Satisfaction of students


Criteria for the verification of the successful
redesign of the course


The Constructive Alignment Model which is used
for the redesign of the course

Source: Biggs, 2003; Felder & Brent, 2003.


Redesigning a course: (1) Learning Outcomes (students) -
Learning Objectives (tutor)

ΑΒΕΤ
criteria
by italics

A model for course provision based on 3 areas: 1. Disciplinary content – content of the
course 2. Disciplinary skills - practical skills associated with the content of the course 3.
Generic skills – informal/transferable skills or abilities

Redesigning a course: (2) Teaching
(strengthening participation & interaction)
 Short lecture technique (vs lectures)
 Brainstorming technique
 Discussion technique
 Question-answer technique
 Giving a speech – presentation of a subject (students)
 Technique of 10-minute exercises in groups of two
 Technique of short written assignments (every week)
 Digital self-evaluation material for the knowledge
evaluation of the content of the course (outside class)
 Research questionnaires (inside/outside class)

Redesigning a course: (3) Assessment
(summative – formative)
Summative assessment:
• Short written-oral assignments (every week)
• Oral performance on disciplinary content & disciplinary skills
Formative assessment:
• 10-minute group exercises (group of two)
• Digital self-evaluation material for the knowledge evaluation
(outside class)
• Peer-assessment technique of written assignments & oral
presentations (inside/outside class)
• Individual feedback for every written (oral) assignment (tutor)
• Self-evaluation of students’ generic/informal skills (before-after
scheme)

Feedback


Treatment group (students of the pilot course)
2009: 1 group, 27 students
2010: 2 groups, 46 students
2011: 2 groups, 44 students

Control group (students of the traditional offered
course)
2009: 57 students
2010: 30 students
2011: 40 students

Pilot Course characteristics

• Meeting: once per week
• Hours: 2 hours/week
• Duration: 13 weeks
• Tutors: 1 professor & 1 coordinator
• Classroom: room (not an amphitheater) of total capacity 30 persons
• Room with circular arrangement
• Technological equipment: WiFi, video-projector, laptop(s), board
• Participation: compulsory
• Submission of assignments: Ε-Class (E-learning platform) with time
constraint
• Groups of two-students: form the groups by lot


Plan of 2nd meeting with topic Information &
Information Systems (2 teaching hours)
Educational Techniques:
• Short lecture (15 minutes)
• Brainstorming (20 minutes)
• Questions-answers (10 minutes)
• Group exercise (10 minutes)
• Discussion about the exercise (10 minutes)
• Short lecture (5 minutes)
• Discussion (15 minutes)
• Overview-short lecture (5 minutes)


Comparison of average performance on disciplinary content & skills for
three years

6.7
Average grades 2011
5.2

6.4 Students of the Pilot course
Average grades 2010
5.4 Students of the Traditional offered
course

6.4
Average grades 2009
5.7

Higher average
5 5.2 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 7 grades for the
students of the
Technical University of Crete, Department pilot course

Results for t-test & Effect size
2009:
• [t (66.33) = 2.12, p <0.05], 26 students pilot course, 52
students traditional course
• Cohen’s d = 0.48, r = 0.23 (medium effect)
2010:
• [t (73) = 2.601, p <0.05], 45 students pilot course, 30
• Cohen’s d = 0.6, r = 0.29 (medium-large effect)
2011:
• [t (79) = 4.029, p <0.00], 42 students pilot course, 39
• Cohen’s d = 0.89, r = 0.41 (large effect)

Self-evaluation of Writing skill (before-after
scheme for the students of the pilot course)
Self-evaluation criteria:
1. Organise a text (report)
2. Use of literature
3. Synthesis & presentation of information in texts
4. Online information retrieval for documentation
purposes
5. Word processing


Writing skill – Statistical & Multicriteria Results of
the self-evaluation survey (year 2010)

Wilcoxon Weight SI index Weight SI index %
Test (before) (before) (after) (after) improvement
-3.924
Organise a text (0.000) 32.6% 16.2% 20% 72.9% 350
-3.848
Use of literature (0.000) 20% 19% 20% 41.5% 118
Synthesis & presentation of -3.646
information (0.000) 20% 25.2% 30.4% 31.8% 26
Information retrieval from -2.358
internet (0.018) 13.3% 55.2% 15.6% 73.5% 33
-3.689
Word processing (0.000) 14.1% 43.1% 14% 72.7% 69
-4.482
Global self-evaluation index (0.000) 26.9% 56.6% 110.4

Strong Weakness Weakness with Significant
Technical University of Crete, Department improvement improvement
of Production Engineering & Management, despite the

Writing skill – comparison of the indices
at the before-after scheme (2010)
80%
73.5%
72.9% 72.7%
70%

60%
55.2% 56.6%
50%

41.5% 43.1%
40% 31.8%

30%
25.2% 26.9%
20%
19%
10% 16.2%

0%
organise a text use of literature synthesis & information word processing Global self-
presentation of retrieval from evaluation index
information internet

SI (before) SI (after)

Self-evaluation of speaking skill (before-after

1. Oral argumentation
2. Formulation of questions-answers in class
3. Participation in discussion in class
4. Speaking in class (speech)
5. Computer-supported presentation of a topic


Speaking skill – Statistical & Multicriteria Results
of the self-evaluation survey (year 2010)
%
Wilcoxon Weight SI index Weight SI index improvem
Test (before) (before) (after) (after) ent
-2.461
Oral argumentation (0.014) 14.6% 51.6% 37% 23.3% (-54.8)
Formulation of questions- -2.953
answers in class (0.003) 30.4% 17.3% 13.5% 51.2% 196
-2.051
Participation in discussion in class (0.040) 19.6% 33.9% 16.5% 71.2% 110
-3.075
Speaking in class (speech) (0.002) 20.6% 18.8% 16.5% 33% 75.5
Computer-supported -3.798
presentation of a topic (0.000) 14.7% 27.2% 16.5% 63.6% 134
-3.162

Strong weakness Weakness with Significant
improvement improvement
Technical University of Crete, Department despite the
Evangelia (Lia) Krassadaki, PhD candidate weaknesses

Speaking skill – comparison of the indices at
the before-after scheme (2010)
80%
71.2%
70%
63.6%
60%

50% 51.6% 51.2%
43.2%
40%
33.9% 33%
30%
28.4%
23.3% 27.2%
20%
17.3%
18.8%
10%

0%
oral argumentation formulation of participation in speaking in class computer-supported Global self-
questions-answers discussion in class (speech) presentation of a evaluation index
topic



Self-evaluation of teamworking skill (before-after
1. Organisation & planning activities with
colleagues
2. Collaboration with colleagues for project
completion
3. Respect for different opinions
4. Consensus politics for teamwork
5. Creative criticism to team members
6. Leading a team
7. Mediating for conflicts among members

Teamworking skill – Statistical & Multicriteria
Results of the self-evaluation survey (year 2010)
%
Wilcoxon Weight SI index Weight SI index improveme
Test (before) (before) (after) (after) nt
Organisation & planning -3.807
activities with colleagues (0.000) 14.3% 58.1% 14.3% 72.3% 24.4
Collaboration with colleagues -2.457
for project completion (0.014) 15.8% 68.9% 14.3% 72.9% 5.7
-1.014
Respect for different opinions (0.311) 14.3% 72.5% 14.3% 75.9% 4.8
Consensus politics for -3.852
teamwork (0.000) 13.1% 59.3% 14.3% 75.9% 28
Creative criticism to team -3.586
members (0.000) 14.3% 60.6% 14.3% 71.8% 18.5
-2.696
Leading a team (0.007) 13.9% 51.1% 14.3% 65.5% 28.2
Mediating for conflicts among -1.272
team members (0.203) 14.3% 67.2% 14.3% 70.9% 5.5
-3.922
Lower room for improvement – higher self-confidence for the teamworking skill

Teamworking skill – comparison of the
indices at the before-after scheme (2010)
90%

80% 72.3% 75.9% 75.9% 71.8%
72.9% 70.9% 76.6%
70% 72.5% 65.5%
66.8% 64.70%
69.6%
60%
58.1%
61.1%
50% 59.3% 51.1%

40%

30%

20%

10%

0%
organisation & collaboration respect for consensus creative leading a team mediating for Global self-
planning of with colleagues different politics for criticism to conflicts among evaluation
activities with for project opinions teamwork team members team members index
colleagues completion



Contribution of different courses at the development of
generic skills of students (a hypothetical example – results
from the pilot course)
Writing skill Speaking skill Teamworking skill
Course % GLOBAL CONTRIBUTI % GLOBAL CONTRIBUTI % GLOBAL CONTRIBUTI
IMPROVEMEN ON IMPROVEME ON IMPROVEMEN ON
T INDEX NT INDEX T INDEX
(2010) 110.4 Substantively 52.1 Slightly 18.4 Moderately
(2011) 140.3 Substantively 277.7 Substantivel 36.3 Substantivel
y y
Low 14.95 112.8 8.95
thresho
ld
High 77.63 195.25 22.63
thresho
ld

The 3-level contribution scale (slightly, moderately, substantively) : a proposal from Felder & Brent (2003)

Implementation of Peer-assessment of
written/oral assignments
Written assignments
• Collect 6 assignments of different quality (coordinator)
• Delete of personal information-details of the authors
• Distribution: copies of 6 assignments & a specially designed
evaluation sheet
• Distribution: same assignments at the years 2010 & 2011
• Distribution: at the beginning of the pilot course as a homework
• When peer-assessment completed a discussion followed for the
findings and results
Oral assignments
• Distribution of the evaluation sheet before the presentations
• Completion of the evaluation sheet after the presentation-speech
• When peer-assessment completed a discussion followed for the
findings


Multicriteria clustering of students as assessors for written
assignments based on the assessed weights of criteria
100%
2010 (38 students)
80%

2011 (19 students)
60%
100%

40%
80%

20% 60%

0% 40%
1 2 3 4 5 6 7 8
20%

0%
ALL (57 students) 1 2 3 4 5 6 7 8
100%

80% Content completeness
Syntax-spelling
60%
Text organisation
40%

20% • Criteria weights: UTASTAR
• Clusters: Global K-means
0%
1 2 3 4 5 6Technical University of Crete, Department
7 8
• Clustering quality:
of Production Engineering & Management, Silhouette coefficient

Population (students-assessors) distribution at the 8
clusters according to criteria weights

54% 9% 38%
5 2 2
3 4% 93% 4%

9% 27% 65%
10 3% 49% 49%

100% 0% 0%

1 44% 46% 10%
The most 29
5
populated group: 33% 34% 33%
almost equal
weight for each of 2% 4% 94%
the 3 criteria


Students’ behavior in peer-assessment: the multicriteria
approach for the 8 groups at this case study


Implementation of integrated feedback policy for
students-tutors

From tutor (coordinator of the pilot course) to
students
• Informative personal comments on written
assignments
• Informative personal comments to oral
presentations
From students to tutor (coordinator of the pilot
course)
• Students’ satisfaction survey
• Teaching techniques preferences survey

Results of the students’ satisfaction survey for the
years 2010 & 2011. (Application of multicriteria
method MUSA)
Criteria Assessed Weight Assessed satisfaction
(%) index-SI (%)
Content comprehension 16.7 92.3
Connection with prior knowledge and 16.7 94
experiences
Quality of the material of the pilot course 16.7 91
Learning & teaching approaches applied 16.7 89.7
Room and recourses used during pilot 16.7 82
course
Coordinator’s preparation for the course 16.7 97
GLOBAL SATISFACTION 97.2

All satisfaction indices
(partial & global) are very
of Production Engineering & Management, high

Teaching techniques preference survey using Conjoint Analysis (for
the development of communication & teamworking skills)
no Factor Level
1 Written assignments Every week
Every two weeks
2 Oral presentations Every week
Every two weeks
3 Completion of a group project during Yes (project technique is applied)
semester No (project technique is not
applied)
4 Discussion through questions-answers Yes (the technique is applied)
No (the technique is not applied)
5 10-minute group exercises by cooperation Yes (the technique is applied)
of two students in every meeting No (the technique is not applied)

The survey is based at a specially designed questionnaire using 10 scenarios via orthogonal design
(8 design, 2 holdout scenarios)

Results of the teaching techniques
preference survey
35%

I 30%

m
25%
p
o 29.55%
20%
r 23.25%
t 15%
21.48%
a
n 10% 13.91%
c
11.81%
e 5%

0%
written assignments oral presentations group project discussion through 10-minute group
questions-answers exercises
Factor
The 2 most preferred techniques


Criteria for the verification of the successful
redesign of the pilot course
• Students’ performance: higher than that of the students
who attended traditional offered course
• Writing skill: improvement despite the weaknesses
• Speaking skill: improvement despite the weaknesses
• Teamworking skill: improvement despite the lower room
for improvement
• Satisfaction survey: high satisfaction partially & globally
• CONCLUSION: successful design of the new pilot course


Some more benefits of the pilot course
design & implementation for 3 years
 Active participation of students at the learning process
 Enhancement of students interaction, who are not well-known, friends, etc
 Connection course’s content with prior learning (knowledge & experience)
 Development of generic skills of students through a course (major trend applied
universally)
 Implementation of Formative Assessment techniques (not for grade)
 Implementation of the peer-assessment technique (new experience for students)
 Implementation of oral presentation technique on a week basis (new experience for
students at this level of studies)
 Determining the behavior of students as assessors in peer-assessment process
 Positive effect of the Learning Objectives of the tutor at the behavior-value system of
the students as assessors
 Determining the criteria significance weights at the peer-assessment process as
opposed to negotiation process among students and the tutor(s).
 Determining the preferences of students for teaching & learning approaches applied


Difficulties for the implementation of active
learning
At general level:
• Low content flow in comparison with the traditional offered course.
• Low number of students per group when active learning is applied, which
practically implies more tutors per course or more time spent by one tutor.
• High initial investment in time for the design of the course, new materials, etc.
• Substantive investment in time during the course which compete other roles of
tutors, like: research, administration tasks, etc.
• Change of the applied low level (or no) feedback. Usually, feedback is an under-
estimated process during teaching.
• etc

At Department level:
• No room with round table for the meetings of the students-tutor.

Difficulties for implementation of active learning
(cont.)
At Greek Universities level:
• Absence of a supportive policy for universities in order to describe the Learning
Outcomes (Objectives), as opposed to content-based curriculum already
applied.
• Staff shortage.
• Educational adequacy of the staff that implements a different technique than
the usual lecture.
• Assessment: an under-estimated process usually synonym to grading.
• Generic-transferable skills: unknown words at Higher Education Level.
• The usual assessment technique: summative. New other forms of assessment
either formative or summative like e-portfolios are unknown.
• etc


Difficulties for implementation of active
learning (cont.)
At methodology level:
• Literature review demonstrates the dominance of statistics due to the great interest in such a field
of researchers coming from education, sociology, etc thus the use of multicriteria methods is a
novelty. Therefore is not easy to ‘compare’ the usefulness of the different approaches applied with
previous research.
• Moreover, is not easy to compare the exact results achieved with other researches due to one more
reason: the uniqueness of each course even if the compared courses deal with the same subject. Of
course, general speaking the results show the same weak and strong points for our students, which
agree in a great extent with literature review.
• Data shortage for testing purposes of methodology and techniques. Each researcher owns data
concerning his/her students but these data are not available for more persons even for research
purposes.
• There is an extended literature about the skills developed during studies. Many synonyms or nearly
synonyms terms are used alternatively, like generic skills, graduate skills, key-skills, transferable
skills, soft skills, etc. Similarly, classification attempts have produced a large number of different
lists of attributes/skills at international level . Mason (2010) says that ‘a definitive agreed set of
skills that students in higher education should be fortunate enough to be born with or expected to
acquire simply does not exist. This is not only true but also desirable’.
• etc


A proposal for implementation
As an extra like the pilot course, offered in parallel
with the traditional form of the course or at a group
of courses where tutors want as volunteers to act
innovatively by applying new methods of teaching
and learning.
Key-point for an ‘innovative’ approach: awareness
of the academic community!!


Pilot implementation of Active Learning at an Engineering Department

More Related Content

What's hot

Viewers also liked

Similar to Pilot implementation of Active Learning at an Engineering Department

More from Krassadaki Lia (Evangelia)

Pilot implementation of Active Learning at an Engineering Department