Paper ID #6204

Using Mini Design Competitions in Capstone

Capt. Joseph Anders Wahlquist, US Air Force Academy

Joseph Wahlquist is an instructor in the department of engineering mechanics at the United States Air
Force Academy. He teaches courses in Mechanical Behavior of Material and leads a Capstone Design
team focusing on Small Unmanned Aerial Systems. He received his MS in Aeronautical engineering
from the Air Force Institute of Technology in Dayton Ohio and a BS in Mechanical Engineering from
Brigham Young University in Provo Utah. He has also worked as a structural engineer on the F-16 and a
mechanical systems engineer for the F-4, F-5, T-37, T-38 and OV-10.

Dr. Daniel D. Jensen, U.S. Air Force Academy

Dr. Dan Jensen is a Professor of Engineering Mechanics at the U.S. Air Force Academy where he has
been since 1997. He received his B.S. (Mechanical Engineering), M.S. (Applied Mechanics) and Ph.D.
(Aerospace Engineering Science) from the University of Colorado at Boulder. He has worked for Texas
Instruments, Lockheed Martin, NASA, University of the Pacific, Lawrence Berkeley National Lab and
MSC Software Corp. His research includes design of Micro Air Vehicles, development of innovative
design methodologies and enhancement of engineering education. Dr Jensen has authored approximately
100 papers and has been awarded over $3 million of research grants.

Dr. Kristin L. Wood, Singapore University of Technology and Design (SUTD)

Lt. Kyle Fitle, United States Air Force
David Carte, United States Air Force

Page 23.1327.1

c American Society for Engineering Education, 2013

 Using Mini Design Competitions in Capstone
Courses to Teach the Design Process

Abstract

For many senior undergraduate engineering students, the capstone design project is their first
experience implementing the design process. As a result many capstone teams do not grasp the
importance of the early stages of the design process. To help students better understand how
these early steps will impact the overall project; capstone students were given the opportunity to
participate in a mini design competition. These students took two weeks to produce classroom
demos which were evaluated by instructors for their usefulness in teaching. In the current
research, this set of students forms the “experimental” group. Other students were given an
overview of the design in a classroom setting, but did not have the opportunity to implement the
process outside of their main capstone experience. This set of students forms the control group.

It was hypothesized that using the students’ time in the design competition would get students
excited for their capstone project, help them be more comfortable using the lab equipment, and
help them better understand the design process. A quiz was administered to assess student
understanding of the design process, motivation, and lab equipment familiarity. The scores from
the experimental and control groups were compared. Qualitative assessment by capstone mentors
indicates that students exposed to the mini design exercise had a more holistic understanding of
the design process. Additionally, quiz scores indicated a slightly more rapid increase in
understanding. However, the design competition did not appear to affect familiarity with lab
equipment and or student motivation. This paper reports on the details of the mini design
competition, the specifics of the assessment instruments and the details of the qualitative and
quantitative assessment results.

Page 23.1327.2
Introduction

The capstone design course offered at many universities provides engineering students with the
opportunity to apply lessons learned throughout their education. These courses are an important
method used to prepare future engineers and help to fulfill many of the requirements specified by
ABET. Specifically these courses meet ABET criteria 5 which states “Students must be prepared
for engineering practice through a curriculum culminating in a major design experience based on
the knowledge and skills acquired in earlier course work and incorporating appropriate
engineering standards and multiple realistic constraints.” [1] Three of the important outcomes of
this capstone experience involve learning about professional ethics, teamwork skills, and design
methodologies. These courses also provide an invaluable opportunity for students to move
beyond passive absorption of the material and apply these skills to an open ended design project.
This paper focuses on efforts to improve the understanding and application of design
methodologies. However, it should be noted that by better understanding and applying this
aspect of the capstone experience students frequently experience beneficial effects in other key
elements of the learning experience.

Capstone engineering experiences across the country are typically either one or two semesters in
length. [2] Normally, if the capstone design project is a one semester course, then a separate
course covering design methodology is taught previous to the capstone course. There are
advantages and disadvantages of each of these approaches. If the design methodologies are
taught in a separate course, the students should be well prepared to make efficient and effective
use of these methods in their one semester capstone design project. However, when capstone
projects are sponsored by industry or when the project is a national competition (such as the SAE
Formula or Baja projects) it can be extremely difficult to complete the project in a single
semester simply due to the time needed to complete a full design. Design process steps such as
customer needs analysis, design requirement development, feasibility and performance analysis,
manufacturing and testing all take significant time. [3] The approach detailed in this paper creates
a hybrid solution between the one and two semester course strategies in that it allows for the full
year long focus on the sponsored or competition project, but provides an initial exposure to the
design tools prior to full immersion in the capstone project.

To make application of design methodologies successful it is imperative that students have a firm
understanding of what they are supposed to be doing and why. Many educators have noted that
students experience a great level of discomfort when presented with ill-defined design problems.
[4]
Students frequently express this displeasure when presented with a situation for which there is
no closed form solution and to which there may be many acceptable solutions. Repetition is an
Page 23.1327.3

invaluable tool to increase student familiarity and proficiency in confronting these ill-defined
problems. Some universities have taken dramatic steps to increase the frequency of projects
which require students to use the design process to confront these ambiguous problems. [5]

Experience with capstone teams has shown that students have particular difficulty with the early
stages of the design process. Frequently these teams fail to properly understand customer needs
and the importance of sufficiently translating these needs into specifications. Only late in the
project do they learn that many of the design solutions they have selected do not sufficiently
meet some of the key customer requirements. Unfortunately, this experience is not limited to
students; industry is strewn with examples of failed products which neglected to address key
customer needs. [6]

Another common problem experienced by capstone teams is that they misunderstand the purpose
of the design tools. Students may see the design tools as simply a means to assign grades or a
series of obstacles they need to overcome before getting to the “real” work of building their
product. All too frequently students “…become so deeply involved in their projects that they fail
to see “the forest for the trees.” [7] This appears to be especially true for the design process steps
of customer needs analysis [6] functional decomposition [8] and concept generation. [9]

To address these shortcomings it is proposed that a brief refresher providing a big picture view of
the design process is beneficial. One method of providing this big picture is through the use of a
mini-design competition. This competition can serve to help students better understand how
design tools are interrelated and emphasize the importance of using these tools when in solving
ill-defined problems. Other institutions have used these mini design projects to improve student
understanding of the design process and have found them to be beneficial.[3] [7] [10] This work
attempts to qualify and quantify the benefit obtained through use of these mini-design projects.

Background

The design process is a methodical system used to understand customer needs, generate
solutions, and evaluate the potential for these solutions to meet the customer needs. While there
are some differences in the way that the design process is taught the overarching concept remains
the same.[11] [12] [13] [14] The design process, used in the Engineering Mechanics department at the
Air Force Academy, consists of 7 major steps. These steps are:

 Identify the Customer Needs

 Derive Specifications from Customer Needs
 Generate Concept Variants
 Select the Most Promising Concept Variants
 Analyze the System

Page 23.1327.4

Prototype the System

 Test the System
Capstone courses at the Air Force Academy are two semester in length and are taught during the
senior year. Teams are composed of 2 to 11 students with the average team size being 6.4.
Projects are funded by private companies, multiple external DOD organizations, the Air Force
Academy, and other federal agencies. Frequently the result that these sponsor agencies find most
valuable is the innovative ideas offered by students. Because the students are relatively new to
the engineering field they frequently approach problems from a different perspective than those
who have spent years working these issues. The heavy emphasis on creative solutions makes it
imperative that students avoid becoming fixated on a single solution. By properly applying the
design process they are able to generate a far greater number of useful concept variants. [15] [16]

Methods

In order to gage the effectiveness of the mini design competition in teaching design
methodologies two groups were used in this study. The first group was given a mini design
competition. This competition was intended to remind students of the design process which they
have used in previous classes and emphasize its utility. Students were given two weeks to
accomplish the task of designing a classroom demonstration to be used in an introductory
engineering course. All students had taken this introductory engineering class in their freshman
or sophomore year and were familiar with the course content. However, the projects were to be
judged by a panel of instructors so it was important for students to understand the utility of the
demonstration from the instructor’s perspective.

Participants were allowed to select their own two person team for the mini design competition.
They were then provided with a handout which briefly reviewed the highlights of the design
process and were informed that at the end of the two weeks they would give a presentation to a
board of faculty members. This handout can be seen in the appendix A. The evaluating faculty
would assess both their final product and the quality of presentation. Additionally, the team who
best met the instructor’s needs for a classroom demonstration was awarded a gift certificate to a
local restaurant. Because previous capstone teams had struggled significantly when customer
needs were unclear the panel’s judging criteria and constraints for the competition were left
purposefully vague. Instructors provided additional guidance only when students put forward
extra effort to understand their customer’s needs. However, students were able to seek continual
feedback from instructors to verify that their concepts would satisfy the customer’s needs as
often as they desired.

The second group acted as a control group for the experiment. These students attended lectures
on the design process and were given team time to work on their primary capstone project but
did not participate in the mini design competition. The two groups were divided by capstone
project with two capstone teams participating in the mini design competition and two capstone
Page 23.1327.5

teams acting as the control. Additional capstone projects at the Air Force Academy did not
participate in this study. One limitation of this study is that the assignment of students to
capstone teams and subsequently the two study groups was not purely random. Capstone team
assignments were based partially on student preference and instructor requests. Another
limitation is the small sample sizes used; the number of students participating in the mini design
competition was 15 and the number in the control group was 18.

Both quantitative and qualitative measures were used to gage student understanding of design
methodologies. To quantify understanding of the design process, quizzes were administered
before and after the mini design competition. The quizzes were also administered at the end of
the first semester in an effort to measure the longitudinal effects of the experience. These
quizzes consisted of multiple choice and short answer questions and covered each of the major
steps in the design process. In addition to the quizzes, the performance of teams who participated
in the mini design competition was discussed with capstone mentors. These mentors had guided
capstone teams in the past that did not participate in mini design competitions and were ideally
situated to compare the performance of teams with and without this introductory experience.

Additionally, students were asked to rate their familiarity with lab equipment and provide a self-
assessment of their level of motivation for the capstone course. It was hypothesized that getting
students into the lab working on a project early in the semester would increase their motivation
and familiarity with the lab resources.

Results

Qualitative assessment of capstone team performance by instructors indicated that the mini
design competition was a valuable use of student time. This sentiment was expressed by one of
the mentors who remarked that “[i]t was apparent that my team retained very little knowledge of
the engineering design process from previous courses prior to this exercise. Instead of teaching
them how these should be done, they actually went through the process with physical objects and
in doing so greatly enhanced their learning. In addition, it was a fun way to get the semester
started.” [17] Also, when compared to previous teams, there was a marked decrease in the amount
of student resistance encountered in the early stages of the design process. Students were more
willing to spend the time to understand customer needs and generate concept variants than in
previous years. Mentors were able to refer back to the mini design competition when guiding the
team through the design process. This was beneficial as it provided context for learning using a
recent shared experience. This type of learning environment appears to fit well within the
constructivist or scaffolding theories of education. [18]

During the time that the experimental group was participating in the design competition the
control group spent roughly equivalent amounts of time attending lectures focused on the design
process and working on their principal capstone project. One faculty concern was that by using
Page 23.1327.6

two weeks of class time to hold the design competition, the teams would not be able to complete
as extensive of a primary capstone project as teams who did not participate in the competition.
However, experience showed that students who completed the design competition were able to
make the same progress as those teams who had not participated in the competition. Capstone
mentors observed that the increase in student understanding and proper use of the design tools
compensated for the condensed timeline.

Students expressed that while intellectually they understand that the design process is critically
important in engineering product development, it is often neglected due to the desire to rapidly
produce a product. One student noted that “participation in a design process exercise greatly
solidified the concepts and procedures employed during product design. Rather than a lecture
that often teaches from a theoretical standpoint, this participatory exercise allowed the us to
conduct our own research and apply these concepts in a hands-on application.” Developing their
own “mini project” allowed the students to inject their own creativity and ideas into the design
process and established the personal investment in a project which helped solidify the learning
experience. Another student said that the mini design “helped me think about the individual steps
in the design process and utilize the concepts of the process in a real-world design scenario.”

Quantitative measures also indicated that student understanding of the design process improved
more rapidly in groups participating in the mini design competition than in lectures alone. The
results of the quiz scores can be seen in Figure 1. Analysis of incoming grades indicated that
there was no statistically significant difference in GPAs. Additionally, the differences in initial
quiz scores between the control and experimental group did not show a significant difference.
This led to the observation that both groups started with a roughly equal understanding of the
design process and learning potential. The quizzes consisted of multiple choice and short answer
questions and partial credit was awarded. Sample questions can be seen in appendix B.

At the end of the mini design competition the design process quiz scores for the two groups
diverged. Assuming a normal distribution, the statistical t-test revealed that there is a 98%
probability that the there is a true difference between the quiz scores of these two groups (p <
0.02). This is well within the normal criteria for “statistical significance” which is normally set
at p<0.05. This increased understanding of the design tools carried through for the rest of the
semester. While quiz scores decreased slightly for both groups, the average quiz scores of those
participating in the design competition remained greater than 8% higher than the control group.
Similar to the results immediately following the design competition there was a 98% probability
that the end of semester scores represented a real difference in understanding.
Page 23.1327.7
 Design Process
100%

90%

80%
Quiz Score

70%

60%

50%
Short Design Comp, n=15
40%
Control, n=18
30%
0 10 20 30 40 50
Lesson

Figure 1. Design process quiz scores

Error bars indicate a standard error for each of the data sets

After students had taken the design process quizzes they were asked to anonymously report their
level of familiarity with lab equipment using a Likert-style scale. The results of these surveys
indicated that, although students became more familiar with lab equipment throughout the
semester, the mini design competition did not play a significant role in improving this metric.
Overall, there was no statistically significant difference between the control and experimental
groups in self-reported lab equipment familiarity at any time in the semester. However, because
this rating was self-reported and subjective in nature, future efforts may shed additional light on
this topic.

Another metric tracked in this study was the students’ self-reported level of motivation. The
results of this survey are shown in Figure 2. At the same time students reported their lab
equipment familiarity they also reported how motivated they felt towards their capstone project.
In both groups the level of motivation decreased throughout the semester, and in both cases the
amount of decrease was similar. Unfortunately, the self-reported motivation level was not equal
at the start of the study and remained unequal throughout the semester. Further investigation into
the source of this discrepancy revealed that this difference was due primarily to team members
participating on the Formula Society of Automotive Engineers (Formula SAE) team. Members
of the Formula SAE capstone team reported significantly higher levels of motivation throughout
the semester than members of any other team. Teams not participating in FSAE showed very
similar trends in motivation level irrespective of their participation in the mini design
competition. This observation infers that motivation level is far more strongly influenced by the
Page 23.1327.8

overall project focus than by the use of a mini design competition.

 Motivation
120%
Self-Reported Motication Level 100%

80%

60%

40%

20% Short Design Comp, n=15

Control, n=18
0%
0 10 20 30 40 50
Lesson

Figure 2. Self-reported motivation level

Error bars indicate a standard error for each of the data sets

After observing the difference in motivation level experienced by the FSAE team, analysis was
performed to investigate whether this team’s understanding of the design process or familiarity
with the lab was significantly different than the other individuals in the control group. While
there are significant limitations to the analysis, due to the small sample sizes, it appears that the
only area in which the FSAE team performed significantly differently than other team was in the
self-reported level of motivation.

Utility of Mini Design Competition

Student Responses
9
8
Number of Respondents

7
6
5
4
3
2
1
Page 23.1327.9

0
Yes Somewhat No
 Figure 3. Student assessment of utility of mini design competition

The final question asked to students who participated in the design competition was whether or
not they felt that the mini design competition had been a worthwhile investment of their time.
The results of this question are shown in Figure 3. Approximately 60% of respondents indicated
that they believed the design competition had been beneficial, 20% indicated that it was
somewhat useful and 20% said that it was not a valuable use of their time.

Conclusion

The results of this study indicate that the use of a mini design competition at the beginning of the
capstone experience is a worthwhile investment of student time. The increased understanding of
design tools, and their relation to the process, appears to have improved the student’s desire to
follow a systematic approach to design. Capstone mentors observed that the vague problem
statement used in the design competition was a valuable tool in preparing student for some of the
more challenging aspects of their primary capstone experience. Students and mentors felt that the
use of competition was an effective way to get students engaged early in the course.

During subsequent applications of the mini design competition, various improvements are
planned. First, to increase student motivation to participate in the design competition, mentors
may exercise the option to tie the competition more directly to student grades. This would be
accomplished by assigning points based on the presentation given at the end of the competition.
This produces the additional benefit of providing a slightly more demanding situation for
students when giving their presentation. The additional stress produced by linking the
presentation to grades helps better prepare students for design review and sponsor presentations.
Second, competition objectives will be selected which more actively engage the students. While
students were reasonably engaged in designing and building classroom demonstrations selection
of a more exciting project is expected to produce more vigorous participation.

In conclusion, the use of a mini design competition at the beginning of a capstone course
produced beneficial effect in student understanding and application of the design process. This
effect was observed both quantitatively via quiz scores and qualitatively through capstone
mentors evaluation of student performance and attitudes.

Acknowledgements

I would like to express my sincere gratitude to fellow capstone mentors who contributed ideas
and helped guide this study particularly Maj Cody Rasmussen, Capt Michael Knauf and Capt
Andrew Radzicki. I would also like to express appreciation to the students who participated in
Page 23.1327.10

this study and Department leadership for supporting innovation in teaching approaches.
This material is based on research sponsored by the United Sates Air Force Academy under
agreement number FA7000-12-2-2005. The US Government is authorized to reproduce and
distribute reprints for Government purposes notwithstanding any copyright notation thereon.

The views and conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies or endorsements, either expressed or
implied, of the United States Air Force Academy or the US government.

Bibliography

[1] ABET, "ABET Criteria for Accrediting Engineering Programs, 2012 - 2013," 7 Jan 2013.
[Online]. Available: http://www.abet.org/.

[2] S. Howe, "Where Are We Now? Statistics on Capstone Courses Nationwide," Advances in
Engineering Education, vol. 2, no. 1, 2010.

[3] K. Wood, D. Jensen, J. Bezedek and K. Otto, "Reverse Engineering and Redesign: Courses
to Incrementally and Systematically Teach Design," Journal of Engineering Education, pp.
363-374, July 2001.

[4] M. Z. Hasan, "Trend Analysis of Capstone Design Projects for Improving Undergraduate
Engineering Education," in ASEE Anual Conference and Exposition, San Antonio, 2012.

[5] J. L. Schiano, "A Four-year Vertically Integrated Design Sequence in Electrical

Engineering," in ASEE Annual Conference and Exposition, San Antonio, 2012.

[6] M. Green, D. Jensen and K. Wood, "Design for Frontier Contexts: Classroom Assesment of
a New Design Methodology with Humanitarian Applications," International Journal of
Engineering Education, vol. 25, no. 5, 2009.

[7] M. Eggermont, R. Brennan and T. Freiheit, "Improving a Capstone Design Course Through
Mindmapping," Advances in Engineering Education, vol. 2, no. 1, 2010.

[8] B. W. Caldwell, G. M. Mocko and J. D. Summers, "An Empirical Study of the

Page 23.1327.11

Expressiveness of the Functional Basis," Artificial Intelligence for Engineering Design,

 Analysis & Manufacturing (AI-EDAM), pp. 273-287, 2011.

[9] C. White, D. Jensen and K. Wood, "From Brainstorming to C-Sketch to Principles of

Historical Innovators: Ideation Techniques to Enhance Student Creativity," Journal of
STEM Education, vol. 13, no. 5, 2012.

[10] K. Wood, D. Frey, R. Crawford, C. White, R. Mohan, C. Dym, S. Kaijima, S. Dritdsas and
D. Jensen, "A Symphony of Designiettes – Exploring the Boundaries of Design Thinking in
Engineering Education," in ASEE Annual Conference and Exposition, San Antonio, 2012.

[11] D. Ullman, The Mechanical Design Process, McGraw Hill, 1997.

[12] K. Ulrich and S. Eppinger, roduct design and Development, McGraw Hill, 2000.

[13] S. Otto and K. Wood, Product Design: Techniques in Reverse Engineering and New
Product Development, Prentice Hall, 2001.

[14] C. Dym, Engineering Design: A Product Based Introduction, Wiley, 2000.

[15] J. Linsey, K. Wood and A. Markman, "Increasing Innovation: Presentation and Evaluation
of the WordTree Design-by-Analogy Method," in ASME Design Theory and Methodology
Conference, New York, 2008.

[16] D. Jensen, P. Knodel, R. Vincent, J. Wood and K. Wood, "Evaluating Ideation using the
Publications Popular Science, Popular Mechanics and Make in Coordination with a New
Patent Search Tool and the 6-3-5 Method," in ASEE Annual Conference, San Antonio,
2010.

[17] M. W. Knauf, Interviewee, Effectiveness of Mini Design. [Interview]. 4 January 2013.

[18] D. Holt-Reynolds, "What do teachers do? Constructivist pedagogies and prospective

teachers’ beliefs about the role of a teacher," Teaching and Teacher Education, vol. 16, pp.
21-32, 2000.

Page 23.1327.12
Appendix A – Design Process Handout

Capstone Mini Design Competition

One of the major difficulties encountered by previous capstone teams was an inadequate
understanding of the design process. To help enhance your understanding of the design process
and get you working in the lab we are going to participate in a miniature design build
competition. The task is to build a classroom demo for EM 220. The instructions and
requirements for the demo are purposefully vague to allow you freedom when creating your
masterpiece. The only firm requirements are for a design review where you will present your
concept and a final project demonstration. The design review and the final project demonstration
will occur on lesson five. Hopefully this will allow you to become acquainted with the design
process so you will understand how the pieces fit together and see the big picture better as we
develop the design tools in more detail throughout the semester. To motivate top performance in
this task you will be divided into two person teams and the winners will receive the pride of
knowing they are better than everyone else on the team (and a gift certificate for dinner).

Below is a VERY brief review of the design process

Customer Needs Development

 Identify the customer
 Gather customer needs
 Group customer needs
 Prioritize customer needs
Page 23.1327.13
Functional Analysis:

Overview of Specs:
Engineering Specifications (specs) are not the same as CN, but are related to them. The specs
are quantifiable representations of what the product must do. Sometimes these are called
“engineering requirements” in the design literature, so we’ll use these 2 terms interchangeably.
These should be based on Customer Needs and they must be measurable (i.e. have units).

Concept Generation:
Brainstorming and Mind Maps are just two of the methods we will use in this class. We will
spend a lot of time on this step because this is an area where cadets can make a significant
contribution to solving real world problems.

Concept Variants:
Morph Matrices can be useful tools to assemble concept Variants

Page 23.1327.14
Decision Matrix:

Analysis:
Perform modeling and simulation to analyze whether or not your ideas will work and how well
they will work.
-----------------------------The Design Review will cover the items listed above---------------------

BUILD and TEST

This is the most time consuming portion of the design process. Building prototypes and testing
them against the specifications allows you to evaluate whether your design will meet the
customer needs or not.

Page 23.1327.15
Appendix B – Design Process Quiz

MULTIPLE CHOICE

1. Is design an iterative Process? Yes / No

2. For a generic hair dryer what is the best example of a customer need?
A. Dries Hair Quickly
B. 50 Watt heating element
C. Pink
D. Weighs less than 1 pound
3. Morph matrices can be used to:
A. Pick the optimal design
B. Develop multiple concept variants
C. Set up a test program
D. Perform stress analysis
4. When selecting which design to manufacture it is important to rate potential designs against:
A. Needs/Specifications
B. Functions
C. Design Intent
D. Cost
5. Brainstorming should be limited to practical ideas in the early phases of designs. T/F

SHORT ANSWER – (no more than one sentence)

6. Who would be the target customer for a hair dryer?

7. What are 3 important qualities of a specification?

8. What is frequently the most time consuming portion of the design process?

9. What is design space and how is it used?

10. How comfortable do you feel with the lab equipment (1=not confident 10=highly confident)?
11. My current motivation level for Capstone is (1=low 10=high)
Page 23.1327.16