Greg Wilson - We Know (but ignore) More Than We Think
0 likes443 views
This document contains the slides from a presentation on best practices for scientific computing. It discusses several key findings from studies on software engineering practices. Some of the main points summarized are: - Early studies found that rigorous code inspections can remove 60-90% of errors before testing begins. Later work refined this by finding the first review and hour matter most. - A classic 1975 study found that most errors are introduced during requirements and design, and errors become more expensive to fix the later they are found. - More recent studies found that an individual's distance in an organization's structure is a better predictor of software quality than their geographic distance.
![2013-03-22
2
Best Practices for Scientific Computing 5
1950: Hill & Doll publish a
case-control study comparing
smokers with non-smokers
Now called the āBritish Doctorsā
study, it ran until 2001
It Took a While...
Best Practices for Scientific Computing 6
#1: Smoking causes
lung cancer
#2: Most people would
rather fail than change
āWhat happens āon averageā is
of no help when one is faced
with a specific patientā¦ā
What They Found
The Cochrane Collaboration (http://www.cochrane.org/)
now archives results from hundreds of medical studies
Best Practices for Scientific Computing 7
ā[Using domain-specific languages] leads to two primary
benefits. The first, and simplest, is improved programmer
productivity... The second...is...communication with
domain experts.ā
ā Martin Fowler,
IEEE Software,
July/August 2009
So Where Are We?
Best Practices for Scientific Computing 8
One of the smartest guys in the industry...
...made two substantive claims of factā¦
Look Closer
...in a peer-reviewed journal...
...without a single citationā¦
ā¦because nobody expected one](https://image.slidesharecdn.com/gregwilson-30yrsofsoftware-130825213548-phpapp01/85/Greg-Wilson-We-Know-but-ignore-More-Than-We-Think-2-320.jpg)
Greg Wilson - We Know (but ignore) More Than We Think
- 1. 2013-03-22 1 What We Actually Know About Programming And What We Ought To Do Next Greg Wilson http://software-carpentry.org March 2013 Best Practices for Scientific Computing 2 You are free to: Copy, share, adapt, or re-mix; Photograph, film, or broadcast; Blog, live-blog, or post video of; This presentation. Provided that: You attribute the work to its author and respect the rights and licenses associated with its components. Best Practices for Scientific Computing 3 Arrr, Matey Seven Yearsā War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy Best Practices for Scientific Computing 4 James Lind (1716-94) 1747: (possibly) the first-ever controlled medical experiment ĆāÆcider ĆāÆsulfuric acid ĆāÆvinegar ĆāÆsea water āāÆoranges ĆāÆbarley water Of course, no-one paid attention until a proper Englishman repeated the experiment in 1794... Oh, the Irony
- 2. 2013-03-22 2 Best Practices for Scientific Computing 5 1950: Hill & Doll publish a case-control study comparing smokers with non-smokers Now called the āBritish Doctorsā study, it ran until 2001 It Took a While... Best Practices for Scientific Computing 6 #1: Smoking causes lung cancer #2: Most people would rather fail than change āWhat happens āon averageā is of no help when one is faced with a specific patientā¦ā What They Found The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies Best Practices for Scientific Computing 7 ā[Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.ā ā Martin Fowler, IEEE Software, July/August 2009 So Where Are We? Best Practices for Scientific Computing 8 One of the smartest guys in the industry... ...made two substantive claims of factā¦ Look Closer ...in a peer-reviewed journal... ...without a single citationā¦ ā¦because nobody expected one
- 3. 2013-03-22 3 Best Practices for Scientific Computing 9 Growing emphasis on empirical studies in software engineering since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Many are flawed or incomplete, but standards are constantly improving A New Hope Best Practices for Scientific Computing 10 Rigorous inspections can remove 60-90% of errors before the first test is run. (Fagan 1975) A Classic Result Best Practices for Scientific Computing 11 Rigorous inspections can remove 60-90% of errors before the first test is run. (Fagan 1975) The first review and hour matter most. (Cohen 2006) A Classic Result Refined Best Practices for Scientific Computing 12 Sackman, Erikson, and Grant (1968): āExploratory experimental studies comparing online and offline programming performance.ā Or 10, or 40, or 100, or whatever other large number pops into the head of someone who canāt be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst. Most Often Misquoted
- 4. 2013-03-22 4 Best Practices for Scientific Computing 13 1.āÆ Study was designed to compare batch vs. interactive, not measure productivity 2.āÆ How was productivity measured, anyway? 3.āÆ Best vs. worst exaggerates any effect 4.āÆ Twelve programmers for an afternoon Pick That Apart Next major study was 54 programmers... ...for up to an hour Best Practices for Scientific Computing 14 Boehm et al (1975): āSome Experience with Automated Aids to the Design of Large-Scale Reliable Software.ā ...and many, many more since 1.āÆ Most errors are introduced during requirements analysis and design 2.āÆ The later they are removed, the most expensive it is to take them out time number/cost Another Classic Result Best Practices for Scientific Computing 15 Pessimists: āIf we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.ā Optimists: āIf we do lots of short iterations, the total cost of fixing bugs will go down.ā That Explains a Lot Best Practices for Scientific Computing 16 Nagappan et al (2007) & Bird et al (2009): Geography has little correlation with software quality Isnāt That Interestingā¦
- 5. 2013-03-22 5 Best Practices for Scientific Computing 17 Nagappan et al (2007) & Bird et al (2009): Distance in the org chart is a much better predictor Isnāt That Interestingā¦ Best Practices for Scientific Computing 18 Are any metrics better at predicting faults/effort than LOC? No. A Few More Results Best Practices for Scientific Computing 19 Do more frequent releases improve software quality? Yes, but it also changes the nature of the bugs. A Few More Results Best Practices for Scientific Computing 20 Are there better ways to teach programming? Yes: media-based instruction and peer instruction. A Few More Results
- 6. 2013-03-22 6 Best Practices for Scientific Computing 21 Sampling Bias I focus on quantitative studies because theyāre what I know best A lot of the best work in this field is using qualitative methods Best Practices for Scientific Computing 22 All Together Now Andy Oram & Greg Wilson (ed): Making Software: What Really Works, and Why We Believe It. O'Reilly, 2010, 978-0596808327. http://neverworkintheory.org Best Practices for Scientific Computing 23 Where to Start? Best Practices for Scientific Computing 24 Where to Start? Many practices can be monitored automatically
- 7. 2013-03-22 7 Best Practices for Scientific Computing 25 Where to Start? But top-down initiatives usually donāt work Best Practices for Scientific Computing 26 1.āÆ How do you identify people with good ideas? Where to Start? Best Practices for Scientific Computing 27 1.āÆ How do you identify people with good ideas? 2.āÆ How do you reward people for good ideas? Where to Start? Best Practices for Scientific Computing 28 1.āÆ How do you identify people with good ideas? 2.āÆ How do you reward people for good ideas? 3.āÆ How do they share those ideas with peers? Where to Start?
- 8. 2013-03-22 8 Best Practices for Scientific Computing 29 1.āÆ How do you identify people with good ideas? 2.āÆ How do you reward people for good ideas? 3.āÆ How do they share those ideas with peers? 4.āÆ How do you tell if it actually worked? Where to Start? Best Practices for Scientific Computing 30 Where to Start? Remember: some changes will be qualitative 1.āÆ How do you identify people with good ideas? 2.āÆ How do you reward people for good ideas? 3.āÆ How do they share those ideas with peers? 4.āÆ How do you tell if it actually worked? Best Practices for Scientific Computing 31 Words to Live By If you build a man a fire, you'll keep him warm for a night. If you set a man on fire, you'll keep him warm for the rest of his life. ā Terry Pratchett Best Practices for Scientific Computing 32 http://software-carpentry.org Thank You