CS5032 L11 validation and reliability testing 2013
Download as PPTX, PDF1 like1,294 views
Critical systems require additional validation processes beyond non-critical systems due to the high costs of failure. Validation costs for critical systems are significantly higher, usually taking over 50% of development costs. Various static analysis techniques can be used for validation, including formal verification, model checking, and automated program analysis. Statistical testing with an accurate operational profile is also used to measure a critical system's reliability.
CS5032 L11 validation and reliability testing 2013
- 1. Critical Systems Assurance Lecture 1 Dependability and Security Assurance 1, 2013 Slide 1
- 2. Validation of critical systems • The verification and validation costs for critical systems involves additional validation processes and analysis than for non-critical systems: – The costs and consequences of failure are high so it is cheaper to find and remove faults than to pay for system failure; – You may have to make a formal case to customers or to a regulator that the system meets its dependability requirements. This dependability case may require specific V & V activities to be carried out. Dependability and Security Assurance 1, 2013 Slide 2
- 3. Validation costs • Because of the additional activities involved, the validation costs for critical systems are usually significantly higher than for non-critical systems. • Normally, V & V costs take up more than 50% of the total system development costs. • The outcome of the validation process is a tangible body of evidence that demonstrates the level of dependability of a software system. Dependability and Security Assurance 1, 2013 Slide 3
- 4. Static analysis • Static analysis techniques are system verification techniques that don’t involve executing a program. • The work on a source representation of the software – either a model or the program code itself. • Inspections and reviews are a form of static analysis • Techniques covered here: – Formal verification – Model checking – Automated program analysis Dependability and Security Assurance 1, 2013 Slide 4
- 5. Verification and formal methods • Formal methods can be used when a mathematical specification of the system is produced. • The specification is based on well-understood mathematical theories such as formal logic and set theory • Used in the development of some critical systems Dependability and Security Assurance 1, 2013 Slide 5
- 6. Formal methods • Formal methods are the ultimate static verification technique that may be used at different stages in the development process: – A formal specification may be developed and mathematically analyzed for consistency. This helps discover specification errors and omissions. – Formal arguments that a program conforms to its mathematical specification may be developed. This is effective in discovering programming and design errors. Dependability and Security Assurance 1, 2013 Slide 6
- 7. Arguments for formal methods • Producing a mathematical specification requires a detailed analysis of the requirements and this is likely to uncover errors. • Concurrent systems can be analysed to discover race conditions that might lead to deadlock. Testing for such problems is very difficult. • They can detect implementation errors before testing when the program is analyzed alongside the specification. Dependability and Security Assurance 1, 2013 Slide 7
- 8. Arguments against formal methods • They require the use of specialised notations that cannot be understood by domain experts. • Very expensive to develop a specification and even more expensive to show that a program meets that specification. • Proofs may contain errors. • It may be possible to reach the same level of confidence in a program more cheaply using other V & V techniques. Dependability and Security Assurance 1, 2013 Slide 8
- 9. Model checking Create an extended finite state model of a system and, using a specialized system (a model checker), check that model for errors. Dependability and Security Assurance 1, 2013 Slide 9
- 10. Model checking • The model checker explores all possible paths through the model and checks that a user-specified property is valid for each path. • Model checking is particularly valuable for verifying concurrent systems, which are hard to test. • Although model checking is computationally very expensive, it is now practical to use it in the verification of small to medium sized critical systems. Dependability and Security Assurance 1, 2013 Slide 10
- 11. Model checking process Dependability and Security Assurance 1, 2013 Slide 11
- 12. Automated static analysis • Static analysers are software tools for source text processing. • They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team. • They are very effective as an aid to program inspections – but inspections can catch errors of understanding that tools can’t. Dependability and Security Assurance 1, 2013 Slide 12
- 13. Levels of static analysis • Characteristic error checking – The static analyzer can check for patterns in the code that are characteristic of errors made by programmers using a particular language. • User-defined error checking – Users of a programming language define error patterns, thus extending the types of error that can be detected. This allows specific rules that apply to a program to be checked. • Assertion checking – Developers include formal assertions in their program and relationships that must hold. The static analyzer symbolically executes the code and highlights potential problems. Dependability and Security Assurance 1, 2013 Slide 13
- 14. Automated static analysis checks Fault class Static analysis check Data faults Variables used before initialization Variables declared but never used Variables assigned twice but never used between assignments Possible array bound violations Undeclared variables Control faults Unreachable code Unconditional branches into loops Input/output faults Variables output twice with no intervening assignment Interface faults Parameter-type mismatches Parameter number mismatches Non-usage of the results of functions Uncalled functions and procedures Storage management faults Unassigned pointers Pointer arithmetic Memory leaks Dependability and Security Assurance 1, 2013 Slide 14
- 15. Use of static analysis • Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler. • Particularly valuable for security checking – the static analyzer can discover areas of vulnerability such as buffer overflows or unchecked inputs. • Static analysis is now routinely used in the development of many safety and security critical systems. Dependability and Security Assurance 1, 2013 Slide 15
- 16. Reliability testing • Reliability validation involves exercising the program to assess whether or not it has reached the required level of reliability. • This cannot normally be included as part of a normal defect testing process because data for defect testing is (usually) atypical of actual usage data. • Reliability measurement therefore requires a specially designed data set that replicates the pattern of inputs to be processed by the system. Dependability and Security Assurance 1, 2013 Slide 16
- 17. Statistical testing • Testing software for reliability rather than fault detection. • Measuring the number of errors allows the reliability of the software to be predicted. Note that, for statistical reasons, more errors than are allowed for in the reliability specification must be induced. • An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached. Dependability and Security Assurance 1, 2013 Slide 17
- 18. Reliability validation activities • Establish the operational profile for the system. • Construct test data reflecting the operational profile. • Test the system and observe the number of failures and the times of these failures. • Compute the reliability after a statistically significant number of failures have been observed. Dependability and Security Assurance 1, 2013 Slide 18
- 19. Reliability measurement Dependability and Security Assurance 1, 2013 Slide 19
- 20. Operational profiles • An operational profile is a set of test data whose frequency matches the actual frequency of these inputs from ‘normal’ usage of the system. A close match with actual usage is necessary otherwise the measured reliability will not be reflected in the actual usage of the system. • It can be generated from real data collected from an existing system or (more often) depends on assumptions made about the pattern of usage of a system. Dependability and Security Assurance 1, 2013 Slide 20
- 21. An operational profile Dependability and Security Assurance 1, 2013 Slide 21
- 22. Operational profile generation • Should be generated automatically whenever possible. • Automatic profile generation is difficult for interactive systems. • May be straightforward for ‘normal’ inputs but it is difficult to predict ‘unlikely’ inputs and to create test data for them. • Pattern of usage of new systems is unknown. • Operational profiles are not static but change as users learn about a new system and change the way that they use it. Dependability and Security Assurance 1, 2013 Slide 22
- 23. Reliability measurement problems • Operational profile uncertainty – The operational profile may not be an accurate reflection of the real use of the system. • High costs of test data generation – Costs can be very high if the test data for the system cannot be generated automatically. • Statistical uncertainty – You need a statistically significant number of failures to compute the reliability but highly reliable systems will rarely fail. • Recognizing failure – It is not always obvious when a failure has occurred as there may be conflicting interpretations of a specification. Dependability and Security Assurance 1, 2013 Slide 23
- 24. Key points • Static analysis is an approach to V & V that examines the source code (or other representation) of a system, looking for errors and anomalies. It allows all parts of a program to be checked, not just those parts that are exercised by system tests. • Model checking is a formal approach to static analysis that exhaustively checks all states in a system for potential errors. • Statistical testing is used to estimate software reliability. It relies on testing the system with a test data set that reflects the operational profile of the software. Dependability and Security Assurance 1, 2013 Slide 24