A number of large, complicated, aerospace programs have apparently come afoul of the verification and validation process in recent years. It is often alleged that these difficulties are the result of bad management and/or the ineffective application of well-known processes. It is becoming increasingly clear that, as usual, reality is much more complicated. In fact, the advent of systems-of-systems embedded in complex socio-technical frameworks, and indeed, the advent of engineered complex systems themselves, requires a new theory of test to match the challenging aspects of these new types of entity. The FAA has advocated such a new theory since at least 2006, has struggled with these challenges, and has experienced many successes in tackling them, although limitations continue to exist. This paper documents the agency's experience with this approach over the past seven years, and reviews several promising methods, including the use of flexible System-of-system assessment platforms (that incorporate high fidelity human in the loop simulation, distributed live virtual and constructive simulation, and fast time simulation modeling,) a technique that allows flexible exercise of the system under test (which must always be full fidelity: only the environment should ever be subject to a greater or lesser degree of simulation) as required to address the challenges posed in each individual situation. In the long term, a different approach to system design is required: verification and validation can never make up for inadequate design (and in the complex system world, we may well only blindly hope that verification and validation will expose the most insidious design flaws.) In the meantime, application of sound traditional methods, use of innovative techniques, a continued pursuit of theoretical understanding, along with integration of verification and validation throughout the life cycle are the best guarantees against undesirable system failures in the field.
展开▼
