Reflections On a Decade of AI (Part 3)

A recent experience with managing the tenth anniversary version of a technical conference (www.iccve2022.org) triggered a process of reflection on the last decade of artificial intelligence. The first article in the series, Reflections On a Decade Of AI (forbes.com), explored the historical background of AI technology, the current successes of Machine Learning (ML) methods, and the contrast with conventional algorithmic design.

The predecessor article, Reflections On a Decade of AI (Part 2) (forbes.com), explored the historical approach taken by safety focused designs for safety validation with a deep dive for airborne systems. As shown in the figure below, initially the use of electronics hardware (HW Paradigm) injected a powerful new capability for system designers, and accompanying safety methodologies were developed to harness the power of electronics. In the last 30 years, software has become available as a powerful new capability, and again safety methodologies had to evolve. Today, we are the starting point of the AI Paradigm, and leveraging this powerful tool requires progress on the safety methodology front.

What exactly is the AI Paradigm ?

The AI Paradigm allows the use of machine learning to build intelligence from data. In the case of autonomy, this data is often from various sensor domains. With these capabilities, sound can be interpreted as language (NLP), video can lead to object recognition (Vision), and there are many more applications. This AI paradigm is built on top of another abstraction called the AI model. Various terms such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Transformer and more are used to describe these models. Fundamentally, these models are containers which can map complex functional relationships from inputs to outputs.

In the AI Paradigm, data streams are used to “train” the underlying model. The objective is to build an algorithm which reflects some underlying relationship, and use that relationship for data streams beyond the training set.

This is literally data generated software!

For many reasons, this capability is very attractive for system designers, but how does it fit into existing safety critical methodologies?

In these systems, safety consists of verifying a network of components which together form the whole design. At the component level, the techniques for verification are specific to the nature of the component. Electronics HW is just a component with hazards, design constraints, and potential safety consequences which must be handled carefully. Electronics SW is a special form of HW component which is also programmable.

With the above thinking in mind, AI components can be defined as simply SW components which are programmed with data.

The above simple statement is very powerful because it allows AI components to inherit a massive amount of the work done in the last 30 years for integrating SW components into safety critical systems. Now, one must “only” manage the issue of data generated “code” vs conventional human programming. Where does this divergence make a difference for safety ? In the world of validation, the impact is on three core functions: coverage analysis, code reviews, and version control.

Coverage Analysis: In conventional software, the software program has been written by a human being (much like the graphic at the start of article). A standard validation technique is to drive tests which expose every part of the program. The rationale is that if the coverage of the structure of the program is very high, one can gain confidence that the software component has been tested well. In AI/ML, the “software” has no structure, so conventional code coverage is not possible.

Code Reviews: Much like writers benefit from editors, programmers benefit from group code reviews. This technique is so effective that it is integrated into the fundamental design and validation procedures for high performance software teams. In AI/ML, there is no code to review in the conventional sense. It likely still makes sense to review the results of AI training in group reviews, but the task is exceedingly difficult because one cannot ask the programmer their rationale for a design transformation.

Version Control: A deep part of the conventional software process is to carefully control the construction and release of safety critical software. Programmers work together to provide quantum updates to the central code base which is then tested before release to the field. In the AI/ML paradigm, the “code” is generated from data. Does this mean that training data-sets must be captured and accumulated from the beginning of the training process? What if the starting point consists of “warmed up” models from other vendors ? Is this viable over a long period of time ? If not, how does one know the exact function which is being captured and tested.

Overall, intelligent test generation is exceedingly difficult in conventional software because one must exercise all the interesting “edge” conditions to validate the design. With AI/ML, this task becomes all the more difficult because of the above issues. Currently, this is an area of significant academic research with interesting proposal such as:

1. Adversarial AI/ML: In this approach, an AI adversary is created which tries to actively break the safety paradigm of the core system.
2. Safety Collars: In this approach, deterministic conventional software forms a collar which the AI/ML component cannot violate.
3. Voting Systems: Divergent components suggest an answer and a voting paradigm selects the final answer.

While the academic work is interesting, commercial solutions have drifted towards powerful automated directed pseudo-random test generators which build scenarios with the objective of more thoroughly exercising AI components. In the automotive realm, the Association for Standardization of Automation and Measurement Systems (ASAM) has defined a testing framework with standards such as OpenSCENARIO V2.0, and high-flying startups such as Foretellix are actively implementing this standard. While specific to automotive, the general techniques apply equally well in other domains of autonomy where AI components are part of the solution.

In conclusion, over the last decade, an AI/ML validation methodology is starting to emerge in design domains with a deep system specification culture.

However, one of the key powers of AI/ML is the ability to build functions without ever really understanding what is being built and why it works. This is such a powerful tool that system designers find it very attractive to employ, but in this circumstance, the obvious question is: if you don’t know how it works, how do you know it is working? This will be the topic of the next article.

Notes:

1. An article in the Journal of Air Traffic Control (ATCA) Journal (atca.org) (Sep, 2022 issue) provides a more detailed view on the methods to move from conventional software to AI/ML components for system validation.
2. AI/ML is a remarkably simple methodology is quite profound. In fact, many scientists have analogized it to the process of evolution for living beings. A very good discussion of the connection of AI with evolution can be found in Leslie Valient’s book “Probably Approximately Correct.”