AAI research group

Applied Artificial Intelligence (AAI) research group

Research theme
Our research concerns artificial intelligence (AI), particularly conversational AI and natural language processing (NLP), mobile robots (including humanoid robots), automated transportation (road, sea, and rail), and computational finance. 

Interpretable AI: Background
A central theme in our work is interpretable artificial intelligence (IAI), i.e. models and systems whose components (and ideally the entire system) are human-interpretable, which is a necessary prerequisite for safe and accountable AI, especially in applications involving high-stakes decisions (for example, in medical applications).

Currently, much work in AI is focused on black box models, especially deep neural networks (DNNs). Such models have been very successful in many different applications, for example image processing and speech processing. However, black box models, including DNNs, also have several drawbacks. For example: (1) Their decision-making is opaque, due to the non-linear nature of their computation and their sheer size; (2) While well-trained DNNs can give excellent results on average, they are also prone to occasional catastrophic (and unpredictable) failures; and (3) DNNs are typically trained on massive amounts of data, meaning that it is difficult or impossible to curate the data, so as to remove unwanted biases in the data. Therefore, during training, DNNs may pick up such biases.

In order to get to terms with the opaque decision-making in black box models, many researchers have considered what is known as Explainable AI, in which one typically builds a secondary model, which is simpler and ideally human-interpretable, and which approximates the black box model. However, it is not always clear to what degree the secondary model is a faithful representation of the (much more complex) original black box model. Moreover, the secondary model, being much simpler than the model it is supposed to explain, may not be able to provide a useful explanation of what the original black box actually does (or how it does it).

By contrast, in interpretable AI, we strive to generate models and system that are human-interpretable by construction, i.e. that consist of human-interpretable primitives. An interpretable model need not be small or simple, but it should consist of components that can be individually understood by a human, and also a allow a human observer to follow the model's reasoning, step-by-step, through a chain of actions taken by such human-interpretable components. An example is our dialogue manager DAISY that not only consists of human-interpretable primitives, but also (by design) is able to generate a human-interpretable explanation of its actions. It should be noted that (unfortunately) many researchers tend to use the (wholly different) terms interpretability and explainability interchangeably, something that creates quite a bit of confusion.

Why is interpretability important? In many cases, it is not. For example, when processing speech, as long as the system is able to accurately determine what a person says, it might not matter very much how the system does so. However, in other applications, especially those that involve high-stakes decision-making, interpretability is key. Such applications occur, for instance, in medicine and healthcare, automated driving, credit scoring, and so on. Despite the importance of interpretability, much current research is entirely focused on black box models, perhaps partly because of a (mythical) belief that only such models can reach top performance. That is not true, but even if it were true, the performance of a model must generally be weighed against other aspects, such as its safety and accountability, two aspects that naturally occur in interpretable models, but are completely alien to DNN-based models.

Now, if interpretability is a key aspect, why is so much research focused exclusively on DNN-based models? There can be many reasons, one being the ease with which one can set up DNN-based systems (e.g. classifiers) due to the availability of many ready-made code libraries for DNNs, primarily in Python. This, in turn, has led to a situation in which alternative models are sometimes not even considered. Even their existence is not always known to end users. Moreover, since the DNN-based models somehow represent the state-of-the-art in (the use of)  computer science, they tend perhaps to get more media coverage than other types of models. Also, for industrial applications, there might be strong financial incentives for using a completely opaque, giant black box, rather than a (non-patentable) much simpler system. It is safe to say that the importance of interpretability has not yet been emphasized sufficiently, even though one of the key aspect of AI-based systems (as measured in interviews with potential end users) is indeed accountability. Finally, interpretability (and related aspects such as accountability, safety, fairness, and so on) is also a central concept in proposed legislation related to AI, both in the EU ("The right to an explanation") and in the US ("The algorithmic accountability act).

Ongoing research projects (Note: These are examples - the page is under construction)

The aim of this project is to develop a fully interpretable, general-purpose dialogue manager for conversational AI.

Wahde, M. and Virgolin, M. "DAISY: An implementation of five core principles for transparent and accountable conversational AI", International Journal of Human-Computer Interaction, pp. 1-18, 2022, https://doi.org/10.1080/10447318.2022.2081762
Wahde, M. and Virgolin, M. "The five Is: Key principles for interpretable and safe conversational AI", in Proc. of the 4th international conference on Computational Intelligence and Intelligent Systems (CIIS2021), pp. 50-54, 2021, https://doi.org/10.1145/3507623.3507632

In this project, we are developing a method for automated trajectory planning for a fleet of autonomous vehicles.

Wahde, M., Bellone, M., and Torabi, S. "A method for real-time dynamic fleet mission planning for autonomous mining", Autonomous Agents and Multi-agent Systems, vol. 33, pp. 564-590, 2019,

Stochastic optimization methods (FFR105, FIM711), 1st quarter (Aug. - Oct.)

Intelligent Agents (TME286), 3rd quarter (Jan. - March)  (The next course starts in January 2020)

Autonomous Robots (TME290), 4th quarter (March - May)

Introduction to Artificial Intelligence, 4th quarter (March-May)

Humanoid robotics (TIF160, FIM800), 1st quarter (Aug. - Oct.)

Current group members
Mattias Wahde
, PhD, Professor, Group leader

Krister Wolff
, PhD, Docent, Vice Head of department

Peter Forsberg
, Adjunct Associate Professor

Ola Benderius
PhD, Docent, Associate Professor
Marco Della Vedova, PhD, Associate Professor
Björnborg Nguyen, PhD student,
Krister Blanch, PhD student

Contact person: Prof. Mattias Wahde, mattias.wahde@chalmers.se