CMU SEI Research Review

Dr. Heim will preview new research focused on a gap in the practice of employing Artificial Intelligence. Modern AI systems most commonly employ Machine Learning (ML) models to make important, domain-relevant inferences. However, ML, almost by its very nature, is called upon to make inferences in the presence of inherent uncertainty. Due in part to uncertainty, state-of-the-art ML models can produce inaccurate inferences in scenarios where humans would reasonably expect high accuracy. Further exacerbating this issue is that many commonly used models do not provide accurate estimates about when they are uncertain about their predictions. As a result, not only can ML inferences be incorrect, but it can be extremely difficult to predict when they will be incorrect. AI system components downstream from an ML model or humans using the model's output to complete a task must then reason with incorrect inferences that they expect to be correct.