A framework for semantics-based situational awareness during mobile robot deployments

Deployment of robots into hazardous environments typically involves a “human–robot teaming” (HRT) paradigm, in which a human supervisor interacts with a remotely operating robot inside the hazardous zone. Situational awareness (SA) is vital for enabling HRT, to support navigation, planning, and decision-making. In this paper, we explore issues of higher-level “semantic” information and understanding in SA. In semi-autonomous or variable-autonomy paradigms, different types of semantic information may be important, in different ways, for both the human operator and an autonomous agent controlling the robot. We propose a generalizable framework for acquiring and combining multiple modalities of semantic-level SA during remote deployments of mobile robots. We demonstrate the framework with an example application of search and rescue (SAR) in disaster-response robotics. We propose a set of “environment semantic indicators” that can reflect a variety of different types of semantic information, such as indicators of risk or signs of human activity (SHA), as the robot encounters different scenes. Based on these indicators, we propose a metric to describe the overall situation of the environment, called “Situational Semantic Richness” (SSR). This metric combines multiple semantic indicators to summarize the overall situation. The SSR indicates whether an information-rich, complex situation has been encountered, which may require advanced reasoning by robots and humans and, hence, the attention of the expert human operator. The framework is tested on a Jackal robot in a mock-up disaster-response environment. Experimental results demonstrate that the proposed semantic indicators are sensitive to changes in different modalities of semantic information in different scenes, and the SSR metric reflects the overall semantic changes in the situations encountered.