Sight & Foresight: Helping First Responders Visualize the Future in AR with Spatial Data

When disaster strikes, the people trained to respond need information to make critical decisions. From incident commanders who need a high-level understanding of the situation to first responders on the ground who need to know when and where to act, the pressure to process and understand all the available data is intense.  

Thankfully, we have the technology to get responders the information they need—when they need it. The latest sensor technology and advanced visualization platforms help them sift through vast amounts of noise to find the most important signals. Then, with a clearer vision, they can drive actions where they’re needed most to keep emergency personnel safe, respond to disasters more efficiently, protect property, and save lives.  

This isn’t a future state either. The innovations necessary to optimize emergency response exist but have traditionally been fragmented and hard to deploy. That’s why when U.Group learned of a tech challenge calling on innovators to develop user-friendly ways to process and package this technology we leapt into action. 

About the CHARIoT Challenge  

The Public Safety Communications Research (PSCR) division of the National Institute of Standards and Technology is a federal laboratory that researches, develops, and evaluates technology for public safety communications. One of the ways they do this is through innovation competitions like the CHARIoT Challenge. This challenge explored the potential for combining augmented reality (AR) interfaces with internet of things (Iot) sensors that stream smart city data to simulate disaster response scenarios.  

U.Group received phase one and phase two awards to develop AR concepts, and were matched with the Florida Hazardous Materials Symposium to collaborate with professionals and learn from their real-world experience and perspective.  

Our Solution to the CHARIoT Challenge: DaaS + AR/VR  

The core of our CHARIoT Challenge solution made heavy use of one of U.Group’s proprietary solutions, the U Data Platform (UDP). UDP is a DaaS product that supplies business intelligence, supply chain information, social media, financial, corporate location, and other data through an array of interfaces. Customers can use it to evaluate supply chain risks, determine links between entities, evaluate corporate intellectual property postures, and map companies in the real world. Users can leverage its spatial and location data to map publicly-available information on company locations, branches, offices, and facilities to the real world. We used this feature to enrich the dataset for the CHARIoT Challenge, allowing us to superimpose new layers of information on top of our AR/VR solution—providing sight (real time visuals) and foresight (future decision-support) to first responders. 

Sight: The A/R Visualization Aspect 

The CHARIoT Challenge focused on four disaster scenarios in the fictional town of Misfortuneville, USA (which bore a striking resemblance to Richmond, VA—home to one of the members of our data team). Knowing we needed to gain a comprehensive understanding of all the information available before working towards our solution, we took a three step approach to problem-solving—starting with tapping into the expertise of experienced emergency response professionals.  

Step 1: Expert Involvement  

We took a deep dive into the use cases for each scenario, and then sought out advice from our public sector partner. Their experience working with an urban search and rescue team in Florida that had deployed for the Hurricane Harvey response in Texas provided invaluable insights. Talking to expert first responders about how they respond to disasters helped prepare us for our next step: exploring the data. 

Step 2: Data Exploration 

With our first responder’s expertise, we then identified and organized the universe of data relevant to the disaster scenario, a flood emergency. We organized and mapped the information to real-world use cases which enabled us to identify an opportunity to develop a predictive model that could help first responders in the field visualize critical data in the context of the physical world and the areas of most critical risk and need. 

Mapping user needs to IoT sensors into themes of traffic, environment, and life signs 

Step 3: AR Interface Design  

With a solid understanding of the human needs and available data, we were able to design a preliminary AR tabletop visualization of the disaster response scenario. We designed the user interface to prioritize and support features incident commanders needed most: 

  • Geographic: a holographic map of the area that would show locations of personnel, buildings, IoT sensor locations, and critical alerts. 
  • IoT Data Filters: the ability to quickly activate or deactivate data layers to accelerate access to specific information needed in any given situation. 
  • Temporal: this is where things got really interesting. We designed a timeline that would enable command to see how data and conditions have evolved over time in the past and present. This led to our asking the question: 
    “What if we could predict with a high degree of confidence what conditions would look like in the future, and where is action most critically needed?”
Our first prototype of an augmented reality interface for emergency flood response

If we could help incident commanders and first responders both quickly find the data they need most when they need it, and identify where action is most critically needed, we could potentially help enable faster decisions on key use cases like: 

  • Recommendations on road closures 
  • Priority notification of buildings and businesses  
  • Staging of personnel and equipment for future response 
  • Awareness of environmental hazards before they become an immediate danger 

Foresight: Critical Decisions Driven by DaaS 

The ability to enable decision support (i.e., foresight) is where DasS comes in. Thanks to UDP, the base data, mapping data, and our user interface, we had a universe of linked, self-consistent data that was useful in its own right. We were able to extend the usefulness of this data by leveraging it with our data science expertise in two ways:  

  1. To create predictions of where the disaster was going to be 
  1. To identify key facilities it might affect 

To predict future states, we examined historical flood data— combined with specific measurements from the challenge’s flood—to identify areas at particular risk of higher water levels. For this, we used techniques like Gaussian process regression to create a geospatial forecasting model that probabilistically interpolated our data, allowing us to create a holistic predictive model for the complete problem area. Similar approaches could work for fires or other spatially-based disasters. 

This approach became truly valuable when we combined it with UDP data, because we were able to map this model onto existing company locations. This allowed us to identify (hypothetically) schools, hospitals, critical infrastructure, and at-risk companies, along with their contact information, to provide first responders and other coordinators with the tools they need to identify and contact those most at risk. This kind of foresight would save lives and reduce individual burden. 

Our Next Digital Transformation Challenge  

The approach we modeled here was quite simple: we took spatially-linked disaster data, layered our DaaS services onto it using geographic locations as the point of reference (including corporate and individual contact information), and enriched it with predictive data science.  

We can apply this paradigm to a wide range of disasters—floods, fires, storms, earthquakes, and more—to provide solutions that allow first responders to prioritize, extend, and expand their reach in urgent, life-threatening situations. 

The basic principles are the same. The goals are the same. The promise is the same: to leverage cross-discipline expertise and tools to bring digital transformation solutions to human problems.

It’s an exciting journey, and we’re excited to continue it. 

DaaS and AR solutions extend beyond disasters. Get in touch to learn how they can be used to solve your challenges.

Get alerted to new job postings, events, and insights by registering for our monthly newsletter.