Phase one: Data modernization
With this partnership in place, an Insight architect and senior systems engineer began implementing a Proof of Concept (POC) to establish critical features and feasibility for the project.
They worked closely with the client’s IT team to pull data from the legacy Oracle database and other peripheral systems into a centralized data lake. Azure Data Factory was leveraged for ingestion, with Databricks used to support processing before landing the data in Delta Lake. The data was then written to Azure SQL and a semantic model was developed through SQL Server Analytics Services, allowing sample reports to be served up through Power BI®.
The POC was successfully completed within four weeks, and IT leaders immediately recognized the value of the unified reporting platform and the ability to bring in real-time data to support data science projects.
With the POC approved, Insight’s team began implementing the full data modernization project at scale. They leveraged Insight’s repeatable Extract, Transform, Load (ETL) framework to automate significant portions of the infrastructure deployment, data cleansing and configuration processes. This dramatically accelerated project timelines and ultimately meant the new data architecture was up and running in a matter of weeks — rather than months.
Phase two: A real-time dashboard
While the modern data architecture greatly improved the way the fire department managed its data, leaders had additional goals to leverage their data in support of daily operations. They wanted to add a real-time channel to the reporting infrastructure to serve up critical information through a centralized dashboard, including:
- The most up-to-date locations and status of equipment (fire trucks, ambulances, ladders, etc.)
- Current locations and status of firefighters, EMT personnel and other relevant fire department employees
- A list of active incidents within the city
The primary data sources for the dashboard included the city’s 911 system, as well as GPS trackers from fire department equipment and personnel. This data is reported at regular intervals to track activity and movement over time. The goal was to display this information on a map which would automatically update with locations and incidents. This view would need to be as accurate and up-to-date as possible to facilitate rapid decision-making on resource and equipment deployments.
Once again, Insight’s data and AI team worked closely with the organization’s internal IT department to develop a POC for this phase of the project. The newly centralized data architecture — combined with the Insight’s automated ETL framework — provided a robust foundation for accelerated development and deployment.
Insight’s team leveraged Azure Event Hubs to ingest event data at rates of up to 100+/second. A Databricks analytics platform with Spark structured streaming was then used to stream events from the ingestion platform into Delta Lake. This served as the high-performance, big data ACID store to capture and store processed data, allowing it to be merged with previous records. Finally, Power BI was used for real-time visualization with automatic updates triggered directly by data changes in Delta Lake.
In just a few weeks, the POC was complete and approved for full production. The full-scale solution was quickly delivered, providing a single-pane-of-glass view of city-wide emergency response data.