Microsoft Fabric Data Architecture

Multi-source intelligence fusion for operational situational awareness

📐 Reference target architecture — demonstrates how this radar demo scales to production on Microsoft Fabric

🏗️ Architecture

Multi-Source Ingestion Architecture

End-to-end data flow from distributed sensor networks to operational intelligence products.

📡 Radar SensorsAERIS-10 network

🌡️ IoT SensorsEnvironmental, perimeter

📷 Camera SystemsCCTV, thermal imaging

🛰️ Satellite ImagerySAR, optical recon

🌐 OSINTOpen source intel feeds

🎯 C4ISR SystemsCommand & control

Azure Event HubManaged Identity · Zero secrets

Fabric EventstreamReal-time ingestion & routing

Eventhouse (KQL Database)Real-time streaming analytics

Lakehouse (Delta Lake)Historical batch analytics

OneLake — Unified Storage LayerSingle copy of truth for all workloads

🗺️ ESRI ArcGISGeospatial operational views

📊 Power BIBI dashboards & reporting

📊 Data Streams

Data Streams Produced

Three core data streams flow from the radar network into Fabric.

📡 Raw Radar Detections

HIGH VOLUME — up to 50 k events/sec

Show Schema ▼

Field	Type	Description
detection_id	string	Unique detection UUID
radar_id	string	Source radar identifier
timestamp	datetime	Detection time (UTC)
range_m	float	Range in metres
azimuth_deg	float	Azimuth angle (°)
elevation_deg	float	Elevation angle (°)
doppler_mps	float	Radial velocity (m/s)
snr_db	float	Signal-to-noise ratio (dB)
rcs_dbsm	float	Radar cross-section (dBsm)
latitude	float	WGS-84 latitude
longitude	float	WGS-84 longitude
altitude_m	float	Altitude (metres MSL)

🎯 Fused Tracks

OPERATIONAL — correlated multi-radar tracks

Show Schema ▼

Field	Type	Description
track_id	string	Fused track UUID
timestamp	datetime	Last update time (UTC)
latitude	float	Fused position lat
longitude	float	Fused position lon
altitude_m	float	Fused altitude (MSL)
heading_deg	float	Track heading (°)
speed_mps	float	Ground speed (m/s)
classification	string	UAV / aircraft / bird / unknown
confidence	float	Classification confidence 0–1
contributing_radars	array	Radar IDs contributing
threat_level	string	LOW / MEDIUM / HIGH / CRITICAL

🔧 Radar Telemetry

HEALTH — system status per radar

Show Schema ▼

Field	Type	Description
radar_id	string	Radar identifier
timestamp	datetime	Telemetry time (UTC)
status	string	OPERATIONAL / DEGRADED / OFFLINE
cpu_pct	float	CPU utilisation %
memory_pct	float	Memory utilisation %
temperature_c	float	Enclosure temperature (°C)
avg_snr_db	float	Average SNR last 60 s
detections_per_min	int	Detection rate
uptime_hours	float	Hours since last restart
firmware_version	string	Running firmware

🗄️ Storage

OneLake Namespace Structure

Logical organisation of all intelligence data within the unified OneLake storage layer.

OneLake/
├── radar-intelligence/
│   ├── raw-detections/          Delta table — partitioned by radar_id, date
│   ├── fused-tracks/            Delta table — partitioned by date
│   └── radar-telemetry/         Delta table — radar health metrics
├── sensor-feeds/
│   ├── iot-environmental/       Temperature, wind, humidity
│   ├── camera-feeds/            Motion events, object detection results
│   └── acoustic-sensors/        Audio classification events
├── geospatial-intelligence/
│   ├── satellite-imagery/       SAR and optical image metadata + shortcuts
│   ├── terrain-data/            SRTM/DEM elevation (GeoTIFF shortcuts)
│   └── airspace-zones/          Classification polygons (GeoJSON)
├── osint-feeds/
│   ├── flight-tracking/         ADS-B transponder data
│   ├── weather-data/            METAR, radar weather, forecasts
│   └── threat-intelligence/     Known threat signatures and patterns
├── reference-data/
│   ├── radar-sites/             Radar locations, models, configurations
│   ├── geofences/               No-fly zones, restricted areas
│   └── unit-dispositions/       Friendly force positions
└── operational-views/
    ├── common-operating-picture/ Latest fused picture across all sources
    ├── detection-history/        Time-windowed aggregates
    └── threat-assessments/       ML-enriched threat scoring

🧬 Semantic Layer

Fabric Ontologies Preview

A shared semantic model that defines entities, relationships, and business logic across all consumers of the intelligence data.

What Are Ontologies?

Fabric ontologies provide a semantic layer on top of OneLake data. They create a shared understanding of domain concepts — such as Radar, Track, Threat, and Airspace Zone — so every downstream consumer works from the same definitions and relationships, regardless of the underlying storage format.

AerialTrack

track_id, position
classification, threat_level

detected_by ↗

↙↓↘

violates escalates_to correlated_with

RadarSite

radar_id, location
model, status

AirspaceZone

zone_id, boundary
restriction_type

ThreatObject

threat_id, category
severity, last_seen

OsintReport

report_id, source
content, timestamp

↓ monitored_by

SensorNode

sensor_id, type
location, status

📦 Entity Definitions

RadarSite — physical radar installation with location, model, and status
AerialTrack — fused track with position, velocity, and classification
ThreatObject — confirmed or suspected hostile entity
AirspaceZone — restricted area, no-fly zone, or controlled sector
SensorNode — any contributing sensor (IoT, camera, acoustic)

🔗 Relationships

Track detected_by RadarSite
Track violates AirspaceZone
ThreatObject correlated_with OsintReport
RadarSite covers AirspaceZone
SensorNode co-located_with RadarSite

⚙️ Business Rules & Lineage

Auto-classify threat level based on speed, altitude, and proximity to restricted zones
Shared vocabulary — ESRI, Power BI, APIs, and C2 systems all use the same entity model
Full lineage: raw sensor → fused track → threat assessment
Consistent definitions eliminate ambiguity across reporting layers

⚡ Processing

Streaming vs. Batch Processing

Two complementary processing paradigms working together to deliver both tactical and strategic intelligence.

⚡ Streaming — Eventhouse / KQL

Capability	Example
Real-time detection alerts	Instant notification when new UAV enters radar coverage
Live track correlation	Match detections from multiple radars within 500 ms window
Anomaly detection	Flag unusual flight patterns — loitering, speed changes, altitude drops
Geofence violation alerts	Trigger when track enters restricted airspace zone
Radar health monitoring	Alert on SNR degradation or missed detections per scan
Sliding window aggregations	Detections-per-minute per radar for operational tempo

🗃️ Batch — Lakehouse / Spark

Capability	Example
Historical pattern analysis	Identify recurring flight routes over weeks/months
ML model training	Train classification models for UAV type identification
Coverage gap analysis	Compute detection probability maps from historical data
Sensor performance trending	Long-term SNR and detection rate analysis per radar
Forensic reconstruction	Replay complete tracks for post-event investigation
Cross-source correlation	Correlate radar tracks with satellite imagery and OSINT

Strategic Value — Continuous Intelligence Loop

⚡ Tactical Edge

Streaming provides real-time awareness and immediate response capability. Every detection is processed in sub-second latency.

🧠 Strategic Advantage

Batch provides pattern recognition, predictive intelligence, and operational optimisation from deep historical analysis.

🔄 Intelligence Loop

Real-time events feed historical models; historical insights improve real-time decision-making. Fabric unifies both in one platform.

🗺️ Geospatial

ESRI ArcGIS — Geospatial Analytics & Operations

Spatial intelligence capabilities embedded directly into the Fabric platform and Power BI reporting layer.

🌐 ArcGIS GeoAnalytics for Microsoft Fabric

Built directly into the Fabric Spark Runtime — no separate installation needed. Adds 160+ spatial SQL functions, track analysis, hot-spot detection, and automatic spatial indexing to any Fabric Spark notebook or pipeline.

Capability	Radar Use Case	API
160+ Spatial SQL Functions	Convert radar lat/lon to geometries, compute distances, spatial predicates	`ST_Point`, `ST_Distance`, `ST_DWithin`
Track & Movement Analysis	Analyse UAV flight patterns, detect loitering, calculate motion statistics	`CalculateMotionStatistics`, `DetectIncidents`, `FindDwellLocations`
Hot Spot Analysis	Identify detection density clusters, find radar coverage cold spots	`FindHotSpots`, `FindPointClusters`
Proximity Analysis	Find nearest radar to each track, compute coverage overlap	`NearestNeighbors`, `GroupByProximity`
Spatial Joins	Enrich tracks with airspace zone info, terrain data, no-fly zone violations	Spatial join predicates
Aggregation	Aggregate detections per grid cell for Power BI heat maps	`AggregatePoints`
Automatic Spatial Indexing	Optimised spatial joins on millions of detection records	Built-in

# ArcGIS GeoAnalytics in Fabric Spark notebook
from geoanalytics_fabric.sql import functions as ST
from geoanalytics_fabric.tools import FindHotSpots, CalculateMotionStatistics

# Load radar tracks from OneLake
tracks_df = spark.read.format("delta").load("Tables/fused_tracks")

# Create geometries from lat/lon
tracks_geo = tracks_df.withColumn("geometry", ST.ST_Point("lon", "lat"))

# Analyse movement patterns — detect loitering
motion_stats = CalculateMotionStatistics() \
    .setTrackFields("track_id") \
    .setTimestampField("timestamp_utc") \
    .run(tracks_geo)

# Find detection hot spots
hot_spots = FindHotSpots() \
    .setBinSize(500, "Meters") \
    .setNeighborhoodSize(2000, "Meters") \
    .run(tracks_geo)

📖 ArcGIS GeoAnalytics Developer Docs · MS Learn: ArcGIS GeoAnalytics for Fabric · ESRI Marketplace

ℹ️ ArcGIS GeoAnalytics for Fabric requires an ESRI subscription and authentication credentials. Some ArcGIS for Power BI features require an ArcGIS organizational account.

📊 ArcGIS for Power BI

Built-in mapping visual — no installation needed. Available to every Power BI user for rich geospatial visualisation of radar operational data.

Smart mapping templates — radar coverage visualisation, cluster maps, heat maps
Demographic & geographic enrichment — overlay Living Atlas layers for population density, infrastructure
Area-based selections — draw operational sectors and filter tracks interactively
Feature information panes — drill-through from map point to full track detail
Layer joins — combine radar data with Living Atlas content (terrain, boundaries, imagery)
Custom basemaps — satellite, topographic, dark, or custom-styled basemaps for C2 displays

📖 ArcGIS for Power BI Documentation

📊 Analytics

Power BI — Business Intelligence

Operational dashboards and management reporting across all levels of command.

📈 Dashboard Capabilities

Real-time dashboard — track counts, radar status, detection rates (DirectQuery from Eventhouse)
Operational KPIs — mean time to detect, track continuity %, multi-radar coverage %
Sensor performance — SNR trends, detection probability per radar per range bin
Incident reporting — drill-down from track → raw detections → individual radar contributions
Management view — fleet readiness, coverage posture, operational tempo over time

🏗️ Semantic Model & Security

Semantic model — built on OneLake tables, provides consistent definitions for all report consumers
Row-level security — different views for operators, analysts, and commanders based on Entra ID roles
DirectQuery — real-time data from Eventhouse without import lag
Composite models — combine real-time streaming data with historical batch aggregates in one report
Scheduled refresh — batch lakehouse tables refreshed at defined intervals for trending reports

🔒 Security

Security & Confidentiality

Defence-grade security posture across authentication, data protection, access control, governance, and network isolation.

🔑

Authentication & Identity

Microsoft Entra ID (Azure AD) for all user and service authentication
Managed Identity for service-to-service — zero secrets in code or config
Conditional Access policies: MFA, device compliance, location-based restrictions
Certificate-based authentication for radar edge devices

🛡️

Data Protection

OneLake encryption at rest — Microsoft-managed or customer-managed keys via Key Vault
TLS 1.3 encryption in transit for all data flows
Data classification labels (Microsoft Purview) — SECRET, RESTRICTED, OFFICIAL
Sensitivity labels enforced across Power BI, OneLake, and exports

👥

Access Control

Workspace-level RBAC: separate workspaces for raw sensor data vs. fused intelligence
Item-level permissions: control who can see raw detections vs. fused tracks
Row-level security in Power BI: operators see their sector, commanders see all
Column-level security: mask exact radar coordinates from lower clearance levels

📋

Governance & Compliance

Microsoft Purview integration for data cataloging and lineage
Audit logging: every data access logged to Entra audit trail
Data residency: Fabric capacity pinned to specific Azure region (e.g., North Europe for Finland)
Retention policies: automatic archival and deletion per data classification

🌐

Network Security

Private endpoints for Fabric workspace — no public internet exposure
Managed VNet integration for Spark workloads
Network isolation between workspaces handling different classification levels
Azure Firewall and NSG rules controlling ingress from edge sensor networks
ExpressRoute or VPN for secure connectivity from radar sites to Azure backbone

🔗 Integration

Integration Points

How the Fabric intelligence platform connects to the broader operational ecosystem.

🔄 Multi-Source Fusion in OneLake

All sensor data — radar, IoT, cameras, satellite, OSINT — lands in OneLake as Delta tables. Spark notebooks execute cross-source fusion to produce the Common Operating Picture, accessible to every consumer without data duplication.

⚡ Event-Driven Pipelines

Eventstream triggers Spark notebooks automatically when new data arrives. Detection spikes invoke anomaly-analysis notebooks; new satellite imagery triggers correlation workflows. No manual scheduling required.

🌐 REST API Layer for C2 Integration

Fused tracks and threat assessments exposed via REST APIs for external Command & Control systems. Fabric SQL endpoints and Eventhouse KQL endpoints enable direct queries from any HTTP client.

🔑 Managed Identity Everywhere

Zero API keys in the architecture. All service-to-service communication uses Managed Identity — Event Hub, Fabric, OneLake, ESRI, Power BI. Secrets never appear in code, config, or deployment pipelines.