Author: John Kemnetz
Microsoft has announced an improved and expanded way to consume Azure Diagnostic Logs: streaming via Event Hubs. Azure Diagnostic Logs are logs emitted by a resource and provide insight into the operation of that resource. The content of these logs varies by resource type; for example, Windows event system logs are one category of Diagnostic Log for VMs, and blob, table, and queue logs are categories of Diagnostic Logs for storage accounts. These differ from Audit Logs, which provide insight into the operations that were performed on resources in your subscription. Previously, you could only store these logs for audit in a Storage account. Now, you can easily set up streaming Diagnostic Logs to Event Hubs using the Portal or the Azure PowerShell Cmdlets.
What Can I Do with Diagnostics Logs and Event Hubs?
Here are just a few ways you might use the streaming capability for Diagnostic Logs:
- Stream logs to 3rd party logging and telemetry systems – Over time, Event Hubs streaming will become the mechanism to pipe your Diagnostic Logs into 3rd party SIEMs and log analytics solutions.
- View service health by streaming “hot path” data to PowerBI – Using Event Hubs, Stream Analytics, and PowerBI, you can easily transform your diagnostics data into near real-time insights on your Azure services. This documentation article gives a great overview of how to set up an Event Hubs, process data with Stream Analytics, and use PowerBI as an output.