Operational Overview

The underlying mechanisms behind the operation of logs and metrics in Azure Monitor are fundamentally different. The information that goes into the log is at the whim of the resource. It's an all-or-nothing proposition when you ingest the log. Metrics, on the other hand, can be chosen at will by the consumer. So they'll be treated separately here.

Logs

The pipeline of a single log entry is simple: the resource generates the log entry, the log entry is dropped into an event hub (assuming you switched that on), the log entry sits time-ordered in the event hub for some retention period, a client (such as this add-on) retrieves logs by periodically reading new events in the event hub. The client may at any time go back in time to read older events by specifying a starting point in the hub. There may be multiple log categories for a resource and these align to multiple event hubs, each of which is subscribed to independently by a client.

The add-on is configured with the Event Hub Namespace, said namespace contains multiple hubs, each of which contains logs of a log category for a resource type. So, for example, all Microsoft.Network/networkSecurityGroup resources drop NetworkSecurityGroupEvent messages into a single event hub in the namespace. I am oversimplifying for clarity at this point.

Saving the details for later, the following must be in place to see logs appear in Splunk:

• Configure each resource with event hub namespace
• Configure the add-on to read logs from the event hub namespace

Not all resources log via Azure Monitor. Here's a list of those that do: Diagnostic Logs

The Activity Log is a subscription-wide log of operations on all resources in the subscription, such as create/update/delete. It is configured separately in the add-on but works the same way as Diagnostic Logs (i.e. there is an event hub in a namespace, etc).

In addition to CRUD actions, the Activity Log also contains Azure Security Center Recommendations and Alerts, Service Health, Security, and Autoscaling events. See here for details.

Configuration

Metrics

The pipeline for a metric is very similar: the metric is generated by the resource, it is stored in an Azure Storage Account for some retention period, a client may read the metric at any time during that retention period using a REST API. Here the similarities end, however. The REST API gives more flexibility in what you read and how much you read. You can specify which metrics you want and over what time period.

The add-on is configured with the Azure Subscription ID. It periodically enumerates the resources in the subscription, determines which ones are being monitored, collects the required metrics from each and then passes them along to Splunk.

Saving the details for later, the following must be in place to see metrics appear in Splunk:

• Configure each resource with Diagnostic Settings (saying "yes, collect metrics") and Tags (saying, "pass these to Splunk")
• Configure the add-on with the subscription id of the resources to be monitored.

Not all Azure resources emit metrics via Azure Monitor. Here's a list: Metrics

Configuration