Enable the Conditional Access Insights and Reporting Workbook

Conditional Access is a vital component of any Azure AD / Microsoft 365 tenancy. It is an extremely flexible and effective tool to help shape and enforce authentication criteria such as MFA and device compliance. While creating Conditional Access Policies can be very straightforward, as they grow, they can become quite complex and it can be difficult to predict how some new policies may affect existing ones. Given how easy it is to configure a policy, it’s also very easy to misconfigure a policy and have a negative impact on user authentication.

To help admins gain insight into the effect of changes to policies, Conditional Access policies can be configured in “Report Only” mode. This allows each authentication to be assessed against a particular policy, with the results logged but without actually taking the specified actions. This is a really cool way to test the impact of changes before moving them into production.

One issue with Report Only policies though, is that assessing the logs can be cumbersome and require either a lot of manual work or some scripting to pull the data out into an easily readable format. In addition to this, the retention of logs in Azure AD is quite short, even under Premium licensing (check out Table 1 below from the Microsoft Documentation).

ReportAzure AD FreeAzure AD Premium P1Azure AD Premium P2
Audit logs7 days30 days30 days
s7 days30 days30 days
Azure AD MFA usage30 days30 days30 days
Table 1: Log Retention in Azure AD under different license tiers

Conditional Access Insights and Reporting Workbook

The Conditional Access Insights and Reporting Workbook is based on an Azure Log Analytics Workspace and allows for both the retention of logs past the Azure AD defaults and provides a nice dashboard to make sign-in logs a lot more user friendly and informative at a glance.

Azure AD Logs are shipped into a Log Analytics Workspace and from there, pre-configured queries help to display the data in a nice, easy to read dashboard (figure 1).

Conditional Access Insights and Reporting dashboard in the Azure portal
Figure 1: Conditional Access Insights and Reporting Workbook

In this post, I will detail the following steps required to configure the Workbook:

  • Set up a new Log Analytics Workspace
  • Configure Azure AD to send Logs to the new workspace
  • View the Insights and Reporting Workbook

Set up a new Log Analytics Workspace

To create a Log Analytics Workspace, open the Azure Portal and search for “Log Analytics Workspaces”. Click “Create” and give the Workspace a name, Resource Group, Subscription and Location (Figure 2), optionally select the pricing tier and then click “Review + Create” to finish creating the Workspace.

Figure 2: Create a new Log Analytics Workspace

Configure Azure AD to Send Logs to the New Workspace

With the Workspace configured, next open the “Diagnostic Settings” page from the Azure Active Directory Admin Portal. Select the option to “Add diagnostic setting” and configure “SignInLogs” and any other logs you may want to retain to send to the new Log Analytics Workspace that you configured (Figure 3) and click “Save”.

Figure 3: Send Azure AD Sign In Logs to the Log Analytics Workspace

This will send all new sign in logs to the Workspace you created.

View the Insights and Reporting Workbook

To view the Workbook, open the “Security” page from the Azure AD Portal, then select “Conditional Access” -> “Insights & Reporting”. The Workbook shows a detailed breakdown of all sign-ins in the environment and can be filtered by Conditional Access Policy, Time range, User, or Application. To view the results of a specific policy, for instance a report only policy, simply change the dropdown in the report to select the policy and the data will be filtered everywhere in the workbook as can be seen in Figure 4 below.

Figure 4: Filtered breakdown of sign-in activity per policy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s