Conditional Access Report-Only Mode, Stale Exclusions: Policies That Aren't Really Enforced

Conditional Access Report-Only Mode and Stale Exclusions: What "Enforced" Actually Means

Conditional access report-only mode and stale exclusions are the two most common reasons a Conditional Access policy shows as "On" in the Microsoft Entra admin center while blocking nothing at all. Both conditions are easy to miss during a quick portal review because the policy still exists, still has the right conditions configured, and still shows up in the policy list:

• The policy — or a change made to it — is sitting in report-only mode, which evaluates sign-ins and logs the outcome but never grants, blocks, or challenges access.
• The policy is enforced, but its exclusion list has grown past the narrow, documented reason it was created for, so the sign-ins that matter most slip past it untouched.

Neither of these is a bug. Both are supported, intentional Microsoft Entra features. The gap is operational: nobody re-checked whether the exception was still needed. Entra ID Conditional Access Gaps: What Misconfigurations Leave Real Exposure covers the broader scope and control landscape — missing policies, workload identities, legacy authentication. This article stays narrow: what happens after a policy exists, when its enforcement state or its exclusion list drifts quietly away from what the tenant believes is protected.

Report-Only Mode: The Silent Non-Control

Report-only is a per-policy state in Conditional Access. Microsoft documents it as letting administrators "test most Conditional Access policies before enabling them" — during sign-in, the system evaluates the policy but does not enforce grant or session controls, and users are never prompted for MFA, device compliance, or blocked by a report-only policy.

Why report-only drifts from a test state into a permanent one

The mechanism that makes this dangerous in practice is not the initial rollout — most teams deliberately stage new policies in report-only for a review window. The risk is drift after that: a policy that was enforced can be quietly pulled back into report-only by anyone with Conditional Access Administrator rights. Microsoft's own documentation is explicit that report-only mode never enforces grant or session controls, regardless of a policy's prior state: "during sign-in, the system evaluates policies in report-only mode but doesn't enforce them." That single toggle, changed during a troubleshooting session or an emergency access exception and never reverted, silently removes a control that every dashboard and policy list still shows as configured.

Microsoft's guidance is to validate changes to an existing enforced policy by cloning it into a report-only copy rather than flipping the original policy's toggle — comparing the copy's results against the live policy, then discarding the copy once satisfied. When that pattern isn't followed, the original policy itself ends up as the thing sitting in report-only, sometimes indefinitely.

Stale and Excessive Exclusions: How Coverage Erodes Over Time

Exclusions exist for real operational reasons: a remote office that can't yet meet a location requirement, a legacy device pending replacement, a break-glass account. Microsoft's own guidance on managing excluded users is explicit that this starts small and grows into a problem:

"Frequently, when you first configure an exclusion, there's a shortlist of users who bypass the policy. Over time, more users get added to the exclusion, and the list grows. At some point, you need to review the list and confirm that each of these users is still eligible for exclusion."

Three ways exclusion lists get worse than they look

1. Individual user or legacy on-premises group exclusions provide no ongoing visibility. Users often don't know they're excluded, and if the exclusion uses an on-premises-synced or dynamic group, admins have limited insight into who is in it or why.
2. Self-service group membership can turn an exclusion into a bypass. If the exclusion group allows self-service join (a legitimate pattern for the "traveling employee" scenario), any user who discovers the group can add themselves to it.
3. Eligibility doesn't get re-checked. A user excluded for a now-replaced legacy device, or a contractor exclusion that outlived the contract, stays excluded indefinitely because nothing forces a re-review.

The compensating control: Assigned groups plus recurring Access Reviews

Microsoft's recommended compensating control is to back every exclusion with a dedicated, Assigned-membership Microsoft Entra security group (not a list of individual users, not a legacy synced group), and attach a recurring Access Review to that group. The documented pattern for a location-based exclusion is a review that runs every week, never ends, requires self-attestation from each member, and automatically removes anyone who doesn't respond. For a legacy-authentication exclusion, the recommended pattern is a recurring review with business-unit owners as reviewers and auto-applied results. Both patterns require a Microsoft Entra ID P2, Microsoft Entra ID Governance, or Enterprise Mobility + Security E5 license — Access Reviews are not available on lower tiers.

Exclusions also have a scope blind spot most reviewers don't check: resource exclusions on "All resources" policies. Microsoft is rolling out an enforcement change, beginning June 15, 2026, that closes exactly this gap. Previously, when an "All resources" policy had any resource exclusion, sign-ins that requested only a narrow set of "baseline scopes" (openid, profile, email, offline_access, User.Read, and a handful of similar directory scopes) were silently exempted from enforcement entirely — even though the admin who built the policy believed "All resources" meant everything. Public clients like the Azure CLI or VS Code, and confidential web apps that only request those scopes, went through unchallenged. Attackers abusing OAuth flows that request minimal scopes, similar in spirit to what happens in device code phishing, are exactly the kind of sign-in this gap would have let through unreviewed. This is a case where Microsoft's own telemetry confirmed that resource-level exclusions were producing broader, unintended coverage gaps than tenants realized — the exact failure mode this article is about, just at the scope layer instead of the user layer.

Detection

Detection here is a correlation exercise across sign-in evidence and policy-change history, not a single alert.

🚨 Danger: A policy with zero reportOnlyFailure or Failure events isn't necessarily well-tuned. It can equally mean the policy's exclusions are so broad that almost nothing is actually being evaluated against it.

Remediation

Close report-only drift

1. Pull every policy's current Enable state and its last change date. Any policy in report-only without a documented review-window end date is the gap — not just "is this a new test," but "does anyone own graduating this to enforced."
2. Alert on On → Report-only audit log events specifically. This transition is rare, deliberate, and dangerous when it happens on a previously enforced policy — it should never occur silently.

Close exclusion drift

3. Replace individual-user and legacy-synced-group exclusions with dedicated, Assigned-membership Entra security groups. This is the prerequisite for the next step; Access Reviews target groups, not ad hoc user lists.
4. Attach a recurring Access Review to every exclusion group, following Microsoft's documented patterns: self-attestation with auto-removal for travel/location exclusions, owner-reviewed with auto-apply for legacy-auth or device exclusions. This requires Entra ID P2 or Entra ID Governance licensing.
5. Review every "All resources" policy that has a resource exclusion against the June 15, 2026 baseline-scopes enforcement rollout. Decide deliberately — enable the new enforcement early via the Baseline scopes settings, or use "Customize behavior" for the specific applications that genuinely need the legacy exemption — instead of letting Microsoft's default rollout decide silently.
6. Re-validate with evidence, not intent. After any report-only-to-enforced graduation or exclusion-group cleanup, confirm the result in the sign-in logs for representative excluded and included accounts, not just in the policy's configuration screen.

How EtcSec Detects This

EtcSec's Azure/Entra checks map directly to the two failure modes in this article: CA_POLICY_REPORT_ONLY flags Conditional Access policies still in report-only mode, CA_EXCESSIVE_EXCLUSIONS and CA_GROUP_EXCLUSION_LARGE flag policies whose exclusion scope has grown large relative to their intent, and CA_USER_EXCLUSIONS_STALE flags user-level exclusions that show no evidence of recent re-review. Because both failure modes are about drift rather than a one-time setup mistake, the value is in re-running the audit on a schedule — a tenant that passed this check six months ago can quietly fail it today without a single policy being deleted or a single setting screen changing.

Related Reading

• Entra ID Conditional Access Gaps: What Misconfigurations Leave Real Exposure
• How to Audit Microsoft Entra ID Security (Azure AD): Practical Review Guide
• MFA Fatigue: Detection and Prevention for Microsoft Entra ID
• Azure Guest Accounts: The Forgotten Attack Surface in Your Tenant
• Device Code Phishing: How OAuth Device Flow Compromises Entra ID Accounts
• Azure Identity Protection: Blocking Leaked Credentials

Primary References

• Conditional Access Policy Insights: Monitoring and Evaluation
• Conditional Access Insights and Reporting workbook
• Manage users excluded from Conditional Access policies
• Enforcement for baseline scopes in Conditional Access
• appliedConditionalAccessPolicy resource type — Microsoft Graph v1.0
• What are Microsoft Entra sign-in logs?