Maester ScubaGear Entra ID Security Baseline Testing: Guide

What Is Maester ScubaGear Entra ID Security Baseline Testing?

Maester ScubaGear Entra ID security baseline testing turns security configuration review into a repeatable assessment: choose documented assertions, collect tenant settings, evaluate them, and retain the results. Maester offers PowerShell test automation; ScubaGear assesses Microsoft 365 configuration against CISA SCuBA baselines. Their value is a review that can be repeated after a change, with the scope and evidence preserved.

A baseline result answers a bounded question. It can establish that an assertion matched the configuration available to the assessment. It does not establish that every authentication path is protected or that no identity attack occurred. The workflow below separates configuration measurement, investigation of results, controlled remediation, and subsequent verification. It includes documented commands and genuine test identifiers, without presenting an example as a tenant assessment we performed.

For the policy coverage questions that precede tool selection, see Conditional Access baseline policy coverage gaps in Entra ID. That article examines the existence of policies for administrators, users, resources, and devices. This guide focuses on making an assessment repeatable and interpreting its output.

How the Tools Work: Pester Tests and SCuBA Policy Evaluation

Maester and its distinct test suites

Maester is an open source PowerShell framework for maintaining security configuration through automated tests. Its repository documents custom Pester tests, result exports, and pipeline integrations. Pester provides the test execution model; Maester adds security tests and reporting around that model. The framework can be extended, so record the actual tests installed rather than treating the product name as a fixed coverage statement. Maester repository README.

The suites have different origins. The Maester suite contains community security configuration tests. The EIDSCA suite draws on Entra ID Security Config Analyzer and checks mitigations for identity attack scenarios. The CISA suite tests Microsoft 365 configuration against SCuBA guidance. Preserve the suite and test identifier alongside each result so a reader can distinguish their sources.

Maester's CISA tests do not establish that Maester executes ScubaGear's OPA engine. Shared baseline guidance also does not establish identical implementations. When comparing findings, use the requirement, the test implementation, and the release under review. Two similarly named tests may inspect different conditions, as the legacy authentication examples below demonstrate.

ScubaGear and baselines by product

ScubaGear belongs to CISA's Secure Cloud Business Applications project. Its documented assessment separates collection from evaluation: PowerShell queries Microsoft 365 APIs, Open Policy Agent evaluates the collected settings against Rego policies, and the tool generates HTML, JSON, and CSV results. The baseline document describes the security expectation; its automated implementation determines what the run can measure. ScubaGear README.

The repository provides separate baseline documents for Entra ID, the security product suite, Exchange Online, Power BI, Power Platform, SharePoint, and Teams. A baseline document's presence is not proof that every release can execute every product assessment. In particular, the v1.8.0 parameter table lists Entra ID as aad and enumerates the accepted product selectors. Check that table before selecting a product. Versioned README, v1.8.0 parameters.

For Entra-focused work, we recommend starting with the aad selector and reviewing its prerequisites. Expand to other products when the required connections, roles, and evidence owners are ready. This is an operational recommendation, rather than a claim that the rest of Microsoft 365 is assessed through Graph alone.

Keep releases separate from evolving documentation

The release references checked on 3 October 2026 for this guide are Maester 2.2.0 and ScubaGear v1.8.0. At that review date, ScubaGear's main README already discusses v2.0.0. Therefore, this guide labels v1.8.0 execution and permission references explicitly; statements about the evolving baseline documentation are cited separately. Before adopting these examples, freeze the module release, dependency versions, and test revision as an assessment specification.

Prepare Graph Read Access Before Running an Assessment

Read access has two dimensions: what the tool does and what its identity is authorized to do. Microsoft Graph distinguishes delegated access, acting on behalf of a signed-in user, from application access without that user. Consent and supported directory roles still matter. Creating an application, granting consent, or configuring authentication is administrative preparation; it should be distinguished from the subsequent configuration reads. Graph permissions reference, Graph PowerShell authentication.

For example, listing Conditional Access policies is a Graph read operation whose documented permission is Policy.Read.All. That endpoint's requirement is not sufficient evidence for granting access to an entire assessment suite: other tests read other resources. List conditionalAccessPolicies.

Maester's default scopes

The 2.2.0 Get-MtGraphScope source defines the default read scopes, including directory, policy, audit, reporting, device management, identity risk, and authentication method access. This is the framework's list, not the minimum for a single Conditional Access test. Directory.Read.All, for example, is broader than reading only the policy collection.

The same source separates optional additions: -Privileged adds ReportSettings.ReadWrite.All, -SendMail adds Mail.Send, and -SendTeamsMessage adds ChannelMessage.Send. Keep those additions outside a default Graph read assessment unless a separately reviewed requirement needs them. Connect-Maester explains the optional connections and privileged scope behavior.

Do not describe Maester as incapable of reading logs. Its default list includes AuditLog.Read.All and ThreatHunting.Read.All; Microsoft's permission definitions describe audit reads and hunting queries. The narrower point is that the baseline tests discussed here are configuration assertions. Their success does not prove that a particular sign-in was blocked.

ScubaGear's Entra prerequisites

ScubaGear v1.8.0 documents Global Reader for interactive Entra assessment and these Graph permissions: Directory.Read.All, Policy.Read.All, PrivilegedAccess.Read.AzureADGroup, PrivilegedEligibilitySchedule.Read.AzureADGroup, RoleManagement.Read.Directory, RoleManagementPolicy.Read.AzureADGroup, and User.Read.All. Review the actual consent process with the tenant administrator before the run. Interactive prerequisites.

Its noninteractive prerequisites list permissions per product. They also explain that SharePoint and Power Platform need permissions with write privileges because of API requirements, while ScubaGear does not use those privileges for assessments. Consequently, an all-products assessment should not be described as using exclusively read permissions. This distinction also applies when reviewing optional Maester service connections.

Run the Documented Commands and Preserve Their Scope

The following commands are documentation examples. They were not executed against a tenant for this article. Provision and review the assessment environment first; then use the matching documentation for the selected release.

Step 1: Prepare Maester tests

The installation guide provides the Pester installation command:

Install-Module Pester -MinimumVersion 5.7.1 -MaximumVersion 5.7.1 -SkipPublisherCheck -Force -Scope CurrentUser

It separately installs Maester:

Install-Module Maester -Scope CurrentUser

That command does not pin the Maester release. Resolve the intended version before using it for reproducible evidence. From the chosen tests directory, the guide installs the supplied tests:

Install-MaesterTests

Step 2: Connect and evaluate

The Graph connection example in Connect-Maester is:

Connect-Maester

The same documentation and the versioned scope helper give this direct Graph equivalent. It uses the scope list supplied by the installed Maester module:

Connect-MgGraph -Scopes (Get-MtGraphScope)

These are alternative connection approaches. The direct Graph example is not a reason to run both. With the tests available, the documented evaluation command is:

Invoke-Maester

The Invoke-Maester reference describes test path selection, tags, exclusions, and report output. Record the selection used in your run. It also gives an explicit HTML output example:

Invoke-Maester -OutputHtmlFile './test-results/TestResults.html'

Choose that report variant when appropriate. It is another assessment invocation, not merely a command that renames an existing report. Missing optional service modules or connections can cause tests to be skipped, as the installation guide explains.

Step 3: Choose ScubaGear products deliberately

The v1.8.0 execution guide describes Windows execution, required dependencies, and importing a downloaded module in PowerShell 5.1. Its literal all-products example is:

Invoke-SCuBA -ProductNames *

This illustrates the documented wildcard; use it only after reviewing the prerequisites for the expanded scope. For this guide's Entra-only recommendation, replace the wildcard with aad, using the documented product parameter. That substitution is an adaptation of the example, not a verbatim command from the execution guide. Preserve the resulting product selection in the run record.

Measure What the Results Actually Establish

Maester supplies test results and report exports. ScubaGear supplies a global HTML report, individual product reports, structured JSON, and result CSV. ScubaGear's JSON can retain run context and provider output, depending on its options; ScubaResults.csv is a narrower result export. Its ActionPlan.csv filters failed SHALL controls and includes space for failure explanations and remediation timelines. It is not the complete assessment ledger. Maester export features, ScubaGear reports.

We recommend reviewing results in separate queues:

These are handling recommendations, not invented result labels for every version of either tool. They prevent a change in the evaluated population from being mistaken for an improvement. A useful operational comparison records which assertions ran successfully, which could not run, and which require another method.

The current SCuBA Entra baseline explicitly limits automated evaluation of Conditional Access Conditions to those specified in its implementations. It also marks some requirements as manual. A passing automated result therefore leaves questions about conditions, exceptions, and operational verification that the assertion does not cover.

For the complementary investigation, read Conditional Access gaps and sign-in evidence. That article examines actual policy application, exclusions, and authentication paths. It complements a baseline report without turning configuration tests into a claim about every sign-in.

Detection Through Recurring Assessment and CI

Recurring baseline assessment detects changes between successfully observed configurations. We recommend retaining the tenant identifier, timestamp, release, test revision, product scope, permissions profile, errors, reports, and exceptions for every run. Compare like with like. If a suite or dependency changes, first identify whether the difference comes from tenant configuration, test logic, or collection coverage.

Maester's GitHub monitoring guide documents workload identity federation with OIDC, application permissions, scheduled execution, and manual workflow dispatch. Its schedule example runs daily at 07:30 UTC. Choose a cadence for your operational needs; the example does not establish an appropriate response interval for every tenant. -NonInteractive controls presentation behavior, including suppressing browser opening, and does not configure authentication. Invoke-Maester parameter reference.

ScubaGear's execution documentation describes certificate-based service principal execution for pipelines and scheduled jobs. The versioned prerequisites should govern the permission setup. We recommend keeping application identifiers and certificate references in the job configuration while managing the actual private key through the organization's approved credential process. Treat provisioning and credential maintenance as separate administrative work.

For either tool, we recommend pinning the module, tests, dependencies, and CI action revision together. A job failure should create an operational investigation rather than reuse yesterday's favorable result as today's evidence. Route configuration regressions to their owners and preserve the reason for any accepted exception. These are workflow recommendations; no pipeline execution or response performance is claimed here.

An assessment schedule produces observations at run times. It cannot prove that the configuration stayed unchanged between them. Keep the recurring assessment alongside the existing security monitoring process, and make the date of the most recent successful collection visible to reviewers.

Remediation: From a Failed Test to a Verified Change

Step 1: Read Maester MT.1009 precisely

MT.1009 checks for a Conditional Access policy blocking other legacy authentication. The implementation inspected selects enabled policies, removes policies containing passwordChange, and seeks a block grant with client type other, included applications All, and included users All. It returns false if none match. An Entra Free licensing condition or a caught error can instead produce a skipped result.

If that test fails in your assessment, inspect the matched data and confirm that the failure represents the intended requirement. The function does not prove that no exclusions exist or that every relevant connection is blocked. Its implementation is a concrete starting point for investigation, not a complete policy simulator. Retain the source revision with the finding so later implementation changes remain distinguishable from remediation.

Step 2: Compare SCuBA MS.AAD.1.1v1 without assuming equivalence

MS.AAD.1.1v1 in the v1.8.0 baseline requires blocking legacy authentication. Its implementation instructions call for identifying existing legacy dependencies, targeting all users and all resources, selecting Exchange ActiveSync and Other clients, and granting Block Access. MT.1009's other test alone does not represent all those instructions. These are Maester and SCuBA identifiers, not CIS control numbers.

We recommend opening a remediation record containing the failed identifier, evaluated policy data, relevant requirement, owner, affected dependencies, and proposed change. If a result is skipped, open a prerequisite investigation first. If it is manual, document the manual method. None of these examples is an observed failure in a tenant assessed for this article.

Step 3: Validate impact before enforcement

Microsoft's legacy authentication blocking guide starts the new policy in report-only, reviews impact, and moves it to On after validation. Follow the documented treatment of emergency access accounts and any necessary staged deployment. Identify legitimate legacy dependencies and plan their migration or an explicitly governed exception before broad enforcement.

We recommend testing representative intended access paths and reviewing the policy impact evidence with their owners. Record exactly when enforcement is enabled. A report-only policy is useful for assessing impact, but it cannot establish that the intended block was enforced. Do not remove the validation step merely to satisfy an automated assertion that looks for enabled state.

Step 4: Re-execute, then schedule the same review

After the approved change, rerun the relevant Maester tests or ScubaGear product assessment with the retained selection and versions. Review the new data as well as the result. Attach both assessments to the remediation record, and separately confirm the intended behavior through the documented policy impact and sign-in review process. Maester execution reference, ScubaGear execution, Microsoft validation procedure.

We recommend adding the resolved assertion to the recurring review, with an owner for regressions and an expiry for exceptions. If the next run loses permissions or skips the test, reopen the evidence question. Closing a remediation record should mean that the defined requirement was rechecked and its operational validation completed.

How EtcSec Detects This

EtcSec's AZ_CIS_BENCHMARK_GAPS is a configuration aggregate containing four independent checks. The implementation read is public/internal/audit/detectors/azure/compliance/standards/cis-benchmark-gaps.go, with Detect() starting at line 68. Each counted condition adds one; Count ranges from 0 to 4. It is not a percentage or the complete CIS Microsoft 365 Foundations Benchmark.

These conditions are visible in the same file at lines 82–178. The warning AZURE_CONDITIONAL_ACCESS_POLICIES_FAILED suppresses the three policy-backed checks. Missing tenant data can leave the Security Defaults check uncounted. Without that warning, an empty policy collection counts the three policy absences. Thus Count = 0 is not proof that all four conditions were successfully assessed.

The function returns its finding even at zero; positive count and IncludeDetails govern attachment of gap details. It reads configuration and collection warnings, without consuming Maester results, ScubaGear reports, or sign-in evidence. Use it to track these four conditions across EtcSec audits, then investigate their scope. Keep full baseline evaluation and operational enforcement validation as distinct parts of the review.