Assess Against CIS Microsoft 365 Foundations Benchmark¶
Introduction¶
CIS-CAT Pro Assessor allows Microsoft 365 auditors and cloud administrators to assess tenants in the Microsoft 365 Commercial Cloud against the CIS Microsoft 365 Foundations Benchmark. The Benchmark's recommendations were validated against Microsoft 365 Commercial Cloud and cover key services, including Exchange Online, SharePoint Online, OneDrive for Business, Microsoft Teams, Power BI (Fabric), and Microsoft Entra ID.
This guide explains how to set up and run assessments of Microsoft 365 Commercial Cloud tenants against the CIS Microsoft 365 Foundations Benchmark.
Info
This assessment is exclusive to Microsoft 365 Commercial Cloud. Assessment of other Microsoft 365 cloud environments (e.g., Microsoft 365 Government Community Cloud (GCC), Microsoft 365 DoD Cloud, etc.) are not supported.
Assessment Process Overview¶
The assessment process follows these high-level steps:
- PowerShell scripts leverage several Microsoft PowerShell modules to collect configuration data from the Microsoft 365 tenant.
- The CIS-CAT Assessor processes the collected data to evaluate compliance with the CIS Microsoft 365 Foundations Benchmark.
- Assessment results are generated in multiple formats, including HTML and JSON.
- Additional evidence is exported to the reports directory in JSON and CSV formats to support remediation efforts.
Setup¶
Prior to assessing against the CIS Microsoft 365 Foundations Benchmark, you need to install all the required dependencies on the system performing the assessment and ensure that the Entra application associated with the Microsoft 365 tenant has the necessary permissions.
Follow all the steps below in order to install and configure everything required to run the assessment successfully.
Install PowerShell 7.5¶
Refer to Install PowerShell 7.5 for installation instructions.
Note
Both PowerShell 5 and PowerShell 7 are required to execute the various modules and scripts used in the assessment process. PowerShell 5 is pre-installed on most Windows systems.
Configure PowerShell Execution Policy¶
By default, Windows systems use the RemoteSigned execution policy, which requires that scripts downloaded from the internet be digitally signed.
To allow the CIS Microsoft 365 Foundations scripts to run unimpeded, you must either remove the Mark of the Web (MOTW) from the script files or adjust the execution policy.
Info
Refer to Microsoft's documentation on managing signed and unsigned scripts for more information.
Remove the Mark of the Web (Recommended)¶
Removing the MOTW allows scripts to run under the default RemoteSigned policy without modifying system-wide execution settings.
1. Run PowerShell 5 or PowerShell 7 as an Administrator.
2. Navigate to the root directory of the CIS-CAT Assessor installation.
3. Run the following commands one at a time to unblock all relevant script files:
# Run these commands one at a time to unblock script files
Get-ChildItem *.ps1 -Recurse | Unblock-File
Get-ChildItem *.psm1 -Recurse | Unblock-File
Get-ChildItem *.psd1 -Recurse | Unblock-File
Warning
Ensure these commands are executed from the correct directory (i.e., the CIS-CAT Assessor root directory). Running them from the wrong location may inadvertently unblock unrelated files outside the CIS-CAT Assessor folder structure.
Modify the Execution Policy¶
As an alternative, you can lower the execution policy using the Set-ExecutionPolicy cmdlet.
Note
Unblocking the files is the preferred method as it does not modify the execution policy.
1. Run PowerShell 7 as Administrator.
2. Execute the following:
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope LocalMachine
Dependencies¶
Ensure that all required PowerShell modules and the necessary Entra application are properly configured.
Create Certificate¶
The CIS Microsoft 365 Foundations Assessment tool uses certificate-based authentication to connect to an Entra application. Before creating the application, generate a certificate for authentication and add it to the local store on the machine performing the assessment. You will need the Certificate Thumbprint for subsequent configuration steps.
Refer to Microsoft's instructions on creating certificates.
Install Modules¶
Before running the assessment, install the required PowerShell modules, create an Entra application with the necessary permissions, assign the Global Reader role, and grant admin consent. The M365Foundations.ps1 script in <Assessor Root>\Setup\M365 automates all of these steps, but if preferred, you can complete them manually.
1. Run PowerShell 7 as an Administrator.
2. Execute the following command:
.\M365Foundations.ps1 -InstallPreReqs
The script will prompt you to install or update the modules.
Note
A complete list of the installed modules and their minimum and tested versions is available in the ModuleRequirements.ps1 file, found in the <Assessor Root>\Setup\M365\Definitions directory.
Create the Entra Application¶
After the modules are installed, the script guides you through creating the Entra application. The workflow follows least-privilege principles and supports separation of duties. The table below lists the minimum roles required for each step. While the initial app creation requires privileged roles, the application is provisioned with only the permissions it needs.
You do not need to check 'Consent on behalf of your organization' when granting Microsoft Graph delegated permissions; this is separate from the process of granting admin consent for application permissions.
| Setup Step | Minimum Required Directory Role | Delegated Permission (Graph) |
|---|---|---|
| Read organization info | Application Administrator | Organization.Read.All |
| Create Application | Application Administrator | Application.ReadWrite.All |
| Assign Global Reader Role | Privileged Role Administrator | RoleManagement.ReadWrite.Directory |
| Grant Admin Consent | Privileged Role Administrator | AppRoleAssignment.ReadWrite.All |
| Create Fabric Security Group | Groups Administrator | Group.ReadWrite.All, GroupMember.ReadWrite.All |
Note
Application creation is a one-time task per tenant.
Supported Workflow
An Application Administrator creates the Entra application and requests permissions. A Privileged Role Administrator then grants admin consent and assigns the Global Reader role in Entra ID. A Global Administrator, or a combination of roles, can perform all steps manually or via the script if separation of duties is not required.
Automated Application Creation¶
During .\M365Foundations.ps1 -InstallPreReqs, the script prompts for inputs. After you enter an application name, it connects to Microsoft Graph, requests permissions based on your selections, and creates the app. When prompted, sign in with an account that has the required directory roles.
1. Create the required Entra application [Y/N]?:
- Required directory role:
Application AdministratororCloud Application Administrator - Graph permissions requested:
Application.ReadWrite.All
2. Grant admin consent and assign the Global Reader directory role [Y/N]?:
- Required directory role:
Privileged Role Administrator - Graph permissions requested:
RoleManagement.ReadWrite.Directory
3. Create a security group for read-only access to Microsoft Fabric [Y/N]?:
- Required directory role:
Groups Administrator - Graph permissions requested:
Group.ReadWrite.All,GroupMember.ReadWrite.All
4. Provide a name for the application and security group:
- Example:
CIS-CAT Assessor - Example:
Fabric-RO
5. Automatically generate the M365Config.psd1 configuration file [Y/N]?:
- What the script reads: Tenant ID, Default Domain, Initial Domain
- Graph permissions requested:
Organization.Read.All - The Certificate Thumbprint will be requested and the remaining parameters will be populated automatically.
Warning
If you choose not to perform the automated steps, then they must be completed manually. These steps are outlined in the Manual Configuration Steps section below.
Microsoft Fabric Access¶
A security group and modifications to Fabric tenant settings are required to provide read-only access to Microsoft Fabric data. The script can create the security group and add the service principal as a member. You must then manually enable two specific tenant settings in the Fabric admin portal and assign the security group to each.
1. Navigate to the Microsoft Fabric portal.
2. Enable the setting Service principals can call Fabric public APIs and add the security group previously created.
3. Enable the setting Service principals can access read-only admin APIs and add the security group previously created.
Note
Modifying tenant settings in the admin center requires Fabric Administrator or higher privileges. Fabric tenant settings cannot be audited without this step completed.
Upload Certificate¶
The previously created certificate must be associated with the Entra application in the Entra admin center.
1. Navigate to the Microsoft Entra admin center.
2. Go to Entra ID > App registrations > All applications.
3. Select the application you created.
4. Go to Certificates & secrets > Certificate > Upload certificate.
5. Browse to the exported .cer file and upload it.
Run Assessment¶
To run an assessment, open a PowerShell 7 terminal as Administrator and navigate to the <Assessor Root>\Setup\M365 folder:
cd <Assessor Root>\Setup\M365
There are a number of ways to customize how the assessment is run (e.g., assessment profile, reporting output, etc.).
Example 1: Run an assessment in interactive mode; defaults to an ARF report:
.\M365Foundations.ps1
Example 2: Run an assessment using the E3 Level 1 profile and output an HTML formatted report omitting the ARF report. Additional report outputs using switches like -json or -txt can also be specified.
.\M365Foundations.ps1 -ProfileName 'E3 Level 1' -Html -NoARF
To see all available parameters and additional examples, run the following command:
Get-Help .\M365Foundations.ps1 -Full
The full set of CIS-CAT Assessor parameters can be passed through the script.
Reports and Output¶
Reports are saved to the <Assessor Root>\Reports directory. HTML reports include PASS/FAIL states for all recommendations flagged as Automated.
Additionally, a subfolder is created containing exported evidence in JSON and CSV format to support remediation efforts.
Note
Evidence files are included primarily for complex recommendations that involve a large number of objects. For example, a recommendation that outputs a list of users will generate a corresponding CSV or JSON file, making it easier for administrators to review and act on the data programmatically.
Tailoring¶
Tailoring is currently not supported. Work is being done to add this to future Automated Assessment Content releases.
Manual Configuration Steps¶
If you prefer to create the Entra application and assign the required permissions manually, follow the steps below.
Create Entra Application¶
In Entra ID, create a new application registration. See Microsoft's documentation on creating an application registration for detailed instructions.
1. In Entra ID, go to App registrations > All applications.
2. Select the application you created.
3. Select API Permissions and add the permissions as Application permissions (not delegated) as shown in the table below for each of the three Microsoft APIs (Microsoft Graph, Office 365 Exchange Online, and SharePoint):
| Product | API Permissions | API Name |
|---|---|---|
| Entra ID | AccessReview.Read.All | Microsoft.Graph |
| Entra ID | AuditLog.Read.All | |
| Entra ID | Directory.Read.All | |
| Entra ID | Policy.Read.All | |
| Entra ID | RoleManagement.Read.All | |
| Intune | DeviceManagementConfiguration.Read.All | |
| Intune | DeviceManagementServiceConfig.Read.All | |
| Microsoft 365 | OrgSettings-AppsAndServices.Read.All | |
| Microsoft 365 | OrgSettings-Forms.Read.All | |
| Exchange Online | Exchange.ManageAsApp | Office 365 Exchange Online |
| SharePoint Online | Sites.FullControl.All | SharePoint |
Note
The Office 365 Exchange Online API can be found under APIs my organization uses, when adding permissions
Grant Admin Consent¶
Grant admin consent for the permissions assigned to the application.
1. Navigate to the Microsoft Entra admin center.
2. Go to Entra ID > App registrations > All applications.
3. Select the application you created.
4. Select API Permissions and then Grant admin consent for your organization.
Assign Global Reader Role¶
To allow full data collection from Microsoft Teams and Exchange Online, the service principal must be assigned the Global Reader role.
1. Navigate to Entra ID > Roles & admins.
2. Locate and select the Global Reader role from the list.
3. Select Add assignments, then search for the name of the application you created earlier.
Note
The UI may initially display only Users and Groups. Enterprise applications may not populate until you first search for the application by name. Once found, it will appear as a selectable option.
Fabric Security Group and Tenant settings¶
To provide read-only access to Microsoft Fabric, create a security group and add the service principal as a member.
1. Navigate to Entra ID > Groups.
2. Select New group and choose Security for the Group type.
3. Enter a name and assign the new service principal as a member.
4. Navigate to the Microsoft Fabric portal.
5. Enable the setting Service principals can call Fabric public APIs and add the security group created above.
6. Enable the setting Service principals can access read-only admin APIs and add the security group created above.
Note
Modifying tenant settings requires Fabric Administrator or higher privileges.
Certificate¶
Follow the steps in the Upload Certificate section above to upload the certificate to the Entra application.
Configure M365Config.psd1¶
1. Under <Assessor Root>\config, open the M365Config.psd1 file in a text editor.
2. Edit the five parameters and include the required information.
| Parameter | Description |
|---|---|
| CertificateThumbprint | The thumbprint of the certificate used for authentication. |
| ApplicationId | The Application (client) ID of the registered application. The ID can be found in the application overview. |
| TenantId | The Entra tenant ID. |
| Tenant | The primary domain name or onmicrosoft.com domain of the Entra ID tenant. |
| SharePointURL | The Unique SharePoint admin URL for the organization, i.e., |
Note
The assessment script will not execute if default values are left in the configuration file.
Troubleshooting¶
Log files specific to Microsoft 365 assessments are stored in the <Assessor Root>\logs\M365Foundations directory. If you encounter issues during the assessment process, review these logs for error messages and other diagnostic details. You can also submit a Product Technical Support ticket with the logs attached for further assistance.
Warning
When sending in logs to Product Support, do not include your tenant information as it could contain user emails.
The tenant data used during an assessment can be exported in its entirety as a JSON file. This export contains all configuration data collected from the tenant and can be used for manual review, troubleshooting, or manual diffs.
To export, add the -ExportTenantConfig switch when running the assessment script:
.\M365Foundations.ps1 -ExportTenantConfig