7bc2ab8be6
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 3s
931 lines
40 KiB
Markdown
931 lines
40 KiB
Markdown
---
|
|
tags:
|
|
- Exchange Server
|
|
- Database Availability Group
|
|
- Maintenance
|
|
- Windows Server
|
|
- PowerShell
|
|
---
|
|
|
|
## Purpose
|
|
This workflow applies Exchange Server Subscription Edition (SE), Windows Server, and approved prerequisite updates to a three-member database availability group (DAG) by updating one DAG member at a time. The procedure drains client and transport activity, moves active mailbox databases, places the target member into maintenance mode, installs updates, validates the updated member, and restores the intended database placement before the next cycle begins.
|
|
|
|
All organization names, hostnames, FQDNs, DAG names, database names, and example values in this page describe the fictional Bunny Lab environment. The examples use the `bunny-lab.io` DNS namespace and do not identify another organization.
|
|
|
|
!!! warning "Update One DAG Member at a Time"
|
|
Only one DAG member may be in maintenance mode at a time. Do not begin the next cycle until the previous member has returned to service, all database copies are healthy, all copy and replay queues have drained, transport queues are clear, and replication health passes across the entire DAG.
|
|
|
|
## Assumptions and Risk Boundaries
|
|
- The Exchange organization is running Exchange Server SE on three Mailbox servers in one DAG.
|
|
- Every mailbox database has at least two healthy passive copies before maintenance begins.
|
|
- A current backup and a tested Exchange recovery path exist. DAG replication provides availability, but it is not a substitute for backup.
|
|
- The operator has the Exchange and local administrative permissions required by the selected update. A CU that extends the schema or prepares Active Directory may require additional directory permissions before the first server is upgraded.
|
|
- The administrative shell host remains online and is not the current maintenance target.
|
|
- The Exchange DAG maintenance scripts are available through `$ExScripts`, and the administrative shell host has the Failover Clustering management tools installed.
|
|
- The current Exchange release notes, prerequisites, known issues, and update-specific manual actions have been reviewed.
|
|
- The required Exchange update media and Windows updates are approved and staged before the maintenance window begins.
|
|
- Any load balancer, monitoring platform, backup platform, mail gateway, or third-party application that targets an individual Exchange server has an established drain and restore procedure.
|
|
- All DAG members are returned to the same Exchange CU, SU, and HU level during the rolling update window.
|
|
|
|
!!! danger "Do Not Use a Snapshot as the Only Recovery Plan"
|
|
Do not begin the rolling update without an Exchange-aware backup and documented recovery method. If an update fails, keep the affected server isolated in maintenance mode and repair that server before continuing to another DAG member.
|
|
|
|
## Example Bunny Lab Environment
|
|
### Exchange Topology
|
|
| **Object** | **Example Value** |
|
|
| :--- | :--- |
|
|
| Organization | `Bunny Lab` |
|
|
| Active Directory DNS domain | `bunny-lab.io` |
|
|
| DAG | `BL-DAG-01` |
|
|
| Exchange version | Exchange Server Subscription Edition |
|
|
| Update staging directory | `C:\ExchangeUpdates` |
|
|
| Support scripts directory | `C:\Scripts` |
|
|
|
|
### DAG Members
|
|
| **Short Name** | **FQDN** | **Normal Role** |
|
|
| :--- | :--- | :--- |
|
|
| `EXCH-SE-01` | `EXCH-SE-01.bunny-lab.io` | DAG member and primary administrative shell host |
|
|
| `EXCH-SE-02` | `EXCH-SE-02.bunny-lab.io` | DAG member and alternate administrative shell host |
|
|
| `EXCH-SE-03` | `EXCH-SE-03.bunny-lab.io` | DAG member |
|
|
|
|
Exchange cmdlets in this page normally use the Exchange server object name, such as `EXCH-SE-01`. Commands that require an FQDN, including `Redirect-Message -Target`, use the corresponding `bunny-lab.io` FQDN.
|
|
|
|
### Intended Database Placement
|
|
| **Active Database** | **Intended Active Server** | **Passive Copy Servers** |
|
|
| :--- | :--- | :--- |
|
|
| `BL-MBX-01` | `EXCH-SE-01` | `EXCH-SE-02`, `EXCH-SE-03` |
|
|
| `BL-ARC-01` | `EXCH-SE-01` | `EXCH-SE-02`, `EXCH-SE-03` |
|
|
| `BL-MBX-02` | `EXCH-SE-02` | `EXCH-SE-01`, `EXCH-SE-03` |
|
|
| `BL-ARC-02` | `EXCH-SE-02` | `EXCH-SE-01`, `EXCH-SE-03` |
|
|
| `BL-MBX-03` | `EXCH-SE-03` | `EXCH-SE-01`, `EXCH-SE-02` |
|
|
|
|
### Rolling Upgrade Plan
|
|
| **Cycle** | **Administrative Shell Host** | **Maintenance Target** | **Transport Redirect Target** | **Temporary Database Placement** |
|
|
| ---: | :--- | :--- | :--- | :--- |
|
|
| `1` | `EXCH-SE-01` | `EXCH-SE-03` | `EXCH-SE-01.bunny-lab.io` | `BL-MBX-03` to `EXCH-SE-01` |
|
|
| `2` | `EXCH-SE-01` | `EXCH-SE-02` | `EXCH-SE-03.bunny-lab.io` | `BL-MBX-02` to `EXCH-SE-01`; `BL-ARC-02` to `EXCH-SE-03` |
|
|
| `3` | `EXCH-SE-02` | `EXCH-SE-01` | `EXCH-SE-03.bunny-lab.io` | `BL-MBX-01` to `EXCH-SE-02`; `BL-ARC-01` to `EXCH-SE-03` |
|
|
|
|
This order is specific to the fictional topology above. In another environment, choose an order that preserves quorum, keeps an administrative shell host available, and distributes active databases across healthy remaining members.
|
|
|
|
## Prepare the Exchange Updates
|
|
### Determine the Required Exchange Build
|
|
From Exchange Management Shell on a healthy DAG member, record the base Exchange build stored in Active Directory:
|
|
|
|
```powershell
|
|
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
|
|
|
$DagMembers | ForEach-Object {
|
|
Get-ExchangeServer -Identity $_
|
|
} | Format-Table Name, Edition, AdminDisplayVersion -Auto
|
|
```
|
|
|
|
`AdminDisplayVersion` identifies the base release or CU, but it does not reliably identify the installed SU or HU. Query the local `ExSetup.exe` file on every DAG member to record the full installed file version:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
|
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
|
|
|
[PSCustomObject]@{
|
|
Server = $env:COMPUTERNAME
|
|
ProductVersion = $VersionInfo.ProductVersion
|
|
FileVersion = $VersionInfo.FileVersion
|
|
}
|
|
} | Format-Table -Auto
|
|
```
|
|
|
|
Compare the results with the current Microsoft build table and the release article for the intended update. Install the latest supported CU required for the target release, then install the latest applicable SU or HU according to that release article. Do not install an SU or HU that was built for a different CU.
|
|
|
|
!!! warning "Release-Specific Instructions Take Precedence"
|
|
Review the selected update's prerequisites, Active Directory preparation requirements, known issues, and manual post-installation actions before changing the first DAG member. This generic workflow does not replace release-specific Microsoft instructions.
|
|
|
|
### Run the Exchange Health Checker
|
|
Download and stage the current Microsoft Exchange Health Checker script before the maintenance window. From an elevated PowerShell session on the administrative shell host, run it against every DAG member:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
& "C:\Scripts\HealthChecker.ps1" -Server $_
|
|
}
|
|
```
|
|
|
|
Resolve update-blocking findings before continuing. Preserve the generated reports with the maintenance record so the pre-update and post-update states can be compared.
|
|
|
|
### Stage Update Media
|
|
Create the local staging directory on every DAG member:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
|
New-Item -Path "C:\ExchangeUpdates" -ItemType Directory -Force | Out-Null
|
|
}
|
|
```
|
|
|
|
Copy the approved Exchange CU media, SU or HU package, prerequisite installers, and any required scripts to `C:\ExchangeUpdates` on every DAG member. Use the exact package linked by the applicable Microsoft release article and verify that the file is complete before the maintenance window begins.
|
|
|
|
## Select the Current Upgrade Cycle
|
|
Set these variables in Exchange Management Shell on the administrative shell host before running the common workflow. Run only the block for the current cycle.
|
|
|
|
### Cycle 1: Update `EXCH-SE-03`
|
|
Run from Exchange Management Shell on `EXCH-SE-01`:
|
|
|
|
```powershell
|
|
$DagName = "BL-DAG-01"
|
|
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
|
$AdminHost = "EXCH-SE-01"
|
|
$TargetServer = "EXCH-SE-03"
|
|
$RedirectTargetFqdn = "EXCH-SE-01.bunny-lab.io"
|
|
```
|
|
|
|
### Cycle 2: Update `EXCH-SE-02`
|
|
Run from Exchange Management Shell on `EXCH-SE-01` after Cycle 1 is fully validated:
|
|
|
|
```powershell
|
|
$DagName = "BL-DAG-01"
|
|
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
|
$AdminHost = "EXCH-SE-01"
|
|
$TargetServer = "EXCH-SE-02"
|
|
$RedirectTargetFqdn = "EXCH-SE-03.bunny-lab.io"
|
|
```
|
|
|
|
### Cycle 3: Update `EXCH-SE-01`
|
|
Run from Exchange Management Shell on `EXCH-SE-02` after Cycle 2 is fully validated:
|
|
|
|
```powershell
|
|
$DagName = "BL-DAG-01"
|
|
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
|
$AdminHost = "EXCH-SE-02"
|
|
$TargetServer = "EXCH-SE-01"
|
|
$RedirectTargetFqdn = "EXCH-SE-03.bunny-lab.io"
|
|
```
|
|
|
|
Confirm that the current Exchange Management Shell session is running on `$AdminHost` and that `$AdminHost` is not equal to `$TargetServer`:
|
|
|
|
```powershell
|
|
[PSCustomObject]@{
|
|
CurrentComputer = $env:COMPUTERNAME
|
|
AdminHost = $AdminHost
|
|
TargetServer = $TargetServer
|
|
}
|
|
```
|
|
|
|
Do not continue if `CurrentComputer` does not match `AdminHost`, or if the administrative shell host is not fully healthy.
|
|
|
|
### Capture a Pre-Update Service Baseline
|
|
Capture the Exchange-related service state on each server before the first maintenance action. Run this block once for the current target server from an elevated PowerShell session:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
Get-CimInstance Win32_Service |
|
|
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" } |
|
|
Select-Object Name, DisplayName, State, StartMode, ExitCode |
|
|
Export-Csv -Path "C:\ExchangeUpdates\PreUpdate-Services.csv" -NoTypeInformation
|
|
}
|
|
```
|
|
|
|
Do not replace this baseline with a service list copied from another organization. Exchange service startup modes can differ because of installed roles, enabled protocols, product version, and local design decisions.
|
|
|
|
## Validate DAG Health Before Each Cycle
|
|
### Check Database Copy Health
|
|
From the administrative shell host, check every database copy:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Sort-Object Name |
|
|
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
|
```
|
|
|
|
The preflight passes only when:
|
|
|
|
- Every active copy reports `Mounted`
|
|
- Every passive copy reports `Healthy`
|
|
- No copy reports `Failed`, `Suspended`, `Disconnected`, `ServiceDown`, or another unresolved failure state
|
|
- Every copy queue is `0`
|
|
- Every replay queue is `0`, or is low and demonstrably draining before any state-changing action
|
|
- `ContentIndexState` matches the expected Exchange SE state; `NotApplicable` is normal for the BigFunnel search architecture and is not, by itself, a failure
|
|
|
|
### Check Replication Health
|
|
Run replication health against every DAG member:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
Test-ReplicationHealth -Identity $_
|
|
}
|
|
```
|
|
|
|
Every applicable check must return `Passed`, and the `Error` field must be blank. Investigate any failure before moving a database or entering maintenance mode.
|
|
|
|
### Check Cluster State and Primary Active Manager
|
|
Confirm every cluster node is up and identify the current Primary Active Manager (PAM):
|
|
|
|
```powershell
|
|
Get-ClusterNode | Format-Table Name, State, NodeWeight, DynamicWeight -Auto
|
|
|
|
Get-DatabaseAvailabilityGroup -Identity $DagName -Status |
|
|
Format-List Name, PrimaryActiveManager, OperationalServers
|
|
```
|
|
|
|
All DAG members must be operational before the cycle begins. The maintenance script will move critical DAG functionality away from the target and pause its cluster node.
|
|
|
|
### Check Exchange Service Health
|
|
Check Exchange service health on the target and the administrative shell host:
|
|
|
|
```powershell
|
|
Test-ServiceHealth -Server $TargetServer
|
|
Test-ServiceHealth -Server $AdminHost
|
|
```
|
|
|
|
Resolve any required service failure before continuing.
|
|
|
|
### Check Active Database Placement
|
|
List the currently mounted copies:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" } |
|
|
Sort-Object ActiveDatabaseCopy, Name |
|
|
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
|
```
|
|
|
|
Compare the result with the intended placement table. A database may be temporarily active on another healthy member because of an earlier event, but its current state and all candidate copies must be understood before the maintenance move begins.
|
|
|
|
### Check Transport Queues
|
|
Inspect transport queues on the target before draining it:
|
|
|
|
```powershell
|
|
Get-Queue -Server $TargetServer |
|
|
Sort-Object MessageCount -Descending |
|
|
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
|
```
|
|
|
|
Investigate growing, retrying, or unreachable queues before maintenance. Redirecting a queue does not correct an underlying transport or name-resolution failure.
|
|
|
|
!!! warning "Preflight Stop Conditions"
|
|
Stop the cycle if any database copy is failed or suspended, replication health does not pass, a required Exchange service is unavailable, a cluster node is down, quorum is at risk, transport queues are persistently growing, the update prerequisites are unresolved, or the recovery path is unavailable.
|
|
|
|
## Move Active Databases Away From the Target
|
|
### Confirm Candidate Copies
|
|
Before each move, confirm that the chosen destination server holds a healthy passive copy with drained queues:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Sort-Object Name |
|
|
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ActiveDatabaseCopy -Auto
|
|
```
|
|
|
|
Confirm the selected destination copy reports `Healthy` with `CopyQueueLength` and `ReplayQueueLength` equal to `0` before moving the active database.
|
|
|
|
!!! warning "Run Only the Current Cycle's Move Block"
|
|
The following move blocks are cycle-specific. Do not run move commands for a different maintenance target.
|
|
|
|
### Cycle 1 Database Move
|
|
Move `BL-MBX-03` from `EXCH-SE-03` to `EXCH-SE-01`:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-03" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
|
```
|
|
|
|
### Cycle 2 Database Moves
|
|
Distribute the two active databases from `EXCH-SE-02` across the remaining healthy members:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-02" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
|
Move-ActiveMailboxDatabase -Identity "BL-ARC-02" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
|
```
|
|
|
|
### Cycle 3 Database Moves
|
|
Distribute the two active databases from `EXCH-SE-01` across the remaining healthy members:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-01" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
|
Move-ActiveMailboxDatabase -Identity "BL-ARC-01" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
|
```
|
|
|
|
### Validate the Database Moves
|
|
Confirm that no active database remains on the target:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" } |
|
|
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```text
|
|
<no output>
|
|
```
|
|
|
|
Review each move result and confirm that `Status` is `Succeeded`, `NumberOfLogsLost` is `0`, and `MountStatusAtMoveEnd` is `Mounted`. If any move fails or reports log loss, stop the cycle and investigate before entering maintenance mode.
|
|
|
|
## Place the Target DAG Member Into Maintenance Mode
|
|
### Drain External Client Traffic
|
|
If the environment uses a load balancer, reverse proxy, monitoring probe, backup job, or third-party connector that targets individual Exchange servers, drain or disable the target member according to that platform's documented procedure. Confirm that healthy remaining members are serving the traffic before continuing.
|
|
|
|
### Drain Hub Transport
|
|
From Exchange Management Shell on the administrative shell host, set the target Hub Transport component to draining:
|
|
|
|
```powershell
|
|
Set-ServerComponentState -Identity $TargetServer -Component HubTransport -State Draining -Requester Maintenance
|
|
```
|
|
|
|
Restart the Microsoft Exchange Transport service on the target to initiate queue draining:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
Restart-Service MSExchangeTransport
|
|
}
|
|
```
|
|
|
|
### Run the DAG Maintenance Script
|
|
From Exchange Management Shell on the administrative shell host, run the Exchange-provided DAG maintenance script:
|
|
|
|
```powershell
|
|
Set-Location $ExScripts
|
|
|
|
.\StartDagServerMaintenance.ps1 -ServerName $TargetServer -MoveComment "Rolling Exchange update" -PauseClusterNode
|
|
```
|
|
|
|
The script blocks database activation, pauses the target cluster node, moves any remaining active databases, and moves critical DAG functionality away from the target. The command can take time without producing continuous console output.
|
|
|
|
### Redirect Pending Transport Messages
|
|
Redirect messages still pending on the target to the healthy server selected for the current cycle:
|
|
|
|
```powershell
|
|
Redirect-Message -Server $TargetServer -Target $RedirectTargetFqdn -Confirm:$false
|
|
```
|
|
|
|
### Set the Server-Wide Maintenance State
|
|
Place the target server into Exchange server-wide maintenance mode:
|
|
|
|
```powershell
|
|
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Inactive -Requester Maintenance
|
|
```
|
|
|
|
### Validate Maintenance Mode
|
|
Check effective Exchange component states:
|
|
|
|
```powershell
|
|
Get-ServerComponentState -Identity $TargetServer |
|
|
Format-Table Component, State -Auto
|
|
```
|
|
|
|
`ServerWideOffline` must report `Inactive`. In the standard maintenance state, only `Monitoring` and `RecoveryActionsEnabled` should remain `Active`; investigate any other component that remains active before rebooting the server.
|
|
|
|
Confirm the database activation policy is blocked:
|
|
|
|
```powershell
|
|
Get-MailboxServer -Identity $TargetServer |
|
|
Format-List Name, DatabaseCopyAutoActivationPolicy
|
|
```
|
|
|
|
Confirm that the target cluster node is paused:
|
|
|
|
```powershell
|
|
Get-ClusterNode -Name $TargetServer |
|
|
Format-List Name, State
|
|
```
|
|
|
|
Confirm that no active databases remain on the target:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" } |
|
|
Format-Table Name, Status, ActiveDatabaseCopy -Auto
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```text
|
|
<no output>
|
|
```
|
|
|
|
Confirm that the target transport queues have drained:
|
|
|
|
```powershell
|
|
Get-Queue -Server $TargetServer |
|
|
Sort-Object MessageCount -Descending |
|
|
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
|
```
|
|
|
|
Confirm that the PAM is hosted by another DAG member:
|
|
|
|
```powershell
|
|
Get-DatabaseAvailabilityGroup -Identity $DagName -Status |
|
|
Format-List PrimaryActiveManager, OperationalServers
|
|
```
|
|
|
|
!!! warning "Do Not Reboot Until Every Maintenance Gate Passes"
|
|
Do not install updates or reboot the target until it has no mounted databases, its database activation policy is `Blocked`, its cluster node is `Paused`, `ServerWideOffline` is `Inactive`, its transport queues are drained, and the PAM is hosted by another member.
|
|
|
|
## Install Exchange and Windows Updates
|
|
### Reboot Before Installing the Exchange Update
|
|
A clean reboot before Exchange Setup or an Exchange SU or HU reduces failures caused by pending file handles and services that do not stop cleanly. From the administrative shell host, reboot the target:
|
|
|
|
```powershell
|
|
Restart-Computer -ComputerName $TargetServer -Force
|
|
```
|
|
|
|
Wait until the server is reachable, then revalidate that maintenance state persisted:
|
|
|
|
```powershell
|
|
Get-ServerComponentState -Identity $TargetServer |
|
|
Where-Object { $_.Component -eq "ServerWideOffline" } |
|
|
Format-Table Server, Component, State -Auto
|
|
|
|
Get-ClusterNode -Name $TargetServer |
|
|
Format-List Name, State
|
|
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" }
|
|
```
|
|
|
|
Do not launch the update if `ServerWideOffline` is not `Inactive`, the cluster node is not `Paused`, or a database mounted on the target after reboot.
|
|
|
|
### Install an Exchange CU or Build Upgrade
|
|
Mount the correct Exchange installation media on the target server. From an elevated Command Prompt on the target, run Setup by using the full media path so Windows does not invoke the installed `Setup.exe` from the Exchange binary directory.
|
|
|
|
=== "Diagnostic Data Off"
|
|
|
|
```cmd
|
|
D:\Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF
|
|
```
|
|
|
|
=== "Diagnostic Data On"
|
|
|
|
```cmd
|
|
D:\Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
|
|
```
|
|
|
|
Replace `D:` with the actual mounted-media drive. Wait for Exchange Setup to complete successfully and review `C:\ExchangeSetupLogs\ExchangeSetup.log` before continuing.
|
|
|
|
!!! danger "A CU Upgrade Is Not Reversible by Uninstalling It"
|
|
Do not treat a CU as an ordinary removable patch. If the CU fails, keep the server in maintenance mode and repair or recover that server by following Microsoft Exchange recovery guidance.
|
|
|
|
### Install an Exchange SU or HU
|
|
Run the exact update package specified by the applicable release article from an elevated Command Prompt on the target server:
|
|
|
|
```cmd
|
|
C:\ExchangeUpdates\<EXCHANGE_UPDATE_FILENAME>.exe
|
|
```
|
|
|
|
Replace `<EXCHANGE_UPDATE_FILENAME>` with the staged package name. Do not launch the installer from a non-elevated shell or by double-clicking it in File Explorer.
|
|
|
|
Do not stop IIS, WMI, Windows Event Log, or other Windows services preemptively unless the release article or a matching Microsoft troubleshooting procedure explicitly instructs you to do so. Wait for the installer to report successful completion before continuing.
|
|
|
|
### Install Approved Windows Updates
|
|
Install the approved Windows Server updates while the target remains in maintenance mode. Apply prerequisites in the order required by the Exchange release notes, and continue rebooting as required until the server has no remaining approved updates or pending restart.
|
|
|
|
### Perform the Final Maintenance Reboot
|
|
After Exchange and Windows updates have completed, reboot the target one final time:
|
|
|
|
```powershell
|
|
Restart-Computer -ComputerName $TargetServer -Force
|
|
```
|
|
|
|
Wait for Windows, Active Directory connectivity, the Cluster service, and Exchange services to initialize before beginning return-to-service validation.
|
|
|
|
## Return the Updated DAG Member to Service
|
|
### Verify the Installed Exchange Build
|
|
From Exchange Management Shell on the administrative shell host, confirm the base build:
|
|
|
|
```powershell
|
|
Get-ExchangeServer -Identity $TargetServer |
|
|
Format-Table Name, Edition, AdminDisplayVersion -Auto
|
|
```
|
|
|
|
Query the local `ExSetup.exe` file on the target to identify the installed SU or HU file version:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
|
|
|
[PSCustomObject]@{
|
|
Server = $env:COMPUTERNAME
|
|
ProductVersion = $VersionInfo.ProductVersion
|
|
FileVersion = $VersionInfo.FileVersion
|
|
}
|
|
}
|
|
```
|
|
|
|
Compare both values with the intended Microsoft build. An unchanged `AdminDisplayVersion` does not, by itself, prove that an SU or HU failed to install.
|
|
|
|
### Validate Exchange Services
|
|
From the administrative shell host, run:
|
|
|
|
```powershell
|
|
Test-ServiceHealth -Server $TargetServer
|
|
```
|
|
|
|
Inspect Exchange-related services on the target:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
Get-CimInstance Win32_Service |
|
|
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" } |
|
|
Select-Object Name, DisplayName, State, StartMode, ExitCode
|
|
} | Format-Table -Auto
|
|
```
|
|
|
|
Compare the result with `C:\ExchangeUpdates\PreUpdate-Services.csv` from that same server. Do not enable a service merely because it is stopped; protocol services and role-specific services may intentionally be manual or stopped.
|
|
|
|
!!! warning "Keep the Server in Maintenance Mode During Repair"
|
|
If required Exchange services are disabled, fail to start, or report dependency errors, keep the target in maintenance mode. Repair the service state and complete the failed update before running the return-to-service commands.
|
|
|
|
### Remove the Server-Wide Maintenance State
|
|
From Exchange Management Shell on the administrative shell host, restore the server-wide component state:
|
|
|
|
```powershell
|
|
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Active -Requester Maintenance
|
|
```
|
|
|
|
### Run the DAG Return-to-Service Script
|
|
Run the Exchange-provided DAG maintenance exit script:
|
|
|
|
```powershell
|
|
Set-Location $ExScripts
|
|
|
|
.\StopDagServerMaintenance.ps1 -ServerName $TargetServer
|
|
```
|
|
|
|
The script resumes the cluster node, restores database activation policy, and resumes database copies hosted by the member.
|
|
|
|
### Resume Hub Transport
|
|
Return Hub Transport to active state:
|
|
|
|
```powershell
|
|
Set-ServerComponentState -Identity $TargetServer -Component HubTransport -State Active -Requester Maintenance
|
|
```
|
|
|
|
Restart transport on the target:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
Restart-Service MSExchangeTransport
|
|
}
|
|
```
|
|
|
|
### Validate the Restored Member
|
|
Confirm Exchange component states:
|
|
|
|
```powershell
|
|
Get-ServerComponentState -Identity $TargetServer |
|
|
Format-Table Component, State -Auto
|
|
```
|
|
|
|
Confirm database activation is unrestricted:
|
|
|
|
```powershell
|
|
Get-MailboxServer -Identity $TargetServer |
|
|
Format-List Name, DatabaseCopyAutoActivationPolicy
|
|
```
|
|
|
|
Confirm the cluster node is up:
|
|
|
|
```powershell
|
|
Get-ClusterNode -Name $TargetServer |
|
|
Format-List Name, State
|
|
```
|
|
|
|
Run service and replication health:
|
|
|
|
```powershell
|
|
Test-ServiceHealth -Server $TargetServer
|
|
Test-ReplicationHealth -Identity $TargetServer
|
|
```
|
|
|
|
Inspect transport queues:
|
|
|
|
```powershell
|
|
Get-Queue -Server $TargetServer |
|
|
Sort-Object MessageCount -Descending |
|
|
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
|
```
|
|
|
|
Do not restore the target to a load balancer or other external traffic source until required Exchange components are active, required services are running, the cluster node is up, replication health passes, and transport queues are processing normally.
|
|
|
|
### Restore External Client Traffic
|
|
Return the target to its load balancer pool, monitoring platform, backup schedule, mail gateway, and any other system that was intentionally drained. Confirm the external health checks recognize the server as healthy before restoring active mailbox databases to it.
|
|
|
|
## Restore the Intended Database Placement
|
|
### Wait for Database Copies to Catch Up
|
|
Before moving an active database back to the updated member, confirm every copy is healthy and all queues have drained:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Sort-Object Name |
|
|
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
|
```
|
|
|
|
Do not activate a copy on the updated member while it is `Failed`, `Suspended`, `Disconnected`, `ServiceDown`, or building a persistent queue.
|
|
|
|
!!! warning "Run Only the Current Cycle's Restore Block"
|
|
The following restore blocks are cycle-specific. Do not move databases for a different cycle.
|
|
|
|
### Cycle 1 Database Restore
|
|
Move `BL-MBX-03` back to `EXCH-SE-03`:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-03" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
|
```
|
|
|
|
### Cycle 2 Database Restore
|
|
Move both databases back to `EXCH-SE-02`:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-02" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
|
Move-ActiveMailboxDatabase -Identity "BL-ARC-02" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
|
```
|
|
|
|
### Cycle 3 Database Restore
|
|
Move both databases back to `EXCH-SE-01`:
|
|
|
|
```powershell
|
|
Move-ActiveMailboxDatabase -Identity "BL-MBX-01" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
|
Move-ActiveMailboxDatabase -Identity "BL-ARC-01" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
|
```
|
|
|
|
### Validate the Restored Placement
|
|
Confirm active database placement:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" } |
|
|
Sort-Object ActiveDatabaseCopy, Name |
|
|
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
|
```
|
|
|
|
Expected Bunny Lab placement:
|
|
|
|
```text
|
|
Name Status ActiveDatabaseCopy CopyQueueLength ReplayQueueLength
|
|
---- ------ ------------------ --------------- -----------------
|
|
BL-ARC-01\EXCH-SE-01 Mounted EXCH-SE-01 0 0
|
|
BL-MBX-01\EXCH-SE-01 Mounted EXCH-SE-01 0 0
|
|
BL-ARC-02\EXCH-SE-02 Mounted EXCH-SE-02 0 0
|
|
BL-MBX-02\EXCH-SE-02 Mounted EXCH-SE-02 0 0
|
|
BL-MBX-03\EXCH-SE-03 Mounted EXCH-SE-03 0 0
|
|
```
|
|
|
|
Temporary queues may appear immediately after activation. Wait until all copy and replay queues return to `0` before declaring the cycle complete.
|
|
|
|
## Validate the Completed Cycle
|
|
Run replication health against every DAG member:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
Test-ReplicationHealth -Identity $_
|
|
}
|
|
```
|
|
|
|
Check all database copies:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Sort-Object Name |
|
|
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
|
```
|
|
|
|
Run the Exchange Health Checker against the updated member:
|
|
|
|
```powershell
|
|
& "C:\Scripts\HealthChecker.ps1" -Server $TargetServer
|
|
```
|
|
|
|
The cycle is complete only when:
|
|
|
|
- The target reports the intended Exchange build
|
|
- Required Exchange services pass `Test-ServiceHealth`
|
|
- The target cluster node is `Up`
|
|
- Database activation policy is `Unrestricted`
|
|
- Required Exchange components are active
|
|
- Every applicable replication-health check returns `Passed`
|
|
- Every active database copy reports `Mounted`
|
|
- Every passive database copy reports `Healthy`
|
|
- Every copy and replay queue is `0`
|
|
- Transport queues are processing normally
|
|
- The intended active database placement is restored
|
|
- External health checks and monitoring report the target as healthy
|
|
- The post-update Health Checker report contains no unresolved update-blocking finding
|
|
|
|
Do not begin the next cycle until all conditions are satisfied. Repeat the common workflow with the next cycle's variables and database move block.
|
|
|
|
## Final DAG Validation
|
|
### Confirm a Consistent Exchange Build
|
|
After all three cycles are complete, confirm the base build recorded for every DAG member:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
Get-ExchangeServer -Identity $_
|
|
} | Format-Table Name, Edition, AdminDisplayVersion -Auto
|
|
```
|
|
|
|
Confirm the local `ExSetup.exe` file version on every member:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
|
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
|
|
|
[PSCustomObject]@{
|
|
Server = $env:COMPUTERNAME
|
|
ProductVersion = $VersionInfo.ProductVersion
|
|
FileVersion = $VersionInfo.FileVersion
|
|
}
|
|
} | Sort-Object Server | Format-Table -Auto
|
|
```
|
|
|
|
All three servers must report the same intended Exchange build unless Microsoft explicitly documents a temporary mixed-build state during the active maintenance window.
|
|
|
|
### Validate Database Health and Placement
|
|
Run:
|
|
|
|
```powershell
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Sort-Object Name |
|
|
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
|
|
|
Get-MailboxDatabaseCopyStatus * |
|
|
Where-Object { $_.Status -eq "Mounted" } |
|
|
Sort-Object ActiveDatabaseCopy, Name |
|
|
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
|
```
|
|
|
|
Confirm every active copy is `Mounted`, every passive copy is `Healthy`, all queues are `0`, and placement matches the Bunny Lab topology table.
|
|
|
|
### Validate Replication, Services, Components, and Cluster State
|
|
Run:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
Test-ServiceHealth -Server $_
|
|
Test-ReplicationHealth -Identity $_
|
|
}
|
|
|
|
Get-ClusterNode |
|
|
Format-Table Name, State, NodeWeight, DynamicWeight -Auto
|
|
|
|
$DagMembers | ForEach-Object {
|
|
Get-ServerComponentState -Identity $_ |
|
|
Where-Object { $_.State -ne "Active" } |
|
|
Select-Object Server, Component, State
|
|
}
|
|
```
|
|
|
|
Investigate every unexpected non-active component state. Protocol components that are deliberately disabled must match the documented Bunny Lab design rather than being assumed healthy merely because they were disabled before the update.
|
|
|
|
### Validate Mail Flow
|
|
From Exchange Management Shell on each source server, test mail flow to another DAG member. The following examples cover all three members:
|
|
|
|
```powershell
|
|
Test-Mailflow -Identity "EXCH-SE-01" -TargetMailboxServer "EXCH-SE-02"
|
|
Test-Mailflow -Identity "EXCH-SE-02" -TargetMailboxServer "EXCH-SE-03"
|
|
Test-Mailflow -Identity "EXCH-SE-03" -TargetMailboxServer "EXCH-SE-01"
|
|
```
|
|
|
|
Each test must return `Success`. Also validate inbound and outbound mail flow through the organization's real mail gateways and confirm the client-access namespaces used by the environment are healthy.
|
|
|
|
### Run the Final Health Checker Audit
|
|
Run the current Exchange Health Checker against every DAG member:
|
|
|
|
```powershell
|
|
$DagMembers | ForEach-Object {
|
|
& "C:\Scripts\HealthChecker.ps1" -Server $_
|
|
}
|
|
```
|
|
|
|
Archive the final reports with the change record.
|
|
|
|
## Troubleshooting
|
|
### The Installer Reports Files in Use or Cannot Stop Services
|
|
Do not select an installer option that ignores locked files, and do not terminate the Windows Event Log service. Exit the installer, confirm the server remains in maintenance mode, reboot it, and rerun the update from an elevated Command Prompt.
|
|
|
|
If the problem persists:
|
|
|
|
- Confirm the update package matches the installed CU
|
|
- Confirm the server was rebooted immediately before the installation attempt
|
|
- Review the selected update's known issues and antivirus exclusion guidance
|
|
- Preserve `C:\ExchangeSetupLogs`
|
|
- Use the Microsoft SetupAssist or Setup Log Reviewer tooling appropriate to the failure
|
|
- Follow the matching procedure in [Fix Failed Exchange Server Updates](https://learn.microsoft.com/en-us/troubleshoot/exchange/client-connectivity/exchange-security-update-issues)
|
|
|
|
Do not reuse process IDs from an earlier attempt or another server. Process IDs are transient, and terminating an unidentified Windows or Exchange process can leave the installation in a worse state.
|
|
|
|
### Exchange Services Are Disabled After the Update
|
|
Compare the current service configuration with `C:\ExchangeUpdates\PreUpdate-Services.csv` from the same server:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
$Baseline = Import-Csv "C:\ExchangeUpdates\PreUpdate-Services.csv"
|
|
$Current = Get-CimInstance Win32_Service |
|
|
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" }
|
|
|
|
foreach ($Before in $Baseline) {
|
|
$After = $Current | Where-Object { $_.Name -eq $Before.Name }
|
|
|
|
if ($After -and ($After.State -ne $Before.State -or $After.StartMode -ne $Before.StartMode)) {
|
|
[PSCustomObject]@{
|
|
Name = $Before.Name
|
|
BeforeState = $Before.State
|
|
AfterState = $After.State
|
|
BeforeStartMode = $Before.StartMode
|
|
AfterStartMode = $After.StartMode
|
|
}
|
|
}
|
|
}
|
|
} | Format-Table -Auto
|
|
```
|
|
|
|
Restore only a service whose required startup mode is confirmed by the server's baseline, current Exchange role, and Microsoft guidance:
|
|
|
|
```powershell
|
|
Set-Service -Name <SERVICE_NAME> -StartupType Automatic
|
|
Start-Service -Name <SERVICE_NAME>
|
|
```
|
|
|
|
If the service fails with dependency error `1068`, inspect its required services:
|
|
|
|
```powershell
|
|
Get-Service -Name <SERVICE_NAME> -RequiredServices |
|
|
Format-Table Name, DisplayName, Status, StartType -Auto
|
|
```
|
|
|
|
Correct the failed dependency before retrying the dependent service. Do not automatically enable IMAP, POP, EdgeSync, or another optional service that was intentionally disabled before the update.
|
|
|
|
### An Exchange Component Remains Inactive
|
|
Inspect the effective and requester-specific component states:
|
|
|
|
```powershell
|
|
Get-ServerComponentState -Identity $TargetServer |
|
|
Format-Table Component, State -Auto
|
|
|
|
Get-ServerComponentState -Identity $TargetServer -Component <COMPONENT_NAME> |
|
|
Format-List Component, State, LocalStates, RemoteStates
|
|
```
|
|
|
|
Correct the requester that actually holds the component inactive. Do not repeatedly issue `Requester Maintenance` commands when the inactive state belongs to `Functional`, `HealthAPI`, or another requester.
|
|
|
|
If a failed Exchange update left `ServerWideOffline`, `Monitoring`, or `RecoveryActionsEnabled` inactive under the `Functional` requester, and the failed installation has already been repaired, restore only the affected states:
|
|
|
|
```powershell
|
|
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Active -Requester Functional
|
|
Set-ServerComponentState -Identity $TargetServer -Component Monitoring -State Active -Requester Functional
|
|
Set-ServerComponentState -Identity $TargetServer -Component RecoveryActionsEnabled -State Active -Requester Functional
|
|
```
|
|
|
|
Rerun `Test-ServiceHealth` and `Test-ReplicationHealth` after the correction.
|
|
|
|
### High Availability Remains Offline After Maintenance Ends
|
|
If a database move fails with an error stating that the `HighAvailability` component is offline, inspect its requester-specific state:
|
|
|
|
```powershell
|
|
Get-ServerComponentState -Identity $TargetServer -Component HighAvailability |
|
|
Format-List Component, State, LocalStates, RemoteStates
|
|
```
|
|
|
|
Confirm `StopDagServerMaintenance.ps1` completed successfully, the cluster node is `Up`, database activation is `Unrestricted`, and `MSExchangeRepl` is running. If every requester reports `Active` but Active Manager state remains stale, restart the replication service on the target:
|
|
|
|
```powershell
|
|
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
|
Restart-Service MSExchangeRepl
|
|
Get-Service MSExchangeRepl
|
|
}
|
|
```
|
|
|
|
Rerun replication health and retry the database move only after every applicable check passes.
|
|
|
|
### Exchange Setup Reports Success but the Build Does Not Change
|
|
Confirm Setup was launched from the mounted media by using an absolute path such as `D:\Setup.exe` or, from PowerShell in the media root, `.\Setup.exe`. Running only `Setup.exe` can invoke the installed copy under the Exchange binary path instead of the intended installation media.
|
|
|
|
Review `C:\ExchangeSetupLogs\ExchangeSetup.log`, correct the launch path, and rerun Setup while the server remains in maintenance mode.
|
|
|
|
### Outlook on the Web or the Exchange Admin Center Fails After the Update
|
|
First confirm the Exchange update completed successfully and required services are running. Review the matching symptom in Microsoft's failed-update guidance rather than running a generic post-update command sequence.
|
|
|
|
For an IIS state that specifically requires a service restart, run from an elevated PowerShell session on the affected server:
|
|
|
|
```powershell
|
|
Restart-Service -Name WAS, W3SVC
|
|
```
|
|
|
|
Do not make `UpdateCas.ps1`, `UpdateConfigFiles.ps1`, `IISADMIN`, ADSI changes, or forced process termination part of the normal update workflow. Use a repair command only when an authoritative troubleshooting procedure identifies the same failure condition and explains the required validation.
|
|
|
|
## Recovery and Stop Conditions
|
|
If a target server cannot be returned to service:
|
|
|
|
- Keep `ServerWideOffline` inactive and keep the cluster node paused
|
|
- Keep database activation blocked on the failed member
|
|
- Leave active databases on healthy DAG members
|
|
- Do not begin maintenance on another member
|
|
- Preserve Exchange Setup logs, Windows event logs, Health Checker reports, and the pre-update service baseline
|
|
- Repair the update or perform the documented Exchange server recovery procedure
|
|
- Revalidate quorum, database redundancy, transport, client access, and backup status before resuming the rolling upgrade
|
|
|
|
The rolling update is complete only after all three members report the intended Exchange build, every required health check passes, the documented database placement is restored, and Bunny Lab mail flow and client access are validated end to end.
|
|
|
|
## Reference Documentation
|
|
- [Manage Database Availability Groups in Exchange Server](https://learn.microsoft.com/en-us/exchange/high-availability/manage-ha/manage-dags)
|
|
- [Upgrade Exchange to the Latest Cumulative Update](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/install-cumulative-updates)
|
|
- [Use Unattended Mode in Exchange Setup](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/deploy-new-installations/unattended-installs)
|
|
- [Exchange Server Build Numbers and Release Dates](https://learn.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates)
|
|
- [Exchange Server Update FAQ](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/security-best-practices/exchange-server-update-faq)
|
|
- [Fix Failed Exchange Server Updates](https://learn.microsoft.com/en-us/troubleshoot/exchange/client-connectivity/exchange-security-update-issues)
|
|
- [Exchange Server Health Checker](https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/)
|