Restructured Documentation
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
This commit is contained in:
+20
@@ -0,0 +1,20 @@
|
||||
---
|
||||
tags:
|
||||
- Rocket.Chat
|
||||
- Communication
|
||||
---
|
||||
|
||||
## Purpose
|
||||
When someone types a message that includes a ticket number (e.g. `T00000000.0000`) we want to replace that text with an API-friendly URL that leverages Markdown language as well.
|
||||
|
||||
From RocketChat, navigate to the "Marketplace" and look for "**Word Replacer**". You can find the application's [GitHub Page](https://github.com/Dimsday/WordReplacer) for additional information / source code review. Proceed to install the application. Once it has been installed, use the following RegEx filter / string in the application's settings:
|
||||
|
||||
```json
|
||||
[{"search": "T(\\d{8}\\.\\d{4})", "replace": "[$&](https://ww15.autotask.net/Autotask/AutotaskExtend/ExecuteCommand.aspx?Code=OpenTicketDetail&TicketNumber=$&)"}]
|
||||
```
|
||||
|
||||
!!! success
|
||||
Now everything should be functional and replacing ticket numbers with valid links that open the ticket in Autotask.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Applications Documentation](<../../../../reference/Applications/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
tags:
|
||||
- Microsoft Exchange
|
||||
- Email
|
||||
---
|
||||
|
||||
## Purpose
|
||||
If you operate an Exchange Database Availability Group (DAG) with 2 or more servers, you may need to do maintenance to one of the members, and during that maintenance, it's possible that one of the databases of the server that was rebooted etc will be out-of-date. In case this happens, it may suspend the database replication to one of the DAG's member servers.
|
||||
|
||||
!!! warning "Exchange Version Context"
|
||||
This page preserves the older database-copy and content-index repair examples. The Exchange SE rolling-update guide documents a different search architecture and service-validation process. Confirm the Exchange version and failure state before selecting a repair command.
|
||||
|
||||
## Checking DAG Database Replication Status
|
||||
You will want to first log into one of the DAG servers and open the *"Exchange Management Shell"*. From there, run the following command to get the status of database replication. An example of the kind of output you would see is below the command.
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * | Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState
|
||||
```
|
||||
|
||||
| **Name** | **Status** | **CopyQueueLength** | **ReplayQueueLength** | **ContentIndexState** |
|
||||
| :--- | ---: | ---: | ---: | ---: |
|
||||
| DB01\MX-DAG-01 | Mounted | 0 | 0 | Healthy |
|
||||
| DB01\MX-DAG-02 | Healthy | 0 | 0 | Healthy |
|
||||
|
||||
!!! info "Example Output Breakdown"
|
||||
In the above example output, you can see that there are two member servers in the DAG, `MX-DAG-01` and `MX-DAG-02`. Then you will see that there is a status of `Mounted`, this means that `MX-DAG-01` is the active production server; this means that it is handling all mailflow and web requests / webmail.
|
||||
|
||||
**CopyQueueLength**: This is a number of database "*transaction logs*" that have taken place since a replica database stopped getting updates. This is the queue of all database transactions that are being copied from the production (mounted) database to replica databases. This data is not immediately written to the replica database(s).
|
||||
|
||||
**CopyReplayLength**: This represents the queue of all data that was successfully copied from the production database to the replica database on the given DAG member that still needs to process on the replica database. The "**CopyQueueLength**" will need to reach zero before the "**CopyReplayLength**" will start making meaningful progress to reaching zero.
|
||||
|
||||
When both the "**CopyQueueLength**" and "**CopyReplayLength**" queues have reached zero, the replica database(s) will have reached 100% parity with the production (active/mounted) database.
|
||||
|
||||
## Changing Active/Mounted DAG Member
|
||||
You may find that you need to perform work on one of the DAG members, and that requires you to failover the responsibility of hosting the Exchange environment to one of the other members of the DAG. You can generally do this with one command, seen below:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "DB01" -ActivateOnServer "MX-DAG-02" -MountDialOverride BestAvailability
|
||||
```
|
||||
|
||||
!!! info "Argument Breakdown"
|
||||
`-MountDialOverride`
|
||||
Specifies how tolerant Exchange should be to database copy health when mounting a database on the target server. This setting controls the level of availability Exchange requires before mounting the mailbox database after the move.
|
||||
|
||||
`-MountDialOverride`
|
||||
Instructs Exchange to mount the database as long as at least one healthy copy is available. This option maximizes uptime by allowing a database to mount even if some copies are unhealthy, prioritizing availability over strict health checks.
|
||||
|
||||
## Troubleshooting
|
||||
You may run into issues where either the `Status` or `ContentIndexState` are either Unhealthy, Suspended, or Failed. If this happens, you need to resume replication of the database from the production active/mounted server to the server that is having issues. In the worst-case, you would re-seed the replica database from-scratch.
|
||||
|
||||
### If `Status` is Unhealthy or Suspended
|
||||
If one of the DAG members has a status of "**Unhealthy**", you can run the following command to attempt to resume replication.
|
||||
|
||||
```powershell
|
||||
Resume-MailboxDatabaseCopy -Identity "DB01\MX-DAG-02"
|
||||
```
|
||||
|
||||
If this fails to cause replication to resume, you can try telling the database to just focus on replication, which tells it to copy the queues and replay them on the replica database, while avoiding interacting with the "**ContentIndexState**" which can be individually fixed in the commands below:
|
||||
|
||||
```powershell
|
||||
Resume-MailboxDatabaseCopy -Identity "DB01\MX-DAG-02" -ReplicationOnly
|
||||
```
|
||||
|
||||
### If `Status` is `ServiceDown`
|
||||
If you see this, it generally means that the Exchange Services for some reason or another are not running. You can remediate this with a powershell script. You will then have to double-check your work to ensure that all "Microsoft Exchange" services that have a startup mode of "Automatic" are running, if not, manually start them, then check on the status of the DAG again to see if the status changes from `ServiceDown` to `Healthy`. Depending on the speed of the Exchange server, it may take a few minutes, 5-10 minutes, for the services to fully initialize and be ready to handle requests. Go get a coffee and come back and check on the status of the DAG at that time.
|
||||
|
||||
[:material-powershell: Start Exchange Services Script](<../../../../scripts/Applications/Email/Microsoft Exchange/Start Exchange Services.md>){ .md-button }
|
||||
|
||||
### If `ContentIndexState` is Unhealthy or Suspended
|
||||
If you see that the "ContentIndexState" is unhappy, you can run the following command to force it to re-seed / rebuild itself. (This is non-destructive this this is happening on a replica database).
|
||||
|
||||
```powershell
|
||||
Update-MailboxDatabaseCopy "DB01\MX05" -CatalogOnly -BeginSeed
|
||||
```
|
||||
|
||||
### If Replica Database is FUBAR
|
||||
If the replica database just is not playing nice, you can take the *nuclear option* of completely rebuilding the replica database.
|
||||
|
||||
!!! warning
|
||||
This will destroy the replica database, so be careful to ensure you have a backup (if possible) before you do this. The following command will completely replace the replica database and replicate the data from the production active/mounted database to the newly-created replica database.
|
||||
|
||||
```powershell
|
||||
Update-MailboxDatabaseCopy -Identity "DB01\MX-DAG-02" -SourceServer "MX-DAG-01"
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Email Documentation](<../../../../reference/Applications/Email/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+933
@@ -0,0 +1,933 @@
|
||||
---
|
||||
tags:
|
||||
- Exchange Server
|
||||
- Database Availability Group
|
||||
- Maintenance
|
||||
- Windows Server
|
||||
- PowerShell
|
||||
---
|
||||
|
||||
## Purpose
|
||||
This workflow applies Exchange Server Subscription Edition (SE), Windows Server, and approved prerequisite updates to a three-member database availability group (DAG) by updating one DAG member at a time. The procedure drains client and transport activity, moves active mailbox databases, places the target member into maintenance mode, installs updates, validates the updated member, and restores the intended database placement before the next cycle begins.
|
||||
|
||||
All organization names, hostnames, FQDNs, DAG names, database names, and example values in this page describe the fictional Bunny Lab environment. The examples use the `bunny-lab.io` DNS namespace and do not identify another organization.
|
||||
|
||||
!!! warning "Update One DAG Member at a Time"
|
||||
Only one DAG member may be in maintenance mode at a time. Do not begin the next cycle until the previous member has returned to service, all database copies are healthy, all copy and replay queues have drained, transport queues are clear, and replication health passes across the entire DAG.
|
||||
|
||||
## Assumptions and Risk Boundaries
|
||||
- The Exchange organization is running Exchange Server SE on three Mailbox servers in one DAG.
|
||||
- Every mailbox database has at least two healthy passive copies before maintenance begins.
|
||||
- A current backup and a tested Exchange recovery path exist. DAG replication provides availability, but it is not a substitute for backup.
|
||||
- The operator has the Exchange and local administrative permissions required by the selected update. A CU that extends the schema or prepares Active Directory may require additional directory permissions before the first server is upgraded.
|
||||
- The administrative shell host remains online and is not the current maintenance target.
|
||||
- The Exchange DAG maintenance scripts are available through `$ExScripts`, and the administrative shell host has the Failover Clustering management tools installed.
|
||||
- The current Exchange release notes, prerequisites, known issues, and update-specific manual actions have been reviewed.
|
||||
- The required Exchange update media and Windows updates are approved and staged before the maintenance window begins.
|
||||
- Any load balancer, monitoring platform, backup platform, mail gateway, or third-party application that targets an individual Exchange server has an established drain and restore procedure.
|
||||
- All DAG members are returned to the same Exchange CU, SU, and HU level during the rolling update window.
|
||||
|
||||
!!! danger "Do Not Use a Snapshot as the Only Recovery Plan"
|
||||
Do not begin the rolling update without an Exchange-aware backup and documented recovery method. If an update fails, keep the affected server isolated in maintenance mode and repair that server before continuing to another DAG member.
|
||||
|
||||
## Example Bunny Lab Environment
|
||||
### Exchange Topology
|
||||
| **Object** | **Example Value** |
|
||||
| :--- | :--- |
|
||||
| Organization | `Bunny Lab` |
|
||||
| Active Directory DNS domain | `bunny-lab.io` |
|
||||
| DAG | `BL-DAG-01` |
|
||||
| Exchange version | Exchange Server Subscription Edition |
|
||||
| Update staging directory | `C:\ExchangeUpdates` |
|
||||
| Support scripts directory | `C:\Scripts` |
|
||||
|
||||
### DAG Members
|
||||
| **Short Name** | **FQDN** | **Normal Role** |
|
||||
| :--- | :--- | :--- |
|
||||
| `EXCH-SE-01` | `EXCH-SE-01.bunny-lab.io` | DAG member and primary administrative shell host |
|
||||
| `EXCH-SE-02` | `EXCH-SE-02.bunny-lab.io` | DAG member and alternate administrative shell host |
|
||||
| `EXCH-SE-03` | `EXCH-SE-03.bunny-lab.io` | DAG member |
|
||||
|
||||
Exchange cmdlets in this page normally use the Exchange server object name, such as `EXCH-SE-01`. Commands that require an FQDN, including `Redirect-Message -Target`, use the corresponding `bunny-lab.io` FQDN.
|
||||
|
||||
### Intended Database Placement
|
||||
| **Active Database** | **Intended Active Server** | **Passive Copy Servers** |
|
||||
| :--- | :--- | :--- |
|
||||
| `BL-MBX-01` | `EXCH-SE-01` | `EXCH-SE-02`, `EXCH-SE-03` |
|
||||
| `BL-ARC-01` | `EXCH-SE-01` | `EXCH-SE-02`, `EXCH-SE-03` |
|
||||
| `BL-MBX-02` | `EXCH-SE-02` | `EXCH-SE-01`, `EXCH-SE-03` |
|
||||
| `BL-ARC-02` | `EXCH-SE-02` | `EXCH-SE-01`, `EXCH-SE-03` |
|
||||
| `BL-MBX-03` | `EXCH-SE-03` | `EXCH-SE-01`, `EXCH-SE-02` |
|
||||
|
||||
### Rolling Upgrade Plan
|
||||
| **Cycle** | **Administrative Shell Host** | **Maintenance Target** | **Transport Redirect Target** | **Temporary Database Placement** |
|
||||
| ---: | :--- | :--- | :--- | :--- |
|
||||
| `1` | `EXCH-SE-01` | `EXCH-SE-03` | `EXCH-SE-01.bunny-lab.io` | `BL-MBX-03` to `EXCH-SE-01` |
|
||||
| `2` | `EXCH-SE-01` | `EXCH-SE-02` | `EXCH-SE-03.bunny-lab.io` | `BL-MBX-02` to `EXCH-SE-01`; `BL-ARC-02` to `EXCH-SE-03` |
|
||||
| `3` | `EXCH-SE-02` | `EXCH-SE-01` | `EXCH-SE-03.bunny-lab.io` | `BL-MBX-01` to `EXCH-SE-02`; `BL-ARC-01` to `EXCH-SE-03` |
|
||||
|
||||
This order is specific to the fictional topology above. In another environment, choose an order that preserves quorum, keeps an administrative shell host available, and distributes active databases across healthy remaining members.
|
||||
|
||||
## Prepare the Exchange Updates
|
||||
### Determine the Required Exchange Build
|
||||
From Exchange Management Shell on a healthy DAG member, record the base Exchange build stored in Active Directory:
|
||||
|
||||
```powershell
|
||||
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
||||
|
||||
$DagMembers | ForEach-Object {
|
||||
Get-ExchangeServer -Identity $_
|
||||
} | Format-Table Name, Edition, AdminDisplayVersion -Auto
|
||||
```
|
||||
|
||||
`AdminDisplayVersion` identifies the base release or CU, but it does not reliably identify the installed SU or HU. Query the local `ExSetup.exe` file on every DAG member to record the full installed file version:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
||||
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
||||
|
||||
[PSCustomObject]@{
|
||||
Server = $env:COMPUTERNAME
|
||||
ProductVersion = $VersionInfo.ProductVersion
|
||||
FileVersion = $VersionInfo.FileVersion
|
||||
}
|
||||
} | Format-Table -Auto
|
||||
```
|
||||
|
||||
Compare the results with the current Microsoft build table and the release article for the intended update. Install the latest supported CU required for the target release, then install the latest applicable SU or HU according to that release article. Do not install an SU or HU that was built for a different CU.
|
||||
|
||||
!!! warning "Release-Specific Instructions Take Precedence"
|
||||
Review the selected update's prerequisites, Active Directory preparation requirements, known issues, and manual post-installation actions before changing the first DAG member. This generic workflow does not replace release-specific Microsoft instructions.
|
||||
|
||||
### Run the Exchange Health Checker
|
||||
Download and stage the current Microsoft Exchange Health Checker script before the maintenance window. From an elevated PowerShell session on the administrative shell host, run it against every DAG member:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
& "C:\Scripts\HealthChecker.ps1" -Server $_
|
||||
}
|
||||
```
|
||||
|
||||
Resolve update-blocking findings before continuing. Preserve the generated reports with the maintenance record so the pre-update and post-update states can be compared.
|
||||
|
||||
### Stage Update Media
|
||||
Create the local staging directory on every DAG member:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
||||
New-Item -Path "C:\ExchangeUpdates" -ItemType Directory -Force | Out-Null
|
||||
}
|
||||
```
|
||||
|
||||
Copy the approved Exchange CU media, SU or HU package, prerequisite installers, and any required scripts to `C:\ExchangeUpdates` on every DAG member. Use the exact package linked by the applicable Microsoft release article and verify that the file is complete before the maintenance window begins.
|
||||
|
||||
## Select the Current Upgrade Cycle
|
||||
Set these variables in Exchange Management Shell on the administrative shell host before running the common workflow. Run only the block for the current cycle.
|
||||
|
||||
### Cycle 1: Update `EXCH-SE-03`
|
||||
Run from Exchange Management Shell on `EXCH-SE-01`:
|
||||
|
||||
```powershell
|
||||
$DagName = "BL-DAG-01"
|
||||
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
||||
$AdminHost = "EXCH-SE-01"
|
||||
$TargetServer = "EXCH-SE-03"
|
||||
$RedirectTargetFqdn = "EXCH-SE-01.bunny-lab.io"
|
||||
```
|
||||
|
||||
### Cycle 2: Update `EXCH-SE-02`
|
||||
Run from Exchange Management Shell on `EXCH-SE-01` after Cycle 1 is fully validated:
|
||||
|
||||
```powershell
|
||||
$DagName = "BL-DAG-01"
|
||||
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
||||
$AdminHost = "EXCH-SE-01"
|
||||
$TargetServer = "EXCH-SE-02"
|
||||
$RedirectTargetFqdn = "EXCH-SE-03.bunny-lab.io"
|
||||
```
|
||||
|
||||
### Cycle 3: Update `EXCH-SE-01`
|
||||
Run from Exchange Management Shell on `EXCH-SE-02` after Cycle 2 is fully validated:
|
||||
|
||||
```powershell
|
||||
$DagName = "BL-DAG-01"
|
||||
$DagMembers = @("EXCH-SE-01", "EXCH-SE-02", "EXCH-SE-03")
|
||||
$AdminHost = "EXCH-SE-02"
|
||||
$TargetServer = "EXCH-SE-01"
|
||||
$RedirectTargetFqdn = "EXCH-SE-03.bunny-lab.io"
|
||||
```
|
||||
|
||||
Confirm that the current Exchange Management Shell session is running on `$AdminHost` and that `$AdminHost` is not equal to `$TargetServer`:
|
||||
|
||||
```powershell
|
||||
[PSCustomObject]@{
|
||||
CurrentComputer = $env:COMPUTERNAME
|
||||
AdminHost = $AdminHost
|
||||
TargetServer = $TargetServer
|
||||
}
|
||||
```
|
||||
|
||||
Do not continue if `CurrentComputer` does not match `AdminHost`, or if the administrative shell host is not fully healthy.
|
||||
|
||||
### Capture a Pre-Update Service Baseline
|
||||
Capture the Exchange-related service state on each server before the first maintenance action. Run this block once for the current target server from an elevated PowerShell session:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
Get-CimInstance Win32_Service |
|
||||
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" } |
|
||||
Select-Object Name, DisplayName, State, StartMode, ExitCode |
|
||||
Export-Csv -Path "C:\ExchangeUpdates\PreUpdate-Services.csv" -NoTypeInformation
|
||||
}
|
||||
```
|
||||
|
||||
Do not replace this baseline with a service list copied from another organization. Exchange service startup modes can differ because of installed roles, enabled protocols, product version, and local design decisions.
|
||||
|
||||
## Validate DAG Health Before Each Cycle
|
||||
### Check Database Copy Health
|
||||
From the administrative shell host, check every database copy:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Sort-Object Name |
|
||||
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
||||
```
|
||||
|
||||
The preflight passes only when:
|
||||
|
||||
- Every active copy reports `Mounted`
|
||||
- Every passive copy reports `Healthy`
|
||||
- No copy reports `Failed`, `Suspended`, `Disconnected`, `ServiceDown`, or another unresolved failure state
|
||||
- Every copy queue is `0`
|
||||
- Every replay queue is `0`, or is low and demonstrably draining before any state-changing action
|
||||
- `ContentIndexState` matches the expected Exchange SE state; `NotApplicable` is normal for the BigFunnel search architecture and is not, by itself, a failure
|
||||
|
||||
### Check Replication Health
|
||||
Run replication health against every DAG member:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
Test-ReplicationHealth -Identity $_
|
||||
}
|
||||
```
|
||||
|
||||
Every applicable check must return `Passed`, and the `Error` field must be blank. Investigate any failure before moving a database or entering maintenance mode.
|
||||
|
||||
### Check Cluster State and Primary Active Manager
|
||||
Confirm every cluster node is up and identify the current Primary Active Manager (PAM):
|
||||
|
||||
```powershell
|
||||
Get-ClusterNode | Format-Table Name, State, NodeWeight, DynamicWeight -Auto
|
||||
|
||||
Get-DatabaseAvailabilityGroup -Identity $DagName -Status |
|
||||
Format-List Name, PrimaryActiveManager, OperationalServers
|
||||
```
|
||||
|
||||
All DAG members must be operational before the cycle begins. The maintenance script will move critical DAG functionality away from the target and pause its cluster node.
|
||||
|
||||
### Check Exchange Service Health
|
||||
Check Exchange service health on the target and the administrative shell host:
|
||||
|
||||
```powershell
|
||||
Test-ServiceHealth -Server $TargetServer
|
||||
Test-ServiceHealth -Server $AdminHost
|
||||
```
|
||||
|
||||
Resolve any required service failure before continuing.
|
||||
|
||||
### Check Active Database Placement
|
||||
List the currently mounted copies:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" } |
|
||||
Sort-Object ActiveDatabaseCopy, Name |
|
||||
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
||||
```
|
||||
|
||||
Compare the result with the intended placement table. A database may be temporarily active on another healthy member because of an earlier event, but its current state and all candidate copies must be understood before the maintenance move begins.
|
||||
|
||||
### Check Transport Queues
|
||||
Inspect transport queues on the target before draining it:
|
||||
|
||||
```powershell
|
||||
Get-Queue -Server $TargetServer |
|
||||
Sort-Object MessageCount -Descending |
|
||||
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
||||
```
|
||||
|
||||
Investigate growing, retrying, or unreachable queues before maintenance. Redirecting a queue does not correct an underlying transport or name-resolution failure.
|
||||
|
||||
!!! warning "Preflight Stop Conditions"
|
||||
Stop the cycle if any database copy is failed or suspended, replication health does not pass, a required Exchange service is unavailable, a cluster node is down, quorum is at risk, transport queues are persistently growing, the update prerequisites are unresolved, or the recovery path is unavailable.
|
||||
|
||||
## Move Active Databases Away From the Target
|
||||
### Confirm Candidate Copies
|
||||
Before each move, confirm that the chosen destination server holds a healthy passive copy with drained queues:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Sort-Object Name |
|
||||
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ActiveDatabaseCopy -Auto
|
||||
```
|
||||
|
||||
Confirm the selected destination copy reports `Healthy` with `CopyQueueLength` and `ReplayQueueLength` equal to `0` before moving the active database.
|
||||
|
||||
!!! warning "Run Only the Current Cycle's Move Block"
|
||||
The following move blocks are cycle-specific. Do not run move commands for a different maintenance target.
|
||||
|
||||
### Cycle 1 Database Move
|
||||
Move `BL-MBX-03` from `EXCH-SE-03` to `EXCH-SE-01`:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-03" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
||||
```
|
||||
|
||||
### Cycle 2 Database Moves
|
||||
Distribute the two active databases from `EXCH-SE-02` across the remaining healthy members:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-02" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
||||
Move-ActiveMailboxDatabase -Identity "BL-ARC-02" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
||||
```
|
||||
|
||||
### Cycle 3 Database Moves
|
||||
Distribute the two active databases from `EXCH-SE-01` across the remaining healthy members:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-01" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
||||
Move-ActiveMailboxDatabase -Identity "BL-ARC-01" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
||||
```
|
||||
|
||||
### Validate the Database Moves
|
||||
Confirm that no active database remains on the target:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" } |
|
||||
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
<no output>
|
||||
```
|
||||
|
||||
Review each move result and confirm that `Status` is `Succeeded`, `NumberOfLogsLost` is `0`, and `MountStatusAtMoveEnd` is `Mounted`. If any move fails or reports log loss, stop the cycle and investigate before entering maintenance mode.
|
||||
|
||||
## Place the Target DAG Member Into Maintenance Mode
|
||||
### Drain External Client Traffic
|
||||
If the environment uses a load balancer, reverse proxy, monitoring probe, backup job, or third-party connector that targets individual Exchange servers, drain or disable the target member according to that platform's documented procedure. Confirm that healthy remaining members are serving the traffic before continuing.
|
||||
|
||||
### Drain Hub Transport
|
||||
From Exchange Management Shell on the administrative shell host, set the target Hub Transport component to draining:
|
||||
|
||||
```powershell
|
||||
Set-ServerComponentState -Identity $TargetServer -Component HubTransport -State Draining -Requester Maintenance
|
||||
```
|
||||
|
||||
Restart the Microsoft Exchange Transport service on the target to initiate queue draining:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
Restart-Service MSExchangeTransport
|
||||
}
|
||||
```
|
||||
|
||||
### Run the DAG Maintenance Script
|
||||
From Exchange Management Shell on the administrative shell host, run the Exchange-provided DAG maintenance script:
|
||||
|
||||
```powershell
|
||||
Set-Location $ExScripts
|
||||
|
||||
.\StartDagServerMaintenance.ps1 -ServerName $TargetServer -MoveComment "Rolling Exchange update" -PauseClusterNode
|
||||
```
|
||||
|
||||
The script blocks database activation, pauses the target cluster node, moves any remaining active databases, and moves critical DAG functionality away from the target. The command can take time without producing continuous console output.
|
||||
|
||||
### Redirect Pending Transport Messages
|
||||
Redirect messages still pending on the target to the healthy server selected for the current cycle:
|
||||
|
||||
```powershell
|
||||
Redirect-Message -Server $TargetServer -Target $RedirectTargetFqdn -Confirm:$false
|
||||
```
|
||||
|
||||
### Set the Server-Wide Maintenance State
|
||||
Place the target server into Exchange server-wide maintenance mode:
|
||||
|
||||
```powershell
|
||||
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Inactive -Requester Maintenance
|
||||
```
|
||||
|
||||
### Validate Maintenance Mode
|
||||
Check effective Exchange component states:
|
||||
|
||||
```powershell
|
||||
Get-ServerComponentState -Identity $TargetServer |
|
||||
Format-Table Component, State -Auto
|
||||
```
|
||||
|
||||
`ServerWideOffline` must report `Inactive`. In the standard maintenance state, only `Monitoring` and `RecoveryActionsEnabled` should remain `Active`; investigate any other component that remains active before rebooting the server.
|
||||
|
||||
Confirm the database activation policy is blocked:
|
||||
|
||||
```powershell
|
||||
Get-MailboxServer -Identity $TargetServer |
|
||||
Format-List Name, DatabaseCopyAutoActivationPolicy
|
||||
```
|
||||
|
||||
Confirm that the target cluster node is paused:
|
||||
|
||||
```powershell
|
||||
Get-ClusterNode -Name $TargetServer |
|
||||
Format-List Name, State
|
||||
```
|
||||
|
||||
Confirm that no active databases remain on the target:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" } |
|
||||
Format-Table Name, Status, ActiveDatabaseCopy -Auto
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
<no output>
|
||||
```
|
||||
|
||||
Confirm that the target transport queues have drained:
|
||||
|
||||
```powershell
|
||||
Get-Queue -Server $TargetServer |
|
||||
Sort-Object MessageCount -Descending |
|
||||
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
||||
```
|
||||
|
||||
Confirm that the PAM is hosted by another DAG member:
|
||||
|
||||
```powershell
|
||||
Get-DatabaseAvailabilityGroup -Identity $DagName -Status |
|
||||
Format-List PrimaryActiveManager, OperationalServers
|
||||
```
|
||||
|
||||
!!! warning "Do Not Reboot Until Every Maintenance Gate Passes"
|
||||
Do not install updates or reboot the target until it has no mounted databases, its database activation policy is `Blocked`, its cluster node is `Paused`, `ServerWideOffline` is `Inactive`, its transport queues are drained, and the PAM is hosted by another member.
|
||||
|
||||
## Install Exchange and Windows Updates
|
||||
### Reboot Before Installing the Exchange Update
|
||||
A clean reboot before Exchange Setup or an Exchange SU or HU reduces failures caused by pending file handles and services that do not stop cleanly. From the administrative shell host, reboot the target:
|
||||
|
||||
```powershell
|
||||
Restart-Computer -ComputerName $TargetServer -Force
|
||||
```
|
||||
|
||||
Wait until the server is reachable, then revalidate that maintenance state persisted:
|
||||
|
||||
```powershell
|
||||
Get-ServerComponentState -Identity $TargetServer |
|
||||
Where-Object { $_.Component -eq "ServerWideOffline" } |
|
||||
Format-Table Server, Component, State -Auto
|
||||
|
||||
Get-ClusterNode -Name $TargetServer |
|
||||
Format-List Name, State
|
||||
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" -and $_.Name -like "*\$TargetServer" }
|
||||
```
|
||||
|
||||
Do not launch the update if `ServerWideOffline` is not `Inactive`, the cluster node is not `Paused`, or a database mounted on the target after reboot.
|
||||
|
||||
### Install an Exchange CU or Build Upgrade
|
||||
Mount the correct Exchange installation media on the target server. From an elevated Command Prompt on the target, run Setup by using the full media path so Windows does not invoke the installed `Setup.exe` from the Exchange binary directory.
|
||||
|
||||
=== "Diagnostic Data Off"
|
||||
|
||||
```cmd
|
||||
D:\Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF
|
||||
```
|
||||
|
||||
=== "Diagnostic Data On"
|
||||
|
||||
```cmd
|
||||
D:\Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
|
||||
```
|
||||
|
||||
Replace `D:` with the actual mounted-media drive. Wait for Exchange Setup to complete successfully and review `C:\ExchangeSetupLogs\ExchangeSetup.log` before continuing.
|
||||
|
||||
!!! danger "A CU Upgrade Is Not Reversible by Uninstalling It"
|
||||
Do not treat a CU as an ordinary removable patch. If the CU fails, keep the server in maintenance mode and repair or recover that server by following Microsoft Exchange recovery guidance.
|
||||
|
||||
### Install an Exchange SU or HU
|
||||
Run the exact update package specified by the applicable release article from an elevated Command Prompt on the target server:
|
||||
|
||||
```cmd
|
||||
C:\ExchangeUpdates\<EXCHANGE_UPDATE_FILENAME>.exe
|
||||
```
|
||||
|
||||
Replace `<EXCHANGE_UPDATE_FILENAME>` with the staged package name. Do not launch the installer from a non-elevated shell or by double-clicking it in File Explorer.
|
||||
|
||||
Do not stop IIS, WMI, Windows Event Log, or other Windows services preemptively unless the release article or a matching Microsoft troubleshooting procedure explicitly instructs you to do so. Wait for the installer to report successful completion before continuing.
|
||||
|
||||
### Install Approved Windows Updates
|
||||
Install the approved Windows Server updates while the target remains in maintenance mode. Apply prerequisites in the order required by the Exchange release notes, and continue rebooting as required until the server has no remaining approved updates or pending restart.
|
||||
|
||||
### Perform the Final Maintenance Reboot
|
||||
After Exchange and Windows updates have completed, reboot the target one final time:
|
||||
|
||||
```powershell
|
||||
Restart-Computer -ComputerName $TargetServer -Force
|
||||
```
|
||||
|
||||
Wait for Windows, Active Directory connectivity, the Cluster service, and Exchange services to initialize before beginning return-to-service validation.
|
||||
|
||||
## Return the Updated DAG Member to Service
|
||||
### Verify the Installed Exchange Build
|
||||
From Exchange Management Shell on the administrative shell host, confirm the base build:
|
||||
|
||||
```powershell
|
||||
Get-ExchangeServer -Identity $TargetServer |
|
||||
Format-Table Name, Edition, AdminDisplayVersion -Auto
|
||||
```
|
||||
|
||||
Query the local `ExSetup.exe` file on the target to identify the installed SU or HU file version:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
||||
|
||||
[PSCustomObject]@{
|
||||
Server = $env:COMPUTERNAME
|
||||
ProductVersion = $VersionInfo.ProductVersion
|
||||
FileVersion = $VersionInfo.FileVersion
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Compare both values with the intended Microsoft build. An unchanged `AdminDisplayVersion` does not, by itself, prove that an SU or HU failed to install.
|
||||
|
||||
### Validate Exchange Services
|
||||
From the administrative shell host, run:
|
||||
|
||||
```powershell
|
||||
Test-ServiceHealth -Server $TargetServer
|
||||
```
|
||||
|
||||
Inspect Exchange-related services on the target:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
Get-CimInstance Win32_Service |
|
||||
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" } |
|
||||
Select-Object Name, DisplayName, State, StartMode, ExitCode
|
||||
} | Format-Table -Auto
|
||||
```
|
||||
|
||||
Compare the result with `C:\ExchangeUpdates\PreUpdate-Services.csv` from that same server. Do not enable a service merely because it is stopped; protocol services and role-specific services may intentionally be manual or stopped.
|
||||
|
||||
!!! warning "Keep the Server in Maintenance Mode During Repair"
|
||||
If required Exchange services are disabled, fail to start, or report dependency errors, keep the target in maintenance mode. Repair the service state and complete the failed update before running the return-to-service commands.
|
||||
|
||||
### Remove the Server-Wide Maintenance State
|
||||
From Exchange Management Shell on the administrative shell host, restore the server-wide component state:
|
||||
|
||||
```powershell
|
||||
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Active -Requester Maintenance
|
||||
```
|
||||
|
||||
### Run the DAG Return-to-Service Script
|
||||
Run the Exchange-provided DAG maintenance exit script:
|
||||
|
||||
```powershell
|
||||
Set-Location $ExScripts
|
||||
|
||||
.\StopDagServerMaintenance.ps1 -ServerName $TargetServer
|
||||
```
|
||||
|
||||
The script resumes the cluster node, restores database activation policy, and resumes database copies hosted by the member.
|
||||
|
||||
### Resume Hub Transport
|
||||
Return Hub Transport to active state:
|
||||
|
||||
```powershell
|
||||
Set-ServerComponentState -Identity $TargetServer -Component HubTransport -State Active -Requester Maintenance
|
||||
```
|
||||
|
||||
Restart transport on the target:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
Restart-Service MSExchangeTransport
|
||||
}
|
||||
```
|
||||
|
||||
### Validate the Restored Member
|
||||
Confirm Exchange component states:
|
||||
|
||||
```powershell
|
||||
Get-ServerComponentState -Identity $TargetServer |
|
||||
Format-Table Component, State -Auto
|
||||
```
|
||||
|
||||
Confirm database activation is unrestricted:
|
||||
|
||||
```powershell
|
||||
Get-MailboxServer -Identity $TargetServer |
|
||||
Format-List Name, DatabaseCopyAutoActivationPolicy
|
||||
```
|
||||
|
||||
Confirm the cluster node is up:
|
||||
|
||||
```powershell
|
||||
Get-ClusterNode -Name $TargetServer |
|
||||
Format-List Name, State
|
||||
```
|
||||
|
||||
Run service and replication health:
|
||||
|
||||
```powershell
|
||||
Test-ServiceHealth -Server $TargetServer
|
||||
Test-ReplicationHealth -Identity $TargetServer
|
||||
```
|
||||
|
||||
Inspect transport queues:
|
||||
|
||||
```powershell
|
||||
Get-Queue -Server $TargetServer |
|
||||
Sort-Object MessageCount -Descending |
|
||||
Format-Table Identity, Status, MessageCount, NextHopDomain -Auto
|
||||
```
|
||||
|
||||
Do not restore the target to a load balancer or other external traffic source until required Exchange components are active, required services are running, the cluster node is up, replication health passes, and transport queues are processing normally.
|
||||
|
||||
### Restore External Client Traffic
|
||||
Return the target to its load balancer pool, monitoring platform, backup schedule, mail gateway, and any other system that was intentionally drained. Confirm the external health checks recognize the server as healthy before restoring active mailbox databases to it.
|
||||
|
||||
## Restore the Intended Database Placement
|
||||
### Wait for Database Copies to Catch Up
|
||||
Before moving an active database back to the updated member, confirm every copy is healthy and all queues have drained:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Sort-Object Name |
|
||||
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
||||
```
|
||||
|
||||
Do not activate a copy on the updated member while it is `Failed`, `Suspended`, `Disconnected`, `ServiceDown`, or building a persistent queue.
|
||||
|
||||
!!! warning "Run Only the Current Cycle's Restore Block"
|
||||
The following restore blocks are cycle-specific. Do not move databases for a different cycle.
|
||||
|
||||
### Cycle 1 Database Restore
|
||||
Move `BL-MBX-03` back to `EXCH-SE-03`:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-03" -ActivateOnServer "EXCH-SE-03" -Confirm:$false
|
||||
```
|
||||
|
||||
### Cycle 2 Database Restore
|
||||
Move both databases back to `EXCH-SE-02`:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-02" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
||||
Move-ActiveMailboxDatabase -Identity "BL-ARC-02" -ActivateOnServer "EXCH-SE-02" -Confirm:$false
|
||||
```
|
||||
|
||||
### Cycle 3 Database Restore
|
||||
Move both databases back to `EXCH-SE-01`:
|
||||
|
||||
```powershell
|
||||
Move-ActiveMailboxDatabase -Identity "BL-MBX-01" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
||||
Move-ActiveMailboxDatabase -Identity "BL-ARC-01" -ActivateOnServer "EXCH-SE-01" -Confirm:$false
|
||||
```
|
||||
|
||||
### Validate the Restored Placement
|
||||
Confirm active database placement:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" } |
|
||||
Sort-Object ActiveDatabaseCopy, Name |
|
||||
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
||||
```
|
||||
|
||||
Expected Bunny Lab placement:
|
||||
|
||||
```text
|
||||
Name Status ActiveDatabaseCopy CopyQueueLength ReplayQueueLength
|
||||
---- ------ ------------------ --------------- -----------------
|
||||
BL-ARC-01\EXCH-SE-01 Mounted EXCH-SE-01 0 0
|
||||
BL-MBX-01\EXCH-SE-01 Mounted EXCH-SE-01 0 0
|
||||
BL-ARC-02\EXCH-SE-02 Mounted EXCH-SE-02 0 0
|
||||
BL-MBX-02\EXCH-SE-02 Mounted EXCH-SE-02 0 0
|
||||
BL-MBX-03\EXCH-SE-03 Mounted EXCH-SE-03 0 0
|
||||
```
|
||||
|
||||
Temporary queues may appear immediately after activation. Wait until all copy and replay queues return to `0` before declaring the cycle complete.
|
||||
|
||||
## Validate the Completed Cycle
|
||||
Run replication health against every DAG member:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
Test-ReplicationHealth -Identity $_
|
||||
}
|
||||
```
|
||||
|
||||
Check all database copies:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Sort-Object Name |
|
||||
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
||||
```
|
||||
|
||||
Run the Exchange Health Checker against the updated member:
|
||||
|
||||
```powershell
|
||||
& "C:\Scripts\HealthChecker.ps1" -Server $TargetServer
|
||||
```
|
||||
|
||||
The cycle is complete only when:
|
||||
|
||||
- The target reports the intended Exchange build
|
||||
- Required Exchange services pass `Test-ServiceHealth`
|
||||
- The target cluster node is `Up`
|
||||
- Database activation policy is `Unrestricted`
|
||||
- Required Exchange components are active
|
||||
- Every applicable replication-health check returns `Passed`
|
||||
- Every active database copy reports `Mounted`
|
||||
- Every passive database copy reports `Healthy`
|
||||
- Every copy and replay queue is `0`
|
||||
- Transport queues are processing normally
|
||||
- The intended active database placement is restored
|
||||
- External health checks and monitoring report the target as healthy
|
||||
- The post-update Health Checker report contains no unresolved update-blocking finding
|
||||
|
||||
Do not begin the next cycle until all conditions are satisfied. Repeat the common workflow with the next cycle's variables and database move block.
|
||||
|
||||
## Final DAG Validation
|
||||
### Confirm a Consistent Exchange Build
|
||||
After all three cycles are complete, confirm the base build recorded for every DAG member:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
Get-ExchangeServer -Identity $_
|
||||
} | Format-Table Name, Edition, AdminDisplayVersion -Auto
|
||||
```
|
||||
|
||||
Confirm the local `ExSetup.exe` file version on every member:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $DagMembers -ScriptBlock {
|
||||
$VersionInfo = (Get-Item (Join-Path $env:ExchangeInstallPath "bin\ExSetup.exe")).VersionInfo
|
||||
|
||||
[PSCustomObject]@{
|
||||
Server = $env:COMPUTERNAME
|
||||
ProductVersion = $VersionInfo.ProductVersion
|
||||
FileVersion = $VersionInfo.FileVersion
|
||||
}
|
||||
} | Sort-Object Server | Format-Table -Auto
|
||||
```
|
||||
|
||||
All three servers must report the same intended Exchange build unless Microsoft explicitly documents a temporary mixed-build state during the active maintenance window.
|
||||
|
||||
### Validate Database Health and Placement
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Sort-Object Name |
|
||||
Format-Table Name, Status, CopyQueueLength, ReplayQueueLength, ContentIndexState -Auto
|
||||
|
||||
Get-MailboxDatabaseCopyStatus * |
|
||||
Where-Object { $_.Status -eq "Mounted" } |
|
||||
Sort-Object ActiveDatabaseCopy, Name |
|
||||
Format-Table Name, Status, ActiveDatabaseCopy, CopyQueueLength, ReplayQueueLength -Auto
|
||||
```
|
||||
|
||||
Confirm every active copy is `Mounted`, every passive copy is `Healthy`, all queues are `0`, and placement matches the Bunny Lab topology table.
|
||||
|
||||
### Validate Replication, Services, Components, and Cluster State
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
Test-ServiceHealth -Server $_
|
||||
Test-ReplicationHealth -Identity $_
|
||||
}
|
||||
|
||||
Get-ClusterNode |
|
||||
Format-Table Name, State, NodeWeight, DynamicWeight -Auto
|
||||
|
||||
$DagMembers | ForEach-Object {
|
||||
Get-ServerComponentState -Identity $_ |
|
||||
Where-Object { $_.State -ne "Active" } |
|
||||
Select-Object Server, Component, State
|
||||
}
|
||||
```
|
||||
|
||||
Investigate every unexpected non-active component state. Protocol components that are deliberately disabled must match the documented Bunny Lab design rather than being assumed healthy merely because they were disabled before the update.
|
||||
|
||||
### Validate Mail Flow
|
||||
From Exchange Management Shell on each source server, test mail flow to another DAG member. The following examples cover all three members:
|
||||
|
||||
```powershell
|
||||
Test-Mailflow -Identity "EXCH-SE-01" -TargetMailboxServer "EXCH-SE-02"
|
||||
Test-Mailflow -Identity "EXCH-SE-02" -TargetMailboxServer "EXCH-SE-03"
|
||||
Test-Mailflow -Identity "EXCH-SE-03" -TargetMailboxServer "EXCH-SE-01"
|
||||
```
|
||||
|
||||
Each test must return `Success`. Also validate inbound and outbound mail flow through the organization's real mail gateways and confirm the client-access namespaces used by the environment are healthy.
|
||||
|
||||
### Run the Final Health Checker Audit
|
||||
Run the current Exchange Health Checker against every DAG member:
|
||||
|
||||
```powershell
|
||||
$DagMembers | ForEach-Object {
|
||||
& "C:\Scripts\HealthChecker.ps1" -Server $_
|
||||
}
|
||||
```
|
||||
|
||||
Archive the final reports with the change record.
|
||||
|
||||
## Troubleshooting
|
||||
### The Installer Reports Files in Use or Cannot Stop Services
|
||||
Do not select an installer option that ignores locked files, and do not terminate the Windows Event Log service. Exit the installer, confirm the server remains in maintenance mode, reboot it, and rerun the update from an elevated Command Prompt.
|
||||
|
||||
If the problem persists:
|
||||
|
||||
- Confirm the update package matches the installed CU
|
||||
- Confirm the server was rebooted immediately before the installation attempt
|
||||
- Review the selected update's known issues and antivirus exclusion guidance
|
||||
- Preserve `C:\ExchangeSetupLogs`
|
||||
- Use the Microsoft SetupAssist or Setup Log Reviewer tooling appropriate to the failure
|
||||
- Follow the matching procedure in [Fix Failed Exchange Server Updates](https://learn.microsoft.com/en-us/troubleshoot/exchange/client-connectivity/exchange-security-update-issues)
|
||||
|
||||
Do not reuse process IDs from an earlier attempt or another server. Process IDs are transient, and terminating an unidentified Windows or Exchange process can leave the installation in a worse state.
|
||||
|
||||
### Exchange Services Are Disabled After the Update
|
||||
Compare the current service configuration with `C:\ExchangeUpdates\PreUpdate-Services.csv` from the same server:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
$Baseline = Import-Csv "C:\ExchangeUpdates\PreUpdate-Services.csv"
|
||||
$Current = Get-CimInstance Win32_Service |
|
||||
Where-Object { $_.Name -like "MSExchange*" -or $_.Name -eq "FMS" }
|
||||
|
||||
foreach ($Before in $Baseline) {
|
||||
$After = $Current | Where-Object { $_.Name -eq $Before.Name }
|
||||
|
||||
if ($After -and ($After.State -ne $Before.State -or $After.StartMode -ne $Before.StartMode)) {
|
||||
[PSCustomObject]@{
|
||||
Name = $Before.Name
|
||||
BeforeState = $Before.State
|
||||
AfterState = $After.State
|
||||
BeforeStartMode = $Before.StartMode
|
||||
AfterStartMode = $After.StartMode
|
||||
}
|
||||
}
|
||||
}
|
||||
} | Format-Table -Auto
|
||||
```
|
||||
|
||||
Restore only a service whose required startup mode is confirmed by the server's baseline, current Exchange role, and Microsoft guidance:
|
||||
|
||||
```powershell
|
||||
Set-Service -Name <SERVICE_NAME> -StartupType Automatic
|
||||
Start-Service -Name <SERVICE_NAME>
|
||||
```
|
||||
|
||||
If the service fails with dependency error `1068`, inspect its required services:
|
||||
|
||||
```powershell
|
||||
Get-Service -Name <SERVICE_NAME> -RequiredServices |
|
||||
Format-Table Name, DisplayName, Status, StartType -Auto
|
||||
```
|
||||
|
||||
Correct the failed dependency before retrying the dependent service. Do not automatically enable IMAP, POP, EdgeSync, or another optional service that was intentionally disabled before the update.
|
||||
|
||||
### An Exchange Component Remains Inactive
|
||||
Inspect the effective and requester-specific component states:
|
||||
|
||||
```powershell
|
||||
Get-ServerComponentState -Identity $TargetServer |
|
||||
Format-Table Component, State -Auto
|
||||
|
||||
Get-ServerComponentState -Identity $TargetServer -Component <COMPONENT_NAME> |
|
||||
Format-List Component, State, LocalStates, RemoteStates
|
||||
```
|
||||
|
||||
Correct the requester that actually holds the component inactive. Do not repeatedly issue `Requester Maintenance` commands when the inactive state belongs to `Functional`, `HealthAPI`, or another requester.
|
||||
|
||||
If a failed Exchange update left `ServerWideOffline`, `Monitoring`, or `RecoveryActionsEnabled` inactive under the `Functional` requester, and the failed installation has already been repaired, restore only the affected states:
|
||||
|
||||
```powershell
|
||||
Set-ServerComponentState -Identity $TargetServer -Component ServerWideOffline -State Active -Requester Functional
|
||||
Set-ServerComponentState -Identity $TargetServer -Component Monitoring -State Active -Requester Functional
|
||||
Set-ServerComponentState -Identity $TargetServer -Component RecoveryActionsEnabled -State Active -Requester Functional
|
||||
```
|
||||
|
||||
Rerun `Test-ServiceHealth` and `Test-ReplicationHealth` after the correction.
|
||||
|
||||
### High Availability Remains Offline After Maintenance Ends
|
||||
If a database move fails with an error stating that the `HighAvailability` component is offline, inspect its requester-specific state:
|
||||
|
||||
```powershell
|
||||
Get-ServerComponentState -Identity $TargetServer -Component HighAvailability |
|
||||
Format-List Component, State, LocalStates, RemoteStates
|
||||
```
|
||||
|
||||
Confirm `StopDagServerMaintenance.ps1` completed successfully, the cluster node is `Up`, database activation is `Unrestricted`, and `MSExchangeRepl` is running. If every requester reports `Active` but Active Manager state remains stale, restart the replication service on the target:
|
||||
|
||||
```powershell
|
||||
Invoke-Command -ComputerName $TargetServer -ScriptBlock {
|
||||
Restart-Service MSExchangeRepl
|
||||
Get-Service MSExchangeRepl
|
||||
}
|
||||
```
|
||||
|
||||
Rerun replication health and retry the database move only after every applicable check passes.
|
||||
|
||||
### Exchange Setup Reports Success but the Build Does Not Change
|
||||
Confirm Setup was launched from the mounted media by using an absolute path such as `D:\Setup.exe` or, from PowerShell in the media root, `.\Setup.exe`. Running only `Setup.exe` can invoke the installed copy under the Exchange binary path instead of the intended installation media.
|
||||
|
||||
Review `C:\ExchangeSetupLogs\ExchangeSetup.log`, correct the launch path, and rerun Setup while the server remains in maintenance mode.
|
||||
|
||||
### Outlook on the Web or the Exchange Admin Center Fails After the Update
|
||||
First confirm the Exchange update completed successfully and required services are running. Review the matching symptom in Microsoft's failed-update guidance rather than running a generic post-update command sequence.
|
||||
|
||||
For an IIS state that specifically requires a service restart, run from an elevated PowerShell session on the affected server:
|
||||
|
||||
```powershell
|
||||
Restart-Service -Name WAS, W3SVC
|
||||
```
|
||||
|
||||
Do not make `UpdateCas.ps1`, `UpdateConfigFiles.ps1`, `IISADMIN`, ADSI changes, or forced process termination part of the normal update workflow. Use a repair command only when an authoritative troubleshooting procedure identifies the same failure condition and explains the required validation.
|
||||
|
||||
## Recovery and Stop Conditions
|
||||
If a target server cannot be returned to service:
|
||||
|
||||
- Keep `ServerWideOffline` inactive and keep the cluster node paused
|
||||
- Keep database activation blocked on the failed member
|
||||
- Leave active databases on healthy DAG members
|
||||
- Do not begin maintenance on another member
|
||||
- Preserve Exchange Setup logs, Windows event logs, Health Checker reports, and the pre-update service baseline
|
||||
- Repair the update or perform the documented Exchange server recovery procedure
|
||||
- Revalidate quorum, database redundancy, transport, client access, and backup status before resuming the rolling upgrade
|
||||
|
||||
The rolling update is complete only after all three members report the intended Exchange build, every required health check passes, the documented database placement is restored, and Bunny Lab mail flow and client access are validated end to end.
|
||||
|
||||
## Reference Documentation
|
||||
- [Manage Database Availability Groups in Exchange Server](https://learn.microsoft.com/en-us/exchange/high-availability/manage-ha/manage-dags)
|
||||
- [Upgrade Exchange to the Latest Cumulative Update](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/install-cumulative-updates)
|
||||
- [Use Unattended Mode in Exchange Setup](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/deploy-new-installations/unattended-installs)
|
||||
- [Exchange Server Build Numbers and Release Dates](https://learn.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates)
|
||||
- [Exchange Server Update FAQ](https://learn.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/security-best-practices/exchange-server-update-faq)
|
||||
- [Fix Failed Exchange Server Updates](https://learn.microsoft.com/en-us/troubleshoot/exchange/client-connectivity/exchange-security-update-issues)
|
||||
- [Exchange Server Health Checker](https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/)
|
||||
|
||||
## Related Documentation
|
||||
- [Related Email Documentation](<../../../../reference/Applications/Email/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
tags:
|
||||
- Microsoft Exchange
|
||||
- Email
|
||||
---
|
||||
|
||||
## Purpose
|
||||
This document is meant to be an abstract guide on what to do before installing Cumulative Updates on Microsoft Exchange Server. There are a few considerations that need to be made ahead of time. This list was put together through shere brute-force while troubleshooting an update issue for a server on 12/16/2024.
|
||||
|
||||
!!! abstract "Overview"
|
||||
We are looking to add an administrative user to several domain security groups, adjust local security policy to put them into the "Manage Auditing and Security Logs" security policy, and run the setup.exe included on the Cumulative Update ISO images within a `SeSecurityPrivilege` operational context.
|
||||
|
||||
## Domain Group Membership
|
||||
You have to be logged in with a domain user that possesses the following domain group memberships, if these group memberships are missing, the upgrade process will fail.
|
||||
|
||||
- `Enterprise Admins`
|
||||
- `Schema Admins`
|
||||
- `Organization Management`
|
||||
|
||||
## User Rights Management
|
||||
You have to be part of the "**Local Policies > User Rights Assignment > "Manage Auditing and Security Logs**" security policy. You can set this via group policy management or locally on the Exchange server via `secpol.msc`. This is required for the "Monitoring Tools" portion of the upgrade.
|
||||
|
||||
It's recommended to reboot the server after making this change to be triple-sure that everything was applied correctly.
|
||||
|
||||
!!! note "Security Policy Only Required on Exchange Server"
|
||||
While the `Enterprise Admins`, `Schema Admins`, and `Organization Management` security group memberships are required on a domain-wide level, the security policy membership for "Manage Auditing and Security Logs" mentioned above is only required on the Exchange Server itself. You can create a group policy that only targets the Exchange Server to add this, or you can make your user a domain-wide member of "Manage Auditing and Security Logs" (Optional). If no existing policies are in-place affecting the Exchange server, you can just use `secpol.msc` to manually add your user to this security policy for the duration of the upgrade/update (or leave it there for future updates).
|
||||
|
||||
## Running Updater within `SeSecurityPrivilege` Operational Context
|
||||
At this point, you would technically be ready to invoke `setup.exe` on the Cumulative Update ISO image to launch the upgrade process, but we are going to go the extra mile to manually "Enable" the `SeSecurityPrivilege` within a Powershell session, then use that same session to invoke the `setup.exe` so the updater runs within that context. This is not really necessary, but something I added as a "hail mary" to make the upgrade successful.
|
||||
|
||||
### Open Powershell ISE
|
||||
The first thing we are going to do, is open the Powershell ISE so we can copy/paste the following powershell script, this script will explicitely enable `SeSecurityPrivilege` for anyone who holds that privilege within the powershell session.
|
||||
|
||||
!!! warning "Run Powershell ISE as Administrator"
|
||||
In order for everything to work correctly, the ISE has to be launched by right-clicking "Run as Administrator", otherwise it is guarenteed that the updater application will fail at some point.
|
||||
|
||||
```powershell title="SeSecurityPrivilege Enablement Script"
|
||||
# Create a Privilege Adjustment
|
||||
$definition = @"
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
public class Privilege
|
||||
{
|
||||
const int SE_PRIVILEGE_ENABLED = 0x00000002;
|
||||
const int TOKEN_ADJUST_PRIVILEGES = 0x0020;
|
||||
const int TOKEN_QUERY = 0x0008;
|
||||
const string SE_SECURITY_NAME = "SeSecurityPrivilege";
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true)]
|
||||
public static extern bool OpenProcessToken(IntPtr ProcessHandle, int DesiredAccess, out IntPtr TokenHandle);
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
public static extern bool LookupPrivilegeValue(string lpSystemName, string lpName, out long lpLuid);
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true)]
|
||||
public static extern bool AdjustTokenPrivileges(IntPtr TokenHandle, bool DisableAllPrivileges, ref TOKEN_PRIVILEGES NewState, int BufferLength, IntPtr PreviousState, IntPtr ReturnLength);
|
||||
|
||||
[StructLayout(LayoutKind.Sequential, Pack = 1)]
|
||||
public struct TOKEN_PRIVILEGES
|
||||
{
|
||||
public int PrivilegeCount;
|
||||
public long Luid;
|
||||
public int Attributes;
|
||||
}
|
||||
|
||||
public static bool EnablePrivilege()
|
||||
{
|
||||
IntPtr tokenHandle;
|
||||
TOKEN_PRIVILEGES tokenPrivileges;
|
||||
|
||||
if (!OpenProcessToken(System.Diagnostics.Process.GetCurrentProcess().Handle, TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, out tokenHandle))
|
||||
return false;
|
||||
|
||||
if (!LookupPrivilegeValue(null, SE_SECURITY_NAME, out tokenPrivileges.Luid))
|
||||
return false;
|
||||
|
||||
tokenPrivileges.PrivilegeCount = 1;
|
||||
tokenPrivileges.Attributes = SE_PRIVILEGE_ENABLED;
|
||||
|
||||
return AdjustTokenPrivileges(tokenHandle, false, ref tokenPrivileges, 0, IntPtr.Zero, IntPtr.Zero);
|
||||
}
|
||||
}
|
||||
"@
|
||||
|
||||
Add-Type -TypeDefinition $definition
|
||||
[Privilege]::EnablePrivilege()
|
||||
```
|
||||
|
||||
### Validate Privilege
|
||||
At this point, we now have a powershell session operating with the `SeSecurityPrivilege` privilege enabled. We want to confirm this by running the following commands:
|
||||
|
||||
```powershell
|
||||
whoami # (1)
|
||||
whoami /priv # (2)
|
||||
```
|
||||
|
||||
1. Output will appear similar to "bunny-lab\nicole.rappe", prefixing the username of the person running the command with the domain they belong to.
|
||||
2. Reference the privilege table seen below to validate the output of this command matches what you see below.
|
||||
|
||||
| **Privilege Name** | **Description** | **State** |
|
||||
| :--- | :--- | :--- |
|
||||
| `SeSecurityPrivilege` | Manage auditing and security log | Enabled |
|
||||
|
||||
### Execute `setup.exe`
|
||||
Finally, at the last stage, we mount the ISO file for the Cumulative Update ISO (e.g. 6.6GB ISO image), and using this powershell session we made above, we navigate to the drive it is running on, and invoke setup.exe, causing it to run under the `SeSecurityPrivilege` operational state.
|
||||
|
||||
```powershell
|
||||
D: <ENTER> # (1)
|
||||
.\Setup.EXE /m:upgrade /IAcceptExchangeServersLicenseTerms_DiagnosticDataON # (2)
|
||||
```
|
||||
|
||||
1. Replace this drive letter with whatever letter was assigned when you mounted the ISO image for the Exchange Updater.
|
||||
2. This launches the Exchange updater application. Be patient and give it time to launch. At this point, you should be good to proceed with the update. You can optionally change the argument to `/IAcceptExchangeServersLicenseTerms_DiagnosticDataOFF` if you do not need diagnostic data.
|
||||
|
||||
!!! success "Ready to Proceed with Updating Exchange"
|
||||
At this point, after doing the three sections above, you should be safe to do the upgrade/update of Microsoft Exchange Server. The installer will run its own readiness checks for other aspects such as IIS Rewrite Modules and will give you a link to download / upgrade it separately, then giving you the option to "**Retry**" after installing the module for the installer to re-check and proceed.
|
||||
|
||||
## Post-Update Health Checks
|
||||
After the update(s) are installed, you will likely want to check to ensure things are healthy and operational, validating mail flow in both directions, running `Get-Queue` to check for backlogged emails, etc.
|
||||
|
||||
!!! note "Under Construction"
|
||||
This section is under construction and will be based on some feedback from others to help build the section out.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Email Documentation](<../../../../reference/Applications/Email/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+329
@@ -0,0 +1,329 @@
|
||||
---
|
||||
tags:
|
||||
- Proxmox Mail Gateway
|
||||
- Mailcow
|
||||
- Email
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Use this workflow when PMG can receive mail but delivery or quarantine release to Mailcow is rejected by sender validation or relay trust settings. It assumes the PMG and Mailcow integration already exists.
|
||||
|
||||
## Scope
|
||||
This workflow describes how to configure Mailcow to trust Proxmox Mail Gateway (PMG) as an upstream SMTP relay after PMG has been placed in front of Mailcow for inbound filtering. It prevents Mailcow from applying sender-domain validation and duplicate spam filtering to messages that PMG has already accepted, including messages released from the PMG quarantine.
|
||||
|
||||
In the current Bunny Lab design, this trust applies to the inbound delivery path from `192.168.3.15` to `192.168.3.61`. Mailcow continues to send outbound mail directly to the Internet and does not relay outbound mail through PMG.
|
||||
|
||||
!!! info "Assumptions"
|
||||
- Mailcow is already deployed at `192.168.3.61`.
|
||||
- PMG is already deployed at `192.168.3.15`.
|
||||
- Public inbound SMTP port `25` is forwarded to PMG.
|
||||
- PMG relays accepted inbound mail to Mailcow on `192.168.3.61:25`.
|
||||
- Mailcow remains responsible for mailbox hosting, authenticated submission, DKIM signing, and outbound delivery.
|
||||
- You have `root` access to the Mailcow host.
|
||||
|
||||
!!! warning "Trust Only the PMG Host"
|
||||
Add only `192.168.3.15/32` to Mailcow's trusted networks. Do not trust the entire `192.168.3.0/24` subnet unless every system on that subnet is authorized to relay mail through Mailcow without authentication.
|
||||
|
||||
## Architecture
|
||||
```text
|
||||
Internet
|
||||
|
|
||||
v
|
||||
pfSense WAN :25
|
||||
|
|
||||
v
|
||||
PMG 192.168.3.15:25
|
||||
|
|
||||
v
|
||||
Mailcow 192.168.3.61:25
|
||||
|
|
||||
v
|
||||
User Mailbox
|
||||
```
|
||||
|
||||
PMG is the primary inbound spam and virus filtering authority. Mailcow should accept the SMTP session from PMG as a trusted relay and avoid repeating edge filtering or rejecting PMG-generated envelope senders.
|
||||
|
||||
## Symptoms
|
||||
Use this workflow when PMG receives and processes mail correctly, but Mailcow rejects or defers the final delivery.
|
||||
|
||||
Common symptoms include:
|
||||
|
||||
- A released PMG quarantine message remains in the PMG deferred queue
|
||||
- Clicking **Flush** in PMG does not deliver the message
|
||||
- PMG repeatedly attempts delivery to `192.168.3.61:25`
|
||||
- Mailcow rejects a PMG-generated envelope sender such as `postmaster@lab-mail-gw-01.bunny-lab.io`
|
||||
- PMG reports an SMTP response similar to:
|
||||
|
||||
```text
|
||||
450 4.1.8 Sender address rejected: Domain not found
|
||||
```
|
||||
|
||||
The visible message sender may be valid even when Mailcow rejects the PMG-generated envelope sender used during quarantine release.
|
||||
|
||||
## Confirm the PMG Source Address
|
||||
Mailcow must trust the source address it actually sees on the SMTP connection. Confirm that address before changing the configuration.
|
||||
|
||||
On the Mailcow host, run:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose logs --since 5m postfix-mailcow
|
||||
```
|
||||
|
||||
Flush or resend a test message from PMG, then locate the connection line.
|
||||
|
||||
Expected connection source:
|
||||
|
||||
```text
|
||||
connect from lab-mail-gw-01[192.168.3.15]
|
||||
```
|
||||
|
||||
If Mailcow sees a different address because of NAT, a load balancer, or another SMTP proxy, use that observed address instead of `192.168.3.15`.
|
||||
|
||||
## Configure Mailcow Forwarding Hosts
|
||||
Configure PMG as a trusted forwarding host in the Mailcow WebUI.
|
||||
|
||||
- Navigate to "**System > Configuration > Options > Forwarding Hosts**"
|
||||
- Add `192.168.3.15`
|
||||
- Set the forwarding-host spam filter option to **Inactive**
|
||||
- Save the configuration
|
||||
|
||||
!!! note "Forwarding Host Behavior"
|
||||
The forwarding-host entry tells Mailcow that PMG is the immediate upstream SMTP relay. Keeping the forwarding-host spam filter inactive avoids applying a second aggressive spam-filtering layer to mail that PMG has already inspected.
|
||||
|
||||
## Configure Postfix Trusted Networks
|
||||
The forwarding-host entry does not replace Postfix `mynetworks`. Add PMG to `mynetworks` so Postfix treats SMTP sessions from PMG as trusted and evaluates `permit_mynetworks` before sender-domain restrictions.
|
||||
|
||||
### Inspect the Existing Trusted Networks
|
||||
Before modifying `mynetworks`, determine the currently active value. Mailcow automatically populates `mynetworks` with its loopback and Docker networks. Defining your own value replaces that automatically generated configuration, so you must preserve the existing entries.
|
||||
|
||||
On the Mailcow host, run:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose exec postfix-mailcow postconf mynetworks
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
mynetworks = 127.0.0.0/8 172.22.1.0/24 [::1]/128
|
||||
```
|
||||
|
||||
At this point, append the PMG address as a single-host CIDR. Do not remove any existing networks.
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
mynetworks = 127.0.0.0/8 172.22.1.0/24 [::1]/128 192.168.3.15/32
|
||||
```
|
||||
|
||||
### Update the Persistent Postfix Override
|
||||
Edit the Postfix override file:
|
||||
|
||||
```sh
|
||||
nano /opt/mailcow-dockerized/data/conf/postfix/extra.cf
|
||||
```
|
||||
|
||||
If `mynetworks` config line already exists, append the PMG address while preserving the existing values. If the key does not exist, create it using the value discovered in the previous step.
|
||||
|
||||
```ini title="/opt/mailcow-dockerized/data/conf/postfix/extra.cf"
|
||||
mynetworks = 127.0.0.0/8 172.22.1.0/24 [::1]/128 192.168.3.15/32
|
||||
```
|
||||
|
||||
!!! warning "Do Not Replace Existing Networks"
|
||||
The `mynetworks` directive replaces Mailcow's automatically generated value. Removing the existing loopback or Docker networks may prevent Postfix from functioning correctly. Always preserve the existing entries and append the PMG address.
|
||||
|
||||
### Restart Mailcow Postfix
|
||||
Apply the persistent override:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose restart postfix-mailcow
|
||||
```
|
||||
|
||||
## Validate the Trusted Relay Configuration
|
||||
Confirm the active Postfix configuration after the restart:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose exec postfix-mailcow postconf mynetworks
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
mynetworks = 127.0.0.0/8 172.22.1.0/24 [::1]/128 192.168.3.15/32
|
||||
```
|
||||
|
||||
Confirm the active SMTP restriction order includes `permit_mynetworks`:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose exec postfix-mailcow postconf \
|
||||
smtpd_sender_restrictions \
|
||||
smtpd_recipient_restrictions \
|
||||
smtpd_relay_restrictions
|
||||
```
|
||||
|
||||
The exact restriction lists may change between Mailcow releases. Confirm that `permit_mynetworks` remains present and that the effective configuration recognizes `192.168.3.15/32` as trusted.
|
||||
|
||||
## Retry Deferred PMG Mail
|
||||
After Mailcow trusts PMG, retry the deferred message from the PMG WebUI.
|
||||
|
||||
- Navigate to "**Administration > Queue Administration > Deferred Mail**"
|
||||
- Select the deferred message
|
||||
- Click **Flush**
|
||||
|
||||
Alternatively, run the following on PMG:
|
||||
|
||||
```sh
|
||||
postqueue -f
|
||||
```
|
||||
|
||||
To retry one specific queue item:
|
||||
|
||||
```sh
|
||||
postsuper -r <QUEUE_ID>
|
||||
postqueue -f
|
||||
```
|
||||
|
||||
## Validate Mail Delivery
|
||||
Monitor Mailcow while PMG retries the message:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose logs -f postfix-mailcow
|
||||
```
|
||||
|
||||
Mailcow should accept the SMTP transaction and return a successful queue response similar to:
|
||||
|
||||
```text
|
||||
250 2.0.0 Ok: queued as <MAILCOW_QUEUE_ID>
|
||||
```
|
||||
|
||||
On PMG, confirm the deferred queue no longer contains the message:
|
||||
|
||||
```sh
|
||||
postqueue -p
|
||||
```
|
||||
|
||||
Confirm the message is present in the destination Mailcow mailbox.
|
||||
|
||||
!!! success "Trusted PMG Delivery Confirmed"
|
||||
The workflow is complete when Mailcow accepts mail from `192.168.3.15`, the PMG deferred queue clears, and the destination mailbox receives the message.
|
||||
|
||||
## Security Boundaries
|
||||
Adding `192.168.3.15/32` to `mynetworks` permits PMG to relay mail through Mailcow without SMTP authentication. This is appropriate only while PMG remains a controlled gateway host.
|
||||
|
||||
Maintain the following boundaries:
|
||||
|
||||
- Restrict the trusted entry to `192.168.3.15/32`
|
||||
- Prevent other hosts from impersonating the PMG source address
|
||||
- Keep Mailcow port `25` restricted to expected SMTP sources where practical
|
||||
- Confirm PMG is not configured as an unrestricted open relay
|
||||
- Continue routing authenticated client submission directly to Mailcow on ports `465` and `587`
|
||||
- Continue routing public inbound SMTP port `25` to PMG rather than directly to Mailcow
|
||||
|
||||
## Troubleshooting
|
||||
### Mailcow Still Returns `450 4.1.8`
|
||||
Confirm Mailcow is using the updated configuration:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose exec postfix-mailcow postconf mynetworks
|
||||
```
|
||||
|
||||
If `192.168.3.15/32` is missing, verify `/opt/mailcow-dockerized/data/conf/postfix/extra.cf` and restart `postfix-mailcow`.
|
||||
|
||||
### Mailcow Sees a Different Source Address
|
||||
Inspect the Mailcow Postfix logs:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose logs --since 5m postfix-mailcow
|
||||
```
|
||||
|
||||
Use the IP address shown in the `connect from ...[IP_ADDRESS]` line. Do not assume Mailcow sees the PMG management address when NAT or an SMTP proxy exists between the systems.
|
||||
|
||||
### PMG Flushes the Message but the Mailbox Does Not Receive It
|
||||
Determine whether Mailcow accepted the message.
|
||||
|
||||
If Mailcow returned `250 2.0.0`, the PMG-to-Mailcow relay succeeded. Continue troubleshooting inside Mailcow by reviewing Postfix, Rspamd, and Dovecot logs.
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose logs --since 10m postfix-mailcow rspamd-mailcow dovecot-mailcow
|
||||
```
|
||||
|
||||
If Mailcow returned a `4xx` or `5xx` response, use the complete SMTP response as the controlling error and resolve that policy or recipient failure before retrying again.
|
||||
|
||||
### Mailcow Applies Spam Filtering Again
|
||||
Confirm the PMG entry under "**System > Configuration > Options > Forwarding Hosts**" has its spam filter set to **Inactive**.
|
||||
|
||||
PMG should remain the primary inbound filtering authority in this deployment.
|
||||
|
||||
### Mail Clients or Outbound Delivery Stop Working
|
||||
This workflow does not change client access or outbound delivery.
|
||||
|
||||
Confirm the existing service ownership remains:
|
||||
|
||||
```text
|
||||
Inbound SMTP: Internet -> PMG -> Mailcow
|
||||
Outbound SMTP: Mailcow -> Internet
|
||||
SMTP Submission: Clients -> Mailcow
|
||||
IMAP and POP3: Clients -> Mailcow
|
||||
Webmail and Admin: Internet -> Traefik -> Mailcow
|
||||
```
|
||||
|
||||
Do not redirect ports `465`, `587`, `993`, `995`, `110`, `143`, or `4190` to PMG.
|
||||
|
||||
## Rollback
|
||||
Remove the PMG forwarding-host entry from the Mailcow WebUI only when PMG is no longer the upstream SMTP gateway.
|
||||
|
||||
Then edit:
|
||||
|
||||
```sh
|
||||
nano /opt/mailcow-dockerized/data/conf/postfix/extra.cf
|
||||
```
|
||||
|
||||
Restore the previous `mynetworks` value:
|
||||
|
||||
```ini title="/opt/mailcow-dockerized/data/conf/postfix/extra.cf"
|
||||
mynetworks = 127.0.0.0/8 172.22.1.0/24 [::1]/128
|
||||
```
|
||||
|
||||
Restart Postfix:
|
||||
|
||||
```sh
|
||||
cd /opt/mailcow-dockerized
|
||||
docker compose restart postfix-mailcow
|
||||
```
|
||||
|
||||
!!! warning "Coordinate the SMTP Path Before Rollback"
|
||||
Do not remove PMG trust while public inbound SMTP still routes through PMG. Mailcow may begin rejecting legitimate messages relayed from PMG.
|
||||
|
||||
## Confirmed Final State
|
||||
The validated Bunny Lab configuration is:
|
||||
|
||||
```text
|
||||
PMG Address: 192.168.3.15
|
||||
Mailcow Address: 192.168.3.61
|
||||
Mailcow Forwarder: 192.168.3.15
|
||||
Forwarder Spam Check: Inactive
|
||||
Postfix mynetworks: 127.0.0.0/8 172.22.1.0/24 [::1]/128 192.168.3.15/32
|
||||
```
|
||||
|
||||
The resulting mail flow is:
|
||||
|
||||
```text
|
||||
Inbound SMTP:
|
||||
Internet -> pfSense WAN :25 -> PMG 192.168.3.15:25 -> Mailcow 192.168.3.61:25
|
||||
|
||||
Outbound SMTP:
|
||||
Mailcow -> Internet
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [PMG and Mailcow Integration](<../../../../deployments/Applications/Email/Proxmox Mail Gateway/Integrate PMG with Mailcow.md>) — Confirm the intended gateway and relay topology before changing trust settings.
|
||||
- [Related Email Documentation](<../../../../reference/Applications/Email/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
tags:
|
||||
- IredMail
|
||||
- SMTP
|
||||
- Email
|
||||
---
|
||||
|
||||
## Purpose
|
||||
You may need to troubleshoot the outgoing SMTP email queue / active sessions in iRedMail for one reason or another. This can provide useful insight into the reason why emails are not being delivered, etc.
|
||||
|
||||
### Overall Queue Backlog
|
||||
You can run the following command to get the complete backlog of all email senders in the queue. This can be useful for tracking the queue's "drainage" over-time.
|
||||
|
||||
```sh
|
||||
# List the total number of queued messages
|
||||
postqueue -p | egrep -c '^[A-F0-9]'
|
||||
|
||||
# Itemize and count the queued messages based on sender.
|
||||
postqueue -p | awk '/^[A-F0-9]/ {id=$1} /from=<[^>]+>/ && $0 !~ /from=<>/ {print id; exit}'
|
||||
```
|
||||
|
||||
!!! example "Example Output"
|
||||
- 10392 problematic@bunny-lab.io
|
||||
- 301 prettybad@bunny-lab.io
|
||||
- 39 infrastructure@bunny-lab.io
|
||||
- 20 nicole.rappe@bunny-lab.io
|
||||
|
||||
### Investigating Individual Emails
|
||||
You can run the following command to list all queued messages: `postqueue -p`. You can then run `postcat -vq <message-ID>` to read detailed information on any specific queued SMTP message:
|
||||
|
||||
```sh
|
||||
postqueue -p
|
||||
postcat -vq 4dgHry5LZnzH6x08 # (1)
|
||||
```
|
||||
|
||||
1. Example message ID gathered from the previous `postqueue -p` command.
|
||||
|
||||
### Attempt to Gracefully Reload Postfix
|
||||
You may want to try to unstick things by gracefully "reloading" the postfix service via `postfix reload`. This will ensure that we don't drop / disconnect / lose all of the active outgoing SMTP sessions in the queue. It may not help resolve issues, but it's worth noting down:
|
||||
|
||||
### Reattempt Delivery
|
||||
You can attempt redelivery via running `postqueue -f` to try to free up the queue. Postfix will immediately re-attempt delivery of all queued messages instead of waiting for their scheduled retry time. It does not override remote rejections or fix underlying delivery errors; it only accelerates the next delivery attempt.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Email Documentation](<../../../../reference/Applications/Email/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+445
@@ -0,0 +1,445 @@
|
||||
---
|
||||
tags:
|
||||
- Rclone
|
||||
- Synchronization
|
||||
- PowerShell
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Configure, run, and recover the documented rclone bisync pair between a local Windows path and a Google Drive remote. The examples use PowerShell and were written for rclone `v1.75.0`.
|
||||
|
||||
!!! danger "Bisync Can Delete or Overwrite Data"
|
||||
Confirm both paths and keep an independent backup before applying changes. Review dry-run output before removing `--dry-run`.
|
||||
|
||||
## Understand Bisync State
|
||||
Bisync is a stateful two-way synchronization command. It retains listings of Path1 and Path2 from the prior successful run and compares those listings against the current state during the next run.
|
||||
|
||||
Bisync does not maintain a file-version archive or historical copies of every changed file. The listings record synchronization state and metadata needed to determine whether a file is new, changed, or deleted relative to the previous run.
|
||||
|
||||
!!! warning "Protect the Bisync Work Directory"
|
||||
Use a stable, explicit `--workdir` for every run of a given bisync pair. Do not change the work directory, reverse the order of Path1 and Path2, delete the listing files, or run the same pair with different filters without understanding that the existing state may no longer be valid.
|
||||
|
||||
On Windows, bisync otherwise stores its state beneath the profile of the account running rclone. This can cause scheduled and interactive executions to use different state directories when they run as different accounts.
|
||||
|
||||
## Define the Example Environment
|
||||
Replace the example values before running the commands in this document.
|
||||
|
||||
```powershell
|
||||
$Rclone = "C:\Path\To\rclone.exe"
|
||||
$LocalPath = "C:\Path\To\Local"
|
||||
$RemotePath = "GoogleDrive:Path/To/Remote"
|
||||
$WorkDir = "C:\Path\To\BisyncState"
|
||||
$LogDir = "C:\Path\To\Logs"
|
||||
$FilterFile = "C:\Path\To\bisync-filters.txt"
|
||||
```
|
||||
|
||||
Create the state and log directories:
|
||||
|
||||
```powershell
|
||||
New-Item -Path $WorkDir,$LogDir -ItemType Directory -Force | Out-Null
|
||||
```
|
||||
|
||||
Create a consistent filters file:
|
||||
|
||||
```text title="C:\Path\To\bisync-filters.txt"
|
||||
- *.lnk
|
||||
```
|
||||
|
||||
The `*.lnk` rule excludes Windows shortcut files at any depth beneath the synchronization root.
|
||||
|
||||
!!! warning "Filter Changes Require a Reviewed Rebaseline"
|
||||
Bisync compares current listings against prior listings. Adding, removing, or changing a filter can make previously tracked files disappear from the new listings and appear to have been deleted.
|
||||
|
||||
Treat filter changes as a state-changing event. Stop scheduled runs, review the new scope, and perform a dry-run resync before committing the change.
|
||||
|
||||
## Validate the Paths
|
||||
Confirm the installed rclone version:
|
||||
|
||||
```powershell
|
||||
& $Rclone version
|
||||
```
|
||||
|
||||
Confirm that the local root already exists:
|
||||
|
||||
```powershell
|
||||
Test-Path -LiteralPath $LocalPath
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
True
|
||||
```
|
||||
|
||||
Confirm that the configured remote path is reachable:
|
||||
|
||||
```powershell
|
||||
& $Rclone lsd $RemotePath
|
||||
```
|
||||
|
||||
!!! warning "Do Not Automatically Create an Unexpected Root"
|
||||
Stop if either path is missing or points to an unexpected location. Automatically creating an empty root can conceal a path, authentication, drive-mount, or configuration failure and can cause bisync to interpret an entire data set as deleted.
|
||||
|
||||
## Configure the Access Check
|
||||
The `--check-access` flag verifies that matching files named `RCLONE_TEST` are visible in the same relative locations on both sides. This provides additional protection against an unavailable mount, incorrect remote root, or incomplete listing being interpreted as mass deletion.
|
||||
|
||||
Create the access-check file at the root of the local path and copy it to the root of the remote path:
|
||||
|
||||
```powershell
|
||||
$AccessFile = Join-Path $LocalPath "RCLONE_TEST"
|
||||
New-Item -Path $AccessFile -ItemType File -Force | Out-Null
|
||||
& $Rclone copyto $AccessFile "$RemotePath/RCLONE_TEST"
|
||||
```
|
||||
|
||||
Confirm that the file appears on both sides:
|
||||
|
||||
```powershell
|
||||
& $Rclone lsf $LocalPath --include "RCLONE_TEST"
|
||||
& $Rclone lsf $RemotePath --include "RCLONE_TEST"
|
||||
```
|
||||
|
||||
Expected result from both commands:
|
||||
|
||||
```text
|
||||
RCLONE_TEST
|
||||
```
|
||||
|
||||
!!! warning "Do Not Delete the Access-Check File"
|
||||
The `RCLONE_TEST` file must remain visible on both sides while `--check-access` is enabled. Bisync will abort when the file cannot be found.
|
||||
|
||||
## Audit the Existing Data
|
||||
Before establishing or rebuilding the bisync state, compare the current file sets without changing either side:
|
||||
|
||||
```powershell
|
||||
& $Rclone check $LocalPath $RemotePath --filter-from $FilterFile --drive-skip-gdocs --combined (Join-Path $LogDir "pre-bisync-check.txt") --log-level INFO --log-file (Join-Path $LogDir "pre-bisync-check.log")
|
||||
```
|
||||
|
||||
Review all source-only, destination-only, differing, and unreadable files before proceeding. Determine whether the expected recovery is to merge both sides, prefer one authoritative side, or preserve selected files manually.
|
||||
|
||||
## Initialize a Bisync Pair
|
||||
A new bisync pair requires an initial resync to establish its Path1 and Path2 listings. A preliminary `sync --update` operation is not required and can unnecessarily delete destination-only files.
|
||||
|
||||
!!! danger "Resync Is Not a Database-Only Repair"
|
||||
A resync reconciles the actual contents of both paths while rebuilding the listings.
|
||||
|
||||
Files that exist on only one side are copied to the other side. When the same relative path contains different files on both sides, `--resync-mode` selects one version as the winner and overwrites the other version.
|
||||
|
||||
The conflict flags used during normal bisync runs do not apply during a resync. A resync does not rename the losing version into a conflict copy.
|
||||
|
||||
### Select the Resync Policy
|
||||
Choose the resync policy according to which data is authoritative:
|
||||
|
||||
| **Mode** | **Use When** |
|
||||
| :--- | :--- |
|
||||
| `path1` | Path1 is authoritative and must win every same-path difference |
|
||||
| `path2` | Path2 is authoritative and must win every same-path difference |
|
||||
| `newer` | Modification times are trustworthy and the newest same-path version should win |
|
||||
| `larger` | File size is a more reliable winner than modification time |
|
||||
|
||||
A bare `--resync` is equivalent to `--resync-mode path1`. Do not use a bare `--resync` unless Path1 is intentionally authoritative.
|
||||
|
||||
The examples below use `--resync-mode newer`. This is appropriate only when both sides provide trustworthy modification times and the intended policy is to preserve the newer same-path file.
|
||||
|
||||
### Preview the Initial Resync
|
||||
Run the initial resync as a dry run:
|
||||
|
||||
```powershell
|
||||
& $Rclone bisync $LocalPath $RemotePath --workdir $WorkDir --resync-mode newer --create-empty-src-dirs --compare size,modtime,checksum --check-access --max-delete 0 --filters-file $FilterFile --drive-skip-gdocs --fix-case --dry-run --verbose --log-file (Join-Path $LogDir "bisync-resync-dry-run.log")
|
||||
```
|
||||
|
||||
Review the complete log for:
|
||||
|
||||
- Files copied from Path1 to Path2
|
||||
- Files copied from Path2 to Path1
|
||||
- Same-path files that would be replaced
|
||||
- Unexpected paths
|
||||
- Access or checksum errors
|
||||
- Proposed deletion behavior
|
||||
|
||||
!!! note "Dry-Run Deletion Messages"
|
||||
A bisync dry run may display confusing deletion messages because the simulated copy that would normally precede the deletion did not actually occur. Review the complete sequence rather than evaluating an isolated deletion line.
|
||||
|
||||
Do not dismiss an unexpected deletion unless the log clearly shows that it is a dry-run artifact associated with a preceding simulated copy.
|
||||
|
||||
The `--max-delete 0` setting blocks ordinary file deletions during this recovery run. It does not prevent a same-path losing version from being overwritten during resync, so a separate backup or snapshot remains required.
|
||||
|
||||
### Perform the Initial Resync
|
||||
After reviewing and approving the dry-run output, repeat the command without `--dry-run`:
|
||||
|
||||
```powershell
|
||||
& $Rclone bisync $LocalPath $RemotePath --workdir $WorkDir --resync-mode newer --create-empty-src-dirs --compare size,modtime,checksum --check-access --max-delete 0 --filters-file $FilterFile --drive-skip-gdocs --fix-case --verbose --log-file (Join-Path $LogDir "bisync-resync-live.log")
|
||||
```
|
||||
|
||||
A successful run should end with:
|
||||
|
||||
```text
|
||||
Bisync successful
|
||||
```
|
||||
|
||||
### Validate the Reconciled Data
|
||||
Compare both sides after the live resync:
|
||||
|
||||
```powershell
|
||||
& $Rclone check $LocalPath $RemotePath --filter-from $FilterFile --drive-skip-gdocs --combined (Join-Path $LogDir "post-resync-check.txt") --log-level INFO --log-file (Join-Path $LogDir "post-resync-check.log")
|
||||
```
|
||||
|
||||
The expected result is:
|
||||
|
||||
```text
|
||||
0 differences found
|
||||
```
|
||||
|
||||
## Run Normal Bisync Operations
|
||||
After the baseline has been established, all normal runs must omit `--resync` and `--resync-mode`.
|
||||
|
||||
Preview the first normal run:
|
||||
|
||||
```powershell
|
||||
& $Rclone bisync $LocalPath $RemotePath --workdir $WorkDir --create-empty-src-dirs --conflict-resolve newer --conflict-loser num --compare size,modtime,checksum --resilient --recover --max-lock 2m --check-access --max-delete 10 --filters-file $FilterFile --drive-skip-gdocs --fix-case --dry-run --log-level INFO --log-file (Join-Path $LogDir "bisync-normal-dry-run.log")
|
||||
```
|
||||
|
||||
A healthy unchanged pair should report:
|
||||
|
||||
```text
|
||||
No changes found
|
||||
Updating listings
|
||||
Bisync successful
|
||||
```
|
||||
|
||||
After reviewing the dry run, perform one controlled live run:
|
||||
|
||||
```powershell
|
||||
& $Rclone bisync $LocalPath $RemotePath --workdir $WorkDir --create-empty-src-dirs --conflict-resolve newer --conflict-loser num --compare size,modtime,checksum --resilient --recover --max-lock 2m --check-access --max-delete 10 --filters-file $FilterFile --drive-skip-gdocs --fix-case --log-level INFO --log-file (Join-Path $LogDir "bisync.log")
|
||||
```
|
||||
|
||||
This normal command can be placed into the scheduled automation after it has completed successfully under the same Windows account and execution context that will run the scheduled task.
|
||||
|
||||
## Understand the Normal Bisync Flags
|
||||
| **Flag** | **Behavior** |
|
||||
| :--- | :--- |
|
||||
| `--workdir` | Stores the prior Path1 and Path2 listings in an explicit, stable directory |
|
||||
| `--create-empty-src-dirs` | Synchronizes the creation and deletion of empty directories |
|
||||
| `--compare size,modtime,checksum` | Uses size, modification time, and checksums when determining file state |
|
||||
| `--conflict-resolve newer` | Prefers the newer file when the same file changed independently on both sides since the prior run |
|
||||
| `--conflict-loser num` | Preserves the losing conflict version under a numbered `.conflictN` name |
|
||||
| `--resilient` | Allows certain less-serious errors to be retried during future runs rather than immediately requiring resync |
|
||||
| `--recover` | Uses a backup listing to recover from some interrupted or ungracefully terminated runs |
|
||||
| `--max-lock 2m` | Allows a stale bisync lock to expire after two minutes |
|
||||
| `--check-access` | Aborts when matching `RCLONE_TEST` files cannot be found on both sides |
|
||||
| `--max-delete 10` | Aborts when more than 10 percent of tracked files appear deleted on either side |
|
||||
| `--filters-file` | Applies one consistent set of exclusions to both bisync listings |
|
||||
| `--drive-skip-gdocs` | Makes native Google Docs, Sheets, Slides, and other Google-native documents invisible to rclone |
|
||||
| `--fix-case` | Allows supported case-only filename corrections between filesystems |
|
||||
| `--log-level INFO` | Records normal operations, changes, warnings, and errors rather than errors alone |
|
||||
|
||||
### Understand Conflict Resolution
|
||||
A bisync conflict occurs when the same relative file is new or changed on both sides compared with the prior successful run and the current file contents are not identical.
|
||||
|
||||
The `--conflict-resolve newer` option does not mean that rclone blindly compares every file and always keeps whichever modification time is newest. It applies specifically to a detected two-sided conflict.
|
||||
|
||||
With `--conflict-loser num`, the winner retains the original filename and the losing version is preserved with a numbered conflict suffix, such as:
|
||||
|
||||
```text
|
||||
Document.docx
|
||||
Document.docx.conflict1
|
||||
```
|
||||
|
||||
This is safer than `--conflict-loser delete`, which permanently removes the losing conflict version.
|
||||
|
||||
### Understand the Deletion Limit
|
||||
The `--max-delete 10` value is a conservative example rather than a universal requirement. Select a threshold appropriate for the size of the data set and the organization's normal deletion patterns.
|
||||
|
||||
A large folder rename may appear as many deletions and many new files and can exceed the configured threshold.
|
||||
|
||||
!!! danger "Do Not Use `--force` as a Recurring Option"
|
||||
The `--force` flag bypasses the `--max-delete` safety check. Do not include it in a normal scheduled command.
|
||||
|
||||
When a legitimate change exceeds the threshold, stop the scheduled operation, review the proposed changes with `--dry-run`, and use a one-time explicitly approved threshold instead of permanently disabling the protection.
|
||||
|
||||
### Understand Google-Native Documents
|
||||
The `--drive-skip-gdocs` flag removes native Google documents from rclone listings. These objects are not downloaded, exported, compared, or synchronized while the flag is enabled.
|
||||
|
||||
This flag does not merely ignore local files with extensions such as `.gdoc` or `.gsheet`. It makes the corresponding native Google Drive objects effectively invisible to rclone.
|
||||
|
||||
Remove this flag only when Google-native documents must be exported and synchronized and the desired export formats have been deliberately configured. Changing this behavior on an established bisync pair requires a reviewed rebaseline because it changes which objects appear in the listings.
|
||||
|
||||
## Repair a Broken Bisync Pair
|
||||
A request for `--resync` does not automatically mean that the data is damaged. Bisync may be unable to locate or trust its prior state because the work directory changed, the command used different paths or filters, the execution account changed, or a prior run ended with a critical error.
|
||||
|
||||
### Stop Automated Runs
|
||||
Stop all scheduled or looping bisync processes before investigating or repairing the state. Confirm that another rclone process is not modifying either path or the bisync listings.
|
||||
|
||||
### Review the Logs
|
||||
Search the most recent log for the first critical error rather than relying only on the final resync message.
|
||||
|
||||
```powershell
|
||||
Get-Content (Join-Path $LogDir "bisync.log") -Tail 300
|
||||
```
|
||||
|
||||
Look for:
|
||||
|
||||
- Authentication or remote-access errors
|
||||
- Missing-path errors
|
||||
- Missing or invalid listing files
|
||||
- Access-check failures
|
||||
- Excessive deletion warnings
|
||||
- File-copy or file-move failures
|
||||
- Lock-file errors
|
||||
- Filter changes
|
||||
- `Bisync aborted`
|
||||
- `Bisync critical error`
|
||||
|
||||
### Verify the Existing Configuration
|
||||
Confirm that the repair command uses:
|
||||
|
||||
- The original Path1 and Path2 in the original order
|
||||
- The original `--workdir`
|
||||
- The original filter rules
|
||||
- The intended rclone configuration file
|
||||
- The correct remote account and storage location
|
||||
- The same Google Docs behavior
|
||||
- The same comparison settings
|
||||
|
||||
List the current bisync state directory:
|
||||
|
||||
```powershell
|
||||
Get-ChildItem -LiteralPath $WorkDir
|
||||
```
|
||||
|
||||
The directory should contain Path1 and Path2 `.lst` files for the configured pair.
|
||||
|
||||
### Test Remote and Local Access
|
||||
Confirm that both roots are present and readable:
|
||||
|
||||
```powershell
|
||||
Test-Path -LiteralPath $LocalPath
|
||||
& $Rclone lsd $RemotePath
|
||||
```
|
||||
|
||||
Confirm that the access-check file remains visible:
|
||||
|
||||
```powershell
|
||||
& $Rclone lsf $LocalPath --include "RCLONE_TEST"
|
||||
& $Rclone lsf $RemotePath --include "RCLONE_TEST"
|
||||
```
|
||||
|
||||
### Compare the Current Data
|
||||
Run a read-only comparison before deciding to resync:
|
||||
|
||||
```powershell
|
||||
& $Rclone check $LocalPath $RemotePath --filter-from $FilterFile --drive-skip-gdocs --combined (Join-Path $LogDir "repair-comparison.txt") --log-level INFO --log-file (Join-Path $LogDir "repair-check.log")
|
||||
```
|
||||
|
||||
Review whether one-sided files are expected, whether one side is authoritative, and whether same-path differences should be resolved manually.
|
||||
|
||||
### Attempt Normal Recovery When Appropriate
|
||||
When the prior run was only interrupted and the normal command already uses `--recover`, a normal controlled run may recover without requiring a resync.
|
||||
|
||||
Run it first as a dry run:
|
||||
|
||||
```powershell
|
||||
& $Rclone bisync $LocalPath $RemotePath --workdir $WorkDir --create-empty-src-dirs --conflict-resolve newer --conflict-loser num --compare size,modtime,checksum --resilient --recover --max-lock 2m --check-access --max-delete 10 --filters-file $FilterFile --drive-skip-gdocs --fix-case --dry-run --log-level INFO --log-file (Join-Path $LogDir "bisync-recovery-dry-run.log")
|
||||
```
|
||||
|
||||
Proceed with a live normal run only when the output is understood and expected.
|
||||
|
||||
### Rebuild the State Only When Required
|
||||
Use a resync only when:
|
||||
|
||||
- The pair has never been initialized
|
||||
- The listings are missing or cannot be trusted
|
||||
- A critical bisync error explicitly requires a resync
|
||||
- The path scope or filter rules intentionally changed
|
||||
- You are deliberately establishing a new authoritative baseline
|
||||
|
||||
Select the correct `--resync-mode`, perform a dry run, review every proposed change, and then follow the initialization and validation sequence documented above.
|
||||
|
||||
!!! danger "Do Not Describe Resync as Non-Destructive"
|
||||
Resync can overwrite a same-path file and can restore files that were intentionally deleted by copying one-sided files back to the other side.
|
||||
|
||||
The `--conflict-resolve` and `--conflict-loser` flags are ignored during resync. Maintain an independent backup and do not proceed until the chosen resync policy is understood.
|
||||
|
||||
## Validate Bidirectional Synchronization
|
||||
After the initial setup or a repair, validate both directions with disposable files.
|
||||
|
||||
- Create a test text file beneath the local root.
|
||||
- Run a normal bisync and confirm that the file appears remotely.
|
||||
- Create a second test text file beneath the remote root.
|
||||
- Run another normal bisync and confirm that the file appears locally.
|
||||
- Delete both test files.
|
||||
- Run another normal bisync and confirm that the deletions propagate as intended.
|
||||
- Confirm that the final run ends with `Bisync successful`.
|
||||
- Run `rclone check` and confirm that no file differences remain.
|
||||
|
||||
Remove all test artifacts when validation is complete.
|
||||
|
||||
## Troubleshooting
|
||||
### Google Drive Reports Shared Drive Not Found
|
||||
An error such as `404: Shared Drive not found` normally means that the authenticated Google account cannot access the configured Shared Drive ID.
|
||||
|
||||
Verify:
|
||||
|
||||
- The correct Google account completed OAuth
|
||||
- The account still has access to the Shared Drive
|
||||
- The configured Shared Drive ID is correct
|
||||
- The Shared Drive was not deleted and recreated under a new ID
|
||||
|
||||
Do not replace the drive ID with another visible Shared Drive merely because authentication succeeded.
|
||||
|
||||
### Bisync Cannot Find Its Listings
|
||||
Confirm that the command uses the original `--workdir` and that the scheduled task runs under the expected account.
|
||||
|
||||
Do not perform an immediate resync when the actual problem is that rclone is looking in the wrong state directory.
|
||||
|
||||
### The Deletion Limit Was Exceeded
|
||||
Stop the operation and determine why so many files appear deleted. Common causes include:
|
||||
|
||||
- An unavailable local mount
|
||||
- An inaccessible remote path
|
||||
- An incorrect synchronization root
|
||||
- A large folder rename
|
||||
- Changed filter rules
|
||||
- An actual mass deletion
|
||||
|
||||
Do not use `--force` until the reported deletions have been independently reviewed and approved.
|
||||
|
||||
### Conflict Files Are Appearing
|
||||
Files ending in `.conflict1`, `.conflict2`, or another numbered suffix indicate that both sides changed independently and bisync preserved the losing version.
|
||||
|
||||
Review the contents, retain the correct version, and remove the obsolete conflict copy after confirming that it is no longer needed.
|
||||
|
||||
### Google Docs Are Missing
|
||||
Native Google Docs are intentionally absent when `--drive-skip-gdocs` is enabled. Remove or change this behavior only as a deliberate configuration change with a reviewed resync.
|
||||
|
||||
### Dry Run Appears to Delete a Newly Copied File
|
||||
Review the complete dry-run sequence. Bisync can display an apparent deletion because the preceding simulated copy did not actually create the file on the other side.
|
||||
|
||||
Do not ignore unrelated or unexplained deletion messages.
|
||||
|
||||
### Google Drive Reports Duplicate Objects
|
||||
Google Drive can contain multiple objects with the same name in one folder. List duplicate names with:
|
||||
|
||||
```powershell
|
||||
& $Rclone dedupe list $RemotePath
|
||||
```
|
||||
|
||||
Use the interactive resolver only after reviewing the file sizes, modification times, and hashes:
|
||||
|
||||
```powershell
|
||||
& $Rclone dedupe interactive $RemotePath
|
||||
```
|
||||
|
||||
When the correct version cannot be determined confidently, rename and preserve both objects rather than deleting one automatically.
|
||||
|
||||
## Reference Documentation
|
||||
- [Rclone Command Overview](https://rclone.org/commands/)
|
||||
- [Rclone Bisync](https://rclone.org/bisync/)
|
||||
- [Rclone Copy](https://rclone.org/commands/rclone_copy/)
|
||||
- [Rclone Sync](https://rclone.org/commands/rclone_sync/)
|
||||
- [Rclone Check](https://rclone.org/commands/rclone_check/)
|
||||
- [Rclone Filtering](https://rclone.org/filtering/)
|
||||
- [Rclone Google Drive Backend](https://rclone.org/drive/)
|
||||
- [Rclone Changelog](https://rclone.org/changelog/)
|
||||
|
||||
## Related Documentation
|
||||
- [Related Files and Collaboration Documentation](<../../../reference/Applications/Files and Collaboration/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
---
|
||||
tags:
|
||||
- DFS
|
||||
- Active Directory
|
||||
- Troubleshooting
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Refresh a DFS Management console that shows inconsistent namespaces or replication groups between member servers. Check Active Directory replication before restarting DFS Replication or clearing the console cache.
|
||||
|
||||
## Repair the Console
|
||||
Sometimes the GUI for managing DFS becomes "inconsistent" whereas the namespaces and replication groups are different between member servers, and may be missing namspaces or missing replication groups. DFS Management is an MMC snap-in. MMC persists per-user console state under `%APPDATA%\Microsoft\MMC\`. If that state gets out of sync (common after service hiccups or server crashes), the snap-in can render partial/incorrect namespace/replication trees even when DFS itself is fine. Deleting the cached dfsmgmt* console forces a fresh enumeration. We will also include a few extra commands for extra thouroughness.
|
||||
|
||||
Before anything, we want to make sure that active directory itself is not having replication issues, as this would be a deeper, more complicated issue. Run the following command on one of your domain controllers:
|
||||
|
||||
```powershell
|
||||
repadmin /syncall /AdeP
|
||||
repadmin /replsummary
|
||||
```
|
||||
|
||||
If AD-level replication is successful and timely, you can proceed to run the commands below (one-line-at-a-time):
|
||||
|
||||
```sh
|
||||
# Pull-Down DFS Configuration from Active Directory & Restart DFS
|
||||
dfsrdiag pollad
|
||||
net stop dfsr
|
||||
net start dfsr
|
||||
|
||||
# Clear DFS Management Snap-In Cache
|
||||
taskkill /im mmc.exe /f
|
||||
del "%appdata%\Microsoft\MMC\dfsmgmt*"
|
||||
dfsmgmt.msc
|
||||
```
|
||||
|
||||
!!! success "DFS Management GUI Restored"
|
||||
At this point, the DFS Management snap-in (should) be successfully showing all of the DFS namespaces and replication groups when you re-open "DFS Management".
|
||||
|
||||
## Related Documentation
|
||||
- [DFS Deployment](<../../../../deployments/Applications/Files and Collaboration/Windows Server/DFS Namespaces with Replication.md>) — Confirm the expected namespace and replication configuration.
|
||||
- [DFS Configuration Report](<../../../../scripts/Applications/Files and Collaboration/DFS/Report DFS Namespaces and Replication.md>) — Inspect the objects after reopening the management console.
|
||||
- [Related Files and Collaboration Documentation](<../../../../reference/Applications/Files and Collaboration/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
---
|
||||
tags:
|
||||
- Applications
|
||||
- Access Another User OneDrive Data
|
||||
---
|
||||
|
||||
## Purpose
|
||||
You may find that you need to access data from a user's personal OneDrive account, and since that data cannot be accessed directly via Office365's Admin Portal, you have to do some legwork in Powershell.
|
||||
|
||||
### Connect to Sharepoint
|
||||
```powershell
|
||||
Install-Module Microsoft.Online.SharePoint.PowerShell
|
||||
Import-Module Microsoft.Online.SharePoint.PowerShell
|
||||
Connect-SPOService -Url https://<companyname>-admin.sharepoint.com # Login with your Office365 Admin Credentials when Prompted
|
||||
```
|
||||
|
||||
### Display List of Personal Sharepoint Sites (OneDrive)
|
||||
```powershell
|
||||
Get-SPOSite -IncludePersonalSite $true -Limit All
|
||||
```
|
||||
|
||||
### Check OneDrive Usage of the Given User
|
||||
```powershell
|
||||
Get-SPOSite -Identity "https://<companyname>-my.sharepoint.com/personal/username_companyname_com" | Select Url, Owner, StorageUsageCurrent, StorageQuota, LastContentModifiedDate
|
||||
```
|
||||
|
||||
### Assign Yourself Permissions to Their OneDrive
|
||||
```powershell
|
||||
Set-SPOUser `
|
||||
-Site "https://<companyname>-my.sharepoint.com/personal/username_companyname_com" `
|
||||
-LoginName "i:0#.f|membership|admin@companyname.com" `
|
||||
-IsSiteCollectionAdmin $true
|
||||
```
|
||||
|
||||
### Navigate to Webpage
|
||||
At this point, you now have permissions to access the OneDrive data, so open a web browser and navigate to the SPOSite URL seen previously, seen below. From here, you can download, upload, and manage the data however you need.
|
||||
|
||||
- `https://<companyname>-my.sharepoint.com/personal/username_companyname_com`
|
||||
|
||||
### Remove Permissions
|
||||
At this point, when the work is done, revoke your permissions to lock-down the OneDrive data once again by running the following command:
|
||||
|
||||
```powershell
|
||||
Set-SPOUser `
|
||||
-Site "https://<companyname>-my.sharepoint.com/personal/username_companyname_com" `
|
||||
-LoginName "i:0#.f|membership|admin@companyname.com" `
|
||||
-IsSiteCollectionAdmin $false
|
||||
```
|
||||
|
||||
### Logout and Cleanup Auth Tokens and Remove Module
|
||||
```powershell
|
||||
Disconnect-SPOService
|
||||
Remove-Module Microsoft.Online.SharePoint.PowerShell
|
||||
# Close Powershell Window
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Files and Collaboration Documentation](<../../../../reference/Applications/Files and Collaboration/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
tags:
|
||||
- Bash
|
||||
- Netcat
|
||||
- File Transfer
|
||||
- Scripting
|
||||
- Linux
|
||||
---
|
||||
|
||||
## Purpose
|
||||
You may find that you need to transfer a file, such as a public SSH key, or some other kind of file between two devices. In this scenario, we assume both devices have the `netcat` command available to them. By putting a network listener on the device recieving the file, then sending the file to that device's IP and port, you can successfully transfer data between computers without needing to set up SSH, FTP, or anything else to establish initial trust between the devices. [Original Reference Material](https://www.youtube.com/shorts/1j17UBGqSog).
|
||||
|
||||
!!! warning
|
||||
The data being transferred will not be encrypted. If you are transferring relatively-safe files such as public SSH keys, etc, this should be fine.
|
||||
|
||||
### Destination Computer
|
||||
Run the following command on the computer that will be recieving the file.
|
||||
|
||||
```sh
|
||||
netcat -l <random-port> > /tmp/OUTPUT-AS-FILE.txt
|
||||
```
|
||||
|
||||
### Source Computer
|
||||
Run the following command on the computer that will be sending the file to the destination computer.
|
||||
|
||||
```sh
|
||||
cat INPUT-DATA.txt | netcat <IP-of-Destination-Computer> <Port-of-Destination-Computer> -q 0
|
||||
```
|
||||
|
||||
!!! info
|
||||
The `-q 0` command argument causes the netcat connection to close itself automatically when the transfer is complete.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Files and Collaboration Documentation](<../../../reference/Applications/Files and Collaboration/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
tags:
|
||||
- Tuya
|
||||
- Networking
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Connect the recorded Tuya devices with the local integration tooling and correlate their DHCP reservations. Obtain the correct local key for each device before using the integration.
|
||||
|
||||
### pfSense DHCP Reservations for Tuya-Based Smart Devices
|
||||
| **Description** | **IP Address** | **MAC Address** | **Hostname** | **Device ID** | **Local Key** |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| Bottom of Stairs | 10.0.0.200 | bcddc29072bf | ESP\_9072BF | 50316010bcddc29072bf | REDACTED |
|
||||
| Right Monitor | 10.0.0.201 | bcddc2901aef | ESP\_901AEF | 50316010bcddc2901aef | REDACTED |
|
||||
| Downstairs Light | 10.0.0.202 | bcddc28fe4c4 | ESP\_8FE4C4 | 74160333bcddc28fe4c4 | REDACTED |
|
||||
| Right TV Light | 10.0.0.203 | b4e62d4bc3fe | ESP\_4BC3FE | 36087764b4e62d4bc3fe | REDACTED |
|
||||
| Nightstand | 10.0.0.204 | b4e62d4bc3cb | ESP\_4BC3CB | 36087764b4e62d4bc3cb | REDACTED |
|
||||
| Top of Stairs | 10.0.0.205 | bcddc2904ed9 | ESP\_904ED9 | 50316010bcddc2904ed9 | REDACTED |
|
||||
| Bathroom | 10.0.0.206 | 2cf432220421 | ESP\_220421 | 105480752cf432220421 | REDACTED |
|
||||
| Front Porch | 10.0.0.207 | bcddc2947aae | ESP\_947AAE | 50316010bcddc2947aae | REDACTED |
|
||||
| Left Monitor | 10.0.0.208 | 2cf43221af1a | ESP\_21AF1A | 105480752cf43221af1a | REDACTED |
|
||||
| Puppy Nook | 10.0.0.209 | cc50e3feaa2b | ESP\_FEAA2B | 76380710cc50e3feaa2b | REDACTED |
|
||||
| TV | 10.0.0.210 | cc50e378bab9 | ESP\_78BAB9 | 35138222cc50e378bab9 | REDACTED |
|
||||
| Left TV Light | 10.0.0.211 | cc50e378916d | ESP\_78916D | 10548075cc50e378916d | REDACTED |
|
||||
| Bedroom Light | 10.0.0.212 | bcddc2907645 | | 50316010bcddc2907645 | REDACTED |
|
||||
| Garden Water Pump | 10.0.0.213 | 98f4abef7c2c | | 2182401498f4abef7c2c | REDACTED |
|
||||
| wifi Water Timer | 10.0.0.214 | | | eb54e6ae7c7536a4bbawyw | REDACTED |
|
||||
| Tech Room Light Strips | 10.0.0.215 | | | eb7dd0deaab376a2ffsqwl | REDACTED |
|
||||
| Irrigation Hub | 10.0.0.216 | 10d5615ab16b | | eb3b374a82a993d252j7v9 | REDACTED |
|
||||
| Front Lawn Sprinkler | | | | ebace67e93f8fde4ccdv7p | 2a72efa9b2f36437 |
|
||||
|
||||
### Misc Color Profile Notes
|
||||
- **1 NAME 3 4 0 255 2 5 1500 8000**
|
||||
- 20 NAME 22 23 29 1000 21 24 2700 6500
|
||||
|
||||
## Related Documentation
|
||||
- [Related Applications Documentation](<../../../reference/Applications/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
tags:
|
||||
- Applications
|
||||
- Workflows
|
||||
- Documentation
|
||||
---
|
||||
|
||||
# Applications
|
||||
## Purpose
|
||||
Find workflows for applications. Follow the subject guide to choose the relevant environment and connect this material to the other document types.
|
||||
|
||||
## Includes
|
||||
- Communication
|
||||
- Email
|
||||
- Files and Collaboration
|
||||
- Home Automation
|
||||
|
||||
## Follow the Subject
|
||||
[Applications](<../../reference/Applications/index.md>) explains the relationships and offers starting points for the documented tasks.
|
||||
Reference in New Issue
Block a user