docs/Scripts/Powershell/rClone.md

Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

!!! warning "Be Mindful of Sync Type" The bisync command is meant to keep multiple locations synced with eachother, while in contrast the sync command forces the source to overwrite the destination. If you just want to dump the source into the destination on top of existing data, use the copy command within rclone.

Usage Overview

There is a lot to keep in mind when using rclone, primarily with the sync command. You can find more information in the Official Documentation

rClone bisync Implementation

Perform bidirectional synchronization between two paths. Bisync provides a bidirectional cloud sync solution in rclone. It retains the locationA and locationB filesystem listings from the prior run. On each successive run it will:

• List files on locationA and locationB, and check for changes on each side. Changes include New, Newer, Older, and Deleted files.
• Propagate changes on Path1 to Path2, and vice-versa.

Example Usage

The following commands illustrate how to use bisync to synchronize a local folder and a remote folder (assumed to be Google Drive). The --drive-skip-gdocs flag simply does not sync Google Drive specific documents back to the local folder, such as *.gsheet, *.gdoc, etc. The --resilient flag means that if there are network interruptions, rclone will attempt to recover on its own automatically to resume where it last left off in the sync.

=== "Initial Sync" You only run this command the first time you sync the local and remote locations. It builds a database between the two locations to effectively "index" every file locally and remotely, so that subsequent syncs can track deletions, additions, and modifications in both locations.

```powershell
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync
```

=== "Subsequent Syncs" You run this command after the initial sync to check for changes, and push those changes to both locations. powershell .\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force

??? example "Self-Testing Powershell Script" You may not be 100% confident in the behavior of rclone, especially with something such as the bisync command in rclone. If that is the case, you can run (and modify to your liking) the following powershell script to validate it behaves as you expect it would.

``` powershell
# Change Directory to rClone Folder
CD "C:\Users\nicole.rappe.BUNNY-LAB\Desktop\rclone-v1.68.1-windows-amd64"

# Clear the console
clear

# Define paths for LOCATION_A and LOCATION_B
$locationA = "D:\RCLONE_TESTING_LAB\LOCATION_A"
$locationB = "D:\RCLONE_TESTING_LAB\LOCATION_B"
$canaryFileA = "$locationA\CANARY.TXT"
$canaryFileB = "$locationB\CANARY.TXT"
$dummyFileA = "$locationA\DUMMY.TXT"
$dummyFileB = "$locationB\DUMMY.TXT"

# Step 1: Add dummy files to LOCATION_A and LOCATION_B to avoid empty directory errors
Write-Host "Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors..."
"DUMMY_DATA" | Out-File -FilePath $dummyFileA -Encoding UTF8
"DUMMY_DATA" | Out-File -FilePath $dummyFileB -Encoding UTF8

# Step 2: Create CANARY file in LOCATION_A
Write-Host "Creating CANARY in LOCATION_A..."
"ORIGINAL_DATA" | Out-File -FilePath $canaryFileA -Encoding UTF8  # Write original data to canary file

# Step 3: Run the initial bisync to sync LOCATION_A and LOCATION_B
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync

# Step 4: Check if the CANARY file exists in LOCATION_B
if (Test-Path $canaryFileB) {
    Write-Host "CANARY file found in LOCATION_B."
} else {
    Write-Host "ERROR: CANARY file not found in LOCATION_B."
    exit 1  # Exit script if file is not found
}

# Step 5: Overwrite data in the CANARY file in LOCATION_B
Write-Host "Updating CANARY file in LOCATION_B with 'UPDATED_DATA'..."
"UPDATED_DATA" | Out-File -FilePath $canaryFileB -Encoding UTF8  # Write updated data to canary file

# Step 6: Run bisync again to sync changes from LOCATION_B back to LOCATION_A
Write-Host "Bi-Syncing LOCATION_B back to LOCATION_A..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force

# Step 7: Verify that UPDATED_DATA is now in the CANARY file in LOCATION_A
$canaryContentA = Get-Content -Path $canaryFileA -Encoding UTF8
if ($canaryContentA -eq "UPDATED_DATA") {
    Write-Host "UPDATED_DATA confirmed in LOCATION_A's CANARY file."
} else {
    Write-Host "ERROR: UPDATED_DATA not found in LOCATION_A's CANARY file."
    exit 1  # Exit script if content is not as expected
}

# Step 8: Delete the CANARY file in LOCATION_A
Write-Host "Deleting CANARY file from LOCATION_A..."
Remove-Item -Path $canaryFileA -Force -ErrorAction SilentlyContinue

# Step 9: Run bisync again to propagate deletion to LOCATION_B
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force

# Step 10: Confirm CANARY file no longer exists in LOCATION_B
if (-not (Test-Path $canaryFileB)) {
    Write-Host "CANARY file successfully deleted from LOCATION_B."
} else {
    Write-Host "ERROR: CANARY file still exists in LOCATION_B."
    exit 1  # Exit script if deletion did not propagate
}

# Step 11: Clean up dummy files in LOCATION_A and LOCATION_B
Write-Host "Cleaning up dummy files from LOCATION_A and LOCATION_B..."
Remove-Item -Path $dummyFileA -Force -ErrorAction SilentlyContinue
Remove-Item -Path $dummyFileB -Force -ErrorAction SilentlyContinue

Write-Host "All steps completed successfully!"
```

!!! example "Example Output"
    ```
    Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors...
    Creating CANARY in LOCATION_A...
    Bi-Syncing LOCATION_A and LOCATION_B...
    CANARY file found in LOCATION_B.
    Updating CANARY file in LOCATION_B with 'UPDATED_DATA'...
    Bi-Syncing LOCATION_B back to LOCATION_A...
    UPDATED_DATA confirmed in LOCATION_A's CANARY file.
    Deleting CANARY file from LOCATION_A...
    Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B...
    CANARY file successfully deleted from LOCATION_B.
    Cleaning up dummy files from LOCATION_A and LOCATION_B...
    All steps completed successfully!
    ```