125 lines
7.3 KiB
Markdown
125 lines
7.3 KiB
Markdown
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
|
|
|
|
!!! warning "Be Mindful of Sync Type"
|
|
The `bisync` command is meant to keep multiple locations synced with eachother, while in contrast the `sync` command forces the source to overwrite the destination. If you just want to dump the source into the destination on top of existing data, use the `copy` command within rclone.
|
|
|
|
## Usage Overview
|
|
There is a lot to keep in mind when using rclone, primarily with the `sync` command. You can find more information in the [Official Documentation](https://rclone.org/commands/)
|
|
|
|
## rClone `bisync` Implementation
|
|
Perform bidirectional synchronization between two paths. Bisync provides a bidirectional cloud sync solution in rclone. It retains the `locationA` and `locationB` filesystem listings from the prior run. On each successive run it will:
|
|
|
|
- List files on `locationA` and `locationB`, and check for changes on each side. Changes include `New`, `Newer`, `Older`, and `Deleted files`.
|
|
- Propagate changes on Path1 to Path2, and vice-versa.
|
|
|
|
### Example Usage
|
|
The following commands illustrate how to use bisync to synchronize a local folder and a remote folder (assumed to be Google Drive). The `--drive-skip-gdocs` flag simply does not sync Google Drive specific documents back to the local folder, such as `*.gsheet`, `*.gdoc`, etc. The `--resilient` flag means that if there are network interruptions, rclone will attempt to recover on its own automatically to resume where it last left off in the sync.
|
|
|
|
=== "Initial Sync"
|
|
You only run this command the first time you sync the local and remote locations. It builds a database between the two locations to effectively "index" every file locally and remotely, so that subsequent syncs can track deletions, additions, and modifications in both locations.
|
|
|
|
```powershell
|
|
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync
|
|
```
|
|
|
|
=== "Subsequent Syncs"
|
|
You run this command after the initial sync to check for changes, and push those changes to both locations.
|
|
```powershell
|
|
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
|
|
```
|
|
|
|
??? example "Self-Testing Powershell Script"
|
|
You may not be 100% confident in the behavior of rclone, especially with something such as the `bisync` command in rclone. If that is the case, you can run (and modify to your liking) the following powershell script to validate it behaves as you expect it would.
|
|
|
|
``` powershell
|
|
# Change Directory to rClone Folder
|
|
CD "C:\Users\nicole.rappe.BUNNY-LAB\Desktop\rclone-v1.68.1-windows-amd64"
|
|
|
|
# Clear the console
|
|
clear
|
|
|
|
# Define paths for LOCATION_A and LOCATION_B
|
|
$locationA = "D:\RCLONE_TESTING_LAB\LOCATION_A"
|
|
$locationB = "D:\RCLONE_TESTING_LAB\LOCATION_B"
|
|
$canaryFileA = "$locationA\CANARY.TXT"
|
|
$canaryFileB = "$locationB\CANARY.TXT"
|
|
$dummyFileA = "$locationA\DUMMY.TXT"
|
|
$dummyFileB = "$locationB\DUMMY.TXT"
|
|
|
|
# Step 1: Add dummy files to LOCATION_A and LOCATION_B to avoid empty directory errors
|
|
Write-Host "Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors..."
|
|
"DUMMY_DATA" | Out-File -FilePath $dummyFileA -Encoding UTF8
|
|
"DUMMY_DATA" | Out-File -FilePath $dummyFileB -Encoding UTF8
|
|
|
|
# Step 2: Create CANARY file in LOCATION_A
|
|
Write-Host "Creating CANARY in LOCATION_A..."
|
|
"ORIGINAL_DATA" | Out-File -FilePath $canaryFileA -Encoding UTF8 # Write original data to canary file
|
|
|
|
# Step 3: Run the initial bisync to sync LOCATION_A and LOCATION_B
|
|
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B..."
|
|
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync
|
|
|
|
# Step 4: Check if the CANARY file exists in LOCATION_B
|
|
if (Test-Path $canaryFileB) {
|
|
Write-Host "CANARY file found in LOCATION_B."
|
|
} else {
|
|
Write-Host "ERROR: CANARY file not found in LOCATION_B."
|
|
exit 1 # Exit script if file is not found
|
|
}
|
|
|
|
# Step 5: Overwrite data in the CANARY file in LOCATION_B
|
|
Write-Host "Updating CANARY file in LOCATION_B with 'UPDATED_DATA'..."
|
|
"UPDATED_DATA" | Out-File -FilePath $canaryFileB -Encoding UTF8 # Write updated data to canary file
|
|
|
|
# Step 6: Run bisync again to sync changes from LOCATION_B back to LOCATION_A
|
|
Write-Host "Bi-Syncing LOCATION_B back to LOCATION_A..."
|
|
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
|
|
|
|
# Step 7: Verify that UPDATED_DATA is now in the CANARY file in LOCATION_A
|
|
$canaryContentA = Get-Content -Path $canaryFileA -Encoding UTF8
|
|
if ($canaryContentA -eq "UPDATED_DATA") {
|
|
Write-Host "UPDATED_DATA confirmed in LOCATION_A's CANARY file."
|
|
} else {
|
|
Write-Host "ERROR: UPDATED_DATA not found in LOCATION_A's CANARY file."
|
|
exit 1 # Exit script if content is not as expected
|
|
}
|
|
|
|
# Step 8: Delete the CANARY file in LOCATION_A
|
|
Write-Host "Deleting CANARY file from LOCATION_A..."
|
|
Remove-Item -Path $canaryFileA -Force -ErrorAction SilentlyContinue
|
|
|
|
# Step 9: Run bisync again to propagate deletion to LOCATION_B
|
|
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B..."
|
|
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
|
|
|
|
# Step 10: Confirm CANARY file no longer exists in LOCATION_B
|
|
if (-not (Test-Path $canaryFileB)) {
|
|
Write-Host "CANARY file successfully deleted from LOCATION_B."
|
|
} else {
|
|
Write-Host "ERROR: CANARY file still exists in LOCATION_B."
|
|
exit 1 # Exit script if deletion did not propagate
|
|
}
|
|
|
|
# Step 11: Clean up dummy files in LOCATION_A and LOCATION_B
|
|
Write-Host "Cleaning up dummy files from LOCATION_A and LOCATION_B..."
|
|
Remove-Item -Path $dummyFileA -Force -ErrorAction SilentlyContinue
|
|
Remove-Item -Path $dummyFileB -Force -ErrorAction SilentlyContinue
|
|
|
|
Write-Host "All steps completed successfully!"
|
|
```
|
|
|
|
!!! example "Example Output"
|
|
```
|
|
Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors...
|
|
Creating CANARY in LOCATION_A...
|
|
Bi-Syncing LOCATION_A and LOCATION_B...
|
|
CANARY file found in LOCATION_B.
|
|
Updating CANARY file in LOCATION_B with 'UPDATED_DATA'...
|
|
Bi-Syncing LOCATION_B back to LOCATION_A...
|
|
UPDATED_DATA confirmed in LOCATION_A's CANARY file.
|
|
Deleting CANARY file from LOCATION_A...
|
|
Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B...
|
|
CANARY file successfully deleted from LOCATION_B.
|
|
Cleaning up dummy files from LOCATION_A and LOCATION_B...
|
|
All steps completed successfully!
|
|
``` |