bunny-lab/docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Scripts/Batch/rClone.md

Browse Source
This commit is contained in:
Nicole Rappe
2024-11-17 23:56:57 -07:00
parent a68436b1f6
commit a7171948bb
1 changed files with 124 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
Scripts/Batch/rClone.md Normal file
View File

@@ -0,0 +1,124 @@
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
!!! warning "Be Mindful of Sync Type"
The `bisync` command is meant to keep multiple locations synced with eachother, while in contrast the `sync` command forces the source to overwrite the destination. If you just want to dump the source into the destination on top of existing data, use the `copy` command within rclone.
## Usage Overview
There is a lot to keep in mind when using rclone, primarily with the `sync` command. You can find more information in the [Official Documentation](https://rclone.org/commands/)
## rClone `bisync` Implementation
Perform bidirectional synchronization between two paths. Bisync provides a bidirectional cloud sync solution in rclone. It retains the `locationA` and `locationB` filesystem listings from the prior run. On each successive run it will:
- List files on `locationA` and `locationB`, and check for changes on each side. Changes include `New`, `Newer`, `Older`, and `Deleted files`.
- Propagate changes on Path1 to Path2, and vice-versa.
### Example Usage
The following commands illustrate how to use bisync to synchronize a local folder and a remote folder (assumed to be Google Drive). The `--drive-skip-gdocs` flag simply does not sync Google Drive specific documents back to the local folder, such as `*.gsheet`, `*.gdoc`, etc. The `--resilient` flag means that if there are network interruptions, rclone will attempt to recover on its own automatically to resume where it last left off in the sync.
**Initial Sync**:
You only run this command the first time you sync the local and remote locations. It builds a database between the two locations to effectively "index" every file locally and remotely, so that subsequent syncs can track deletions, additions, and modifications in either location.
```powershell
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync
```
**Subsequent Syncs**:
You run this command after the initial sync to check for changes, and push those changes to both locations. For example, if a
```powershell
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
```
??? example "Self-Testing Powershell Script"
You may not be 100% confident in the behavior of rclone, especially with something such as the `bisync` command in rclone. If that is the case, you can run (and modify to your liking) the following powershell script to validate it behaves as you expect it would.
``` powershell
# Change Directory to rClone Folder
CD "C:\Users\nicole.rappe.BUNNY-LAB\Desktop\rclone-v1.68.1-windows-amd64"
# Clear the console
clear
# Define paths for LOCATION_A and LOCATION_B
$locationA = "D:\RCLONE_TESTING_LAB\LOCATION_A"
$locationB = "D:\RCLONE_TESTING_LAB\LOCATION_B"
$canaryFileA = "$locationA\CANARY.TXT"
$canaryFileB = "$locationB\CANARY.TXT"
$dummyFileA = "$locationA\DUMMY.TXT"
$dummyFileB = "$locationB\DUMMY.TXT"
# Step 1: Add dummy files to LOCATION_A and LOCATION_B to avoid empty directory errors
Write-Host "Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors..."
"DUMMY_DATA" | Out-File -FilePath $dummyFileA -Encoding UTF8
"DUMMY_DATA" | Out-File -FilePath $dummyFileB -Encoding UTF8
# Step 2: Create CANARY file in LOCATION_A
Write-Host "Creating CANARY in LOCATION_A..."
"ORIGINAL_DATA" | Out-File -FilePath $canaryFileA -Encoding UTF8 # Write original data to canary file
# Step 3: Run the initial bisync to sync LOCATION_A and LOCATION_B
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync
# Step 4: Check if the CANARY file exists in LOCATION_B
if (Test-Path $canaryFileB) {
Write-Host "CANARY file found in LOCATION_B."
} else {
Write-Host "ERROR: CANARY file not found in LOCATION_B."
exit 1 # Exit script if file is not found
}
# Step 5: Overwrite data in the CANARY file in LOCATION_B
Write-Host "Updating CANARY file in LOCATION_B with 'UPDATED_DATA'..."
"UPDATED_DATA" | Out-File -FilePath $canaryFileB -Encoding UTF8 # Write updated data to canary file
# Step 6: Run bisync again to sync changes from LOCATION_B back to LOCATION_A
Write-Host "Bi-Syncing LOCATION_B back to LOCATION_A..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
# Step 7: Verify that UPDATED_DATA is now in the CANARY file in LOCATION_A
$canaryContentA = Get-Content -Path $canaryFileA -Encoding UTF8
if ($canaryContentA -eq "UPDATED_DATA") {
Write-Host "UPDATED_DATA confirmed in LOCATION_A's CANARY file."
} else {
Write-Host "ERROR: UPDATED_DATA not found in LOCATION_A's CANARY file."
exit 1 # Exit script if content is not as expected
}
# Step 8: Delete the CANARY file in LOCATION_A
Write-Host "Deleting CANARY file from LOCATION_A..."
Remove-Item -Path $canaryFileA -Force -ErrorAction SilentlyContinue
# Step 9: Run bisync again to propagate deletion to LOCATION_B
Write-Host "Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B..."
.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force
# Step 10: Confirm CANARY file no longer exists in LOCATION_B
if (-not (Test-Path $canaryFileB)) {
Write-Host "CANARY file successfully deleted from LOCATION_B."
} else {
Write-Host "ERROR: CANARY file still exists in LOCATION_B."
exit 1 # Exit script if deletion did not propagate
}
# Step 11: Clean up dummy files in LOCATION_A and LOCATION_B
Write-Host "Cleaning up dummy files from LOCATION_A and LOCATION_B..."
Remove-Item -Path $dummyFileA -Force -ErrorAction SilentlyContinue
Remove-Item -Path $dummyFileB -Force -ErrorAction SilentlyContinue
Write-Host "All steps completed successfully!"
```
!!! example "Example Output"
```
Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors...
Creating CANARY in LOCATION_A...
Bi-Syncing LOCATION_A and LOCATION_B...
CANARY file found in LOCATION_B.
Updating CANARY file in LOCATION_B with 'UPDATED_DATA'...
Bi-Syncing LOCATION_B back to LOCATION_A...
UPDATED_DATA confirmed in LOCATION_A's CANARY file.
Deleting CANARY file from LOCATION_A...
Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B...
CANARY file successfully deleted from LOCATION_B.
Cleaning up dummy files from LOCATION_A and LOCATION_B...
All steps completed successfully!
```