From a7171948bb7859452b80ac7b79bbf45a3488671f Mon Sep 17 00:00:00 2001 From: Nicole Rappe Date: Sun, 17 Nov 2024 23:56:57 -0700 Subject: [PATCH] Add Scripts/Batch/rClone.md --- Scripts/Batch/rClone.md | 124 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 Scripts/Batch/rClone.md diff --git a/Scripts/Batch/rClone.md b/Scripts/Batch/rClone.md new file mode 100644 index 0000000..70a33b0 --- /dev/null +++ b/Scripts/Batch/rClone.md @@ -0,0 +1,124 @@ +Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols. + +!!! warning "Be Mindful of Sync Type" + The `bisync` command is meant to keep multiple locations synced with eachother, while in contrast the `sync` command forces the source to overwrite the destination. If you just want to dump the source into the destination on top of existing data, use the `copy` command within rclone. + +## Usage Overview +There is a lot to keep in mind when using rclone, primarily with the `sync` command. You can find more information in the [Official Documentation](https://rclone.org/commands/) + +## rClone `bisync` Implementation +Perform bidirectional synchronization between two paths. Bisync provides a bidirectional cloud sync solution in rclone. It retains the `locationA` and `locationB` filesystem listings from the prior run. On each successive run it will: + +- List files on `locationA` and `locationB`, and check for changes on each side. Changes include `New`, `Newer`, `Older`, and `Deleted files`. +- Propagate changes on Path1 to Path2, and vice-versa. + +### Example Usage +The following commands illustrate how to use bisync to synchronize a local folder and a remote folder (assumed to be Google Drive). The `--drive-skip-gdocs` flag simply does not sync Google Drive specific documents back to the local folder, such as `*.gsheet`, `*.gdoc`, etc. The `--resilient` flag means that if there are network interruptions, rclone will attempt to recover on its own automatically to resume where it last left off in the sync. + +**Initial Sync**: +You only run this command the first time you sync the local and remote locations. It builds a database between the two locations to effectively "index" every file locally and remotely, so that subsequent syncs can track deletions, additions, and modifications in either location. +```powershell +.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync +``` + +**Subsequent Syncs**: +You run this command after the initial sync to check for changes, and push those changes to both locations. For example, if a +```powershell +.\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force +``` + +??? example "Self-Testing Powershell Script" + You may not be 100% confident in the behavior of rclone, especially with something such as the `bisync` command in rclone. If that is the case, you can run (and modify to your liking) the following powershell script to validate it behaves as you expect it would. + + ``` powershell + # Change Directory to rClone Folder + CD "C:\Users\nicole.rappe.BUNNY-LAB\Desktop\rclone-v1.68.1-windows-amd64" + + # Clear the console + clear + + # Define paths for LOCATION_A and LOCATION_B + $locationA = "D:\RCLONE_TESTING_LAB\LOCATION_A" + $locationB = "D:\RCLONE_TESTING_LAB\LOCATION_B" + $canaryFileA = "$locationA\CANARY.TXT" + $canaryFileB = "$locationB\CANARY.TXT" + $dummyFileA = "$locationA\DUMMY.TXT" + $dummyFileB = "$locationB\DUMMY.TXT" + + # Step 1: Add dummy files to LOCATION_A and LOCATION_B to avoid empty directory errors + Write-Host "Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors..." + "DUMMY_DATA" | Out-File -FilePath $dummyFileA -Encoding UTF8 + "DUMMY_DATA" | Out-File -FilePath $dummyFileB -Encoding UTF8 + + # Step 2: Create CANARY file in LOCATION_A + Write-Host "Creating CANARY in LOCATION_A..." + "ORIGINAL_DATA" | Out-File -FilePath $canaryFileA -Encoding UTF8 # Write original data to canary file + + # Step 3: Run the initial bisync to sync LOCATION_A and LOCATION_B + Write-Host "Bi-Syncing LOCATION_A and LOCATION_B..." + .\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --resync + + # Step 4: Check if the CANARY file exists in LOCATION_B + if (Test-Path $canaryFileB) { + Write-Host "CANARY file found in LOCATION_B." + } else { + Write-Host "ERROR: CANARY file not found in LOCATION_B." + exit 1 # Exit script if file is not found + } + + # Step 5: Overwrite data in the CANARY file in LOCATION_B + Write-Host "Updating CANARY file in LOCATION_B with 'UPDATED_DATA'..." + "UPDATED_DATA" | Out-File -FilePath $canaryFileB -Encoding UTF8 # Write updated data to canary file + + # Step 6: Run bisync again to sync changes from LOCATION_B back to LOCATION_A + Write-Host "Bi-Syncing LOCATION_B back to LOCATION_A..." + .\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force + + # Step 7: Verify that UPDATED_DATA is now in the CANARY file in LOCATION_A + $canaryContentA = Get-Content -Path $canaryFileA -Encoding UTF8 + if ($canaryContentA -eq "UPDATED_DATA") { + Write-Host "UPDATED_DATA confirmed in LOCATION_A's CANARY file." + } else { + Write-Host "ERROR: UPDATED_DATA not found in LOCATION_A's CANARY file." + exit 1 # Exit script if content is not as expected + } + + # Step 8: Delete the CANARY file in LOCATION_A + Write-Host "Deleting CANARY file from LOCATION_A..." + Remove-Item -Path $canaryFileA -Force -ErrorAction SilentlyContinue + + # Step 9: Run bisync again to propagate deletion to LOCATION_B + Write-Host "Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B..." + .\rclone.exe bisync "$locationA" "$locationB" --create-empty-src-dirs --compare size,modtime,checksum --resilient --log-level ERROR --drive-skip-gdocs --fix-case --force + + # Step 10: Confirm CANARY file no longer exists in LOCATION_B + if (-not (Test-Path $canaryFileB)) { + Write-Host "CANARY file successfully deleted from LOCATION_B." + } else { + Write-Host "ERROR: CANARY file still exists in LOCATION_B." + exit 1 # Exit script if deletion did not propagate + } + + # Step 11: Clean up dummy files in LOCATION_A and LOCATION_B + Write-Host "Cleaning up dummy files from LOCATION_A and LOCATION_B..." + Remove-Item -Path $dummyFileA -Force -ErrorAction SilentlyContinue + Remove-Item -Path $dummyFileB -Force -ErrorAction SilentlyContinue + + Write-Host "All steps completed successfully!" + ``` + + !!! example "Example Output" + ``` + Adding dummy files to LOCATION_A and LOCATION_B to avoid empty directory sync errors... + Creating CANARY in LOCATION_A... + Bi-Syncing LOCATION_A and LOCATION_B... + CANARY file found in LOCATION_B. + Updating CANARY file in LOCATION_B with 'UPDATED_DATA'... + Bi-Syncing LOCATION_B back to LOCATION_A... + UPDATED_DATA confirmed in LOCATION_A's CANARY file. + Deleting CANARY file from LOCATION_A... + Bi-Syncing LOCATION_A and LOCATION_B to remove CANARY file from LOCATION_B... + CANARY file successfully deleted from LOCATION_B. + Cleaning up dummy files from LOCATION_A and LOCATION_B... + All steps completed successfully! + ``` \ No newline at end of file