Documentation Restructure

operations/backups-and-dr/veeam-backup-replication/backup-agent-takeover.md

@@ -0,0 +1,20 @@
+**Purpose**: You may find that you need to adopt a device that was onboarded by a different Veeam Backup & Replication server.  Maybe the old server died, or maybe you are restructuring your backup infrastructure, and want a new server taking over the backup responsibilities for the device.
+
+If this happens, Veeam will complain that the device is managed by a different server.  To circumvent this, perform the following changes in the Windows Registry based on the version of Veeam Backup & Replication you are currently using, then try to Update the Agent / Backup the agent again, and it should be successful after the registry changes are made.
+
+**Reference Material**:
+https://forums.veeam.com/servers-workstations-f49/how-do-we-move-agent-to-associate-with-a-new-veeam-server-t79977.html
+
+=== "VBR v11"
+
+    ```jsx title="HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication"
+    AgentDiscoveryIgnoreOwnership
+    REG_DWORD (32-bit) Value: 1
+    ```
+        
+=== "VBR v12"
+
+    ```jsx title="HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication"
+    ProtectionGroupIgnoreOwnership
+    REG_DWORD (32-bit) Value: 1
+    ```

operations/backups-and-dr/veeam-backup-replication/core-veeam-concepts.md

@@ -0,0 +1,38 @@
+**Purpose**:
+The purpose of this document is to explain the core concepts / terminology of things seen in Veeam Backup & Replication from a relatively high-level.  It's more of a quick-reference guide than a formal education.
+
+## Backup Jobs
+Backup jobs take many forms, but the most common are explained in more detail below.  Note that this is not an exhaustive list of the different kinds of backup jobs, just the ones I am currently most familiar with.
+
+- **Backup**: This is the simplest of the backup job options.  A "Backup" backup job will take a backup of a workstation, server, File Server, specific local files and folders on a device, or a GuestVM running in a hypervisor such as Hyper-V, VMWare ESXi, or ProxmoxVE.  
+- **Backup Copy**: 
+    - This is when you make a copy of backup data stored on the Veeam server, and send it somewhere else, such as an off-site "Service Provider" such as Veeam partners.  
+    - You can also send backup copies to local drives, SMB network shares, NFS shares, File Servers, pretty much anywhere you can send normal backups, but with the key difference being the data is originating from the Veeam backup server itself instead of the original server/VM.
+- **SureBackup**: This is where things get a little more complex.  SureBackup is where you effectively "Verify" your backups by spinning them up inside of a lab environment.  While they are spun up, they are checked to see if they fully boot, they can have antivirus scans, ransomware scans, custom scripts executed, and validate the integrity of the backups. The general core components are listed below:
+    - **Virtual Lab**: The virtual lab is a virtual machine environment that you set up for Veeam to leverage to spin up backups on a hypervisor that you configure, such as a remote Hyper-V server in the same building, or perhaps if you have Hyper-V locally installed on the same server as Veeam itself, you would configure the virtual lab's hypervisor to point to `127.0.0.1` or `localhost`.
+        - The virtual lab will have its own unique virtual networking for the VMs to communicate on, so they don't conflict with the production servers/VMs.
+    - **Application Groups**: Application groups are defined groups of devices that need to be running when the backups are being validated.  For example, in my homelab, I have an application group named `Domain Controllers`, and I put `LAB-DC-01` and `LAB-DC-02` into that application group.  I use this as the application group associated with the Virtual Lab because most of my services are authenticated with Active Directory, and if the DCs were missing during backup verification, a variety of issues would ensue.  When the Backup Verification Lab (Virtual Lab) is launched on the targeted hypervisor, it spins up the application group devices from backups first, ensuring they are running and functional, before the virtual lab starts verifying backup objects designated in the "Linked Jobs", seen in the next section.
+    - **Linked Jobs**: These are the "Backup Jobs" you want to verify in in the virtual lab mentioned above.  If you have a large backup job with a bunch of machines you don't want verified, you can configure "Exclusions" in the SureBackup job settings to exclude those objects/devices from verification.
+## Replication Jobs
+As the name states, Veeam Backup & Replication can also handle replicating Servers/VMs from either their original locations or from a recent backup and push them into a hypervisor for rapid failover/failback functionality.  Very useful for workloads that need to be spun up nearly immediately due to strict RTO requirements.  There are some additional notes regarding replication seen below.
+
+!!! warning "Orchestrate Replication & Failover via Veeam, not the Hypervisor"
+    You want to coordinate anything replication-wise directly in Veeam Backup & Replication, not directly on the hypervisor itself.  While you can do this, it is not only slower, but does not give you the option to failback replicas back into production if you spin up a replica directly on its hypervisor.
+    
+- **Replication Restore Points**: Similar to backups, replicas can have multiple restore points associated with them, so you have more than one option when spinning up a replica in a hypervisor.  
+- **Planned Failover**: A planned failover is when you are scheduling the hypervisor to be offline and simply don't have enough resources to live-migrate it to another cluster host, or you might not even have a virtualization cluster to work with in the first place.  In cases like this, a "Planned Failover" tells Veeam to make a fresh replica right now, then shuts down the production VM on its hypervisor, and spins up the replica on the replica server.  (If you installed Hyper-V on the Veeam server, it would spin up the replica on the backup server itself).
+    - A "Planned Failover" allows you to perform a "**Failback to Production**" when the failover event has concluded.  This means that while the production VM was offline and the replica took over the production load, any changes made such as new files added, applications installed, etc will be replicated back to the production VM when the replica is "Failed back to Production".  **This is the ideal choice in most circumstances**.
+- **Failover Now**: Failover now means that the production hypervisor is likely completely dead, and may need to be re-built, or you simply dont need to replicate changes back to production hypervisor after the failover event has concluded, such as on a low-priority print server.  Any changes made while the replica is operational will be completely lost when the production VM is turned back on again or a restore is pushed back onto a new hypervisor.
+## Backup Infrastructure
+### Backup Repository
+A backup repository is simply a destination to send the backups or backup copies.  It can be anything from direct attached storage to a SMB file share on a NAS, or even off-site storage like Backblaze B2 or Amazon S3.
+    - If you use object storage like Backblaze B2 or Amazon S3, you can configure an "Immutability Period" for backups that are sent to these destinations, meaning if your backup server was hit by ransomware or a malicious actor, neither they nor you could delete the backups in the off-site storage such as Backblaze B2 until the immutability period had passed, such as 7 days, 30 days, or however long you configured.
+        - You can adjust the immutability period after-the-fact, but backups that have already been pushed to a backup repository will be immutable for the time period configured when they were originally uploaded, and attempts to delete them will tell you when you are allowed to delete them.  You won't be able to delete them even from Amazon or Backblaze's own internal tools / websites during this immutability period.
+### Backup Proxy
+A backup "proxy" simply refers to a machine that is running the "**Veeam Backup Transport**" agent on it.  The Veeam Backup & Replication server installs a proxy onto itself, but it also deploys proxies onto workstations, servers, and hypervisors.  These proxies are how the "Veeam Backup & Replication Console" interacts with the devices and performs backups and restores.
+### Service Provider
+Service Providers are not the same as cloud storage providers such as Backblaze B2, Amazon S3, etc.  Service Providers are Veeam "partners" who manage, maintain, and deploy Veeam backup appliances at client environments, as well as providing support to clients within the Veeam ecosystem.  You can also use Service Providers as a cloud backup destination in Veeam Backup & Replication for off-site backups.
+
+## Misc Terminology
+- **Unstructured Data**: This refers to a device such as a windows or linux server that you can use WinRM or SSH to access, and want to backup specific files and folders without backing up the entire device / VM.  This is useful in cases where you cannot install a Veeam Agent or the operating system is unsupported by Veeam, or if the device is not operating under a hypervisor, such as a bare-metal server.
+    - When you add a device to Veeam's "Inventory" via the "Unstructured Data" section, if you want to perform backups on the device, you will have to make a special backup job under "**Backups > File Server**", because Veeam will treat the unstructured data as a file server.

operations/backups-and-dr/veeam-backup-replication/manually-pruning-backups.md

@@ -0,0 +1,26 @@
+**Purpose**:
+There may come a time that you need to free up space in a Veeam Backup & Replication backup repository because you are running out of space.  In these cases, you need to manually trim the older backups in a specific way to ensure this is non-destructive. 
+
+## Manual Removal of Backup Data
+You need to perform these steps to carefully delete the oldest full backup chain w/ incrementals.
+
+- Log into the Veeam Backup & Replication server and locate the local folder hosting the backup repository that is running out of space
+- Locate the oldest "**Full**" backup and delete it along with all of the "**incremental**" backups after it leading to the second-most-recent full backup.
+
+!!! warning "Incremental Backups Affecting the Chain"
+    Be mindful that if you delete the incremental backups but not the full backup associated with those incrementals you will break the backup chain.  In the event this happens, either un-delete the incrementals and try again, or go one-level-deeper and delete the second-oldest full backup and all incrementals to correct the chain's structure.
+
+## Rescan Backup Repository
+At this point, you can just re-scan the backup repository within Veeam so the Veeam database gets updated to notice the missing backup files that you just deleted.  [Rescanning Backup Repositories](https://helpcenter.veeam.com/docs/backup/vsphere/rescanning_backup_repositories.html?ver=120)
+
+- Launch Veeam Backup & Replication Console
+- Navigate to "**Backup Infrastructure > Backup Repositories**"
+- Locate the backup repository you deleted backup files from, then "**Right-click > Rescan**"
+
+## Removing Restore Points from Database
+At this point, you have deleted the backup files and re-scanned the backup repository(s) to ensure that Veeam updated its database to notice the now-missing backup files.  Now you need to tell Veeam to "forget" about the older backups you deleted so they are no longer displayed within Veeam itself.  [Removing Missing Restore Points](https://helpcenter.veeam.com/docs/backup/vsphere/remove_missing_point.html?ver=120)
+
+- Navigate to "**Home > Backups > Disk**"
+- Locate the backup job associated with the device's backup files you deleted
+- Right-click the associated backup job > "**Properties...**"
+- In the Backup Properties window, right-click the missing restore point(s) and click "**Forget**" > "**All Unavailable Backups**"

operations/backups-and-dr/veeam-backup-replication/migrating-vms-to-proxmoxve.md

@@ -0,0 +1,57 @@
+**Purpose**:
+When you migrate virtual machines from Hyper-V (and possibly other platforms) to ProxmoxVE, you may run into several issues, from the disk formats being in `.raw` format instead of `.qcow2`, among other things.  One thing in particular, which is the reason for this document, is that if you migrate Rocky Linux from Hyper-V into ProxmoxVE using Veeam Backup & Replication, it will break the storage system so badly that the operating system will not boot.
+
+
+### Fixing Boot Issues
+Some high-level things to do to fix this are listed below:
+
+- Switch the VM processor type to `host`.
+- The socket and core counts are reversed, so a single socket CPU with 16 cores will appear like 16 sockets with one core each, flip these around to correct this issue.
+- The storage controller needs to be set to `VirtIO iSCSI`
+- The display driver needs set to `Default`
+
+#### Dracut Emergency Shell
+If you start the VM and you reach a "dracut" prompt, then the bootloader got nuked and needs to be regenerated.  Follow the steps below to work through this process:
+
+- Boot from a Rocky Linux 9.5+ installation ISO in the broken Rocky Linux VM
+- Select "**Troubleshooting -->**" in the boot menu
+- Select "**Rescue a Rocky Linux System**"
+- Press through the prompt with value `1` and `Continue` to select the automatic mounting of the detected operating system of the virtual machine
+- Press **<ENTER>** to enter the shell, then run the following commands to fix the booting issues
+```sh
+chroot /mnt/sysroot
+dracut --force --regenerate-all
+grub2-mkconfig -o /boot/grub2/grub.cfg
+exit
+exit
+```
+!!! info "Boot Fix May Trigger Reboot Twice"
+    During the process, you may notice that the VM reboots itself a second-time.  This is normal and can be left alone.  The VM will eventually reach the login screen.  Once you get this far, you can login and fix the networking issues in the VM to get it stabilized.
+
+### Fixing Network Issues
+The VM will lose the adapter name of `eth0` and put something else like `ens18` that needs to be reconfigured manually to get networking functional again:
+
+- Type `ethtool ens18`, and if the link speed is `Unknown!`, then poweroff the VM and switch the network adapter from `VirtIO (paravirtualized)` to `Intel E1000`, then boot the VM back up.
+- Run the following commands to assign the new `ens18` interface as a networking interface for the VM to use:
+```sh
+# Create the Interface (Replace the IP & DNS Variables)
+nmcli connection add type ethernet ifname ens18 con-name ens18 ipv4.method manual ipv4.addresses 192.168.3.21/24 ipv4.gateway 192.168.3.1 ipv4.dns "1.1.1.1 1.0.0.1"
+
+# Bring the Connection Online
+nmcli connection up ens18
+```
+
+!!! success "VM Successfully Fixed"
+    At this point, the virtual machine should be booting, and have network access, bringing it back into production use.
+
+### Convert VM Disk from `.RAW` to `.QCOW2`
+Given that the migration process via Veeam Backup & Replication ignores the destination disk format (at the time of writing this), it is necessary to convert the format of the disk from `.raw` to `.qcow2` so that you can perform things like VM snapshots, which are essential during updates, development, and testing.
+
+Open a shell onto the ProxmoxVE server that is currently holding the VM that you need to convert the disks for, then locate the disks (this is not explained here, yet), and run the following commands to convert them.  
+```sh
+# Convert a Single Disk
+qemu-img convert -f raw -O qcow2 source.raw destination.qcow2
+
+# Convert All Disks in a Given Directory
+find . -type f -name "*.raw" -exec sh -c 'qemu-img convert -f raw -O qcow2 "$1" "${1%.raw}.qcow2"' _ {} \;
+```

operations/backups-and-dr/veeam-backup-replication/migrating-vspc-backup-repositories.md

@@ -0,0 +1,11 @@
+## Purpose
+If you find that you need to migrate cloud backups that are being sent to a server running the Veeam VSCP due to issues like exhausted storage space on existing repositories.
+
+### Migrate Backup Repository Data
+- Log into VSPC website and disable all associated jobs that need to be migrated
+- Log into the endpoint (or veeam backup proxy running at the client location, if there is one) and disable the associated job that we need to migrate the data for.
+- Navigate to the directory structure where the backup is located on-disk, such as `E:\Backups\clientname` and move it to the destination backup repository, such as `F:\Backups\clientname`
+- Log back into Veeam Backup & Replication Console and re-scan the new repositories
+
+### Move Backup Job Location in VSPC Portal
+At this point, we need to migrate point the backup job(s) that were affected to the new location.  This is a job-level change, not company-level change.

operations/backups-and-dr/veeam-backup-replication/retention-best-practices.md

@@ -0,0 +1,17 @@
+**Purpose**:
+This is meant as a high-level generally-speaking best practice retention policy in most use-cases.  This document will generally be pretty bare-bones, but the general idea is the following advanced GFS retention period is generally configured on backup copy jobs, specifically ones that have off-site backups, but can also be used for local backup repositories.
+
+Navigate to Jobs > Backup (or Backup Copy) > (Find a Backup Job) > Right-Click > Edit > Storage (or Target) > "**Keep Certain Full Backups for Archival Purposes**: Checked" > Click on the "**Configure**" button.
+
+Optional: Click the "**Save as Default**" button before clicking the "**OK**" button to make this default behavior for new backup jobs.
+
+| **Description** | **Status** | **Value** |
+| :--- | :--- | :--- |
+| Keep Weekly Full Backups | Enabled | 4 |
+| Keep Monthly Full Backups | Enabled | 3 |
+| Keep Yearly Full Backups | Enabled | 1 (`3 - 7 for Medical HIPAA`) |
+
+!!! note "7 Daily Backups Assumption"
+    This document assumes that you at (least) keep 7 daily backups in the normal backup schedule.  Meaning **7 daily, 4 weekly, 3 monthly, and 1 yearly** backup is maintained at all times.
+
+    **7 daily, 4 weekly, 3 monthly, and 1 yearly**

operations/backups-and-dr/veeam-backup-replication/troubleshooting/failed-to-validate-certificates-of-some-gateways.md

@@ -0,0 +1,29 @@
+### Symptoms
+When you try to run a backup to a remote backup server, the backup job fails and gives the following error:
+
+!!! failure "Error: No cloud gateways are available: failed to validate certificates of some gateways"
+
+### Reason
+This means that the SSL certificate installed on the Veeam Backup & Replication Server being used by the endpoint is expired.  While you can update the `Web UI` and `Server` SSL certificates, the "**Gateway Connect**" certificate is different.  When the certificate is expired, the backup agent stops trusting the backup server and fails all backup jobs to that server until the certificate is updated.
+
+### Resolution Steps
+You need to remotely log into the Veeam Backup & Replication server, doing this via RMM tools, RDP, etc.  Once logged-in, you need to open Veeam Backup & Replication and login.
+
+- At the bottom-left sidebar of Veeam Backup & Replication window, you will see tabs such as `Home`, `Inventory`, `Backup Infrastructure`, `Storage Infrastructure`, `Tape Infrastructure`, and `Cloud Connect`.  Proceed to click on the "**Cloud Connect**" tab in the sidebar.
+- Click on the "**Manage Certificates**" button
+- Click on the "**Select an existing certificate from the certificate store**" radio button
+    - You will be prompted that "*This certificate is used by one or more Cloud Gateways*", simply click on "**Yes**" to proceed.
+    - Look for the current and up-to-date / valid certificate from the certificate list then click "**Next**"
+        - e.g. `*.bunny-lab.io`
+    - You will be given a summary of the changes > Click the "**Finish**" button to finish updating the gateway's certificate.
+
+### Re-Attempt Backup
+At this point, the endpoint should immediately trust the new certificate from the remote backup server (assuming the server is `Managed` and not `Standalone`).  The backups should be running successfully the next time you run them.
+
+!!! info "Standalone Mode"
+    In the event that the device is in-fact standalone, you can run the following command on the device via commandline to tell it to immediately sync the configuration settings of the remote backup server with the local backup agent:
+
+    ```batch
+    ::Connect to the backup server and download current configuration settings.
+    "C:\Program Files\Veeam\Endpoint Backup\Veeam.Agent.Configurator.exe" -syncnow
+    ```