Restructured Documentation
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
This commit is contained in:
@@ -0,0 +1,678 @@
|
||||
---
|
||||
tags:
|
||||
- Virtualization and Storage
|
||||
- Detect and Remove Orphaned VM Disks
|
||||
---
|
||||
|
||||
## Purpose
|
||||
This procedure describes how to identify and safely remove orphaned Proxmox VE virtual machine disks from shared iSCSI-backed LVM storage.
|
||||
|
||||
It is intended for environments where:
|
||||
|
||||
- Proxmox VE is clustered.
|
||||
- Multiple Proxmox nodes access the same shared iSCSI LUN.
|
||||
- The shared storage is exposed to Proxmox as LVM storage.
|
||||
- VM disks are stored as LVM logical volumes.
|
||||
- Some volumes may remain after VM disk deletion, failed migrations, failed resizes, storage UI inconsistencies, or manual recovery work.
|
||||
|
||||
The goal is to reclaim storage space without accidentally deleting disks that are still attached to running or stopped VMs.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
This document focuses on the following storage type:
|
||||
|
||||
```text
|
||||
Proxmox storage type: lvm
|
||||
Backing storage: shared iSCSI
|
||||
Volume group: vg_proxmox_iscsi
|
||||
Storage ID example: iscsi-cluster-lvm
|
||||
```
|
||||
|
||||
Adjust the storage ID and volume group names as needed for your environment.
|
||||
|
||||
---
|
||||
|
||||
## Safety Requirements
|
||||
!!! danger "Never delete based on the Storage UI alone"
|
||||
The Proxmox storage UI may show a volume as belonging to a VM because its name follows the pattern `vm-<vmid>-disk-<n>`. That does not prove the disk is currently attached to the VM.
|
||||
|
||||
```text
|
||||
Always verify against VM configuration files and active QEMU processes before deleting.
|
||||
```
|
||||
|
||||
!!! warning "Run the audit before running any cleanup commands"
|
||||
The audit scripts in this document are read-only. The cleanup commands are destructive. Do not run cleanup commands until the audit output has been reviewed.
|
||||
|
||||
!!! warning "Snapshot volumes require extra caution"
|
||||
Volumes named like the following may be part of a snapshot chain:
|
||||
|
||||
````text
|
||||
```text
|
||||
snap_vm-<vmid>-disk-<n>_<snapshot-name>
|
||||
```
|
||||
|
||||
Do not remove snapshot volumes manually unless you have verified that the VM and snapshot are no longer known to Proxmox, no backing chain references them, and no QEMU process has them open.
|
||||
````
|
||||
|
||||
!!! note "Shared storage does not mean shared config"
|
||||
In some cluster layouts, each node may only show VM config files for VMs assigned to that node. Therefore, an audit run from only one node can falsely report disks from other nodes as orphaned.
|
||||
|
||||
```text
|
||||
Run the confirmation script on every node in the cluster.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Terms
|
||||
| Term | Meaning |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| Attached disk | A disk volume referenced in a VM config, such as `scsi0`, `sata0`, `virtio0`, `efidisk0`, or `tpmstate0`. |
|
||||
| Orphan disk | A storage volume that exists on shared storage but is not referenced by any VM config on any node and is not opened by any active process. |
|
||||
| Volume ID | Proxmox storage identifier, such as `iscsi-cluster-lvm:vm-107-disk-1.qcow2`. |
|
||||
| LV | LVM logical volume, such as `/dev/vg_proxmox_iscsi/vm-107-disk-1.qcow2`. |
|
||||
| Snapshot chain | A chain of qcow2 backing files or Proxmox snapshot volumes. |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Identify Storage Names
|
||||
Run this on any Proxmox node:
|
||||
|
||||
```bash
|
||||
pvesm status
|
||||
cat /etc/pve/storage.cfg
|
||||
vgs
|
||||
```
|
||||
|
||||
Identify the shared iSCSI/LVM storage.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Storage ID: iscsi-cluster-lvm
|
||||
VG name: vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
For the rest of this document, replace these values if your environment differs:
|
||||
|
||||
```bash
|
||||
STORAGE="iscsi-cluster-lvm"
|
||||
VG="vg_proxmox_iscsi"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Run the Storage Orphan Audit
|
||||
Run the following script on one node that can see the shared LVM storage.
|
||||
|
||||
This script does not delete anything.
|
||||
|
||||
```bash
|
||||
cat > /root/pve-iscsi-orphan-audit.sh <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -u
|
||||
|
||||
STORAGE="iscsi-cluster-lvm"
|
||||
VG="vg_proxmox_iscsi"
|
||||
OUT="/root/pve-iscsi-orphan-audit-$(hostname)-$(date +%Y%m%d-%H%M%S).txt"
|
||||
|
||||
{
|
||||
echo "===== PVE ISCSI ORPHAN AUDIT ====="
|
||||
echo "Host: $(hostname)"
|
||||
echo "Date: $(date)"
|
||||
echo "Storage: ${STORAGE}"
|
||||
echo "VG: ${VG}"
|
||||
|
||||
echo
|
||||
echo "===== STORAGE STATUS ====="
|
||||
pvesm status 2>&1 | egrep "^(Name|${STORAGE})" || true
|
||||
vgs "${VG}" 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== ALL VOLUMES IN ${STORAGE} ====="
|
||||
pvesm list "${STORAGE}" 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== ALL LVs IN ${VG} ====="
|
||||
lvs -a -o lv_name,lv_path,lv_size,lv_attr,devices "${VG}" 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== LOCAL VM CONFIG FILES ====="
|
||||
for conf in /etc/pve/qemu-server/*.conf; do
|
||||
[ -e "$conf" ] || continue
|
||||
echo
|
||||
echo "----- $conf -----"
|
||||
cat "$conf"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "===== REFERENCE ANALYSIS - LOCAL CONFIG FILES ONLY ====="
|
||||
printf '%-55s | %-8s | %-10s | %-8s | %-8s | %s\n' \
|
||||
"volume" "vmid" "referenced" "open" "size" "path"
|
||||
|
||||
{
|
||||
pvesm list "${STORAGE}" 2>/dev/null | awk 'NR>1 {print $1}' | sed "s#^${STORAGE}:##"
|
||||
lvs --noheadings -o lv_name "${VG}" 2>/dev/null | awk '{print $1}'
|
||||
} | sort -u | while read -r vol; do
|
||||
[ -n "$vol" ] || continue
|
||||
|
||||
case "$vol" in
|
||||
vm-*-disk-*|snap_vm-*-disk-*) ;;
|
||||
*) continue ;;
|
||||
esac
|
||||
|
||||
vmid="unknown"
|
||||
if [[ "$vol" =~ ^vm-([0-9]+)-disk- ]]; then
|
||||
vmid="${BASH_REMATCH[1]}"
|
||||
elif [[ "$vol" =~ ^snap_vm-([0-9]+)-disk- ]]; then
|
||||
vmid="${BASH_REMATCH[1]}"
|
||||
fi
|
||||
|
||||
ref="no"
|
||||
if grep -R -Fq "$vol" /etc/pve/qemu-server 2>/dev/null; then
|
||||
ref="yes"
|
||||
fi
|
||||
|
||||
open="no"
|
||||
if lsof 2>/dev/null | grep -Fq "$vol"; then
|
||||
open="yes"
|
||||
fi
|
||||
|
||||
size="$(lvs --noheadings -o lv_size "${VG}/${vol}" 2>/dev/null | awk '{$1=$1;print}')"
|
||||
path="$(lvs --noheadings -o lv_path "${VG}/${vol}" 2>/dev/null | awk '{$1=$1;print}')"
|
||||
|
||||
printf '%-55s | %-8s | %-10s | %-8s | %-8s | %s\n' \
|
||||
"$vol" "$vmid" "$ref" "$open" "$size" "$path"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "===== DONE ====="
|
||||
} | tee "$OUT"
|
||||
|
||||
echo
|
||||
echo "Saved audit to: $OUT"
|
||||
EOF
|
||||
|
||||
chmod +x /root/pve-iscsi-orphan-audit.sh
|
||||
/root/pve-iscsi-orphan-audit.sh
|
||||
```
|
||||
|
||||
The script writes a file similar to:
|
||||
|
||||
```text
|
||||
/root/pve-iscsi-orphan-audit-<node>-<timestamp>.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to Read the First Audit
|
||||
The most important section is:
|
||||
|
||||
```text
|
||||
REFERENCE ANALYSIS - LOCAL CONFIG FILES ONLY
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
volume | vmid | referenced | open | size
|
||||
vm-107-disk-0.qcow2 | 107 | no | no | 4.00m
|
||||
vm-107-disk-1.qcow2 | 107 | no | no | 256.04g
|
||||
vm-107-disk-2.qcow2 | 107 | yes | no | 4.00m
|
||||
vm-107-disk-3.qcow2 | 107 | yes | no | 256.04g
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| `referenced=yes` | The volume appears in a local VM config file. Do not delete. |
|
||||
| `referenced=no` | The volume does not appear in local VM configs. It may be orphaned, but confirm across all nodes first. |
|
||||
| `open=yes` | A process has the volume open. Do not delete. |
|
||||
| `open=no` | No process on this node has the volume open. Still confirm across all nodes. |
|
||||
|
||||
!!! warning "Local reference analysis is not enough"
|
||||
If a VM runs on another cluster node, its config may not appear on the node where you ran the audit. This can make valid disks look orphaned.
|
||||
|
||||
```text
|
||||
Continue to Phase 3 before deleting anything.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Run Cluster-Wide Confirmation
|
||||
Run the following script on **every Proxmox node** in the cluster.
|
||||
|
||||
This script is read-only.
|
||||
|
||||
```bash
|
||||
cat > /root/pve-cluster-vm-confirm.sh <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -u
|
||||
|
||||
STORAGE="iscsi-cluster-lvm"
|
||||
VG="vg_proxmox_iscsi"
|
||||
OUT="/root/pve-cluster-vm-confirm-$(hostname)-$(date +%Y%m%d-%H%M%S).txt"
|
||||
|
||||
{
|
||||
echo "===== NODE ====="
|
||||
hostname
|
||||
date
|
||||
|
||||
echo
|
||||
echo "===== CLUSTER RESOURCES - VMS ====="
|
||||
pvesh get /cluster/resources --type vm 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== LOCAL QM LIST ====="
|
||||
qm list 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== QEMU CONFIG FILES PRESENT ====="
|
||||
ls -la /etc/pve/qemu-server/ 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== QEMU CONFIG FILE CONTENTS ====="
|
||||
for conf in /etc/pve/qemu-server/*.conf; do
|
||||
[ -e "$conf" ] || continue
|
||||
echo
|
||||
echo "----- $conf -----"
|
||||
cat "$conf"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "===== ALL STORAGE VOLUMES ====="
|
||||
pvesm list "${STORAGE}" 2>&1 || true
|
||||
|
||||
echo
|
||||
echo "===== ALL LVs ====="
|
||||
lvs -a -o lv_name,lv_path,lv_size,lv_attr,devices "${VG}" 2>&1 || true
|
||||
|
||||
} | tee "$OUT"
|
||||
|
||||
echo
|
||||
echo "Saved to: $OUT"
|
||||
EOF
|
||||
|
||||
chmod +x /root/pve-cluster-vm-confirm.sh
|
||||
/root/pve-cluster-vm-confirm.sh
|
||||
```
|
||||
|
||||
Collect the output file from each node.
|
||||
|
||||
Example for a three-node cluster:
|
||||
|
||||
```text
|
||||
/root/pve-cluster-vm-confirm-cluster-node-01-YYYYMMDD-HHMMSS.txt
|
||||
/root/pve-cluster-vm-confirm-cluster-node-02-YYYYMMDD-HHMMSS.txt
|
||||
/root/pve-cluster-vm-confirm-cluster-node-03-YYYYMMDD-HHMMSS.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to Read the Cluster Confirmation
|
||||
For each suspicious volume, search all three outputs.
|
||||
|
||||
Example candidate:
|
||||
|
||||
```text
|
||||
vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
Check whether it appears in any VM config:
|
||||
|
||||
```bash
|
||||
grep -R "vm-107-disk-1.qcow2" /etc/pve/qemu-server/ || true
|
||||
```
|
||||
|
||||
If reviewing output files manually, look for config lines such as:
|
||||
|
||||
```text
|
||||
scsi0: iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
sata0: iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
virtio0: iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
efidisk0: iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
tpmstate0: iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
If a volume appears in any of those lines, it is attached to a VM and must not be deleted.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Classify Candidate Volumes
|
||||
Use the following decision table.
|
||||
|
||||
| Condition | Classification | Action |
|
||||
| --------------------------------------------------------------------- | ------------------- | ---------------------------------------------- |
|
||||
| Volume appears in any VM config on any node | In use | Do not delete |
|
||||
| Volume is opened by QEMU or another process | In use or unsafe | Do not delete |
|
||||
| Volume is a `snap_vm-*` snapshot volume | Snapshot-chain item | Inspect snapshot/backing chain before deletion |
|
||||
| Volume does not appear in any VM config and is not open | Orphan candidate | Eligible for final verification |
|
||||
| VMID no longer exists in cluster resources and disk is not referenced | Strong orphan | Eligible for cleanup |
|
||||
|
||||
---
|
||||
|
||||
## Example: Valid VM Disks
|
||||
If VM `107` has this config:
|
||||
|
||||
```text
|
||||
efidisk0: iscsi-cluster-lvm:vm-107-disk-2.qcow2
|
||||
sata0: iscsi-cluster-lvm:vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
Then these disks are valid and must not be deleted:
|
||||
|
||||
```text
|
||||
vm-107-disk-2.qcow2
|
||||
vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
If storage also contains:
|
||||
|
||||
```text
|
||||
vm-107-disk-0.qcow2
|
||||
vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
and neither appears in any config file on any node, those are orphan candidates.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Final Verification Before Deletion
|
||||
For each candidate volume, run the following checks on a node that can see the shared storage.
|
||||
|
||||
Replace the volume name as appropriate.
|
||||
|
||||
```bash
|
||||
VOL="vm-107-disk-1.qcow2"
|
||||
STORAGE="iscsi-cluster-lvm"
|
||||
VG="vg_proxmox_iscsi"
|
||||
|
||||
echo "===== Check all cluster config references ====="
|
||||
grep -R "$VOL" /etc/pve/qemu-server/ || true
|
||||
|
||||
echo
|
||||
echo "===== Check Proxmox storage listing ====="
|
||||
pvesm list "$STORAGE" | grep "$VOL" || true
|
||||
|
||||
echo
|
||||
echo "===== Check LVM volume ====="
|
||||
lvs -a -o lv_name,lv_path,lv_size,lv_attr,devices "$VG" | grep "$VOL" || true
|
||||
|
||||
echo
|
||||
echo "===== Check whether open by any process ====="
|
||||
lsof | grep "$VOL" || true
|
||||
|
||||
echo
|
||||
echo "===== Check qemu-img metadata if device path exists ====="
|
||||
LVPATH="$(lvs --noheadings -o lv_path "${VG}/${VOL}" 2>/dev/null | awk '{$1=$1;print}')"
|
||||
if [ -n "$LVPATH" ] && [ -e "$LVPATH" ]; then
|
||||
qemu-img info --backing-chain "$LVPATH"
|
||||
else
|
||||
echo "LV path missing or inactive: $LVPATH"
|
||||
fi
|
||||
```
|
||||
|
||||
Safe deletion pattern:
|
||||
|
||||
```text
|
||||
grep -R ... no output
|
||||
pvesm list ... shows the volume
|
||||
lvs ... shows the volume
|
||||
lsof ... no output
|
||||
qemu-img info ... no unexpected backing file dependency
|
||||
```
|
||||
|
||||
!!! danger "Stop if grep finds a reference"
|
||||
If the candidate volume appears in any `/etc/pve/qemu-server/*.conf` file, do not delete it.
|
||||
|
||||
!!! danger "Stop if lsof finds a process"
|
||||
If `lsof` shows the volume is open, do not delete it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Cleanup Commands
|
||||
## Preferred Method: Proxmox Storage Layer
|
||||
Use `pvesm free` first.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
pvesm free iscsi-cluster-lvm:vm-107-disk-0.qcow2
|
||||
pvesm free iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
Then verify:
|
||||
|
||||
```bash
|
||||
pvesm list iscsi-cluster-lvm | grep "vm-107-disk" || true
|
||||
lvs -a -o lv_name,lv_size,lv_attr,devices vg_proxmox_iscsi | grep "vm-107-disk" || true
|
||||
vgs vg_proxmox_iscsi
|
||||
pvesm status | egrep '^(Name|iscsi-cluster-lvm)'
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
vm-107-disk-0.qcow2 gone
|
||||
vm-107-disk-1.qcow2 gone
|
||||
vm-107-disk-2.qcow2 still present
|
||||
vm-107-disk-3.qcow2 still present
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fallback Method: Direct LVM Removal
|
||||
Only use this if `pvesm free` refuses and the final verification confirms the volume is not referenced and not open.
|
||||
|
||||
```bash
|
||||
lvremove /dev/vg_proxmox_iscsi/vm-107-disk-0.qcow2
|
||||
lvremove /dev/vg_proxmox_iscsi/vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
Then refresh device nodes and verify:
|
||||
|
||||
```bash
|
||||
vgscan --mknodes
|
||||
udevadm settle
|
||||
|
||||
pvesm list iscsi-cluster-lvm | grep "vm-107-disk" || true
|
||||
lvs -a -o lv_name,lv_size,lv_attr,devices vg_proxmox_iscsi | grep "vm-107-disk" || true
|
||||
vgs vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
!!! warning "Prefer pvesm free over lvremove"
|
||||
`pvesm free` lets Proxmox remove the volume through its storage abstraction. Use direct `lvremove` only when Proxmox refuses and the orphan status is already proven.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Post-Cleanup Validation
|
||||
After deleting orphan volumes, validate storage and VM health.
|
||||
|
||||
```bash
|
||||
pvesm status
|
||||
vgs vg_proxmox_iscsi
|
||||
lvs -a -o lv_name,lv_size,lv_attr,devices vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
Check the affected VM’s config:
|
||||
|
||||
```bash
|
||||
qm config 107
|
||||
```
|
||||
|
||||
Confirm the VM still starts or remains healthy:
|
||||
|
||||
```bash
|
||||
qm status 107
|
||||
```
|
||||
|
||||
If the VM is running, confirm its active QEMU process only references expected disks:
|
||||
|
||||
```bash
|
||||
ps auxww | grep "kvm -id 107" | grep -o "/dev/vg_proxmox_iscsi/[^ ,\"]*" | sort -u
|
||||
```
|
||||
|
||||
Expected example:
|
||||
|
||||
```text
|
||||
/dev/vg_proxmox_iscsi/vm-107-disk-2.qcow2
|
||||
/dev/vg_proxmox_iscsi/vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Snapshot Volume Handling
|
||||
Snapshot volumes require additional review.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
snap_vm-105-disk-0_Fresh_Install.qcow2
|
||||
snap_vm-106-disk-0_Fresh_Install_FullyUpdated.qcow2
|
||||
```
|
||||
|
||||
Before deleting a snapshot volume, check:
|
||||
|
||||
```bash
|
||||
qm config <vmid>
|
||||
qm listsnapshot <vmid>
|
||||
grep -R "snap_vm-<vmid>" /etc/pve/qemu-server/ || true
|
||||
qemu-img info --backing-chain /dev/vg_proxmox_iscsi/vm-<vmid>-disk-<n>.qcow2
|
||||
```
|
||||
|
||||
If the VM still has a `parent:` line or `qm listsnapshot` shows the snapshot, remove it through Proxmox first:
|
||||
|
||||
```bash
|
||||
qm delsnapshot <vmid> <snapshot-name>
|
||||
```
|
||||
|
||||
Only consider manual removal if:
|
||||
|
||||
- the VM no longer references the snapshot,
|
||||
- no backing chain references the snapshot volume,
|
||||
- no QEMU process has it open,
|
||||
- and Proxmox cannot delete it normally.
|
||||
|
||||
!!! danger "Do not manually delete active snapshot-chain volumes"
|
||||
Deleting an active snapshot backing volume can corrupt the VM disk chain.
|
||||
|
||||
---
|
||||
|
||||
## Example Cleanup Walkthrough
|
||||
## Scenario
|
||||
VM `107` has this config:
|
||||
|
||||
```text
|
||||
efidisk0: iscsi-cluster-lvm:vm-107-disk-2.qcow2
|
||||
sata0: iscsi-cluster-lvm:vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
Storage contains:
|
||||
|
||||
```text
|
||||
vm-107-disk-0.qcow2
|
||||
vm-107-disk-1.qcow2
|
||||
vm-107-disk-2.qcow2
|
||||
vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
`disk-0` and `disk-1` do not appear in any config and are not open by any process.
|
||||
|
||||
## Verify
|
||||
```bash
|
||||
grep -R "vm-107-disk-0.qcow2" /etc/pve/qemu-server/ || true
|
||||
grep -R "vm-107-disk-1.qcow2" /etc/pve/qemu-server/ || true
|
||||
|
||||
lsof | grep "vm-107-disk-0.qcow2" || true
|
||||
lsof | grep "vm-107-disk-1.qcow2" || true
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
no output
|
||||
```
|
||||
|
||||
## Delete
|
||||
```bash
|
||||
pvesm free iscsi-cluster-lvm:vm-107-disk-0.qcow2
|
||||
pvesm free iscsi-cluster-lvm:vm-107-disk-1.qcow2
|
||||
```
|
||||
|
||||
## Validate
|
||||
```bash
|
||||
pvesm list iscsi-cluster-lvm | grep "vm-107-disk"
|
||||
lvs -a -o lv_name,lv_size,lv_attr,devices vg_proxmox_iscsi | grep "vm-107-disk"
|
||||
vgs vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
Expected remaining volumes:
|
||||
|
||||
```text
|
||||
vm-107-disk-2.qcow2
|
||||
vm-107-disk-3.qcow2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Technician Checklist
|
||||
Use this checklist before removing any orphan disk.
|
||||
|
||||
- [ ] I ran the storage orphan audit.
|
||||
- [ ] I ran the cluster confirmation script on every Proxmox node.
|
||||
- [ ] I confirmed the candidate volume is not referenced in any VM config.
|
||||
- [ ] I confirmed the candidate volume is not open by any process.
|
||||
- [ ] I confirmed the candidate volume is not part of an active snapshot chain.
|
||||
- [ ] I confirmed the VMID relationship is understood.
|
||||
- [ ] I used `pvesm free` first.
|
||||
- [ ] I used `lvremove` only if Proxmox refused and the volume was proven orphaned.
|
||||
- [ ] I validated storage state after cleanup.
|
||||
- [ ] I validated the affected VM still references only expected disks.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Commands
|
||||
## List shared storage volumes
|
||||
```bash
|
||||
pvesm list iscsi-cluster-lvm
|
||||
```
|
||||
|
||||
## List LVs
|
||||
```bash
|
||||
lvs -a -o lv_name,lv_path,lv_size,lv_attr,devices vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
## Search VM configs
|
||||
```bash
|
||||
grep -R "vm-<vmid>-disk-<n>" /etc/pve/qemu-server/ || true
|
||||
```
|
||||
|
||||
## Check open files
|
||||
```bash
|
||||
lsof | grep "vm-<vmid>-disk-<n>" || true
|
||||
```
|
||||
|
||||
## Check image metadata
|
||||
```bash
|
||||
qemu-img info --backing-chain /dev/vg_proxmox_iscsi/vm-<vmid>-disk-<n>.qcow2
|
||||
```
|
||||
|
||||
## Delete via Proxmox
|
||||
```bash
|
||||
pvesm free iscsi-cluster-lvm:vm-<vmid>-disk-<n>.qcow2
|
||||
```
|
||||
|
||||
## Delete via LVM fallback
|
||||
```bash
|
||||
lvremove /dev/vg_proxmox_iscsi/vm-<vmid>-disk-<n>.qcow2
|
||||
```
|
||||
|
||||
## Verify storage usage
|
||||
```bash
|
||||
pvesm status
|
||||
vgs vg_proxmox_iscsi
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
tags:
|
||||
- Proxmox
|
||||
- Virtualization and Storage
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Sometimes in some very specific situations, you will find that an LVM / VG just won't come online in ProxmoxVE. If this happens, you can run the following commands (and replace the placeholder location) to manually bring the storage online.
|
||||
|
||||
```sh
|
||||
lvchange -an local-vm-storage/local-vm-storage
|
||||
lvchange -an local-vm-storage/local-vm-storage_tmeta
|
||||
lvchange -an local-vm-storage/local-vm-storage_tdata
|
||||
vgchange -ay local-vm-storage
|
||||
```
|
||||
|
||||
!!! info "Be Patient"
|
||||
It can take some time for everything to come online.
|
||||
|
||||
!!! success
|
||||
If you see something like this: `6 logical volume(s) in volume group "local-vm-storage" now active`, then you successfully brought the volume online.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
tags:
|
||||
- Proxmox
|
||||
- Virtualization and Storage
|
||||
---
|
||||
|
||||
## Purpose
|
||||
The purpose of this document is to outline common tasks that you may need to run in your cluster to perform various tasks.
|
||||
|
||||
## Delete Node from Cluster
|
||||
Sometimes you may need to delete a node from the cluster if you have re-built it or had issues and needed to destroy it. In these instances, you would run the following command (assuming you have a 3-node quorum in your cluster).
|
||||
|
||||
```sh
|
||||
pvecm delnode promox-node-01
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
---
|
||||
tags:
|
||||
- Proxmox
|
||||
- Veeam
|
||||
- Backup
|
||||
- Disaster Recovery
|
||||
---
|
||||
|
||||
## Purpose
|
||||
When you migrate virtual machines from Hyper-V (and possibly other platforms) to ProxmoxVE, you may run into several issues, from the disk formats being in `.raw` format instead of `.qcow2`, among other things. One thing in particular, which is the reason for this document, is that if you migrate Rocky Linux from Hyper-V into ProxmoxVE using Veeam Backup & Replication, it will break the storage system so badly that the operating system will not boot.
|
||||
|
||||
### Fixing Boot Issues
|
||||
Some high-level things to do to fix this are listed below:
|
||||
|
||||
- Switch the VM processor type to `host`.
|
||||
- The socket and core counts are reversed, so a single socket CPU with 16 cores will appear like 16 sockets with one core each, flip these around to correct this issue.
|
||||
- The storage controller needs to be set to `VirtIO iSCSI`
|
||||
- The display driver needs set to `Default`
|
||||
|
||||
#### Dracut Emergency Shell
|
||||
If you start the VM and you reach a "dracut" prompt, then the bootloader got nuked and needs to be regenerated. Follow the steps below to work through this process:
|
||||
|
||||
- Boot from a Rocky Linux 9.5+ installation ISO in the broken Rocky Linux VM
|
||||
- Select "**Troubleshooting -->**" in the boot menu
|
||||
- Select "**Rescue a Rocky Linux System**"
|
||||
- Press through the prompt with value `1` and `Continue` to select the automatic mounting of the detected operating system of the virtual machine
|
||||
- Press **<ENTER>** to enter the shell, then run the following commands to fix the booting issues
|
||||
|
||||
```sh
|
||||
chroot /mnt/sysroot
|
||||
dracut --force --regenerate-all
|
||||
grub2-mkconfig -o /boot/grub2/grub.cfg
|
||||
exit
|
||||
exit
|
||||
```
|
||||
|
||||
!!! info "Boot Fix May Trigger Reboot Twice"
|
||||
During the process, you may notice that the VM reboots itself a second-time. This is normal and can be left alone. The VM will eventually reach the login screen. Once you get this far, you can login and fix the networking issues in the VM to get it stabilized.
|
||||
|
||||
### Fixing Network Issues
|
||||
The VM will lose the adapter name of `eth0` and put something else like `ens18` that needs to be reconfigured manually to get networking functional again:
|
||||
|
||||
- Type `ethtool ens18`, and if the link speed is `Unknown!`, then poweroff the VM and switch the network adapter from `VirtIO (paravirtualized)` to `Intel E1000`, then boot the VM back up.
|
||||
- Run the following commands to assign the new `ens18` interface as a networking interface for the VM to use:
|
||||
|
||||
```sh
|
||||
# Create the Interface (Replace the IP & DNS Variables)
|
||||
nmcli connection add type ethernet ifname ens18 con-name ens18 ipv4.method manual ipv4.addresses 192.168.3.21/24 ipv4.gateway 192.168.3.1 ipv4.dns "1.1.1.1 1.0.0.1"
|
||||
|
||||
# Bring the Connection Online
|
||||
nmcli connection up ens18
|
||||
```
|
||||
|
||||
!!! success "VM Successfully Fixed"
|
||||
At this point, the virtual machine should be booting, and have network access, bringing it back into production use.
|
||||
|
||||
### Convert VM Disk from `.RAW` to `.QCOW2`
|
||||
Given that the migration process via Veeam Backup & Replication ignores the destination disk format (at the time of writing this), it is necessary to convert the format of the disk from `.raw` to `.qcow2` so that you can perform things like VM snapshots, which are essential during updates, development, and testing.
|
||||
|
||||
Open a shell onto the ProxmoxVE server that is currently holding the VM that you need to convert the disks for, then locate the disks (this is not explained here, yet), and run the following commands to convert them.
|
||||
|
||||
```sh
|
||||
# Convert a Single Disk
|
||||
qemu-img convert -f raw -O qcow2 source.raw destination.qcow2
|
||||
|
||||
# Convert All Disks in a Given Directory
|
||||
find . -type f -name "*.raw" -exec sh -c 'qemu-img convert -f raw -O qcow2 "$1" "${1%.raw}.qcow2"' _ {} \;
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
tags:
|
||||
- Virtualization and Storage
|
||||
- Repair iSCSI Connections After Reboot
|
||||
---
|
||||
|
||||
## Purpose
|
||||
You may enounter an issue where when you reboot your ProxmoxVE cluster node, it will fail to start virtual machines because of an error related to being unable to see the underlying LVM disks for the GuestVM. This is generally bandaid-fixed by running `iscsiadm -m node --login` on the node, which makes it reconnect to the cluster's iSCSI storage. This is not a viable long-term solution.
|
||||
|
||||
### Configure Automatic Startup of iSCSI Services
|
||||
Run these commands on every ProxmoxVE cluster node:
|
||||
|
||||
```sh
|
||||
systemctl enable --now iscsid
|
||||
systemctl enable --now open-iscsi
|
||||
```
|
||||
|
||||
### Discover Targets (To Create Necessary iSCSI Records)
|
||||
```sh
|
||||
iscsiadm -m discovery -t sendtargets -p 192.168.3.3:3260
|
||||
```
|
||||
|
||||
### Configure Automatic iSCSI Target Connection Behavior
|
||||
Run these commands on every ProxmoxVE cluster node:
|
||||
|
||||
```sh
|
||||
iscsiadm -m node \
|
||||
-T iqn.2026-01.io.bunny-lab:storage:iscsi-cluster-storage \
|
||||
-p 192.168.3.3 \
|
||||
--op update \
|
||||
-n node.startup \
|
||||
-v automatic
|
||||
|
||||
iscsiadm -m node \
|
||||
-T iqn.2026-01.io.bunny-lab:storage:iscsi-cluster-storage \
|
||||
-p 192.168.3.3 \
|
||||
--op update \
|
||||
-n node.conn[0].startup \
|
||||
-v automatic
|
||||
```
|
||||
|
||||
### Verify Configuration
|
||||
You will want to ensure that `node.startup = automatic` and `node.conn[0].startup = automatic` when you run the following command.
|
||||
|
||||
```sh
|
||||
iscsiadm -m node -o show | grep -E 'node.name|node.conn\[0\].address|node.startup|node.conn\[0\].startup'
|
||||
```
|
||||
|
||||
### Login/Mount iSCSI Targets
|
||||
```sh
|
||||
iscsiadm -m node \
|
||||
-T iqn.2026-01.io.bunny-lab:storage:iscsi-cluster-storage \
|
||||
-p 192.168.3.3:3260 \
|
||||
--login
|
||||
```
|
||||
|
||||
!!! success "Example Output"
|
||||
If everything worked correctly, you should see output like the example below:
|
||||
|
||||
```sh
|
||||
node.name = iqn.2026-01.io.bunny-lab:storage:iscsi-cluster-storage
|
||||
node.startup = automatic
|
||||
node.conn[0].address = 192.168.3.3
|
||||
node.conn[0].startup = automatic
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
tags:
|
||||
- Proxmox
|
||||
- Virtualization and Storage
|
||||
---
|
||||
|
||||
## Purpose
|
||||
There are a few steps you have to take when upgrading ProxmoxVE from 8.4.1+ to 9.0+. The process is fairly straightforward, so just follow the instructions seen below.
|
||||
|
||||
!!! info "GuestVM Assumptions"
|
||||
It is assumed that if you are running a ProxmoxVE cluster, you will migrate all GuestVMs to another cluster node. If this is a standalone ProxmoxVE server, you will shut down all GuestVMs safely before proceeding.
|
||||
|
||||
!!! warning "Perform `pve8to9` Readiness Check"
|
||||
It's critical that you run the `pve8to9` command to ensure that your ProxmoxVE server meets all of the requirements and doesn't have any failures or potentially server-breaking warnings. If the `pve8to9` command is unknown, then run `apt update && apt dist-upgrade` in the shell then try again. Warnings should be addressed ad-hoc, but *CPU Microcode warnings can be safely ignored*.
|
||||
|
||||
**Example pve8to9 Summary Output**:
|
||||
|
||||
```sh
|
||||
= SUMMARY =
|
||||
|
||||
TOTAL: 48
|
||||
PASSED: 39
|
||||
SKIPPED: 8
|
||||
WARNINGS: 1
|
||||
FAILURES: 0
|
||||
```
|
||||
|
||||
### Update Repositories from `bookworm` to `trixie`
|
||||
```sh
|
||||
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
|
||||
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list.d/pve-install-repo.list
|
||||
apt update
|
||||
```
|
||||
|
||||
### Upgrade to ProxmoxVE 9.0
|
||||
!!! warning "Run Upgrade Commands in iLO/iDRAC/IPMI"
|
||||
At this point, its very likely that if you are using SSH, it may unexpectedly have the session terminated, so you absolutely want to use a local or remote console to the server to run the commands below, both to ensure you maintain access to the console, as well as to see if any issues arise during POST after the reboot.
|
||||
|
||||
```sh
|
||||
apt dist-upgrade -y
|
||||
reboot
|
||||
```
|
||||
|
||||
!!! note "Disable `pve-enterprise` Repository"
|
||||
At this point, the ProxmoxVE server should be running on v9.0+, you will want to disable the `pve-enterprise` repository as it will goof up future updates if you don't disable it.
|
||||
|
||||
## Related Documentation
|
||||
- [Related Proxmox Documentation](<../../../reference/Virtualization and Storage/Proxmox/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
Reference in New Issue
Block a user