docs/Servers/Virtualization/Proxmox/Configuring iSCSI-based Cluster Storage.md

## Purpose
This document describes the **end-to-end procedure** for creating a **thick-provisioned iSCSI-backed shared storage target** on **TrueNAS CORE**, and consuming it from a **Proxmox VE cluster** using **shared LVM**.

This approach is intended to:
- Replace NFS-backed VM storage
- Provide SAN-style block semantics
- Enable Proxmox-native snapshot functionality (LVM volume chains)
- Avoid third-party plugins or middleware
- Be fully reproducible via CLI

---

## Assumptions
- TrueNAS **CORE** (not SCALE)
- ZFS pool already exists and is healthy
- SSH service is enabled on TrueNAS
- Proxmox VE nodes have network connectivity to TrueNAS
- iSCSI traffic is on a reliable, low-latency network (10GbE recommended)
- All VM workloads are drained from at least one Proxmox node for maintenance

!!! warning "Important"
    `volblocksize` **cannot be changed after zvol creation**. Choose carefully.

---

## Target Architecture

```
ZFS Pool
 └─ Zvol (Thick / Reserved)
    └─ iSCSI Extent
       └─ Proxmox LVM PV
          └─ Shared VG
             └─ VM Disks
```

---

## Step 1 – Create a Dedicated Zvol for Proxmox

### Variables
Adjust as needed before execution.

```
POOL_NAME="CLUSTER-STORAGE"
ZVOL_NAME="iscsi-proxmox"
ZVOL_SIZE="14T"
VOLBLOCKSIZE="16K"
```

### Create the Zvol (Thick-Provisioned)
```
zfs create -V ${ZVOL_SIZE} \
  -o volblocksize=${VOLBLOCKSIZE} \
  -o compression=lz4 \
  -o refreservation=${ZVOL_SIZE} \
  ${POOL_NAME}/${ZVOL_NAME}
```

!!! note
    The `refreservation` enforces **true thick provisioning** and prevents overcommit.

---

## Step 2 – Configure iSCSI Target (TrueNAS CLI)

### Enable iSCSI Service
```
service iscsitarget start
sysrc iscsitarget_enable=YES
```

---

### Create iSCSI Portal
```
ctladm create -p 0.0.0.0
```

---

### Create Target
```
ctladm create -t proxmox-target
```

---

### Create Extent
```
ctladm create -b block \
  -l /dev/zvol/${POOL_NAME}/${ZVOL_NAME} \
  proxmox-extent
```

---

### Associate Target + Extent
```
ctladm add -t proxmox-target -l proxmox-extent
```

!!! tip
    At this point, the LUN is live and can be discovered by initiators.

---

## Step 3 – Connect from Proxmox VE Nodes

Perform the following **on each Proxmox node**.

### Install iSCSI Utilities
```
apt update
apt install -y open-iscsi lvm2
```

---

### Discover Target
```
iscsiadm -m discovery -t sendtargets -p <TRUENAS_IP>
```

---

### Log In
```
iscsiadm -m node --login
```

---

### Verify Device
```
lsblk
```

---

## Step 4 – Create Shared LVM (One Node Only)

!!! warning "Important"
    **Only run LVM creation on ONE node**. All other nodes will only scan.

### Initialize Physical Volume
```
pvcreate /dev/sdX
```

---

### Create Volume Group
```
vgcreate vg_proxmox_iscsi /dev/sdX
```

---

## Step 5 – Register Storage in Proxmox

### Rescan LVM (Other Nodes)
```
pvscan
vgscan
```

---

### Add Storage (GUI or CLI)

**Datacenter → Storage → Add → LVM**

- ID: `iscsi-lvm`
- Volume Group: `vg_proxmox_iscsi`
- Content: `Disk image`
- Shared: ✔️

---

## Step 6 – Validation

- Snapshot create / revert / delete
- Live migration between nodes
- PBS backup and restore test

!!! success
    If all validation tests pass, the storage is production-ready.

---

## Step 7 – Decommission NFS (After Cutover)

```
zfs destroy CLUSTER-STORAGE/NFS-STORAGE
```

---

## Step 8 – Expand iSCSI Storage (No Downtime)

### Expand Zvol (TrueNAS)
```
zfs set volsize=16T CLUSTER-STORAGE/iscsi-proxmox
zfs set refreservation=16T CLUSTER-STORAGE/iscsi-proxmox
```

---

### Rescan on Proxmox Nodes
```
pvresize /dev/sdX
```

---

## Final Notes
- Maintain ≥15–20% free pool capacity
- Monitor pool usage and fragmentation
- Prefer PBS restores for large migrations