**Purpose**: There is a way to incorporate ProxmoxVE and TrueNAS more deeply using SSH, simplifying the deployment of virtual disks/volumes passed into GuestVMs in ProxmoxVE.  Using ZFS over iSCSI will give you the following non-exhaustive list of benefits:

- Automatically make Zvols in a ZFS Storage Pool
- Automatically bind device-based iSCSI Extents/LUNs to the Zvols
- Allow TrueNAS to handle VM snapshots directly
- Simplify the filesystem overhead of using TrueNAS and iSCSI with ProxmoxVE

!!! note "Environment Assumptions"
    This document assumes you are running at least 2 ProxmoxVE nodes.  For the sake of the example, it will assume they are named `proxmox-node-01` and `proxmox-node-02`.  We will also assume you are using TrueNAS Core.  TrueNAS SCALE (should work) in the same way, but there may be minor operational / setup differences between the two different deployments of TrueNAS.

    Secondly, this guide assumes the ProxmoxVE cluster nodes and TrueNAS server exist on the same network `192.168.101.0/24`.

## ZFS over iSCSI Operational Flow
``` mermaid
sequenceDiagram
    participant ProxmoxVE as ProxmoxVE Cluster
    participant TrueNAS as TrueNAS Core (inc. iSCSI & ZFS Storage)
    participant Snapshot

    ProxmoxVE->>TrueNAS: Cluster VM node connects via SSH to create ZVol for VM
    TrueNAS->>TrueNAS: Create ZVol in ZFS storage pool
    TrueNAS->>TrueNAS: Bind ZVol to iSCSI LUN
    ProxmoxVE->>TrueNAS: Connect to iSCSI & attach ZVol as VM storage
    ProxmoxVE->>TrueNAS: Connect via SSH to create VM snapshot of ZVol
    TrueNAS->>Snapshot: Create Snapshot of ZVol/VM
```

## All ProxmoxVE Cluster Nodes
### Configure SSH Key Exchange
The first step is creating SSH trust between the ProxmoxVE cluster nodes and the TrueNAS storage appliance.  You will leverage the ProxmoxVE `shell` to run the following commands.

``` sh
apt-get install librest-client-perl git unzip # (4)
mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.168.101.100_id_rsa # (1)
ssh-copy-id -i /etc/pve/priv/zfs/192.168.101.100_id_rsa.pub root@192.168.101.100 # (2)
ssh -i /etc/pve/priv/zfs/192.168.101.100_id_rsa root@192.168.101.100 # (3)
iscsiadm --mode discovery --op update --type sendtargets --portal 192.168.101.100
iscsiadm -m node -T iqn.2005-10.org.moon-storage-01.ctl:proxmox-zfs-storage -p 192.168.101.100 -l
iscsiadm -m node # (5)
iscsiadm -m session # (6)
```
    
1.  Do not set a password.  It will break the automatic functionality.
2.  Send the SSH key to the TrueNAS server.
3.  Connect to the TrueNAS server at least once to finish establishing the connection.
4.  Some prerequisites that need to be installed.
5.  Making sure that it's actually communicating with the iSCSI target successfully or not.
6.  Making sure that it's actually communicating with the iSCSI target successfully or not.

### Install & Configure Storage Provider
Now you need to set up the storage provider in TrueNAS via a series of "patches" provided by the community.  You will run the commands below within a ProxmoxVE shell, then when finished, log out of the ProxmoxVE WebUI, clear the browser cache for ProxmoxVE, then log back in.  This will have added a new storage provider called `FreeNAS-API` under the `ZFS over iSCSI` storage type.

``` sh
cd /tmp
wget https://github.com/TheGrandWazoo/freenas-proxmox/archive/refs/tags/v2.2.zip
unzip v2.2.zip
cd freenas-proxmox
patch -b /usr/share/pve-manager/js/pvemanagerlib.js < pve-manager/js/pvemanagerlib.js.patch
patch -b /usr/share/perl5/PVE/Storage/ZFSPlugin.pm < perl5/PVE/Storage/ZFSPlugin.pm.patch
patch -b /usr/share/pve-docs/api-viewer/apidoc.js < pve-docs/api-viewer/apidoc.js.patch
cp perl5/PVE/Storage/LunCmd/FreeNAS.pm /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd
```

## Primary ProxmoxVE Cluster Node
From this point, we are ready to add the shared storage provider to the cluster via the primary node in the cluster.  This is not strictly required, just simplifies the documentation.

Navigate to **"Datacenter (BUNNY-CLUSTER) > Storage > Add > ZFS over iSCSI"**

| **Field** | **Value** | **Additional Notes** |
| :--- | :--- | :--- |
| ID | `bunny-zfs-over-iscsi` | Friendly Name |
| Portal | `192.168.101.100` | IP Address of iSCSI Portal |
| Pool | `PROXMOX-ZFS-STORAGE` | This is the ZFS Storage Pool you will use to store GuestVM Disks |
| ZFS Block Size | `4k` | |
| Target | `iqn.2005-10.org.moon-storage-01.ctl:proxmox-zfs-storage` | The iSCSI Target |
| Target Group | `<Leave Blank>` | |
| Enable | `<Checked>` | |
| iSCSI Provider | `FreeNAS-API` | |
| Thin-Provision | `<Checked>` | |
| Write Cache | `<Checked>` | |
| API use SSL | `<Unchecked>` | Disabled unless you have SSL Enabled on TrueNAS |
| API Username | `root` | This is the account that is allowed to make ZFS zvols / datasets |
| API IPv4 Host | `192.168.101.100` | iSCSI Portal Address |
| API Password | `<Root Password of TrueNAS Box>` | |
| Nodes | `proxmox-node-01,proxmox-node-02` | All ProxmoxVE Cluster Nodes |