## Initial Installation / Configuration
Proxmox Virtual Environment is an open source server virtualization management solution based on QEMU/KVM and LXC. You can manage virtual machines, containers, highly available clusters, storage and networks with an integrated, easy-to-use web interface or via CLI.

!!! note
    This document assumes you have a storage server that hosts both ISO files via CIFS/SMB share, and has the ability to set up an iSCSI LUN (VM & Container storage).  This document assumes that you are using a TrueNAS Core server to host both of these services.

### Create the first Node
You will need to download the [Proxmox VE 8.1 ISO Installer](https://www.proxmox.com/en/downloads) from the Official Proxmox Website.  Once it is downloaded, you can use [Balena Etcher](https://etcher.balena.io/#download-etcher) or [Rufus](https://rufus.ie/en/) to deploy Proxmox onto a server.

!!! warning
    If you are virtualizing Proxmox under a Hyper-V environment, you will need to follow the [Official Documentation](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/enable-nested-virtualization) to ensure that nested virtualization is enabled.  An example is listed below:
    ```
    Set-VMProcessor -VMName <VMName> -ExposeVirtualizationExtensions $true # (1)
    Get-VMNetworkAdapter -VMName <VMName> | Set-VMNetworkAdapter -MacAddressSpoofing On # (2)
    ``` 

    1.  This tells Hyper-V to allow the GuestVM to behave as a hypervisor, nested under Hyper-V, allowing the virtualization functionality of the Hypervisor's CPU to be passed-through to the GuestVM.
    2.  This tells Hyper-V to allow your GuestVM to have multiple nested virtual machines with their own independant MAC addresses.  This is useful when using nested Virtual Machines, but is also a requirement when you set up a [Docker Network](https://docs.bunny-lab.io/Containers/Docker/Docker%20Networking/) leveraging MACVLAN technology.

### Networking
You will need to set a static IP address, in this case, it will be an address within the 20GbE network.  You will be prompted to enter these during the ProxmoxVE installation.  Be sure to set the hostname to something that matches the following FQDN: `proxmox-node-01.MOONGATE.local`.

| Hostname        | IP Address      | Subnet Mask         | Gateway | DNS Server | iSCSI Portal IP   |
| --------------- | --------------- | ------------------- | ------- | ---------- | ----------------- |
| proxmox-node-01 | 192.168.101.200 | 255.255.255.0 (/24) | None    | 1.1.1.1    | 192.168.101.100   |
| proxmox-node-01 | 192.168.103.200 | 255.255.255.0 (/24) | None    | 1.1.1.1    | 192.168.103.100   |
| proxmox-node-02 | 192.168.102.200 | 255.255.255.0 (/24) | None    | 1.1.1.1    | 192.168.102.100   |
| proxmox-node-02 | 192.168.104.200 | 255.255.255.0 (/24) | None    | 1.1.1.1    | 192.168.104.100   |

### iSCSI Initator Configuration
You will need to add the iSCSI initiator from the proxmox node to the allowed initiator list in TrueNAS Core under "**Sharing > Block Shares (iSCSI) > Initiators Groups**"

In this instance, we will reference Group ID: `2`.  We need to add the iniator to the "**Allowed Initiators (IQN)**" section. This also includes the following networks that are allowed to connect to the iSCSI portal:

- `192.168.101.0/24`
- `192.168.102.0/24`
- `192.168.103.0/24`
- `192.168.104.0/24`

To get the iSCSI Initiator IQN of the current Proxmox node, you need to navigate to the Proxmox server's webUI, typically located at `https://<IP>:8006` then log in with username `root` and whatever you set the password to during initial setup when the ISO image was mounted earlier.

- On the left-hand side, click on the name of the server node (e.g. `proxmox-node-01` or `proxmox-node-02`)
- Click on "**Shell**" to open a CLI to the server
- Run the following command to get the iSCSI Initiator (IQN) name to give to TrueNAS Core for the previously-mentioned steps:
``` sh
cat /etc/iscsi/initiatorname.iscsi | grep "InitiatorName=" | sed 's/InitiatorName=//'
```

!!! example
    Output of this command will look something like `iqn.1993-08.org.debian:01:b16b0ff1778`.

## Disable Enterprise Subscription functionality
You will likely not be paying for / using the enterprise subscription, so we are going to disable that functionality and enable unstable builds.  The unstable builds are surprisingly stable, and should not cause you any issues.

Add Unstable Update Repository:
```jsx title="/etc/apt/sources.list"
# Add to the end of the file
# Non-Production / Unstable Updates
deb https://download.proxmox.com/debian bookworm pve-no-subscription
```

!!! warning
    Please note the reference to `bookworm` in both the sections above and below this notice, this may be different depending on the version of ProxmoxVE you are deploying.  Please reference the version indicated by the rest of the entries in the sources.list file to know which one to use in the added line section.

Comment-Out Enterprise Repository:
```jsx title="/etc/apt/sources.list.d/pve-enterprise.list"
# deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
```

Pull / Install Available Updates:
``` sh
apt-get update
apt dist-upgrade
reboot
```

## NIC Teaming
You will need to set up NIC teaming to configure a LACP LAGG.  This will add redundancy and a way for devices outside of the 20GbE backplane to interact with the server.  

- Ensure that all of the network interfaces appear as something similar to the following:
```jsx title="/etc/network/interfaces"
iface eno1 inet manual
iface eno2 inet manual
# etc
```

- Adjust the network interfaces to add a bond:
```jsx title="/etc/network/interfaces"
iface eno1 inet manual
iface eno2 inet manual

iface bond0 inet manual
        bond-slaves eno1 eno2 # Interfaces to assign to the bond
        bond-miimon 100 # Enable Bond Monitoring - Suggested to Configure based on Proxmox Guidance
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 192.168.3.4/24
        gateway 192.168.3.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
```

!!! warning
    Be sure to include both interfaces for the (Dual-Port) 10GbE connections in the network configuration.  Final example document will be updated at a later point in time once the production server is operational.

- Reboot the server again to make the networking changes take effect fully.  Use iLO / iDRAC / IPMI if you have that functionality on your server in case your configuration goes errant and needs manual intervention / troubleshooting to re-gain SSH control of the proxmox server.