docs/Servers/Automation/Ansible/AWX/AWX Kerberos Implementation.md

## Kerberos Implementation
You may find that you need to be able to run playbooks on domain-joined Windows devices using Kerberos.  You need to go through some extra steps to set this up after you have successfully fully deployed AWX Operator into Kubernetes.

### Configure Windows Devices
You will need to prepare the Windows devices to allow them to be remotely controlled by Ansible playbooks.  Run the following powershell script on all of the devices that will be managed by the Ansible AWX environment.

- [WinRM Prerequisite Setup Script](https://docs.bunny-lab.io/Docker%20%26%20Kubernetes/Servers/AWX/AWX%20Operator/Enable%20Kerberos%20WinRM/)

### Create an AWX Instance Group
At this point, we need to make an "Instance Group" for the AWX Execution Environments that will use both a Keytab file and custom DNS servers defined by configmap files created below.  Reference information was found [here](https://github.com/kurokobo/awx-on-k3s/blob/main/tips/use-kerberos.md#create-container-group).  This group allows for persistence across playbooks/templates, so that if you establish a Kerberos authentication in one playbook, it will persist through the entire job's workflow.

Create the following files in the `/awx` folder on the AWX Operator server you deployed earlier when setting up the Kubernetes Cluster and deploying AWX Operator into it so we can later mount them into the new Execution Environment we will be building.

=== "Custom DNS Records"

    ```yaml title="/awx/custom_dns_records.yml"
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: custom-dns
      namespace: awx
    data:
      custom-hosts: |
        192.168.3.25 LAB-DC-01.bunny-lab.io LAB-DC-01
        192.168.3.26 LAB-DC-02.bunny-lab.io LAB-DC-02
        192.168.3.4 VIRT-NODE-01.bunny-lab.io VIRT-NODE-01
        192.168.3.5 BUNNY-NODE-02.bunny-lab.io BUNNY-NODE-02
    ```

=== "Kerberos Keytab File"

    ```ini title="/awx/krb5.conf"
    [libdefaults]
        default_realm = BUNNY-LAB.IO
        dns_lookup_realm = false
        dns_lookup_kdc = false

    [realms]
        BUNNY-LAB.IO = {
            kdc = 192.168.3.25
            kdc = 192.168.3.26
            admin_server = 192.168.3.25
        }

    [domain_realm]
        192.168.3.25 = BUNNY-LAB.IO
        192.168.3.26 = BUNNY-LAB.IO
        .bunny-lab.io = BUNNY-LAB.IO
        bunny-lab.io = BUNNY-LAB.IO
    ```

Then we apply these configmaps to the AWX namespace with the following commands: 
``` sh
cd /awx
kubectl -n awx create configmap awx-kerberos-config --from-file=/awx/krb5.conf
kubectl apply -f custom_dns_records.yml
```

- Open AWX UI and click on "**Instance Groups**" under the "**Administration**" section, then press "**Add > Add container group**".
- Enter a descriptive name as you like (e.g. `Kerberos`) and click the toggle "**Customize Pod Specification**".
- Put the following YAML string in "**Custom pod spec**" then press the "**Save**" button
```yaml title="Custom Pod Spec"
apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: default
  automountServiceAccountToken: false
  initContainers:
    - name: init-hosts
      image: busybox
      command:
        - sh
        - '-c'
        - cat /etc/custom-dns/custom-hosts >> /etc/hosts
      volumeMounts:
        - name: custom-dns
          mountPath: /etc/custom-dns
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 250m
          memory: 100Mi
      volumeMounts:
        - name: awx-kerberos-volume
          mountPath: /etc/krb5.conf
          subPath: krb5.conf
  volumes:
    - name: awx-kerberos-volume
      configMap:
        name: awx-kerberos-config
    - name: custom-dns
      configMap:
        name: custom-dns
```

### Job Template & Inventory Examples
At this point, you need to adjust your exist Job Template(s) that need to communicate via Kerberos to domain-joined Windows devices to use the "Instance Group" of "**Kerberos**" while keeping the same Execution Environment you have been using up until this point.  This will change the Execution Environment to include the Kerberos Keytab file in the EE at playbook runtime.  When the playbook has completed running, (or if you are chain-loading multiple playbooks in a workflow job template), it will cease to exist.  The kerberos keytab data will be regenerated at the next runtime.

Also add the following variables to the job template you have associated with the playbook below:
``` yaml
---
kerberos_user: nicole.rappe@BUNNY-LAB.IO
kerberos_password: <DomainPassword>
```

You will want to ensure your inventory file is configured to use Kerberos Authentication as well, so the following example is a starting point:
```ini
virt-node-01 ansible_host=virt-node-01.bunny-lab.io
bunny-node-02 ansible_host=bunny-node-02.bunny-lab.io

[virtualizationHosts]
virt-node-01
bunny-node-02

[virtualizationHosts:vars]
ansible_connection=winrm
ansible_port=5986
ansible_winrm_transport=kerberos
ansible_winrm_scheme=https
ansible_winrm_server_cert_validation=ignore
#kerberos_user=nicole.rappe@BUNNY-LAB.IO #Optional, if you define this in the Job Template, it is not necessary here.
#kerberos_password=<DomainPassword> #Optional, if you define this in the Job Template, it is not necessary here.
```
!!! failure "Usage of Fully-Quality Domain Names"
    It is **critical** that you define Kerberos-authenticated devices with fully qualified domain names.  This is just something I found out from 4+ hours of troubleshooting.  If the device is Linux or you are using NTLM authentication instead of Kerberos authentication, you can skip this warning.  If you do not define the inventory using FQDNs, it will fail to run the commands against the targeted device(s).

    In this example, the host is defined via FQDN: `virt-node-01 ansible_host=virt-node-01.bunny-lab.io`

### Kerberos Connection Playbook
At this point, you need a playbook that you can run in a Workflow Job Template (to keep things modular and simplified) to establish a connection to an Active Directory Domain Controller via Kerberos before running additional playbooks/templates against the actual devices.  

You can visualize the connection workflow below:

``` mermaid
graph LR
    A[Update AWX Project] --> B[Update Project Inventory]
    B --> C[Establish Kerberos Connection]
    C --> D[Run Playbook against Windows Device]
```

The following playbook is an example pulled from https://git.bunny-lab.io

!!! note "Playbook Redundancies"
    I have several areas where I could optimize this playbook and remove redundancies.  I just have not had enough time to iterate through it deeply-enough to narrow down exact things I can remove, so for now, it will remain as-is, since it functions as-expected with the example below.

```yaml title="Establish_Kerberos_Connection.yml"
---
- name: Generate Kerberos Ticket to Communicate with Domain-Joined Windows Devices
  hosts: localhost
  vars:
    kerberos_password: "{{ lookup('env', 'KERBEROS_PASSWORD') }}"  # Alternatively, you can set this as an environment variable
    # BE SURE TO PASS "kerberos_user: nicole.rappe@BUNNY-LAB.IO" and "kerberos_password: <domain_admin_password>" to the template variables when running this playbook in a template.

  tasks:
    - name: Generate the keytab file
      ansible.builtin.shell: |
        ktutil <<EOF
        addent -password -p {{ kerberos_user }} -k 1 -e aes256-cts
        {{ kerberos_password }}
        wkt /tmp/krb5.keytab
        quit
        EOF                
      environment:
        KRB5_CONFIG: /etc/krb5.conf
      register: generate_keytab_result

    - name: Ensure keytab file was generated successfully
      fail:
        msg: "Failed to generate keytab file"
      when: generate_keytab_result.rc != 0

    - name: Keytab successfully generated
      ansible.builtin.debug:
        msg: "Keytab successfully generated at /tmp/krb5.keytab"
      when: generate_keytab_result.rc == 0

    - name: Acquire Kerberos ticket using keytab
      ansible.builtin.shell: |
        kinit -kt /tmp/krb5.keytab {{ kerberos_user }}                
      environment:
        KRB5_CONFIG: /etc/krb5.conf
      register: kinit_result

    - name: Ensure Kerberos ticket was acquired successfully
      fail:
        msg: "Failed to acquire Kerberos ticket"
      when: kinit_result.rc != 0

    - name: Kerberos ticket successfully acquired
      ansible.builtin.debug:
        msg: "Kerberos ticket successfully acquired for user {{ kerberos_user }}"
      when: kinit_result.rc == 0
```