Restructured Documentation
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
Automatic Documentation Deployment / Sync Docs to https://kb.bunny-lab.io (push) Successful in 8s
This commit is contained in:
@@ -0,0 +1,220 @@
|
||||
---
|
||||
tags:
|
||||
- Ansible
|
||||
- AWX
|
||||
- Kerberos
|
||||
- Automation
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Configure the AWX execution environment and workflow used to authenticate to domain-joined Windows targets with Kerberos. The AWX Operator deployment and Windows remote-management prerequisites must already be in place.
|
||||
|
||||
## Kerberos Implementation
|
||||
You may find that you need to be able to run playbooks on domain-joined Windows devices using Kerberos. You need to go through some extra steps to set this up after you have successfully fully deployed AWX Operator into Kubernetes.
|
||||
|
||||
### Configure Windows Devices
|
||||
You will need to prepare the Windows devices to allow them to be remotely controlled by Ansible playbooks. Run the following powershell script on all of the devices that will be managed by the Ansible AWX environment.
|
||||
|
||||
- [WinRM Prerequisite Setup Script](<../../Identity and Certificates/Windows/Enable WinRM over HTTPS.md>)
|
||||
|
||||
### Create an AWX Instance Group
|
||||
At this point, we need to make an "Instance Group" for the AWX Execution Environments that will use both a Keytab file and custom DNS servers defined by configmap files created below. Reference information was found [here](https://github.com/kurokobo/awx-on-k3s/blob/main/tips/use-kerberos.md#create-container-group). This group allows for persistence across playbooks/templates, so that if you establish a Kerberos authentication in one playbook, it will persist through the entire job's workflow.
|
||||
|
||||
Create the following files in the `/awx` folder on the AWX Operator server you deployed earlier when setting up the Kubernetes Cluster and deploying AWX Operator into it so we can later mount them into the new Execution Environment we will be building.
|
||||
|
||||
=== "Custom DNS Records"
|
||||
|
||||
```yaml title="/awx/custom_dns_records.yml"
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: custom-dns
|
||||
namespace: awx
|
||||
data:
|
||||
custom-hosts: |
|
||||
192.168.3.25 LAB-DC-01.bunny-lab.io LAB-DC-01
|
||||
192.168.3.26 LAB-DC-02.bunny-lab.io LAB-DC-02
|
||||
192.168.3.4 VIRT-NODE-01.bunny-lab.io VIRT-NODE-01
|
||||
192.168.3.5 BUNNY-NODE-02.bunny-lab.io BUNNY-NODE-02
|
||||
```
|
||||
|
||||
=== "Kerberos Keytab File"
|
||||
|
||||
```ini title="/awx/krb5.conf"
|
||||
[libdefaults]
|
||||
default_realm = BUNNY-LAB.IO
|
||||
dns_lookup_realm = false
|
||||
dns_lookup_kdc = false
|
||||
|
||||
[realms]
|
||||
BUNNY-LAB.IO = {
|
||||
kdc = 192.168.3.25
|
||||
kdc = 192.168.3.26
|
||||
admin_server = 192.168.3.25
|
||||
}
|
||||
|
||||
[domain_realm]
|
||||
192.168.3.25 = BUNNY-LAB.IO
|
||||
192.168.3.26 = BUNNY-LAB.IO
|
||||
.bunny-lab.io = BUNNY-LAB.IO
|
||||
bunny-lab.io = BUNNY-LAB.IO
|
||||
```
|
||||
|
||||
Then we apply these configmaps to the AWX namespace with the following commands:
|
||||
|
||||
```sh
|
||||
cd /awx
|
||||
kubectl -n awx create configmap awx-kerberos-config --from-file=/awx/krb5.conf
|
||||
kubectl apply -f custom_dns_records.yml
|
||||
```
|
||||
|
||||
- Open AWX UI and click on "**Instance Groups**" under the "**Administration**" section, then press "**Add > Add container group**".
|
||||
- Enter a descriptive name as you like (e.g. `Kerberos`) and click the toggle "**Customize Pod Specification**".
|
||||
- Put the following YAML string in "**Custom pod spec**" then press the "**Save**" button
|
||||
|
||||
```yaml title="Custom Pod Spec"
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
namespace: awx
|
||||
spec:
|
||||
serviceAccountName: default
|
||||
automountServiceAccountToken: false
|
||||
initContainers:
|
||||
- name: init-hosts
|
||||
image: busybox
|
||||
command:
|
||||
- sh
|
||||
- '-c'
|
||||
- cat /etc/custom-dns/custom-hosts >> /etc/hosts
|
||||
volumeMounts:
|
||||
- name: custom-dns
|
||||
mountPath: /etc/custom-dns
|
||||
containers:
|
||||
- image: quay.io/ansible/awx-ee:latest
|
||||
name: worker
|
||||
args:
|
||||
- ansible-runner
|
||||
- worker
|
||||
- '--private-data-dir=/runner'
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 100Mi
|
||||
volumeMounts:
|
||||
- name: awx-kerberos-volume
|
||||
mountPath: /etc/krb5.conf
|
||||
subPath: krb5.conf
|
||||
volumes:
|
||||
- name: awx-kerberos-volume
|
||||
configMap:
|
||||
name: awx-kerberos-config
|
||||
- name: custom-dns
|
||||
configMap:
|
||||
name: custom-dns
|
||||
```
|
||||
|
||||
### Job Template and Inventory Examples
|
||||
At this point, you need to adjust your exist Job Template(s) that need to communicate via Kerberos to domain-joined Windows devices to use the "Instance Group" of "**Kerberos**" while keeping the same Execution Environment you have been using up until this point. This will change the Execution Environment to include the Kerberos Keytab file in the EE at playbook runtime. When the playbook has completed running, (or if you are chain-loading multiple playbooks in a workflow job template), it will cease to exist. The kerberos keytab data will be regenerated at the next runtime.
|
||||
|
||||
Also add the following variables to the job template you have associated with the playbook below:
|
||||
|
||||
```yaml
|
||||
---
|
||||
kerberos_user: nicole.rappe@BUNNY-LAB.IO
|
||||
kerberos_password: <DomainPassword>
|
||||
```
|
||||
|
||||
You will want to ensure your inventory file is configured to use Kerberos Authentication as well, so the following example is a starting point:
|
||||
|
||||
```ini
|
||||
virt-node-01 ansible_host=virt-node-01.bunny-lab.io
|
||||
bunny-node-02 ansible_host=bunny-node-02.bunny-lab.io
|
||||
|
||||
[virtualizationHosts]
|
||||
virt-node-01
|
||||
bunny-node-02
|
||||
|
||||
[virtualizationHosts:vars]
|
||||
ansible_connection=winrm
|
||||
ansible_port=5986
|
||||
ansible_winrm_transport=kerberos
|
||||
ansible_winrm_scheme=https
|
||||
ansible_winrm_server_cert_validation=ignore
|
||||
#kerberos_user=nicole.rappe@BUNNY-LAB.IO #Optional, if you define this in the Job Template, it is not necessary here.
|
||||
#kerberos_password=<DomainPassword> #Optional, if you define this in the Job Template, it is not necessary here.
|
||||
```
|
||||
|
||||
!!! failure "Usage of Fully-Quality Domain Names"
|
||||
It is **critical** that you define Kerberos-authenticated devices with fully qualified domain names. This is just something I found out from 4+ hours of troubleshooting. If the device is Linux or you are using NTLM authentication instead of Kerberos authentication, you can skip this warning. If you do not define the inventory using FQDNs, it will fail to run the commands against the targeted device(s).
|
||||
|
||||
In this example, the host is defined via FQDN: `virt-node-01 ansible_host=virt-node-01.bunny-lab.io`
|
||||
|
||||
### Kerberos Connection Playbook
|
||||
At this point, you need a playbook that you can run in a Workflow Job Template (to keep things modular and simplified) to establish a connection to an Active Directory Domain Controller via Kerberos before running additional playbooks/templates against the actual devices.
|
||||
|
||||
You can visualize the connection workflow below:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Update AWX Project] --> B[Update Project Inventory]
|
||||
B --> C[Establish Kerberos Connection]
|
||||
C --> D[Run Playbook against Windows Device]
|
||||
```
|
||||
|
||||
The following playbook is an example pulled from https://git.bunny-lab.io
|
||||
|
||||
!!! note "Playbook Redundancies"
|
||||
I have several areas where I could optimize this playbook and remove redundancies. I just have not had enough time to iterate through it deeply-enough to narrow down exact things I can remove, so for now, it will remain as-is, since it functions as-expected with the example below.
|
||||
|
||||
```yaml title="Establish_Kerberos_Connection.yml"
|
||||
---
|
||||
- name: Generate Kerberos Ticket to Communicate with Domain-Joined Windows Devices
|
||||
hosts: localhost
|
||||
vars:
|
||||
kerberos_password: "{{ lookup('env', 'KERBEROS_PASSWORD') }}" # Alternatively, you can set this as an environment variable
|
||||
# BE SURE TO PASS "kerberos_user: nicole.rappe@BUNNY-LAB.IO" and "kerberos_password: <domain_admin_password>" to the template variables when running this playbook in a template.
|
||||
|
||||
tasks:
|
||||
- name: Generate the keytab file
|
||||
ansible.builtin.shell: |
|
||||
ktutil <<EOF
|
||||
addent -password -p {{ kerberos_user }} -k 1 -e aes256-cts
|
||||
{{ kerberos_password }}
|
||||
wkt /tmp/krb5.keytab
|
||||
quit
|
||||
EOF
|
||||
environment:
|
||||
KRB5_CONFIG: /etc/krb5.conf
|
||||
register: generate_keytab_result
|
||||
|
||||
- name: Ensure keytab file was generated successfully
|
||||
fail:
|
||||
msg: "Failed to generate keytab file"
|
||||
when: generate_keytab_result.rc != 0
|
||||
|
||||
- name: Keytab successfully generated
|
||||
ansible.builtin.debug:
|
||||
msg: "Keytab successfully generated at /tmp/krb5.keytab"
|
||||
when: generate_keytab_result.rc == 0
|
||||
|
||||
- name: Acquire Kerberos ticket using keytab
|
||||
ansible.builtin.shell: |
|
||||
kinit -kt /tmp/krb5.keytab {{ kerberos_user }}
|
||||
environment:
|
||||
KRB5_CONFIG: /etc/krb5.conf
|
||||
register: kinit_result
|
||||
|
||||
- name: Ensure Kerberos ticket was acquired successfully
|
||||
fail:
|
||||
msg: "Failed to acquire Kerberos ticket"
|
||||
when: kinit_result.rc != 0
|
||||
|
||||
- name: Kerberos ticket successfully acquired
|
||||
ansible.builtin.debug:
|
||||
msg: "Kerberos ticket successfully acquired for user {{ kerberos_user }}"
|
||||
when: kinit_result.rc == 0
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
- [Related AWX Documentation](<../../../reference/Automation/AWX/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
tags:
|
||||
- Ansible
|
||||
- AWX
|
||||
- Gitea
|
||||
- Automation
|
||||
---
|
||||
|
||||
## Purpose
|
||||
Once AWX is deployed, you will want to connect Gitea at https://git.bunny-lab.io. The reason for this is so we can pull in our playbooks, inventories, and templates automatically into AWX, making it more stateless overall and more resilient to potential failures of either AWX or the underlying Kubernetes Cluster hosting it.
|
||||
|
||||
## Obtain Gitea Token
|
||||
You already have this documented in Vaultwarden's password notes for awx.bunny-lab.io, but in case it gets lost, go to the [Gitea Token Page](https://git.bunny-lab.io/user/settings/applications) to set up an application token with read-only access for AWX, with a descriptive name.
|
||||
|
||||
## Create Gitea Credentials
|
||||
Before you make move on and make the project, you need to associate the Gitea token with an AWX "Credential". Navigate to **Resources > Credentials > Add**
|
||||
|
||||
| **Field** | **Value** |
|
||||
| :--- | :--- |
|
||||
| Credential Name | `git.bunny-lab.io` |
|
||||
| Description | `Gitea` |
|
||||
| Organization | `Default` *(Click the Magnifying Lens)* |
|
||||
| Credential Type | `Source Control` |
|
||||
| Username | `Gitea Username` *(e.g. `nicole`)* |
|
||||
| Password | `<Gitea Token>` |
|
||||
|
||||
## Create an AWX Project
|
||||
In order to link AWX to Gitea, you have to connect the two of them together with an AWX "Project". Navigate to **Resources > Projects > Add**
|
||||
|
||||
**Project Variables**:
|
||||
|
||||
| **Field** | **Value** |
|
||||
| :--- | :--- |
|
||||
| Project Name | `Bunny-Lab` |
|
||||
| Description | `Homelab Environment` |
|
||||
| Organization | `Default` |
|
||||
| Execution Environment | `AWX EE (latest)` *(Click the Magnifying Lens)* |
|
||||
| Source Control Type | `Git` |
|
||||
|
||||
**Gitea-specific Variables**:
|
||||
|
||||
| **Field** | **Value** |
|
||||
| :--- | :--- |
|
||||
| Source Control URL | `https://git.bunny-lab.io/GitOps/awx.bunny-lab.io.git` |
|
||||
| Source Control Branch/Tag/Commit | `main` |
|
||||
| Source Control Credential | `git.bunny-lab.io` *(Click the Magnifying Lens)* |
|
||||
|
||||
## Add Playbooks
|
||||
AWX automatically imports any playbooks it finds from the project, and makes them available for templates operating within the same project-space. (e.g. "Bunny-Lab"). This means no special configuration is needed for the playbooks.
|
||||
|
||||
## Create an Inventory
|
||||
You will want to associate an inventory with the Gitea project now. Navigate to **Resources > Inventories > Add**
|
||||
|
||||
| **Field** | **Value** |
|
||||
| :--- | :--- |
|
||||
| Inventory Name | `Homelab` |
|
||||
| Description | `Homelab Inventory` |
|
||||
| Organization | `Default` |
|
||||
|
||||
### Add Gitea Inventory Source
|
||||
Now you will want to connect this inventory to the inventory file(s) hosted in the aforementioned Gitea repository. Navigate to **Resources > Inventories > Homelab > Sources > Add**
|
||||
|
||||
| **Field** | **Value** |
|
||||
| :--- | :--- |
|
||||
| Source Name | `git.bunny-lab.io` |
|
||||
| Description | `Gitea` |
|
||||
| Execution Environment | `AWX EE (latest)` *(Click the Magnifying Lens)* |
|
||||
| Source | `Sourced from a Project` |
|
||||
| Project | `Bunny-Lab` |
|
||||
| Inventory File | `inventories/homelab.ini` |
|
||||
|
||||
!!! info "Overwriting Existing Inventory Data"
|
||||
You want to make sure that the checkboxes for "**Overwrite**" and "**Overwrite Variables**" are checked. This ensures that if devices and/or group variables are removed from the inventory file in Gitea, they will also be removed from the inventory inside AWX.
|
||||
|
||||
## Webhooks
|
||||
Optionally, set up webhooks in Gitea to trigger inventory updates in AWX upon changes in the repository. This section is not documented yet, but will eventually be documented.
|
||||
|
||||
## Related Documentation
|
||||
- [Related AWX Documentation](<../../../reference/Automation/AWX/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
tags:
|
||||
- Ansible
|
||||
- AWX
|
||||
- Automation
|
||||
---
|
||||
|
||||
## Purpose
|
||||
This document records the procedure for repair upgrades beyond awx operator 2.10.0. Follow the environment assumptions and commands below.
|
||||
|
||||
## Upgrading from 2.10.0 to 2.19.1+
|
||||
There is a known issue with upgrading / install AWX Operator beyond version 2.10.0, because of how the PostgreSQL database upgrades from 13.0 to 15.0, and has changed permissions. The following workflow will help get past that and adjust the permissions in such a way that allows the upgrade to proceed successfully. If this is a clean installation, you can also perform this step if the fresh install of 2.19.1 is not working yet. (It wont work out of the box because of this bug). `The developers of AWX seem to just not care about this issue, and have not implemented an official fix themselves at this time).
|
||||
|
||||
### Create a Temporary Pod to Adjust Permissions
|
||||
We need to create a pod that will mount the PostgreSQL PVC, make changes to permissions, then destroy the v15.0 pod to have the AWX Operator automatically regenerate it.
|
||||
|
||||
```yaml title="/awx/temp-pod.yml"
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: temp-pod
|
||||
namespace: awx
|
||||
spec:
|
||||
containers:
|
||||
- name: temp-container
|
||||
image: busybox
|
||||
command: ['sh', '-c', 'sleep 3600']
|
||||
volumeMounts:
|
||||
- mountPath: /var/lib/pgsql/data
|
||||
name: postgres-data
|
||||
volumes:
|
||||
- name: postgres-data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgres-15-awx-postgres-15-0
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
```sh
|
||||
# Deploy Temporary Pod
|
||||
kubectl apply -f /awx/temp-pod.yaml
|
||||
|
||||
# Open a Shell in the Temporary Pod
|
||||
kubectl exec -it temp-pod -n awx -- sh
|
||||
|
||||
# Adjust Permissions of the PostgreSQL 15.0 Database Folder
|
||||
chown -R 26:root /var/lib/pgsql/data
|
||||
exit
|
||||
|
||||
# Delete the Temporary Pod
|
||||
kubectl delete pod temp-pod -n awx
|
||||
|
||||
# Delete the Crashlooped PostgreSQL 15.0 Pod to Regenerate It
|
||||
kubectl delete pod awx-postgres-15-0 -n awx
|
||||
|
||||
# Track the Migration
|
||||
kubectl get pods -n awx
|
||||
kubectl logs -n awx awx-postgres-15-0
|
||||
```
|
||||
|
||||
!!! warning "Be Patient"
|
||||
This upgrade may take a few minutes depending on the speed of the node it is running on. Be patient and wait until the output looks something similar to this:
|
||||
|
||||
```text
|
||||
root@awx:/awx# kubectl get pods -n awx
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
awx-migration-24.6.1-bh5vb 0/1 Completed 0 9m55s
|
||||
awx-operator-controller-manager-745b55d94b-2dhvx 2/2 Running 0 25m
|
||||
awx-postgres-15-0 1/1 Running 0 12m
|
||||
awx-task-7946b46dd6-7z9jm 4/4 Running 0 10m
|
||||
awx-web-9497647b4-s4gmj 3/3 Running 0 10m
|
||||
```
|
||||
|
||||
If you see a migration pod, like seen in the above example, you can feel free to delete it with the following command: `kubectl delete pod awx-migration-24.6.1-bh5vb -n awx`.
|
||||
|
||||
## Related Documentation
|
||||
- [Related AWX Documentation](<../../../reference/Automation/AWX/index.md>) — Find the connected deployments, procedures, and references for this subject.
|
||||
Reference in New Issue
Block a user