Operator fails to deploy postgres DB #2072
Description
Please confirm the following
- I agree to follow this project's code of conduct.
- I have checked the current issues for duplicates.
- I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.
Bug Summary
The AWX operator controller deployment stalls during setup. It seems to be running into an issue with creating the status update on the postgres stateful set.
AWX Operator version
2.19.1
AWX version
24.6.1
Kubernetes platform
openshift
Kubernetes/Platform version
4.19.0-okd-scos.12
Modifications
no
Steps to reproduce
I have a pretty simple OKD installation running across 6 nodes. The only running workloads in the cluster are KubeVirt (no active VMs), Rook, CertManager, and Node Health Check.
I used helm via terraform to deploy AWX. I followed the instructions for the helm chart verbatim, with the only modifications being:
# values.yaml
AWX:
enabled: true
spec:
service_type: clusterip
ingress_type: Route
projects_persistence: true
postgres_storage_class: ceph-block # Ceph RBG
rbac:
create: truePS: I would have opened this issue in the awx-operator-helm repo, but that didn't seem like the correct place.
Expected results
This should have simply spun up an AWX instance; however, it just stalls.
Actual results
This is the error I'm getting in the logs of the awx-controller-manager deployment:
--------------------------- Ansible Task StdOut -------------------------------
TASK [installer : Create Database if no database is specified] *****************
task path: /opt/ansible/roles/installer/tasks/database_configuration.yml:187
-------------------------------------------------------------------------------
{
"level": "info",
"ts": "2025-08-15T02:41:05Z",
"logger": "logging_event_handler",
"msg": "[playbook task start]",
"name": "awx",
"namespace": "awx",
"gvk": "awx.ansible.com/v1beta1, Kind=AWX",
"event_type": "playbook_on_task_start",
"job": "8619662123133655771",
"EventData.Name": "installer : Create Database if no database is specified"
}
{
"level": "info",
"ts": "2025-08-15T02:41:06Z",
"logger": "proxy",
"msg": "Read object from cache",
"resource": {
"IsResourceRequest": true,
"Path": "/apis/apps/v1/namespaces/awx/statefulsets/awx-postgres-15",
"Verb": "get",
"APIPrefix": "apis",
"APIGroup": "apps",
"APIVersion": "v1",
"Namespace": "awx",
"Resource": "statefulsets",
"Subresource": "",
"Name": "awx-postgres-15",
"Parts": ["statefulsets", "awx-postgres-15"]
}
}
{
"level": "info",
"ts": "2025-08-15T02:41:06Z",
"logger": "proxy",
"msg": "Read object from cache",
"resource": {
"IsResourceRequest": true,
"Path": "/apis/apps/v1/namespaces/awx/statefulsets/awx-postgres-15",
"Verb": "get",
"APIPrefix": "apis",
"APIGroup": "apps",
"APIVersion": "v1",
"Namespace": "awx",
"Resource": "statefulsets",
"Subresource": "",
"Name": "awx-postgres-15",
"Parts": ["statefulsets", "awx-postgres-15"]
}
}
--------------------------- Ansible Task StdOut -------------------------------
TASK [Create Database if no database is specified] ********************************
fatal: [localhost]: FAILED! => {
"changed": false,
"msg": "Failed to apply object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"StatefulSet.apps \\\\\"awx-postgres-15\\\\\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than \\'replicas\\', \\'ordinals\\', \\'template\\', \\'updateStrategy\\', \\'persistentVolumeClaimRetentionPolicy\\' and \\'minReadySeconds\\' are forbidden\",\"reason\":\"Invalid\",\"details\":{\"name\":\"awx-postgres-15\",\"group\":\"apps\",\"kind\":\"StatefulSet\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: updates to statefulset spec for fields other than \\'replicas\\', \\'ordinals\\', \\'template\\', \\'updateStrategy\\', \\'persistentVolumeClaimRetentionPolicy\\' and \\'minReadySeconds\\' are forbidden\",\"field\":\"spec\"}]},\"code\":422}\\n'",
"reason": "Unprocessable Entity"
}
-------------------------------------------------------------------------------
{
"level": "error",
"ts": "2025-08-15T02:41:06Z",
"logger": "logging_event_handler",
"msg": "",
"name": "awx",
"namespace": "awx",
"gvk": "awx.ansible.com/v1beta1, Kind=AWX",
"event_type": "runner_on_failed",
"job": "8619662123133655771",
"EventData.Task": "Create Database if no database is specified",
"EventData.TaskArgs": "",
"EventData.FailedTaskPath": "/opt/ansible/roles/installer/tasks/database_configuration.yml:187",
"error": "[playbook task failed]",
"stacktrace": "github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"
}Additional information
Here's a closeup of the failure:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "StatefulSet.apps \"awx-postgres-15\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden",
"reason": "Invalid",
"details": {
"name": "awx-postgres-15",
"group": "apps",
"kind": "StatefulSet",
"causes": [
{
"reason": "FieldValueForbidden",
"message": "Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden",
"field": "spec"
}
]
},
"code": 422
}Operator Logs
This all ends with:
--------------------------- Ansible Task StdOut -------------------------------
TASK [Remove PostgreSQL statefulset for upgrade] ********************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'create_statefulset_result.error == 422' failed. The error was: error while evaluating conditional (create_statefulset_result.error == 422): 'dict object' has no attribute 'error'. 'dict object' has no attribute 'error'\n\nThe error appears to be in '/opt/ansible/roles/installer/tasks/database_configuration.yml': line 210, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Remove PostgreSQL statefulset for upgrade\n ^ here\n"}
-------------------------------------------------------------------------------
{"level":"error","ts":"2025-08-15T02:51:50Z","logger":"logging_event_handler","msg":"","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"1538230428873722731","EventData.Task":"Remove PostgreSQL statefulset for upgrade","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/installer/tasks/database_configuration.yml:210","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"}
{"level":"error","ts":"2025-08-15T02:51:51Z","logger":"runner","msg":"\u001b[0;34mansible-playbook [core 2.15.8]\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = ['/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/local/lib/python3.9/site-packages/ansible\u001b[0m\r\n\u001b[0;34m ansible collection location = /opt/ansible/.ansible/collections:/usr/share/ansible/collections\u001b[0m\r\n\u001b[0;34m executable location = /usr/local/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 3.9.18 (main, Sep 22 2023, 17:58:34) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3)\u001b[0m\r\n\u001b[0;34m jinja version = 3.1.3\u001b[0m\r\n\u001b[0;34m libyaml = True\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\u001b[0;34mSkipping callback 'awx_display', as we already have a stdout callback.\u001b[0m\n\u001b[0;34mSkipping callback 'default', a...
----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) -----