OpenStack Client Command-Line Interface (CLI) is not deleting network agent

Found additional issues related to this:

ubuntu@jump:~$ openstack network agent list --host compute01 -c ID -c "Agent Type" -f value
ab155dd0-f478-55e9-ae72-487757766cf5 OVN Metadata agent
compute01 OVN Controller agent

# These are the two OVN agents running on this host

ubuntu@jump:~$ openstack network agent set --disable ab155dd0-f478-55e9-ae72-487757766cf5
BadRequestException: 400: Client Error for url: http://10.2.80.100:9696/v2.0/agents/ab155dd0-f478-55e9-ae72-487757766cf5, Bad agent request: OVN agent status cannot be updated.

# Okay so it seems network agent set can't be used to disable OVN agents

#
# ... Now I have stopped OVN on compute01 and even shut down the host
#

ubuntu@jump:~$ openstack network agent delete compute01
ubuntu@jump:~$ openstack network agent delete ab155dd0-f478-55e9-ae72-487757766cf5
ubuntu@jump:~$
# Delete commands worked fine, and since the agents are gone I assumed it would be done

ubuntu@jump:~$ openstack network agent list --host compute01 -c ID -c "Agent Type" -f value
ab155dd0-f478-55e9-ae72-487757766cf5 OVN Metadata agent
compute01 OVN Controller agent

# Agents are still listed...

ubuntu@jump:~$ openstack network agent delete ab155dd0-f478-55e9-ae72-487757766cf5
Failed to delete network agent with ID 'ab155dd0-f478-55e9-ae72-487757766cf5': ResourceNotFound: 404: Client Error for url: http://10.2.80.100:9696/v2.0/agents/ab155dd0-f478-55e9-ae72-487757766cf5, Agent ab155dd0-f478-55e9-ae72-487757766cf5 could not be found.
1 of 1 network agents failed to delete.

# They are listed but not actually there?

ubuntu@jump:~$ openstack network agent delete compute01
Failed to delete network agent with ID 'compute01': ResourceNotFound: 404: Client Error for url: http://10.2.80.100:9696/v2.0/agents/compute01, Agent compute01 could not be found.
1 of 1 network agents failed to delete.

# Same for both

ubuntu@jump:~$ openstack network agent list --host compute01 -c ID -c "Agent Type" -f value
ab155dd0-f478-55e9-ae72-487757766cf5 OVN Metadata agent
compute01 OVN Controller agent

# But still listed...

These issues seem to be related to how the API is dealing with OVN in general. Running the commands with --debug shows that the HTTP response is actually 204, some version of OK:

Network client initialized using OpenStack SDK: <openstack.network.v2._proxy.Proxy object at 0x7fa447fcfa00>
REQ: curl -g -i -X DELETE http://10.2.80.100:9696/v2.0/agents/compute01 -H "User-Agent: openstacksdk/3.3.0 keystoneauth1/5.7.0 python-requests/2.32.3 CPython/3.10.12" -H "X-Auth-Token: {SHA256}ae9d469bc0ac653b8fb0246bd4f22005e339075a6a9087c18b674d4a91148bbd"
Starting new HTTP connection (1): 10.2.80.100:9696
http://10.2.80.100:9696 "DELETE /v2.0/agents/compute01 HTTP/11" 204 0
RESP: [204] date: Sat, 28 Sep 2024 09:53:37 GMT x-openstack-request-id: req-91ec41b7-fd4f-451c-8fd7-c2867bad6728
RESP BODY: Omitted, Content-Type is set to None. Only text/plain, application/json responses have their bodies logged.
DELETE call to network for http://10.2.80.100:9696/v2.0/agents/compute01 used request id req-91ec41b7-fd4f-451c-8fd7-c2867bad6728
clean_up DeleteNetworkAgent:
END return value: 0

even though the agents are not actually removed.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-openstackclient/+/950304

Hello:

I'll comment here what I wrote in the patch.

There are several disconnected documents that explain this behaviour.

* Neutron API [1]:
** "Agents that won’t be used anymore can be removed. Before deleting agents via API, the agent should be stopped/disabled." The OVN agent must be first deleted (ovn-controller stopped, OVN metadata agent deleted) before deleting in from the API.

* A partial description of an OVN agent deletion: [2]

* A related description of when the OVN agent deletion can be used: [3]

In any case, in order to avoid this kind of misunderstandings, I'll propose a doc patch for Neutron, in the OVN section.

Regards.

[1]https://docs.openstack.org/api-ref/network/v2/#delete-agent
[2]https://github.com/openstack/neutron/blob/master/doc/source/admin/config-services-agent.rst#l2-agents
[3]https://github.com/openstack/neutron/blob/master/doc/source/admin/ovn/troubleshooting.rst#duplicated-or-deleted-ovn-agents