2023-11-02 08:58:11.412 7 INFO nova.compute.manager [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] [instance: 32056bc3-151f-42d4-817a-a1ce27ffc32f] Detaching volume a9fd2417-4c53-4774-a356-e5fd5de00708
2023-11-02 08:58:11.488 7 INFO nova.virt.block_device [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] [instance: 32056bc3-151f-42d4-817a-a1ce27ffc32f] Attempting to driver detach volume a9fd2417-4c53-4774-a356-e5fd5de00708 from mountpoint /dev/vdb
2023-11-02 08:58:11.551 7 INFO nova.virt.libvirt.driver [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] Successfully detached device vdb from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f from the persistent domain config.
2023-11-02 08:58:11.702 7 INFO nova.virt.libvirt.driver [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] Successfully detached device vdb from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f from the live domain config.
2023-11-02 08:58:12.019 7 ERROR nova.volume.cinder [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] Delete attachment failed for attachment 58cec798-7cd5-47d2-be28-21562fb03a09. Error: ConflictNovaUsingAttachment: Detach volume from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f using the Compute API (HTTP 409) (Request-ID: req-1edde727-4d4f-43dc-ac2f-23e7541372df) Code: 409: cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f using the Compute API (HTTP 409) (Request-ID: req-1edde727-4d4f-43dc-ac2f-23e7541372df)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server [req-2e227db3-4734-4564-bdc7-6e85c6626c2d f56d235f80fd44f5ac5cc7d55648e931 e33d7625f80645c1bf52b9f84ca50dcd - default default] Exception during message handling: cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f using the Compute API (HTTP 409) (Request-ID: req-1edde727-4d4f-43dc-ac2f-23e7541372df)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     _emit_versioned_exception_notification(
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7215, in detach_volume
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     do_detach_volume(context, volume_id, instance, attachment_id)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7212, in do_detach_volume
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     self._detach_volume(context, bdm, instance,
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 7163, in _detach_volume
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     driver_bdm.detach(context, instance, self.volume_api, self.driver,
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 462, in detach
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     self._do_detach(context, instance, volume_api, virt_driver,
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/block_device.py", line 453, in _do_detach
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 397, in wrapper
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 448, in wrapper
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/retrying.py", line 49, in wrapped_f
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/retrying.py", line 206, in call
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return attempt.get(self._wrap_exception)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/retrying.py", line 247, in get
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     raise value
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/retrying.py", line 200, in call
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 900, in attachment_delete
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     LOG.error('Delete attachment failed for attachment '
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/volume/cinder.py", line 891, in attachment_delete
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     cinderclient(
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/api_versions.py", line 421, in substitution
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return method.func(obj, *args, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 45, in delete
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/base.py", line 313, in _delete
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 233, in delete
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 215, in _cs_request
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinderclient/client.py", line 201, in request
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance 32056bc3-151f-42d4-817a-a1ce27ffc32f using the Compute API (HTTP 409) (Request-ID: req-1edde727-4d4f-43dc-ac2f-23e7541372df)
2023-11-02 08:58:12.092 7 ERROR oslo_messaging.rpc.server