No active metadata service found

Bug #1844557 reported by Adam Israel
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MicroStack
Triaged
Medium
Unassigned

Bug Description

Occasionally, I'll launch an instance and find this in the console log:

[ 624.792849] cloud-init[853]: 2019-09-18 15:58:58,506 - util.py[WARNING]: No active metadata service found
[ 624.820715] cloud-init[853]: 2019-09-18 15:58:58,577 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f61c1f28da0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]

The error will repeat many times until it finally gives up, and the instance finishes booting but without any of its cloud-init data applied.

Restarting microstack seems to fix the error for a while.

The most interesting thing I can find in the neutron-metadata-agent.log:

neutron-metadata-agent.log:2019-09-18 14:35:23.215 16373 WARNING oslo.service.loopingcall [-] Function 'neutron.agent.metadata.agent.UnixDomainMetadataProxy._report_state' run outlasted interval by 42.12 sec
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 87575a9026274065a933b67e1df29d1f
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 278, in _report_state
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent use_call=self.agent_state.get('start_flag'))
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent return method(context, 'report_state', **kwargs)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent retry=self.retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent retry=retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent call_monitor_timeout)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 87575a9026274065a933b67e1df29d1f

Pen Gale (pengale)
Changed in microstack:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
David Coronel (davecore) wrote :

I had a similar issue which I posted details of in https://bugs.launchpad.net/microstack/+bug/1852206

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.