Occasionally, I'll launch an instance and find this in the console log:
[ 624.792849] cloud-init[853]: 2019-09-18 15:58:58,506 - util.py[WARNING]: No active metadata service found
[ 624.820715] cloud-init[853]: 2019-09-18 15:58:58,577 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f61c1f28da0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]
The error will repeat many times until it finally gives up, and the instance finishes booting but without any of its cloud-init data applied.
Restarting microstack seems to fix the error for a while.
The most interesting thing I can find in the neutron-metadata-agent.log:
neutron-metadata-agent.log:2019-09-18 14:35:23.215 16373 WARNING oslo.service.loopingcall [-] Function 'neutron.agent.metadata.agent.UnixDomainMetadataProxy._report_state' run outlasted interval by 42.12 sec
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID 87575a9026274065a933b67e1df29d1f
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 278, in _report_state
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent use_call=self.agent_state.get('start_flag'))
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/neutron/agent/rpc.py", line 97, in report_state
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent return method(context, 'report_state', **kwargs)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent retry=self.retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent retry=retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent call_monitor_timeout)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent File "/snap/microstack/153/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id)
neutron-metadata-agent.log:2019-09-18 14:36:23.222 16373 ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID 87575a9026274065a933b67e1df29d1f
I had a similar issue which I posted details of in https:/ /bugs.launchpad .net/microstack /+bug/1852206