metadata agent throwing AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id' with latest release

Fix proposed to branch: master
Review: https://review.openstack.org/73904

Not trunk only. The fix for neutron was backported to stable/havana, so stable is broken too.

Reviewed: https://review.openstack.org/73904
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=4c16ec1eca50e2e5f561420141ad268c2970c99e
Submitter: Jenkins
Branch: master

commit 4c16ec1eca50e2e5f561420141ad268c2970c99e
Author: Robert Collins <email address hidden>
Date: Mon Feb 17 16:24:36 2014 +1300

    Workaround broken Neutron requirements.

    Neutron have introduced another hard dependency on unreleased
    python-neutronclient :(.

    Change-Id: I10702098192c0393a91efbd707c61f5b0e627de6
    Partial-Bug: 1280941

Update:

As per @markmcclain, there's some delay publishing a new signed package to PyPI (keys republishing is in progress), that's why we haven't seen a new client release yet. As quick as we get this done, a new package will be pushed to PyPI (by @mark), and requirements.txt for global and neutron will be updated (by me).

I've sent the following related fixes:
https://review.openstack.org/#/c/75014/
https://review.openstack.org/#/c/75009/

Fix proposed to branch: master
Review: https://review.openstack.org/75240

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/75410

Reviewed: https://review.openstack.org/75240
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff2f4dd4363919d49e07eedb25123ac804a7cdc3
Submitter: Jenkins
Branch: master

commit ff2f4dd4363919d49e07eedb25123ac804a7cdc3
Author: Justin Lund <email address hidden>
Date: Thu Feb 20 18:53:39 2014 -0500

    Sync minimum requirements

    Change-Id: I1a0653058f1f8085918d7c43147d6b2631acb013
    Closes-bug:1280941

We are facing following issue with the metadata agent.

Errors in neutron-metadata-agent.log

neutron.agent.metadata.agent AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id'

Errors at the vm side

HTTPError: HTTP Error 500: Internal Server Error
2014-02-22 19:24:30,169 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/boto/utils.py", line 219, in retry_url
    r = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
2014-02-22 19:25:34,285 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/boto/utils.py", line 219, in retry_url
    r = opener.open(req)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

The original rdo installed version was python-neutronclient-2.3.1-2.el6.noarch and we tried to upgrade this to python-neutronclient-2.3.3-1.el6.noarch.rpm

Hi, is it solved in stable havana? After I updated code today from yum. I'm facing the same problem....

It's not solved in havana. See: https://review.openstack.org/#/c/75410/ Jenkins task failed because gate is almost broken for neutron stable (not master).

Adil, please report the issue downstream, in RDO.

In RDO, they should either package new client release (2.3.4) or backport this particular fix to their code.

Added to release notes https://wiki.openstack.org/wiki/ReleaseNotes/2013.2.2#Neutron

Fix is being backported to python-neutronclient-2.3.1-3 in RDO

SRU'ing the required fix for neutronclient to Ubuntu 13.10 and the Havana Cloud Archive.

Hello Robert, or anyone else affected,

Accepted python-neutronclient into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-neutronclient/1:2.3.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I confirm proposed package solves the problem:

cloud-init start-local running: Fri, 07 Mar 2014 19:14:22 +0000. up 5.24 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.100.2 255.255.255.0 fa:16:3e:30:b4:f7
ci-info: route-0: 0.0.0.0 192.168.100.1 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.100.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 07 Mar 2014 19:14:23 +0000. up 6.82 seconds
found data source: DataSourceEc2
2014-03-07 19:14:27,960 - __init__.py[WARNING]: Unhandled non-multipart userdata ''
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.

While previously it was:

cloud-init start-local running: Fri, 07 Mar 2014 19:08:32 +0000. up 5.00 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.100.2 255.255.255.0 fa:16:3e:0f:ed:de
ci-info: route-0: 0.0.0.0 192.168.100.1 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.100.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 07 Mar 2014 19:08:34 +0000. up 6.35 seconds
2014-03-07 19:08:36,915 - utils.py[ERROR]: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/boto/utils.py", line 177, in retry_url
    resp = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

Confirmed verification of fix resolved original problem.

The verification of the Stable Release Update for python-neutronclient has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package python-neutronclient - 1:2.3.0-0ubuntu1.1

---------------
python-neutronclient (1:2.3.0-0ubuntu1.1) saucy-proposed; urgency=medium

  * debian/patches/fix-get-auth-info.patch: Fix regression introduced by
    stable/havanna update. (LP: #1280941)
 -- Chuck Short <email address hidden> Fri, 07 Mar 2014 12:04:11 -0500

CA update for havana

--
 python-neutronclient (1:2.3.0-0ubuntu1.1~cloud0) precise-havana; urgency=low

   * New update for the Ubuntu Cloud Archive.

 python-neutronclient (1:2.3.0-0ubuntu1.1) saucy-proposed; urgency=medium

   * debian/patches/fix-get-auth-info.patch: Fix regression introduced by
     stable/havanna update. (LP: #1280941)

This bug was fixed in the package python-neutronclient - 1:2.3.4-0ubuntu1

---------------
python-neutronclient (1:2.3.4-0ubuntu1) trusty; urgency=medium

  * New upstream point release:
    - [02baef4] Fix get_auth_info() for pre-authenticated clients
      (LP: #1280941).
 -- James Page <email address hidden> Thu, 13 Mar 2014 11:02:06 +0000

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/82154

Reviewed: https://review.openstack.org/82154
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a75435dda8154914f6d6355d9ea6368b53a737e3
Submitter: Jenkins
Branch: stable/havana

commit a75435dda8154914f6d6355d9ea6368b53a737e3
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Feb 21 14:50:51 2014 +0100

    Updated neutronclient dependency

    This is needed by: 15a912b1ca3c24ba8851b8b77d6de8027e120d78

    Change-Id: If0eab1b5dd102cc5ceb7720a8465ee47a708839d
    Closes-Bug: #1280941

TripleO should revert their workaround and update requirements.txt from globals to fix the issue.

@ihar please do submit a patch :). Note that the requirements.txt in question is neutrons one.

@robert, the patch for requirements in neutron already arrived to both stable and master, tripleo workaround is not needed now.

Aha, tripleo already reverted the workaround. So nothing should be done in tripleo, we may mark the bug as fixed.