Monday, April 3, 2017

Debugging OpenStack Undercloud Neutron issue

I had deployed Red Hat OpenStack Platform 9 (Mitaka) on my Dell box using Virtual Machines.


[root@openstack ~]# virsh list --all
Id Name State
----------------------------------------------------
1 undercloud running
2 overcloud-controller1 running
3 overcloud-controller2 running
4 overcloud-controller3 running
5 overcloud-ceph1 running
6 overcloud-ceph2 running
7 overcloud-ceph3 running
8 overcloud-compute1 running
9 overcloud-compute2 running

I had powered off the box for a while, but when I powered it up again and restated the VM’s, I got the following error message on the undercloud.

[stack@undercloud ~]$ nova list
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-37bff437-8574-4568-a232-d4dc94fc6b37)
[stack@undercloud ~]$

[stack@undercloud ~]$ sudo tail -f /var/log/nova/nova-api.log
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler return self.session.request(url, method, **kwargs)
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler return func(*args, **kwargs)
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 452, in request
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler resp = send(**kwargs)
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 496, in _send_request
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler raise exceptions.ConnectFailure(msg)
2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler ConnectFailure: Unable to establish connection to http://172.16.0.1:9696/v2.0/ports.json?fixed_ips=ip_address%3D172.16.0.34 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler
2017-04-03 17:19:43.977 24101 INFO nova.metadata.wsgi.server [req-8d1d4599-6689-4a72-b40f-c5eb6396f594 - - - - -] 172.16.0.34 "GET /latest/meta-data/ HTTP/1.1" status: 500 len: 367 time: 0.0051761

As per https://docs.openstack.org/mitaka/config-reference/firewalls-default-ports.html port 9696 is for the Neutron service.

Using netstat figured that the service was indeed down.

[root@undercloud 1]# netstat -nltp | grep 9696
[root@undercloud 1]#

Test you can connect to the following locally on the undercloud
[root@undercloud 1]# curl http://172.16.0.1:8774/v2/ | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:9292/ | python -m json.tool
[root@undercloud 1]#curl http://172.16.0.1:8777/ | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:9696/ | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:6385/ | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:8004/v1/ | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:5000/v2.0 | python -m json.tool
[root@undercloud 1]# curl http://172.16.0.1:8774/v3 | python -m json.tool
You may see authentication errors for some of them. As long as they are reachable that's ok. Next check if the neutron-openvswitch-agent is running

[root@undercloud ~]# systemctl status neutron-openvswitch-agent
● neutron-openvswitch-agent.service - OpenStack Neutron Open vSwitch Agent
Loaded: loaded (/usr/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: disabled)
 Active: active (running) since Mon 2017-04-03 17:32:19 EDT; 47min ago
Process: 15579 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS)
Main PID: 15591 (neutron-openvsw)
CGroup: /system.slice/neutron-openvswitch-agent.service ├─15591 /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/...
├─15852 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
├─15857 /usr/bin/python2 /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
├─16086 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
├─16090 /usr/bin/python2 /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json
└─16115 /bin/ovsdb-client monitor Interface name,ofport,external_ids --format=json

Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15893]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-br br-int -- set Bridge br-in..._type=system
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15901]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set-fail-mode br-int secure
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15905]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge br-int protocols=OpenFlow10
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15929]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-br br-ctlplane -- set Bridge ..._type=system

Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15938]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set-fail-mode br-ctlplane standalone

Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15943]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge br-ctlplane protocols=OpenFlow10

Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15947]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- del-controller br-ctlplane

Apr 03 17:32:21 undercloud.redhat.local ovs-vsctl[15986]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Interface int-br-ctlplane options:peer=phy-br-ctlplane

Apr 03 17:32:21 undercloud.redhat.local ovs-vsctl[15991]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Interface phy-br-ctlplane options:peer=int-br-ctlplane

Apr 03 17:32:21 undercloud.redhat.local sudo[16086]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface na...--
format=json
Hint: Some lines were ellipsized, use -l to show in full.

Next, stop and restart the networking service on the undercloud



[root@undercloud ~]# openstack-service stop neutron

[root@undercloud ~]# systemctl status neutron-openvswitch-agent
● neutron-openvswitch-agent.service - OpenStack Neutron Open vSwitch Agent
Loaded: loaded (/usr/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2017-04-03 18:21:45 EDT; 9s ago
Process: 15591 ExecStart=/usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log (code=exited, status=0/SUCCESS)
Process: 15579 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS)
Main PID: 15591 (code=exited, status=0/SUCCESS)

Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15905]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge br-int protocols=OpenFlow10
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15929]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-br br-ctlplane -- set Bridge ..._type=system
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15938]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set-fail-mode br-ctlplane standalone
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15943]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge br-ctlplane protocols=OpenFlow10
Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15947]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- del-controller br-ctlplane
Apr 03 17:32:21 undercloud.redhat.local ovs-vsctl[15986]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Interface int-br-ctlplane options:peer=phy-br-ctlplane
Apr 03 17:32:21 undercloud.redhat.local ovs-vsctl[15991]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Interface phy-br-ctlplane options:peer=int-br-ctlplane
Apr 03 17:32:21 undercloud.redhat.local sudo[16086]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface na...--format=json
Apr 03 18:21:45 undercloud.redhat.local systemd[1]: Stopping OpenStack Neutron Open vSwitch Agent...
Apr 03 18:21:45 undercloud.redhat.local systemd[1]: Stopped OpenStack Neutron Open vSwitch Agent.
Hint: Some lines were ellipsized, use -l to show in full.

[root@undercloud ~]# openstack-service start neutron
[root@undercloud ~]#


The Neutron service on the Undercloud is now running listening on port 9696

[stack@undercloud ~]$ sudo netstat -nltp | grep 9696
tcp 0 0 172.16.0.1:9696 0.0.0.0:* LISTEN 11367/python2
[stack@undercloud ~]$
[stack@undercloud ~]$ nova list +--------------------------------------+-------------------------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+----------------------+ | e2ce3df4-630a-43d4-9595-1f6f3d7cc144 | overcloud-cephstorage-0 | ACTIVE | - | Running | ctlplane=172.16.0.36 |
| 63e790c4-1027-497f-b5fd-f477d98d5fac | overcloud-cephstorage-1 | ACTIVE | - | Running | ctlplane=172.16.0.39 |
| 91888b64-7e5d-43ce-afc2-a25e0e478156 | overcloud-cephstorage-2 | ACTIVE | - | Running | ctlplane=172.16.0.37 |
| b2600b35-9008-429d-bab6-f79fc006276f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=172.16.0.38 |
| 3ccf168c-f2d9-42d4-bec1-1e036b8a6df3 | overcloud-compute-1 | ACTIVE | - | Running | ctlplane=172.16.0.31 |
| cda92d97-e6c7-4386-a1f8-0ef0ed63dca9 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=172.16.0.40 |
| c6df9d03-2237-4b21-abb4-ea00723b54eb | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=172.16.0.35 |
| 958677f2-857f-4d02-8e56-944dde7adb94 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=172.16.0.34 | +--------------------------------------+-------------------------+--------+------------+-------------+----------------------+ [stack@undercloud ~]$