I had deployed Red Hat OpenStack Platform 9 (Mitaka) on my Dell box using Virtual Machines.
I had powered off the box for a while, but when I powered it up again and restated the VM’s, I got the following error message on the undercloud.
As per https://docs.openstack.org/mitaka/config-reference/firewalls-default-ports.html port 9696 is for the Neutron service.
Using netstat figured that the service was indeed down.
You may see authentication errors for some of them. As long as they are reachable that's ok.
Next check if the neutron-openvswitch-agent is running
Next, stop and restart the networking service on the undercloud
The Neutron service on the Undercloud is now running listening on port 9696
| [root@openstack ~]# virsh list --all Id Name State ---------------------------------------------------- 1 undercloud running 2 overcloud-controller1 running 3 overcloud-controller2 running 4 overcloud-controller3 running 5 overcloud-ceph1 running 6 overcloud-ceph2 running 7 overcloud-ceph3 running 8 overcloud-compute1 running 9 overcloud-compute2 running |
I had powered off the box for a while, but when I powered it up again and restated the VM’s, I got the following error message on the undercloud.
| [stack@undercloud ~]$ nova list ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-37bff437-8574-4568-a232-d4dc94fc6b37) [stack@undercloud ~]$ |
| [stack@undercloud ~]$ sudo tail -f /var/log/nova/nova-api.log 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler return self.session.request(url, method, **kwargs) 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler return func(*args, **kwargs) 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 452, in request 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler resp = send(**kwargs) 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 496, in _send_request 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler raise exceptions.ConnectFailure(msg) 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler ConnectFailure: Unable to establish connection to http://172.16.0.1:9696/v2.0/ports.json?fixed_ips=ip_address%3D172.16.0.34 2017-04-03 17:19:43.975 24101 ERROR nova.api.metadata.handler 2017-04-03 17:19:43.977 24101 INFO nova.metadata.wsgi.server [req-8d1d4599-6689-4a72-b40f-c5eb6396f594 - - - - -] 172.16.0.34 "GET /latest/meta-data/ HTTP/1.1" status: 500 len: 367 time: 0.0051761 |
As per https://docs.openstack.org/mitaka/config-reference/firewalls-default-ports.html port 9696 is for the Neutron service.
Using netstat figured that the service was indeed down.
| [root@undercloud 1]# netstat -nltp | grep 9696 [root@undercloud 1]# Test you can connect to the following locally on the undercloud [root@undercloud 1]# curl http://172.16.0.1:8774/v2/ | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:9292/ | python -m json.tool [root@undercloud 1]#curl http://172.16.0.1:8777/ | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:9696/ | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:6385/ | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:8004/v1/ | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:5000/v2.0 | python -m json.tool [root@undercloud 1]# curl http://172.16.0.1:8774/v3 | python -m json.tool |
| [root@undercloud ~]# systemctl status neutron-openvswitch-agent ● neutron-openvswitch-agent.service - OpenStack Neutron Open vSwitch Agent Loaded: loaded (/usr/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-04-03 17:32:19 EDT; 47min ago Process: 15579 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS) Main PID: 15591 (neutron-openvsw) CGroup: /system.slice/neutron-openvswitch-agent.service ├─15591 /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/... ├─15852 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf ├─15857 /usr/bin/python2 /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf ├─16086 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json ├─16090 /usr/bin/python2 /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json └─16115 /bin/ovsdb-client monitor Interface name,ofport,external_ids --format=json Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15893]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-br br-int -- set Bridge br-in..._type=system Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15901]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set-fail-mode br-int secure Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15905]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge br-int protocols=OpenFlow10 Apr 03 17:32:20 undercloud.redhat.local ovs-vsctl[15929]: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist add-br br-ctlplane -- set Bridge ..._type=system
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15938]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json --
set-fail-mode br-ctlplane standalone
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15943]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge
br-ctlplane protocols=OpenFlow10
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15947]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json --
del-controller br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local ovs-vsctl[15986]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set
Interface int-br-ctlplane options:peer=phy-br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local ovs-vsctl[15991]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set
Interface phy-br-ctlplane options:peer=int-br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local sudo[16086]: neutron : TTY=unknown ; PWD=/ ;
USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
ovsdb-client monitor Interface na...--
format=json
Hint: Some lines
were ellipsized, use -l to show in full.
|
Next, stop and restart the networking service on the undercloud
| [root@undercloud ~]#
openstack-service stop neutron
[root@undercloud ~]#
systemctl status neutron-openvswitch-agent
●
neutron-openvswitch-agent.service - OpenStack Neutron Open vSwitch
Agent
Loaded: loaded
(/usr/lib/systemd/system/neutron-openvswitch-agent.service; enabled;
vendor preset: disabled)
Active: inactive
(dead) since Mon 2017-04-03 18:21:45 EDT; 9s ago
Process: 15591
ExecStart=/usr/bin/neutron-openvswitch-agent --config-file
/usr/share/neutron/neutron-dist.conf --config-file
/etc/neutron/neutron.conf --config-file
/etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir
/etc/neutron/conf.d/common --config-dir
/etc/neutron/conf.d/neutron-openvswitch-agent --log-file
/var/log/neutron/openvswitch-agent.log (code=exited,
status=0/SUCCESS)
Process: 15579
ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited,
status=0/SUCCESS)
Main PID: 15591
(code=exited, status=0/SUCCESS)
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15905]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge
br-int protocols=OpenFlow10
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15929]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- --may-exist
add-br br-ctlplane -- set Bridge ..._type=system
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15938]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json --
set-fail-mode br-ctlplane standalone
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15943]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set Bridge
br-ctlplane protocols=OpenFlow10
Apr 03 17:32:20
undercloud.redhat.local ovs-vsctl[15947]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json --
del-controller br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local ovs-vsctl[15986]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set
Interface int-br-ctlplane options:peer=phy-br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local ovs-vsctl[15991]: ovs|00001|vsctl|INFO|Called
as /bin/ovs-vsctl --timeout=10 --oneline --format=json -- set
Interface phy-br-ctlplane options:peer=int-br-ctlplane
Apr 03 17:32:21
undercloud.redhat.local sudo[16086]: neutron : TTY=unknown ; PWD=/ ;
USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
ovsdb-client monitor Interface na...--format=json
Apr 03 18:21:45
undercloud.redhat.local systemd[1]: Stopping OpenStack Neutron Open
vSwitch Agent...
Apr 03 18:21:45
undercloud.redhat.local systemd[1]: Stopped OpenStack Neutron Open
vSwitch Agent.
Hint: Some lines
were ellipsized, use -l to show in full.
[root@undercloud ~]#
openstack-service start neutron
[root@undercloud ~]#
|
The Neutron service on the Undercloud is now running listening on port 9696
| [stack@undercloud ~]$ sudo netstat -nltp | grep 9696 tcp 0 0 172.16.0.1:9696 0.0.0.0:* LISTEN 11367/python2 [stack@undercloud ~]$ |
| [stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| e2ce3df4-630a-43d4-9595-1f6f3d7cc144 | overcloud-cephstorage-0 | ACTIVE | - | Running | ctlplane=172.16.0.36 | | 63e790c4-1027-497f-b5fd-f477d98d5fac | overcloud-cephstorage-1 | ACTIVE | - | Running | ctlplane=172.16.0.39 | | 91888b64-7e5d-43ce-afc2-a25e0e478156 | overcloud-cephstorage-2 | ACTIVE | - | Running | ctlplane=172.16.0.37 | | b2600b35-9008-429d-bab6-f79fc006276f | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=172.16.0.38 | | 3ccf168c-f2d9-42d4-bec1-1e036b8a6df3 | overcloud-compute-1 | ACTIVE | - | Running | ctlplane=172.16.0.31 | | cda92d97-e6c7-4386-a1f8-0ef0ed63dca9 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=172.16.0.40 | | c6df9d03-2237-4b21-abb4-ea00723b54eb | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=172.16.0.35 | | 958677f2-857f-4d02-8e56-944dde7adb94 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=172.16.0.34 | +--------------------------------------+-------------------------+--------+------------+-------------+----------------------+ [stack@undercloud ~]$ |
