OpenStack: Difference between revisions

From Network Security Wiki
Content added Content deleted
 
(58 intermediate revisions by the same user not shown)
Line 3: Line 3:
<br />
<br />


= Installation =
= Basics =


* Services:
== Basics ==
Nova => Compute => EC2
Neutron => Network => VPC
Swift => Storage => S3
Glance => Image Services =>
RabbitMQ => Message Queue
Keystone
Horizon

= Installation using PackStack =
Source: [https://www.tecmint.com/openstack-installation-guide-rhel-centos/ tecmint.com]

== Topology ==

Mgmt PG-747
[]--------------[OS]-----------------[]
ens160 ens192
10.10.30.8/23 No IP address, Promiscuous mode

== Initial Steps ==


Requirement:
Requirement:
Line 11: Line 30:
16 GB RAM
16 GB RAM
100 GB HDD
100 GB HDD
2 Virtual NICs

Interface Config(ens192):
* Do not change the HWADDR and UUID keys
* Do not assign IP address
* If the OS VM is deployed on top of a vCenter cloud, change the value of Promiscous Mode, MAC address Changes and Forged Transmists to Accept on the PG used for the Provider Network Mapping interface of the VM

nano /etc/sysconfig/network-scripts/ifcfg-ens192

DEVICE=ens192
TYPE=Ethernet
ONBOOT="yes"
BOOTPROTO="none"


Remove unneeded services:
Remove unneeded services:
Line 28: Line 60:


Install NTP
Install NTP
yum install ntpdate
yum install ntpdate

Install MariaDB:
yum install mariadb-server
systemctl start mariadb.service

Do not set the password for root user, else set it to blank again:
mysqladmin -u root -p<present-password> password


== Preparing OpenStack Queens Installation ==
== Preparing OpenStack Queens Installation ==
Line 35: Line 74:
yum install https://www.rdoproject.org/repos/rdo-release.rpm
yum install https://www.rdoproject.org/repos/rdo-release.rpm


On CentOS 7, the Extras repository includes the RPM that actives the OpenStack repository.<br/>
Setup the OpenStack repository:
Extras is already enabled, so you can easily install the RPM to setup the OpenStack repository:
yum install -y centos-release-openstack-queens
yum install -y centos-release-openstack-queens
yum update -y
yum update -y


Packstack represents a utility which facilitates the deployment on multiple nodes for different components of OpenStack via SSH connections and Puppet modulesL
Packstack facilitates the deployment on multiple nodes for different components of OpenStack via SSH connections and Puppet modules:
yum install openstack-packstack
yum install openstack-packstack


Generate an answer file for Packstack with the default configurations which will be later edited with the required parameters in order to deploy a standalone installation of Openstack (single node):
Generate an answer file to deploy a standalone installation of Openstack (single node):
packstack --gen-answer-file='date +"%d.%m.%y"'.conf
packstack --gen-answer-file='date +"%d.%m.%y"'.conf
ls


Edit the generated answer configuration file with a text editor:
Edit the generated answer configuration file with a text editor:
Line 52: Line 89:
CONFIG_NTP_SERVERS=0.ro.pool.ntp.org
CONFIG_NTP_SERVERS=0.ro.pool.ntp.org
CONFIG_PROVISION_DEMO=n
CONFIG_PROVISION_DEMO=n
CONFIG_KEYSTONE_ADMIN_PW=your_password # for Admin user
CONFIG_KEYSTONE_ADMIN_PW=your_password # For Admin user
CONFIG_HEAT_INSTALL=y # Install OpenStack Orchestration (HEAT)
CONFIG_HORIZON_SSL=y # Access OpenStack dashboard via HTTP with SSL enabled
CONFIG_HORIZON_SSL=y # Access OpenStack dashboard via HTTP with SSL enabled
CONFIG_MARIADB_PW=mypassword1234 # The root password for MySQL server
CONFIG_MARIADB_PW=mypassword1234 # The root password for MySQL server
Line 65: Line 103:
== Start Installation using Packstack Answer File ==
== Start Installation using Packstack Answer File ==


Using above Answer file:
packstack --answer-file 13.04.16.conf
packstack --answer-file 13.04.16.conf


Default setup without Answer file:
packstack --allinone

Credentials can be found (if you haven’t set it in the answers.txt file) in below file:
cat keystonerc_admin

= Installation using DevStack =

Source: [https://www.techrepublic.com/article/how-to-install-openstack-on-a-single-ubuntu-server-virtual-machine techrepublic.com]
{{UC}}

= Verification =

source keystonerc_admin
openstack user list
openstack project list
openstack service list
openstack catalog list

neutron router-port-list adminrouter
neutron router-port-list --column id --column status --column fixed_ips adminrouter
neutron port-show 43c9ff00-07ce-4069-a734-4e6d5e795e6a

= Running CirrOS =
https://download.cirros-cloud.net/
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

openstack image create "cirros-raw" --disk-format raw --container-format bare --public --file ~/cirros-0.4.0-x86_64-disk.img

openstack image list
<pre style="width: 1500px; overflow-x: scroll;">
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+
| b26d5360-5e10-4482-a52b-41782b7e1332 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 | active |
| 7d95c733-3279-47f3-bbc2-047e1d3cb3b7 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 | active |
| 614e8dda-97c3-43c4-a986-0395357ff9d9 | cirros-raw | active |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+
</pre>

Create an instance:
Use vip4 Networks for it as it is a Server

neutron agent-list
<pre style="width: 1500px; overflow-x: scroll;">
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| 293e5768-18ff-4733-aeaa-3faeff3f0165 | DHCP agent | cloud.centos.lan | nova | :-) | True | neutron-dhcp-agent |
| 2ee38f4f-351f-4a86-8f37-635ca07cc6f1 | Open vSwitch agent | cloud.centos.lan | | :-) | True | neutron-openvswitch-agent |
| cc037a05-0873-4fbd-8b44-85ec046ca264 | Metadata agent | cloud.centos.lan | | :-) | True | neutron-metadata-agent |
| eec3cee5-b3a1-4f98-85e7-89d6f5584d77 | L3 agent | cloud.centos.lan | nova | :-) | True | neutron-l3-agent |
| f8f85b4e-73f0-4636-9bef-a7cb68394c63 | Metering agent | cloud.centos.lan | | :-) | True | neutron-metering-agent |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
</pre>

= Troubleshooting =

== RabbitMQ Troubleshooting ==

Check if RabbitMQ is running or not:
rabbitmqctl status
rabbitmqctl environment

Check if any crash file exists here:
cd /var/lib/rabbitmq/

Restart the RabbitMQ service on the first controller node:
service rabbitmq-server stop
service rabbitmq-server start

If the service refuses to stop, then run the pkill command to stop the service, then restart the service:
pkill -KILL -u rabbitmq
service rabbitmq-server start

Verify RabbitMQ processes are running:
ps -ef | grep rabbitmq
rabbitmqctl list_queues
rabbitmqctl list_queues 2>&amp;1 | grep -i error

If there are errors, run the cluster_status command to make sure there are no partitions:
rabbitmqctl cluster_status

== Instance not creating ==
Check Nova logs:
/var/log/nova/nova-api.log ==> API Call logs
/var/log/nova/nova-conductor.log
/var/log/nova/nova-compute.log

Check Nova Services
> nova service-list

<pre style="width: 1500px; overflow-x: scroll;">
+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 4cd953d4-3f77-4c63-9c91-ec61023cd8de | nova-conductor | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:57:43.000000 | - | False |
| dd640ca4-cc3e-4aac-af2f-47f9a79aa7fc | nova-scheduler | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:57:42.000000 | - | False |
| ec1b8ae1-ac8d-42c4-b78d-b08337e7c190 | nova-consoleauth | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:57:44.000000 | - | False |
| cecd523e-eaa0-44cc-a43a-c8986be62d5c | nova-compute | cloud.centos.lan | nova | enabled | up | 2018-09-20T13:57:37.000000 | - | False |
+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+
</pre>

Check Service
nova list

+--------------------------------------+--------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+----------+
| df5f815b-ff04-4890-9092-0a140527ae7c | Avi-se-dpogu | BUILD | scheduling | NOSTATE | |
| d7a4b63d-7d6e-447c-9569-d7ef80724881 | Avi-se-vljym | BUILD | scheduling | NOSTATE | |
+--------------------------------------+--------------+--------+------------+-------------+----------+

Nova Show Instance
nova show 20193e58-2c5b-44c6-a98f-a44e2001934f

Compute Service List
> openstack compute service list
+----+------------------+------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+------------------+----------+---------+-------+----------------------------+
| 3 | nova-conductor | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:55:03.000000 |
| 4 | nova-scheduler | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:55:02.000000 |
| 5 | nova-consoleauth | cloud.centos.lan | internal | enabled | up | 2018-09-20T13:55:04.000000 |
| 6 | nova-compute | cloud.centos.lan | nova | enabled | up | 2018-09-20T13:55:07.000000 |
+----+------------------+------------------+----------+---------+-------+----------------------------+

Check System Services:
systemctl list-unit-files

Restart Nova Compute Service
service openstack-nova-compute restart

Check Images in Glance
<pre style="width: 1500px; overflow-x: scroll;">
> glance image-list
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+
| ID | Name |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+
| 7d95c733-3279-47f3-bbc2-047e1d3cb3b7 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 |
| 18227c90-22fa-4bf1-8497-7a4419a00473 | cirros-qcow2 |
| 614e8dda-97c3-43c4-a986-0395357ff9d9 | cirros-raw |
| 4fc86d84-2fa0-47bc-919e-efd2d8f1031d | cirros-web |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+
</pre>

Check image details
<pre style="width: 1500px; overflow-x: scroll;">
> glance image-show 4fc86d84-2fa0-47bc-919e-efd2d8f1031d
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 097e8befc3def6267eeab69e3db0f220 |
| container_format | bare |
| created_at | 2018-09-21T09:57:41Z |
| disk_format | qcow2 |
| id | 4fc86d84-2fa0-47bc-919e-efd2d8f1031d |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-web |
| owner | d9b6b382a3ed45d1aa0f6c8dacf93176 |
| protected | False |
| size | 20709376 |
| status | active |
| tags | [] |
| updated_at | 2018-09-21T09:57:41Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+
</pre>


> openstack hypervisor list
+----+---------------------+-----------------+------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+------------+-------+
| 1 | cloud.centos.lan | QEMU | 10.10.30.8 | up |
+----+---------------------+-----------------+------------+-------+


> openstack compute service list
+----+------------------+------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+------------------+----------+---------+-------+----------------------------+
| 3 | nova-conductor | cloud.centos.lan | internal | enabled | up | 2018-09-25T04:40:19.000000 |
| 4 | nova-scheduler | cloud.centos.lan | internal | enabled | up | 2018-09-25T04:40:20.000000 |
| 5 | nova-consoleauth | cloud.centos.lan | internal | enabled | up | 2018-09-25T04:40:17.000000 |
| 6 | nova-compute | cloud.centos.lan | nova | enabled | up | 2018-09-25T04:40:15.000000 |
+----+------------------+------------------+----------+---------+-------+----------------------------+


> openstack service list
+----------------------------------+------------+----------------+
| ID | Name | Type |
+----------------------------------+------------+----------------+
| 0a438bbc780f44f9b66371cfa39a542d | placement | placement |
| 19b41de10cba46f58a11187fbdae3eaf | gnocchi | metric |
| 36c9559264b74544a51e0d4b1e7deb3d | heat-cfn | cloudformation |
| 4014b4d990164799832407480d985b59 | ceilometer | metering |
| 66969875070d46c09e67e20fbb169b51 | swift | object-store |
| 6a86847bb0074c399bdd8db17a0d765b | heat | orchestration |
| 73f00b0163f5483faac9e89291c2f414 | neutron | network |
| 854a73ec550d40ef8aef57f0961df5c6 | cinder | volume |
| 8a2d2ffae2f54d3cbab3b1b5611c1119 | cinderv3 | volumev3 |
| a111492334684364ae305d1bb61d8455 | aodh | alarming |
| a951b997549848e7ac93c01ccaf23aab | glance | image |
| e08b87f75da24ba59ca9046e91fea51c | nova | compute |
| f073ee492e0f4349810d5a2b9e3d94f8 | keystone | identity |
| f27c21ecca90427298f3a9af62013602 | cinderv2 | volumev2 |
+----------------------------------+------------+----------------+

> openstack endpoint list | grep -i placement
<pre style="width: 1500px; overflow-x: scroll;">
| 07802b28b40f4cfd8308b4351a5092ee | RegionOne | placement | placement | True | public | http://10.10.30.8:8778/placement |
| 2fa60fbbb5214f45953540a071efaffe | RegionOne | placement | placement | True | internal | http://10.10.30.8:8778/placement |
| b76395b1badb4a2bb462f35e35b71522 | RegionOne | placement | placement | True | admin | http://10.10.30.8:8778/placement |
</pre>

> netstat -an | grep -i 8778
tcp 0 0 0.0.0.0:8778 0.0.0.0:* LISTEN
tcp 1 0 10.10.30.8:59868 10.10.30.8:8778 CLOSE_WAIT
tcp 0 0 10.10.30.8:8778 10.10.30.8:58828 TIME_WAIT
tcp 0 0 10.10.30.8:8778 10.10.30.8:59868 FIN_WAIT2

> neutron agent-list
<pre style="width: 1500px; overflow-x: scroll;">
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| 293e5768-18ff-4733-aeaa-3faeff3f0165 | DHCP agent | cloud.centos.lan | nova | :-) | True | neutron-dhcp-agent |
| 2ee38f4f-351f-4a86-8f37-635ca07cc6f1 | Open vSwitch agent | cloud.centos.lan | | :-) | True | neutron-openvswitch-agent |
| cc037a05-0873-4fbd-8b44-85ec046ca264 | Metadata agent | cloud.centos.lan | | :-) | True | neutron-metadata-agent |
| eec3cee5-b3a1-4f98-85e7-89d6f5584d77 | L3 agent | cloud.centos.lan | nova | :-) | True | neutron-l3-agent |
| f8f85b4e-73f0-4636-9bef-a7cb68394c63 | Metering agent | cloud.centos.lan | | :-) | True | neutron-metering-agent |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
</pre>

> nano /etc/nova/nova.conf
virt_type = qemu




> ip netns
qdhcp-4d8df480-f1ec-413a-8500-a2e05caba55d (id: 4)
qdhcp-208f9e3c-5e66-47a1-95d3-02c9860d122a (id: 3)
qdhcp-98e9a9df-2698-44cd-95b1-905fbbbe1165 (id: 1)
qdhcp-bd58da20-f41b-483f-bee0-55f5e51b7d9a (id: 2)
qrouter-73240828-df62-4c02-b214-be6ee9572bda (id: 0)


> ip netns exec qrouter-73240828-df62-4c02-b214-be6ee9572bda bash


> ping 10.0.3.5
PING 10.0.3.5 (10.0.3.5) 56(84) bytes of data.
64 bytes from 10.0.3.5: icmp_seq=1 ttl=64 time=4.44 ms
64 bytes from 10.0.3.5: icmp_seq=2 ttl=64 time=0.879 ms
^C
--- 10.0.3.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.879/2.661/4.444/1.783 ms


== Aodh Exception ==
Check Aodh Logs:
cat /var/log/aodh/evaluator.log


If you see "Table 'aodh.alarm' doesn't exist" error try to initialize aodh database:
aodh-dbsync




Latest revision as of 17:57, 7 December 2018


Basics

  • Services:
Nova      =>   Compute         =>   EC2
Neutron   =>   Network         =>   VPC
Swift     =>   Storage         =>   S3
Glance    =>   Image Services  => 
RabbitMQ  =>   Message Queue
Keystone
Horizon

Installation using PackStack

Source: tecmint.com

Topology

            Mgmt    PG-747
[]--------------[OS]-----------------[]
          ens160    ens192
   10.10.30.8/23    No IP address, Promiscuous mode

Initial Steps

Requirement: 

CentOS VM
16 GB RAM
100 GB HDD
2 Virtual NICs

Interface Config(ens192):

  • Do not change the HWADDR and UUID keys
  • Do not assign IP address
  • If the OS VM is deployed on top of a vCenter cloud, change the value of Promiscous Mode, MAC address Changes and Forged Transmists to Accept on the PG used for the Provider Network Mapping interface of the VM
nano /etc/sysconfig/network-scripts/ifcfg-ens192

DEVICE=ens192
TYPE=Ethernet
ONBOOT="yes"
BOOTPROTO="none"

Remove unneeded services: 

systemctl stop postfix firewalld NetworkManager
systemctl disable postfix firewalld NetworkManager
systemctl mask NetworkManager
yum remove postfix NetworkManager NetworkManager-libnm

Completely disable Selinux policy 

setenforce 0
getenforce
vi /etc/selinux/config
SELINUX=disabled

Set hostname 

hostnamectl set-hostname cloud.centos.lan

Install NTP 

yum install ntpdate

Install MariaDB: 

yum install mariadb-server
systemctl start mariadb.service

Do not set the password for root user, else set it to blank again: 

mysqladmin -u root -p<present-password> password

Preparing OpenStack Queens Installation

OpenStack will be deployed with the help of PackStack package provided by rdo repository (RPM Distribution of OpenStack): 

yum install https://www.rdoproject.org/repos/rdo-release.rpm

Setup the OpenStack repository: 

yum install -y centos-release-openstack-queens
yum update -y

Packstack facilitates the deployment on multiple nodes for different components of OpenStack via SSH connections and Puppet modules: 

yum install  openstack-packstack

Generate an answer file to deploy a standalone installation of Openstack (single node): 

packstack --gen-answer-file='date +"%d.%m.%y"'.conf

Edit the generated answer configuration file with a text editor: 

vi 03.09.18.conf

CONFIG_NTP_SERVERS=0.ro.pool.ntp.org
CONFIG_PROVISION_DEMO=n
CONFIG_KEYSTONE_ADMIN_PW=your_password          # For Admin user
CONFIG_HEAT_INSTALL=y                           # Install OpenStack Orchestration (HEAT)
CONFIG_HORIZON_SSL=y                            # Access OpenStack dashboard via HTTP with SSL enabled
CONFIG_MARIADB_PW=mypassword1234                # The root password for MySQL server
CONFIG_NAGIOS_INSTALL=n                         # Disable Nagios

Enable Root SSH login by uncomment below line: 

vi /etc/ssh/sshd_config
PermitRootLogin yes

systemctl restart sshd

Start Installation using Packstack Answer File

Using above Answer file: 

packstack --answer-file 13.04.16.conf

Default setup without Answer file: 

packstack --allinone

Credentials can be found (if you haven’t set it in the answers.txt file) in below file: 

cat keystonerc_admin

Installation using DevStack

Source: techrepublic.com

        This section is under construction.

Verification

source keystonerc_admin 
openstack user list
openstack project list 
openstack service list
openstack catalog list

neutron router-port-list adminrouter
neutron router-port-list --column id --column status --column fixed_ips adminrouter
neutron port-show 43c9ff00-07ce-4069-a734-4e6d5e795e6a

Running CirrOS

https://download.cirros-cloud.net/
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

openstack image create "cirros-raw" --disk-format raw --container-format bare --public --file ~/cirros-0.4.0-x86_64-disk.img 

openstack image list

+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+
| ID                                   | Name                                                                                                        | Status |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+
| b26d5360-5e10-4482-a52b-41782b7e1332 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 | active |
| 7d95c733-3279-47f3-bbc2-047e1d3cb3b7 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 | active |
| 614e8dda-97c3-43c4-a986-0395357ff9d9 | cirros-raw                                                                                                  | active |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+--------+

Create an instance: 

Use vip4 Networks for it as it is a Server

neutron agent-list

+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host             | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| 293e5768-18ff-4733-aeaa-3faeff3f0165 | DHCP agent         | cloud.centos.lan | nova              | :-)   | True           | neutron-dhcp-agent        |
| 2ee38f4f-351f-4a86-8f37-635ca07cc6f1 | Open vSwitch agent | cloud.centos.lan |                   | :-)   | True           | neutron-openvswitch-agent |
| cc037a05-0873-4fbd-8b44-85ec046ca264 | Metadata agent     | cloud.centos.lan |                   | :-)   | True           | neutron-metadata-agent    |
| eec3cee5-b3a1-4f98-85e7-89d6f5584d77 | L3 agent           | cloud.centos.lan | nova              | :-)   | True           | neutron-l3-agent          |
| f8f85b4e-73f0-4636-9bef-a7cb68394c63 | Metering agent     | cloud.centos.lan |                   | :-)   | True           | neutron-metering-agent    |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+

Troubleshooting

RabbitMQ Troubleshooting

Check if RabbitMQ is running or not: 

rabbitmqctl status
rabbitmqctl environment

Check if any crash file exists here: 

cd /var/lib/rabbitmq/

Restart the RabbitMQ service on the first controller node: 

service rabbitmq-server stop
service rabbitmq-server start

If the service refuses to stop, then run the pkill command to stop the service, then restart the service: 

pkill -KILL -u rabbitmq
service rabbitmq-server start

Verify RabbitMQ processes are running: 

ps -ef | grep rabbitmq
rabbitmqctl list_queues
rabbitmqctl list_queues 2>&1 | grep -i error

If there are errors, run the cluster_status command to make sure there are no partitions: 

rabbitmqctl cluster_status

Instance not creating

Check Nova logs: 

/var/log/nova/nova-api.log               ==>  API Call logs
/var/log/nova/nova-conductor.log
/var/log/nova/nova-compute.log

Check Nova Services 

> nova service-list

+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| Id                                   | Binary           | Host             | Zone     | Status  | State | Updated_at                 | Disabled Reason | Forced down |
+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+
| 4cd953d4-3f77-4c63-9c91-ec61023cd8de | nova-conductor   | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:57:43.000000 | -               | False       |
| dd640ca4-cc3e-4aac-af2f-47f9a79aa7fc | nova-scheduler   | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:57:42.000000 | -               | False       |
| ec1b8ae1-ac8d-42c4-b78d-b08337e7c190 | nova-consoleauth | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:57:44.000000 | -               | False       |
| cecd523e-eaa0-44cc-a43a-c8986be62d5c | nova-compute     | cloud.centos.lan | nova     | enabled | up    | 2018-09-20T13:57:37.000000 | -               | False       |
+--------------------------------------+------------------+------------------+----------+---------+-------+----------------------------+-----------------+-------------+

Check Service 

nova list

+--------------------------------------+--------------+--------+------------+-------------+----------+
| ID                                   | Name         | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+----------+
| df5f815b-ff04-4890-9092-0a140527ae7c | Avi-se-dpogu | BUILD  | scheduling | NOSTATE     |          |
| d7a4b63d-7d6e-447c-9569-d7ef80724881 | Avi-se-vljym | BUILD  | scheduling | NOSTATE     |          |
+--------------------------------------+--------------+--------+------------+-------------+----------+

Nova Show Instance 

nova show 20193e58-2c5b-44c6-a98f-a44e2001934f

Compute Service List 

> openstack compute service list
+----+------------------+------------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host             | Zone     | Status  | State | Updated At                 |
+----+------------------+------------------+----------+---------+-------+----------------------------+
|  3 | nova-conductor   | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:55:03.000000 |
|  4 | nova-scheduler   | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:55:02.000000 |
|  5 | nova-consoleauth | cloud.centos.lan | internal | enabled | up    | 2018-09-20T13:55:04.000000 |
|  6 | nova-compute     | cloud.centos.lan | nova     | enabled | up    | 2018-09-20T13:55:07.000000 |
+----+------------------+------------------+----------+---------+-------+----------------------------+

Check System Services: 

systemctl list-unit-files

Restart Nova Compute Service 

service openstack-nova-compute restart

Check Images in Glance 

> glance image-list
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+
| ID                                   | Name                                                                                                        |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+
| 7d95c733-3279-47f3-bbc2-047e1d3cb3b7 | Avi-SE-17.2.12-9128-cloud-5f5b017e-dbf9-4ab2-a1cb-b5de3bed6fc2-cluster-c820efe7-2e24-4a14-9a7f-c2df55477599 |
| 18227c90-22fa-4bf1-8497-7a4419a00473 | cirros-qcow2                                                                                                |
| 614e8dda-97c3-43c4-a986-0395357ff9d9 | cirros-raw                                                                                                  |
| 4fc86d84-2fa0-47bc-919e-efd2d8f1031d | cirros-web                                                                                                  |
+--------------------------------------+-------------------------------------------------------------------------------------------------------------+

Check image details 

>  glance image-show 4fc86d84-2fa0-47bc-919e-efd2d8f1031d
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | 097e8befc3def6267eeab69e3db0f220     |
| container_format | bare                                 |
| created_at       | 2018-09-21T09:57:41Z                 |
| disk_format      | qcow2                                |
| id               | 4fc86d84-2fa0-47bc-919e-efd2d8f1031d |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | cirros-web                           |
| owner            | d9b6b382a3ed45d1aa0f6c8dacf93176     |
| protected        | False                                |
| size             | 20709376                             |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2018-09-21T09:57:41Z                 |
| virtual_size     | None                                 |
| visibility       | public                               |
+------------------+--------------------------------------+



> openstack hypervisor list
+----+---------------------+-----------------+------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP    | State |
+----+---------------------+-----------------+------------+-------+
|  1 | cloud.centos.lan    | QEMU            | 10.10.30.8 | up    |
+----+---------------------+-----------------+------------+-------+



> openstack compute service list
+----+------------------+------------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host             | Zone     | Status  | State | Updated At                 |
+----+------------------+------------------+----------+---------+-------+----------------------------+
|  3 | nova-conductor   | cloud.centos.lan | internal | enabled | up    | 2018-09-25T04:40:19.000000 |
|  4 | nova-scheduler   | cloud.centos.lan | internal | enabled | up    | 2018-09-25T04:40:20.000000 |
|  5 | nova-consoleauth | cloud.centos.lan | internal | enabled | up    | 2018-09-25T04:40:17.000000 |
|  6 | nova-compute     | cloud.centos.lan | nova     | enabled | up    | 2018-09-25T04:40:15.000000 |
+----+------------------+------------------+----------+---------+-------+----------------------------+



> openstack service list 
+----------------------------------+------------+----------------+
| ID                               | Name       | Type           |
+----------------------------------+------------+----------------+
| 0a438bbc780f44f9b66371cfa39a542d | placement  | placement      |
| 19b41de10cba46f58a11187fbdae3eaf | gnocchi    | metric         |
| 36c9559264b74544a51e0d4b1e7deb3d | heat-cfn   | cloudformation |
| 4014b4d990164799832407480d985b59 | ceilometer | metering       |
| 66969875070d46c09e67e20fbb169b51 | swift      | object-store   |
| 6a86847bb0074c399bdd8db17a0d765b | heat       | orchestration  |
| 73f00b0163f5483faac9e89291c2f414 | neutron    | network        |
| 854a73ec550d40ef8aef57f0961df5c6 | cinder     | volume         |
| 8a2d2ffae2f54d3cbab3b1b5611c1119 | cinderv3   | volumev3       |
| a111492334684364ae305d1bb61d8455 | aodh       | alarming       |
| a951b997549848e7ac93c01ccaf23aab | glance     | image          |
| e08b87f75da24ba59ca9046e91fea51c | nova       | compute        |
| f073ee492e0f4349810d5a2b9e3d94f8 | keystone   | identity       |
| f27c21ecca90427298f3a9af62013602 | cinderv2   | volumev2       |
+----------------------------------+------------+----------------+

> openstack endpoint list | grep -i placement 

 | 07802b28b40f4cfd8308b4351a5092ee | RegionOne | placement    | placement      | True    | public    | http://10.10.30.8:8778/placement             |
 | 2fa60fbbb5214f45953540a071efaffe | RegionOne | placement    | placement      | True    | internal  | http://10.10.30.8:8778/placement             |
 | b76395b1badb4a2bb462f35e35b71522 | RegionOne | placement    | placement      | True    | admin     | http://10.10.30.8:8778/placement             |

> netstat -an | grep -i 8778 
tcp        0      0 0.0.0.0:8778            0.0.0.0:*               LISTEN     
tcp        1      0 10.10.30.8:59868        10.10.30.8:8778         CLOSE_WAIT 
tcp        0      0 10.10.30.8:8778         10.10.30.8:58828        TIME_WAIT  
tcp        0      0 10.10.30.8:8778         10.10.30.8:59868        FIN_WAIT2

> neutron agent-list

+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host             | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+
| 293e5768-18ff-4733-aeaa-3faeff3f0165 | DHCP agent         | cloud.centos.lan | nova              | :-)   | True           | neutron-dhcp-agent        |
| 2ee38f4f-351f-4a86-8f37-635ca07cc6f1 | Open vSwitch agent | cloud.centos.lan |                   | :-)   | True           | neutron-openvswitch-agent |
| cc037a05-0873-4fbd-8b44-85ec046ca264 | Metadata agent     | cloud.centos.lan |                   | :-)   | True           | neutron-metadata-agent    |
| eec3cee5-b3a1-4f98-85e7-89d6f5584d77 | L3 agent           | cloud.centos.lan | nova              | :-)   | True           | neutron-l3-agent          |
| f8f85b4e-73f0-4636-9bef-a7cb68394c63 | Metering agent     | cloud.centos.lan |                   | :-)   | True           | neutron-metering-agent    |
+--------------------------------------+--------------------+------------------+-------------------+-------+----------------+---------------------------+

> nano /etc/nova/nova.conf
virt_type = qemu



> ip netns
qdhcp-4d8df480-f1ec-413a-8500-a2e05caba55d (id: 4)
qdhcp-208f9e3c-5e66-47a1-95d3-02c9860d122a (id: 3)
qdhcp-98e9a9df-2698-44cd-95b1-905fbbbe1165 (id: 1)
qdhcp-bd58da20-f41b-483f-bee0-55f5e51b7d9a (id: 2)
qrouter-73240828-df62-4c02-b214-be6ee9572bda (id: 0)

> ip netns exec qrouter-73240828-df62-4c02-b214-be6ee9572bda bash

> ping 10.0.3.5
PING 10.0.3.5 (10.0.3.5) 56(84) bytes of data.
64 bytes from 10.0.3.5: icmp_seq=1 ttl=64 time=4.44 ms
64 bytes from 10.0.3.5: icmp_seq=2 ttl=64 time=0.879 ms
^C
--- 10.0.3.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.879/2.661/4.444/1.783 ms

Aodh Exception

Check Aodh Logs: 

cat /var/log/aodh/evaluator.log

If you see "Table 'aodh.alarm' doesn't exist" error try to initialize aodh database: 

aodh-dbsync



References





{{#widget:DISQUS |id=networkm |uniqid=OpenStack |url=https://aman.awiki.org/wiki/OpenStack }}

Retrieved from "https://aman.awiki.org/wiki/OpenStack?oldid=2811"