Control regional access to you service on OpenShift running on AWS

In the last blog we talked about how we can find out the IP address of the clients accessing our service running on OpenShift, and how we can block them.

Once you have our content created as a service running on OpenShift on AWS, its delivery to the end customers becomes a matter of utmost importance. This is where AWS CloudFront can help. CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. 

CloudFront not only delivers both static and dynamic content using a single distribution. It can also control users in selected countries from accessing our content by providing either a whitelist (countries where they can access your content) or a blacklist (countries where they cannot).

In this blog we will look at how to create an AWS CloudFront Distribution, and how we can restrict countries from accessing our Service running on OpenShift. 

For this blog I have deployed OpenShift on AWS using the IPI (Installer Provisioned Installer), and for the service I created a blog application, which is designed to show the various features of OpenShift. The blog application is implemented using Python and Django.

To learn more about installing the Blog application from the source, please visit https://learn.openshift.com/introduction/deploying-python/ 

As I created the application using the web console, a Route was automatically created for the application and it will be exposed outside of the cluster. The URL to access the blog application is as: http://blog-django-py-myproject.apps.mytest.ocp4-test-mshetty.com

Let's look at how we can create an AWS CloudFront Distribution, and use it to blacklist certain countries from accessing our blog service.

1. We start by creating a AWS CloudFront Distribution by going to CloudFront from the AWS console.

2. Next, create a new Distribution by clicking on the “Create Distribution” button.

3. In the “Select a delivery method for your content.” select the “Get Started” button under the Web distribution.

4. When you click on the Origin Domain Name field you’ll see the S3 buckets used for the registry, and all the Elastic Load Balancers, ignore and that and enter “blog-django-py-myproject.apps.mytest.ocp4-test-mshetty.com”. I left all the other values in the form as default for this post, and clicked on the “Create Distribution” at the bottom of the page. 

5. The CloudFront Distribution is created when the State is Enabled, and the Status changes from “In Progress” to “Deployed”.

6. We can now use the CloudFront Domain name “d2ol5va9tfpqgt.cloudfront.net” to access our service by entering it into the URL of the browser.

7. Now to blacklist a country click on the “Distribution Settings” button, and then go to the “Restrictions” tab and click on Edit.

8. Next, we can test this by connecting to a VPN server in that country or with LocaBrowser, by trying to access the service using the CloudFront domain name.

Using the CloudFront domain name is fine, but wouldn’t it be nice to use your own domain name (for example, www.example.com) in the files’ URLs instead. Check out the link for information on using alternate domain names.

AWS WAF for additional rules and resources:

If you want to apply the country blacklist to your other resources on AWS, or you want to apply additional security rules to your service running on OpenShift, you could use AWS WAF to create the rules, and apply them to your CloudFront distributions or other Regional resources (Application Load Balancer, API Gateway, AWS AppSync).

As you see, I have the rule that blocks countries along with three additional rules, that make up the AWS WAF Web ACL called myACL. The Web ACL is then applied to the my CloudFront Distribution resource type.

You see below that the AWS WAF Web ACL is associated with the CloudFront resource used for our OpenShift service.

There we go! It is really that simple to deliver and block access from certain regions to services running on OpenShift on AWS.

Adding node to an OpenShift cluster running on AWS

This post talks about scaling an OpenShift 3.11 cluster deployed on AWS using the Quick Start deployment guide.

The OpenShift Cluster was deployed using 8 m5.xlarge instances as follows:
Master = 3 x m5.xlarge
Etcd = 3 x m5.xlarge
Worker = 1 x m5.xlarge
Bastion/Ansible = 1 x m5.xlarge

The new node was created by right-clicking the existing worker node, and using "Launch More Like This" from the menu.

After the new node is up and running, make sure you can ssh to it from the Bastion/Ansible node.

Next, log into the Bastion/Ansible node and edit the /etc/ansible/hosts file to add the new_node information.

    new_nodes:

      hosts:

        ip-10-0-66-172.us-east-2.compute.internal:

          instance_id: i-092768fa7ac7d9caa

          openshift_node_group_name: node-config-compute-infra

    nodes:

      hosts:

        ip-10-0-23-92.us-east-2.compute.internal: *id001

        ip-10-0-60-39.us-east-2.compute.internal: *id002

        ip-10-0-73-252.us-east-2.compute.internal: 

          instance_id: i-0cc9f39f54c8c87e6

          openshift_node_group_name: node-config-compute-infra

        ip-10-0-95-180.us-east-2.compute.internal: *id003

Next, run the Ansible playbook as follows:

[root@ip-10-0-130-225 ~]# cd /usr/share/ansible/openshift-ansible

[root@ip-10-0-130-225 openshift-ansible]# ansible-playbook -i /etc/ansible/hosts playbooks/openshift-node/scaleup.yml

PLAY [Populate config host groups] *************************************************************************************************************************************************************************************

TASK [Load group name mapping variables] *******************************************************************************************************************************************************************************

Thursday 05 September 2019  18:47:11 +0000 (0:00:00.062)       0:00:00.062 **** 

ok: [localhost]

TASK [Evaluate groups - g_nfs_hosts is single host] ********************************************************************************************************************************************************************

Thursday 05 September 2019  18:47:11 +0000 (0:00:00.026)       0:00:00.089 **** 

skipping: [localhost]

TASK [Evaluate oo_all_hosts] *******************************************************************************************************************************************************************************************

Thursday 05 September 2019  18:47:11 +0000 (0:00:00.020)       0:00:00.109 **** 

ok: [localhost] => (item=ip-10-0-23-92.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-95-180.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-60-39.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-73-252.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-69-149.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-2-243.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-57-57.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-66-172.us-east-2.compute.internal)

TASK [Evaluate oo_masters] *********************************************************************************************************************************************************************************************

Thursday 05 September 2019  18:47:11 +0000 (0:00:00.087)       0:00:00.197 **** 

ok: [localhost] => (item=ip-10-0-23-92.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-95-180.us-east-2.compute.internal)

ok: [localhost] => (item=ip-10-0-60-39.us-east-2.compute.internal)

It will take while before it completes....

TASK [openshift_storage_glusterfs : Generate topology file] ************************************************************************************************************************************************************

Thursday 05 September 2019  19:28:49 +0000 (0:00:00.080)       0:41:37.833 **** 

skipping: [ip-10-0-23-92.us-east-2.compute.internal]

TASK [openshift_storage_glusterfs : Place heketi topology on heketi Pod] ***********************************************************************************************************************************************

Thursday 05 September 2019  19:28:49 +0000 (0:00:00.072)       0:41:37.906 **** 

skipping: [ip-10-0-23-92.us-east-2.compute.internal]

TASK [openshift_storage_glusterfs : Load heketi topology] **************************************************************************************************************************************************************

Thursday 05 September 2019  19:28:49 +0000 (0:00:00.075)       0:41:37.981 **** 

skipping: [ip-10-0-23-92.us-east-2.compute.internal]

TASK [openshift_storage_glusterfs : Delete temp directory] *************************************************************************************************************************************************************

Thursday 05 September 2019  19:28:49 +0000 (0:00:00.218)       0:41:38.200 **** 

ok: [ip-10-0-23-92.us-east-2.compute.internal]

PLAY RECAP *************************************************************************************************************************************************************************************************************

ip-10-0-2-243.us-east-2.compute.internal : ok=14   changed=1    unreachable=0    failed=0   

ip-10-0-23-92.us-east-2.compute.internal : ok=67   changed=2    unreachable=0    failed=0   

ip-10-0-57-57.us-east-2.compute.internal : ok=14   changed=1    unreachable=0    failed=0   

ip-10-0-60-39.us-east-2.compute.internal : ok=32   changed=1    unreachable=0    failed=0   

ip-10-0-66-172.us-east-2.compute.internal : ok=152  changed=82   unreachable=0    failed=0   

ip-10-0-69-149.us-east-2.compute.internal : ok=14   changed=1    unreachable=0    failed=0   

ip-10-0-95-180.us-east-2.compute.internal : ok=32   changed=1    unreachable=0    failed=0   

localhost                  : ok=22   changed=0    unreachable=0    failed=0   

INSTALLER STATUS *******************************************************************************************************************************************************************************************************

Initialization              : Complete (0:02:42)

Node Bootstrap Preparation  : Complete (0:30:10)

Node Join                   : Complete (0:00:20)

Thursday 05 September 2019  19:28:49 +0000 (0:00:00.150)       0:41:38.351 **** 

=============================================================================== 

openshift_node : install needed rpm(s) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1018.88s

container_runtime : Install Docker ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 331.95s

openshift_node : Install iSCSI storage plugin dependencies ---------------------------------------------------------------------------------------------------------------------------------------------------- 244.13s

openshift_node : Install node, clients, and conntrack packages ------------------------------------------------------------------------------------------------------------------------------------------------ 125.15s

Ensure openshift-ansible installer package deps are installed ------------------------------------------------------------------------------------------------------------------------------------------------- 119.25s

openshift_node : Install NFS storage plugin dependencies ------------------------------------------------------------------------------------------------------------------------------------------------------ 116.75s

openshift_node : Install dnsmasq ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 99.21s

openshift_node : Install Ceph storage plugin dependencies ------------------------------------------------------------------------------------------------------------------------------------------------------ 97.82s

os_firewall : Install iptables packages ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 93.99s

openshift_node : Install GlusterFS storage plugin dependencies ------------------------------------------------------------------------------------------------------------------------------------------------- 60.62s

openshift_excluder : Install docker excluder - yum ------------------------------------------------------------------------------------------------------------------------------------------------------------- 42.84s

openshift_repos : Ensure libselinux-python is installed -------------------------------------------------------------------------------------------------------------------------------------------------------- 15.66s

nickhammond.logrotate : nickhammond.logrotate | Install logrotate ---------------------------------------------------------------------------------------------------------------------------------------------- 15.06s

openshift_manage_node : Wait for sync DS to set annotations on all nodes --------------------------------------------------------------------------------------------------------------------------------------- 10.75s

os_firewall : need to pause here, otherwise the iptables service starting can sometimes cause ssh to fail ------------------------------------------------------------------------------------------------------ 10.08s

openshift_repos : refresh cache --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.30s

container_runtime : Get current installed Docker version -------------------------------------------------------------------------------------------------------------------------------------------------------- 5.08s

Approve node certificates when bootstrapping -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.57s

container_runtime : restart container runtime ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.54s

tuned : Ensure files are populated from templates --------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.47s

Login and check if the new node has been added to the cluster.

[root@ip-10-0-130-225 openshift-ansible]# oc login -u system:admin

Logged into "https://Red-H-OpenS-1OZOCWK21Q1XP-9b6c91fba5af361e.elb.us-east-2.amazonaws.com:443" as "system:admin" using existing credentials.

You have access to the following projects and can switch between them with 'oc project <projectname>':

  * default

    kube-public

    kube-service-catalog

    kube-system

    management-infra

    openshift

    openshift-ansible-service-broker

    openshift-console

    openshift-infra

    openshift-logging

    openshift-monitoring

    openshift-node

    openshift-sdn

    openshift-template-service-broker

    openshift-web-console

Using project "default".

[root@ip-10-0-130-225 openshift-ansible]# oc get nodes 

NAME                                        STATUS    ROLES           AGE       VERSION

ip-10-0-23-92.us-east-2.compute.internal    Ready     master          13h       v1.11.0+d4cacc0

ip-10-0-60-39.us-east-2.compute.internal    Ready     master          13h       v1.11.0+d4cacc0

ip-10-0-66-172.us-east-2.compute.internal   Ready     compute,infra   3m        v1.11.0+d4cacc0

ip-10-0-73-252.us-east-2.compute.internal   Ready     compute,infra   13h       v1.11.0+d4cacc0

ip-10-0-95-180.us-east-2.compute.internal   Ready     master          13h       v1.11.0+d4cacc0

Move any hosts that you defined in the new_nodes section to the appropriate section ie. nodes. By moving these hosts, subsequent playbook runs that use this inventory file treat the nodes correctly. You can keep the empty new_nodes section. 

    new_masters: {}

    new_nodes: {}

    nodes:

      hosts:

        ip-10-0-23-92.us-east-2.compute.internal: *id001

        ip-10-0-60-39.us-east-2.compute.internal: *id002

        ip-10-0-73-252.us-east-2.compute.internal:

          instance_id: i-0cc9f39f54c8c87e6

          openshift_node_group_name: node-config-compute-infra

        ip-10-0-66-172.us-east-2.compute.internal:

          instance_id: i-092768fa7ac7d9caa

          openshift_node_group_name: node-config-compute-infra

        ip-10-0-95-180.us-east-2.compute.internal: *id003

    provision_in_progress: {}

No chicken or egg with Ansible!!!

With Ansible being agentless we don't have a what came first, chicken or the egg, issue with it. What I mean by this is, in most automation/configuration tools which require an agent to be installed, you would have to first install the agent on the host that you are going to manage.

In the case of Ansible you don't have to install anything on the host that you wish to manage. As long as you can ssh onto the host, you are good to go.

So consider a brand you RHEL server or VM that you created, and that you can ssh onto it as root. Here are the steps that you would need to do to run Ansible playbooks as user 'mango', or any other fruit of your choice.

1. Get the root password of the managed host.
2. Check to see if you can ssh to the managed host using root password.
3. Create ssh key using ssh-keygen on the control host for root.
4. Copy the ssh key to the managed host using ssh-copy-id
5. If you want to run the Ansible playbooks as user 'mango' then create a user 'mango' on the remote host using Ansible playbook shown below.
6. Make can entry for that user under /etc/sudoers.d so that the user can run commands that need root priviliges.

The Ansible playbook below would do exactly that for you:

---

- name: Adding a new user

  hosts: all

  tasks:

    - name: Create user mango

      user: 

        name: mango 

        append: yes 

        state: present

        createhome: yes

    - name: Add file mango to the sudoers.d directory

      copy:

        content: "mango ALL=(ALL) NOPASSWD: ALL"

        dest: /etc/sudoers.d/mango

    - name: Set up authorized keys for the mango user

      authorized_key: 

        user=mango 

        key="{{ item }}"

      with_file:

        - /home/devops/.ssh/id_rsa.pub

...

Hardware requirements for Red Hat OpenStack and Red Ceph Storage

I recently worked with a couple of our OEM Partners who wanted to create a Reference Architecture with Red Hat OpenStack, and Red Hat Ceph Storage. One of the first questions that come up during the conversation is the hardware requirement for a Reference Architecture.

The below presentation covers not only the minimum requirements for an OpenStack deployment with Ceph Storage, but also the hardware(storage node) requirements for various Ceph workloads like IOPS-Optimized, Throughput-Optimized, and Cost/Capacity Optimized workloads.


  

Ceph for Glance in Red Hat OpenStack

In the previous post we saw the deployment of OpenStack Platform 10 with Ceph.

Lets log into the Controller Node, and look at all the services running.

[root@overcloud-controller-0 ~]# openstack service list

+----------------------------------+------------+----------------+

| ID                               | Name       | Type           |

+----------------------------------+------------+----------------+

| 316642206a194dc58cb4528adbc90f5d | heat-cfn   | cloudformation |

| 593f9fde36634401977d2806a9e6f6f5 | gnocchi    | metric         |

| 6fd62024660942ec8d6451429f03d336 | ceilometer | metering       |

| 762a53b0efd54947b3d6d2d6405440b0 | cinderv2   | volumev2       |

| 7783685d418845b9acc1bad50fbbfe7d | nova       | compute        |

| 93987d0d0fea4f02a73e7825c9b4adfe | glance     | image          |

| 9a9990c7780945aca10fbeba19cc4729 | aodh       | alarming       |

| a13c6596df864b17a33dc674a278c31c | cinderv3   | volumev3       |

| bdb018992e1146b499d5b56d8fdfce29 | heat       | orchestration  |

| d0e8d3c58784485888d5b36d71af39a4 | keystone   | identity       |

| d7a05ca55d2349d2b1f4dbc5fb79fb2d | cinder     | volume         |

| e238bbf05e8547e7b4b57ee4dba52a63 | neutron    | network        |

| fb32e87e977742899426a848077b09db | swift      | object-store   |

+----------------------------------+------------+----------------+

Lets now look at all the Ceph related packages installed on the Controller Node.

[root@overcloud-controller-0 ~]# yum list installed | grep -i ceph

ceph-base.x86_64                    1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

ceph-common.x86_64                  1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

ceph-mon.x86_64                     1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

ceph-osd.x86_64                     1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-osd-signed

ceph-radosgw.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-tools-signed

ceph-selinux.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

fcgi.x86_64                         2.4.0-27.el7cp     @rhos-10.0-ceph-2.0-mon-signed

gperftools-libs.x86_64              2.4-8.el7          @rhos-10.0-ceph-2.0-mon-signed

leveldb.x86_64                      1.12.0-7.el7cp     @rhos-10.0-ceph-2.0-mon-signed

libbabeltrace.x86_64                1.2.4-4.el7cp      @rhos-10.0-ceph-2.0-mon-signed

libcephfs1.x86_64                   1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

librados2.x86_64                    1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

librbd1.x86_64                      1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

librgw2.x86_64                      1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

lttng-ust.x86_64                    2.4.1-4.el7cp      @rhos-10.0-ceph-2.0-mon-signed

puppet-ceph.noarch                  2.3.0-5.el7ost     @rhos-10.0-signed        

python-cephfs.x86_64                1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

python-flask.noarch                 1:0.10.1-5.el7     @rhos-10.0-ceph-2.0-mon-signed

python-jinja2.noarch                2.7.2-2.el7cp      @rhos-10.0-ceph-2.0-mon-signed

python-netifaces.x86_64             0.10.4-3.el7ost    @rhos-10.0-ceph-2.0-tools-signed

python-rados.x86_64                 1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

python-rbd.x86_64                   1:10.2.7-28.el7cp  @rhos-10.0-ceph-2.0-mon-signed

                                                       @rhos-10.0-ceph-2.0-tools-signed

                                                       @rhos-10.0-ceph-2.0-tools-signed

userspace-rcu.x86_64                0.7.16-1.el7cp     @rhos-10.0-ceph-2.0-mon-signed

Lets check the Ceph cluster status(as user admin) before we begin doing anything. I'll ignore the HEALTH_WARN for the time being. These users are Ceph users and not linux system users. The Ceph users are created using the "ceph auth get-or-create <ceph-user-name>" command

[root@overcloud-controller-0 ~]# ceph -s

    cluster 56a3d310-7ba4-11e7-a540-525400ccb563

     health HEALTH_WARN

            224 pgs degraded

            224 pgs stuck degraded

            224 pgs stuck unclean

            224 pgs stuck undersized

            224 pgs undersized

            recovery 56/84 objects degraded (66.667%)

     monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}

            election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0

     osdmap e20: 2 osds: 2 up, 2 in

            flags sortbitwise,require_jewel_osds

      pgmap v7155: 224 pgs, 6 pools, 346 kB data, 28 objects

            86552 kB used, 30613 MB / 30697 MB avail

            56/84 objects degraded (66.667%)

                 224 active+undersized+degraded

Lets verify if the required Ceph configurations are created to integrate with Glance. We will see that a Ceph user "client.openstack" with read permission for monitor, and read, write, execute permission for the created pool.

[root@overcloud-controller-0 ~]# ceph osd pool ls

rbd

metrics

images

backups

volumes

vms

[root@overcloud-controller-0 ~]# ceph auth list

installed auth entries:

osd.0

key: AQAAWI5Z0ygRCBAA4p3emGMKLyvpUxuViBK28w==

caps: [mon] allow profile osd

caps: [osd] allow *

osd.1

key: AQACWI5ZIFwIAxAA+59beKFKKmvdg6K6XiBGKg==

caps: [mon] allow profile osd

caps: [osd] allow *

client.admin

key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==

caps: [mds] allow *

caps: [mon] allow *

caps: [osd] allow *

client.bootstrap-osd

key: AQCVu4hZAAAAABAAFCkRGc+Rx6MS8iJx1nrsoQ==

caps: [mon] allow profile bootstrap-osd

client.openstack

key: AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==

caps: [mon] allow r

caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics

The pool created for Glance is "images". Lets look into some details of the "images" pool.

[root@overcloud-controller-0 ~]# ceph osd pool stats images

pool images id 2

  nothing is going on

Next, in the /etc/ceph directory we see the keyring for the user "client.openstack" and the ceph.conf files.

[root@overcloud-controller-0 ~]# ls -l /etc/ceph/

total 16

-rw-------. 1 root root 129 Aug 12 01:15 ceph.client.admin.keyring

-rw-r--r--. 1 root root 262 Aug 12 01:15 ceph.client.openstack.keyring

-rw-r--r--. 1 root root 561 Aug 12 01:15 ceph.conf

-rw-r--r--. 1 root root  92 Jul  5 20:22 rbdmap

Lets look into the keyring file for the "openstack" user.

[root@overcloud-controller-0 ~]# cat /etc/ceph/ceph.client.openstack.keyring 

[client.openstack]

key = AQCVu4hZAAAAABAAZIqVy2txgF4DfJUyU/6N6A==

caps mon = "allow r"

caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics"

We will no see if we can check the Ceph cluster status as user "openstack".

[root@overcloud-controller-0 ~]# ceph --id openstack -s

    cluster 56a3d310-7ba4-11e7-a540-525400ccb563

     health HEALTH_WARN

            224 pgs degraded

            224 pgs stuck degraded

            224 pgs stuck unclean

            224 pgs stuck undersized

            224 pgs undersized

            recovery 56/84 objects degraded (66.667%)

     monmap e1: 3 mons at {overcloud-controller-0=172.17.3.21:6789/0,overcloud-controller-1=172.17.3.20:6789/0,overcloud-controller-2=172.17.3.15:6789/0}

            election epoch 10, quorum 0,1,2 overcloud-controller-2,overcloud-controller-1,overcloud-controller-0

     osdmap e20: 2 osds: 2 up, 2 in

            flags sortbitwise,require_jewel_osds

      pgmap v7159: 224 pgs, 6 pools, 346 kB data, 28 objects

            86552 kB used, 30613 MB / 30697 MB avail

            56/84 objects degraded (66.667%)

                 224 active+undersized+degraded

  client io 68 B/s rd, 0 op/s rd, 0 op/s wr

Now, just for kicks lets try getting the cluster status as user "foo".

[root@overcloud-controller-0 ~]# ceph --id foo -s

2017-08-16 00:26:11.418444 7f5b36807700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.foo.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory

2017-08-16 00:26:11.418458 7f5b36807700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication

2017-08-16 00:26:11.418459 7f5b36807700  0 librados: client.foo initialization error (2) No such file or directory

Error connecting to cluster: ObjectNotFound

Next, we'll see how Glance is configured to use "rbd" for backend. We can see below that rbd_storage_pool is "images", and "rbd_store_user" is openstack.

[root@overcloud-controller-0 ~]# cat /etc/glance/glance-api.conf | grep rbd

#         * rbd

stores = glance.store.http.Store,glance.store.rbd.Store

#     * rbd

# Allowed values: file, filesystem, http, https, swift, swift+http, swift+https, swift+config, rbd, sheepdog, cinder, vsphere

default_store = rbd

#rbd_store_chunk_size = 8

#rbd_store_pool = images

rbd_store_pool = images

# section in rbd_store_ceph_conf.

#     * rbd_store_ceph_conf

#rbd_store_user = <None>

rbd_store_user = openstack

#     * rbd_store_user

#rbd_store_ceph_conf = /etc/ceph/ceph.conf

#         * rbd

Next, we can see the status of "openstack-glance-api"

[root@overcloud-controller-0 ~]# systemctl status openstack-glance-api

● openstack-glance-api.service - OpenStack Image Service (code-named Glance) API server

   Loaded: loaded (/usr/lib/systemd/system/openstack-glance-api.service; enabled; vendor preset: disabled)

   Active: active (running) since Sat 2017-08-12 01:48:39 UTC; 3 days ago

 Main PID: 190781 (glance-api)

   CGroup: /system.slice/openstack-glance-api.service

           ├─190781 /usr/bin/python2 /usr/bin/glance-api

           ├─190894 /usr/bin/python2 /usr/bin/glance-api

           ├─190895 /usr/bin/python2 /usr/bin/glance-api

           ├─190896 /usr/bin/python2 /usr/bin/glance-api

           └─190897 /usr/bin/python2 /usr/bin/glance-api

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Lets download a cirros image, and then upload it onto Glance.

[root@overcloud-controller-0 ~]# curl -o /tmp/cirros.qcow2 \

>  http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 12.6M  100 12.6M    0     0   297k      0  0:00:43  0:00:43 --:--:--  590k

[root@overcloud-controller-0 ~]# openstack image create  --disk-format qcow2 --container-format bare --public  --file /tmp/cirros.qcow2 cirros

+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Field            | Value                                                                                                                                                                                                      |

+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| checksum         | ee1eca47dc88f4879d8a229cc70a07c6                                                                                                                                                                           |

| container_format | bare                                                                                                                                                                                                       |

| created_at       | 2017-08-16T01:38:46Z                                                                                                                                                                                       |

| disk_format      | qcow2                                                                                                                                                                                                      |

| file             | /v2/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/file                                                                                                                                                       |

| id               | 9b8972f9-6116-4bbe-97de-b416a74cdad2                                                                                                                                                                       |

| min_disk         | 0                                                                                                                                                                                                          |

| min_ram          | 0                                                                                                                                                                                                          |

| name             | cirros                                                                                                                                                                                                     |

| owner            | f0ac0df6e1be446394f28ad66fb40f3c                                                                                                                                                                           |

| properties       | direct_url='rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe-97de-b416a74cdad2/snap', locations='[{u'url': u'rbd://56a3d310-7ba4-11e7-a540-525400ccb563/images/9b8972f9-6116-4bbe-     |

|                  | 97de-b416a74cdad2/snap', u'metadata': {}}]'                                                                                                                                                                |

| protected        | False                                                                                                                                                                                                      |

| schema           | /v2/schemas/image                                                                                                                                                                                          |

| size             | 13287936                                                                                                                                                                                                   |

| status           | active                                                                                                                                                                                                     |

| tags             |                                                                                                                                                                                                            |

| updated_at       | 2017-08-16T01:38:53Z                                                                                                                                                                                       |

| virtual_size     | None                                                                                                                                                                                                       |

| visibility       | public                                                                                                                                                                                                     |

+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Using the rbd command we will see that the image name is the "id" of the Glance image. 

[root@overcloud-controller-0 ~]# rbd --id openstack -p images ls

9b8972f9-6116-4bbe-97de-b416a74cdad2

The "ceph df" command gives a good detail of all the ceph pools and the usage of each of them.

[root@overcloud-cephstorage-0 ceph]# ceph df

GLOBAL:

    SIZE       AVAIL      RAW USED     %RAW USED 

    30697M     30582M         115M          0.38 

POOLS:

    NAME        ID     USED       %USED     MAX AVAIL     OBJECTS 

    rbd         0           0         0        10194M           0 

    metrics     1        609k         0        10194M          47 

    images      2      12976k      0.04        10194M           7 

    backups     3           0         0        10194M           0 

    volumes     4           0         0        10194M           0 

    vms         5           0         0        10194M           0 

For more details on the osd. The ceph-osd did not start for some reason on overcloud-cephstorage-1, I need to root cause that.

[root@overcloud-cephstorage-0 ceph]# ceph osd df tree

ID WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE VAR  PGS TYPE NAME                        

-1 0.02917        - 30697M   115M 30582M 0.38 1.00   0 root default                     

-2 0.01459        - 15348M 59464k 15290M 0.38 1.00   0     host overcloud-cephstorage-2 

 0 0.01459  1.00000 15348M 59464k 15290M 0.38 1.00 224         osd.0                    

-3 0.01459        - 15348M 59228k 15291M 0.38 1.00   0     host overcloud-cephstorage-0 

 1 0.01459  1.00000 15348M 59228k 15291M 0.38 1.00 224         osd.1                    

              TOTAL 30697M   115M 30582M 0.38                                           

MIN/MAX VAR: 1.00/1.00  STDDEV: 0

I have 1 x HDD on each of the Ceph Storage nodes dedicated to Ceph, so for that reason we have both data and journal on the same disk. The output of "fdisk -l" clearly shows that the first partition of vdb is used for data, while the second partition is used for journal.

[root@overcloud-cephstorage-0 ~]# fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x000451ac

   Device Boot      Start         End      Blocks   Id  System

/dev/vda1            2048        4095        1024   83  Linux

/dev/vda2   *        4096    41943006    20969455+  83  Linux

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: gpt

Disk identifier: EF24FDED-84D2-4723-A855-092A9438F6F7

#         Start          End    Size  Type            Name

 1     10487808     41943006     15G  unknown         ceph data

 2         2048     10487807      5G  unknown         ceph journal