Centos7 Upgrading Kernel to 3.19

Centos 7 is the latest release from the community.  Centos7 is shipped with 3.10 Kernel. Many times people may need the latest Kernel. This article telling about how to upgrade from 3.10 to latest upstream kernel.

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

# yum --enablerepo=elrepo-kernel install kernel-ml

Once the installation is finished, make sure you have got new kernel.

# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux, with Linux 3.19.0-1.el7.elrepo.x86_64

CentOS Linux, with Linux 3.10.0-123.el7.x86_64

CentOS Linux, with Linux 3.10.0-123.20.1.el7.x86_64

CentOS Linux, with Linux 0-rescue-b6ea1092b61e4ac7a546ba4454bd63d0

If the CentOS kernel version 3.19.x is listed first, then set grub to use 0 which will set the system to boot off the newer kernel. You might want to leave some of the older 3.10.x kernels on the system in case you decide to revert to the stock CentOS 7 kernel, but you can always modify the default kernel to use later on.

# grub2-set-default 0

Update the grub2 config so it's aware of the changes we just made.

# grub2-mkconfig -o /boot/grub2/grub.cfg

# reboot

Check the new kernal.

# uname -a

Linux compute1 3.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Feb 9 09:57:29 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

3 Node MySQL Cluster using Percona

Percona XtraDB Cluster is an active/active high availability and high scalability open source solution for MySQL clustering. It integrates Percona Server and Percona XtraBackup with the Galera library of MySQL high availability solutions in a single package which enables you to create a cost-effective MySQL high availability cluster. 

Here following a 3 node mysql cluster using Percona. The infrastructure is as follows.

All the Nodes are running on Centos 6.6. The IPAddress / Hostnames  of the Nodes are as follows. Turn off IPTABLES in all these nodes, OR allow traffic between these nodes.

192.168.1.1      cluster-node1

192.168.1.2      cluster-node2

192.168.1.3      cluster-node3

Add these entries in each and every nodes /etc/hosts files.

#vim /etc/hosts

192.168.1.1      cluster-node1

192.168.1.2      cluster-node2

192.168.1.3      cluster-node3

Disable SELINUX

Set Up SSH Keys and place into authorized_keys so all the id_rsa.pub values are in authorized_keys on all servers.

# ssh-keygen -t rsa

# cd /root/.ssh/

# cp id_rsa.pub authorized_keys 

# chmod 600  /root/.ssh/authorized_keys

# chmod 700  /root/.ssh/

Install the Needed Repositories for Percona & Socat.

# yum -y install http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm

# yum -y install http://mirror.pnl.gov/epel/6/x86_64/epel-release-6-8.noarch.rpm

# wget http://www.percona.com/downloads/RPM-GPG-KEY-percona /etc/pki/rpm-gpg/RPM-GPG-KEY-percona 

# wget http://www.percona.com/downloads/RPM-GPG-KEY-percona /etc/pki/rpm-gpg/RPM-GPG-KEY-percona 

# yum -y install socat

Remove the following applications to avoid the dependencies, Later we can bring them back.

# rpm -e mysql-libs postfix cronie redhat-lsb-core redhat-lsb-printing redhat-lsb-graphics libcgroup numad redhat-lsb sysstat crontabs cronie-anacron redhat-lsb-compat

Install the Percona Cluster Packages (Repeat the following Commands in all the Nodes)

#yum -y install Percona-XtraDB-Cluster-full-56

Start the Server.

[root@cluster-node1 ~]#  /etc/init.d/mysql start

Starting MySQL (Percona XtraDB Cluster)......... SUCCESS! 

[root@cluster-node2 ~]# /etc/init.d/mysql start

Starting MySQL (Percona XtraDB Cluster)......... SUCCESS!

[root@cluster-node3 ~]# /etc/init.d/mysql start

Starting MySQL (Percona XtraDB Cluster)......... SUCCESS! 

Run the following Commands

#mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'" 

#mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"

#mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"

Bring in Back the removed packages

#yum -y install postfix cronie redhat-lsb-core redhat-lsb-printing redhat-lsb-graphics libcgroup numad redhat-lsb sysstat crontabs cronie-anacron redhat-lsb-compat

Create the my.cnf file in first node and insert the following contents

#vim /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

user=mysql

# Path to Galera library

wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3

# cluster-node 1 192.168.1.1

# cluster-nod3 2 192.168.1.2

# cluster-nod3 3 192.168.1.3

wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3

# In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

# MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

# This changes how InnoDB auto increment locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

# Node #1 address

wsrep_node_address=192.168.1.1

# SST method

#wsrep_sst_method=xtrabackup

 wsrep_sst_method=rsync      #

# wsrep_sst_method=rsync_wan  #

# wsrep_sst_method=mysqldump  # SLOW

# Cluster name

wsrep_cluster_name=percona_cluster

# Authentication for SST method

wsrep_sst_auth="root:password" 

# server_id

server_id=3232235553  #SELECT INET_ATON('192.168.1.1')

#[client]

socket=/var/lib/mysql/mysql.sock

Start the First Clustering Node.

#/etc/init.d/mysql start --wsrep-cluster-address="gcomm://"

Starting MySQL (Percona XtraDB Cluster)...................................... SUCCESS!

Read the grastate.dat file . (It will be in /var/lib/mysql)

[root@cluster-node1 mysql]# cat grastate.dat
# GALERA saved state
version: 2.1
uuid:    68aa64c3-a966-11e4-96dd-b6bab6cb5af6
seqno:   -1

cert_index:

Check the Cluster Status.

mysql>select @@hostname\G show global status like 'wsrep_cluster_size' \G

*************************** 1. row ***************************
@@hostname: controller-node1
1 row in set (0.00 sec)

*************************** 1. row ***************************
Variable_name: wsrep_cluster_size
        Value: 3
1 row in set (0.00 sec)

#Now copy the following Contents in /etc/my.cnf file of Second Node

#vim /etc/my.cnf
[mysqld]

datadir=/var/lib/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
# cluster-node 1 192.168.1.1
# cluster-nod3 2 192.168.1.2
# cluster-nod3 3 192.168.1.3
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB auto increment locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=192.168.1.2

# SST method
#wsrep_sst_method=xtrabackup
 wsrep_sst_method=rsync      #
# wsrep_sst_method=rsync_wan  #
# wsrep_sst_method=mysqldump  # SLOW

# Cluster name
wsrep_cluster_name=percona_cluster

# Authentication for SST method
wsrep_sst_auth="root:password" 

# server_id
server_id=3232235553  #SELECT INET_ATON('192.168.1.2')

#[client]

socket=/var/lib/mysql/mysql.sock

Start the Cluster in Second Node.

[root@cluster-node2]#/etc/init.d/mysql start

Starting MySQL (Percona XtraDB Cluster)........................... SUCCESS!

Now check the Status in Both Nodes.

First Node

mysql> select @@hostname\G show global status like 'wsrep_cluster_size' \G

*************************** 1. row ***************************

@@hostname: controller-node1

1 row in set (0.00 sec)

*************************** 1. row ***************************

Variable_name: wsrep_cluster_size

        Value: 2

1 row in set (0.00 sec)

Second Node

mysql> select @@hostname\G show global status like 'wsrep_cluster_size' \G

*************************** 1. row ***************************

@@hostname: controller-node2

1 row in set (0.00 sec)

*************************** 1. row ***************************

Variable_name: wsrep_cluster_size

        Value: 2

1 row in set (0.00 sec)

Now, We are adding 3 Node to the Cluster. 

#Now copy the following Contents in /etc/my.cnf file of Third  Node

#vim /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
# cluster-node 1 192.168.1.1
# cluster-nod3 2 192.168.1.2
# cluster-nod3 3 192.168.1.3
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB auto increment locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=192.168.1.3

# SST method
#wsrep_sst_method=xtrabackup
 wsrep_sst_method=rsync      #
# wsrep_sst_method=rsync_wan  #
# wsrep_sst_method=mysqldump  # SLOW

# Cluster name
wsrep_cluster_name=percona_cluster

# Authentication for SST method
wsrep_sst_auth="root:password" 

# server_id
server_id=3232235553  #SELECT INET_ATON('192.168.1.3')

#[client]

socket=/var/lib/mysql/mysql.sock

[root@cluster-node3]#/etc/init.d/mysql start

Starting MySQL (Percona XtraDB Cluster)........................... SUCCESS!

Now Check the Cluster status in Each nodes. 

First Node

mysql> select @@hostname\G show global status like 'wsrep_cluster_size' \G
*************************** 1. row ***************************
@@hostname: controller-node1
1 row in set (0.00 sec)

*************************** 1. row ***************************
Variable_name: wsrep_cluster_size
        Value: 3
1 row in set (0.00 sec)


Second Node
mysql> select @@hostname\G show global status like 'wsrep_cluster_size' \G
*************************** 1. row ***************************
@@hostname: controller-node2
1 row in set (0.00 sec)

*************************** 1. row ***************************
Variable_name: wsrep_cluster_size
        Value: 3

1 row in set (0.00 sec)

Third Node
mysql> select @@hostname\G show global status like 'wsrep_cluster_size' \G
*************************** 1. row ***************************
@@hostname: controller-node2
1 row in set (0.00 sec)

*************************** 1. row ***************************
Variable_name: wsrep_cluster_size
        Value: 3

1 row in set (0.00 sec)

You can see that, the Value has changed to 3. All are Good Now.

Testing.

Create a Test Database in First Node, You can see that it will be replicated in Another nodes.

CommunicationError: Error communicating with http://XX.XX.XX.XXX:9292 [Errno 111] ECONNREFUSED

Some times, when you trying to deploy a machine, you may ended up with the following error. In my case, the infrastructure is as follows.

One Controller Node, Two compute Nodes, One cinder and One Neutron node. All are Centos 6.6 with Icehouse Release.

2013-10-30 12:34:03.574 15487 TRACE nova.api.openstack CommunicationError: Error communicating with http://192.168.72.100:9292 [Errno 111] ECONNREFUSED

Even your glance_host configuration parameter in the /etc/cinder/cinder.conf is right, you will face this issue.  To resolve this issue, add the following in the /etc/nova/nova.conf file of Controller node.

image_service=nova.image.glance.GlanceImageService

glance_api_servers=192.168.72.145:9292  #IPis the Glance Host

Restart the Nova scheduler and API services in Controller node.

You are good to Go !!!

VirtualInterfaceCreateException: Virtual Interface creation failed : Openstack Compute Error

Some times, you may face this issue in your KVM hypervisor.  This is because libvirt wait for neutron to confirm plugging before boot. This makes the libvirt driver use the instance event mechanism to wait for neutron to confirm that VIF plugging is complete before  actually starting the VM. Which will put the vm status to error and in the compute.log you can see the following error.

TRACE nova.compute.manager [instance: 2a9464d0-0fed-423d-b855-a56f7a5f85a9] VirtualInterfaceCreateException: Virtual Interface creation failed

Inorder to prevent this, you have to add the following parametres in the nova.conf of the compute node.

vif_plugging_is_fatal: false
vif_plugging_timeout: 0

And restart the compute Node.

/etc/init.d/openstack-nova-compute restart

The error will disappear.

Tested with Openstack Icehouse, KVM having Kernel 3.18 and a separate node for Neutron.

Over-Committing And Ballooning in KVM

Over-Committing the resource is a vital part in Virtualization. Idea of over-committment is to allocate more memory to guest machines than the host really has. Suppose a 2GB host is running two guest operating system who is having 2GB memory, in this case we can say its over committed.

In Ballooning process, Host have a right to take the free memory from Guest machines if its there. Consider the above said example, where suppose one virtual machine is using 1GB and other 1GB is free, host can ask the guest to return the free memory. Guest will give it back and host can allocate somewhere else. And if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again. 

virtio_balloon - This is the driver who open a communication pathway between Host and the Guest virtual machines. Hypervisor (Host) uses this communication pathway to over-committ the resource from Guest machines. If you are not loaded the virtio_balloon driver, you are not able to do the following.

1. The guest operating system ignores memory requests from the hypervisor.
2. The hypervisor cannot over-commit the memory.

The main problem is, this ballooning process is not automatic. it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. In 2013 October KVM forum, Redhat introduced Automatic Ballooning. You can find the presented slides here, and its available with from Kernel 3.10 onwards. I would say, its still experimental and need more feedback.

Adding Aggregater in Openstack

People may have little bit of confusion about Host Aggregates and Availabaility Zones in Nova. Here I am trying to explain how these things are worked together. All configuration options aer based on Icehouse release of Openstack.

A host aggregate is a grouping of hosts with associated metadata. A host can be in more than one host aggregate.  The concept of host aggregates is only exposed to cloud administrators. A host aggregate may be visible to users in the form of an availability zone. When you create a host aggregate, you have the option of providing an availability zone name.

The following will create a Availability zone called Ezone-IND with a Name of enterprise.

[root@controller ~]# nova aggregate-create enterprise Ezone-IND
+----+------------+-------------------+-------+-------------------------------+
| Id | Name       | Availability Zone | Hosts | Metadata                      |
+----+------------+-------------------+-------+-------------------------------+
| 3  | enterprise | Ezone-IND         |       | 'availability_zone=Ezone-IND' |
+----+------------+-------------------+-------+-------------------------------+

 List the available hypervisor list to add under this zone.

[root@controller ~]# nova hypervisor-list
+----+----------------------+
| ID | Hypervisor hostname  |
+----+----------------------+
| 1  | compute              |
| 4  | domain-c15(Cluster1) |
| 5  | compute1             |
+----+----------------------+

Now, we are going to add hypervisor "compute1" under Ezone-IND

[root@controller ~]# nova aggregate-add-host 3 compute1
Host compute1 has been successfully added for aggregate 3
+----+------------+-------------------+------------+-------------------------------+
| Id | Name       | Availability Zone | Hosts      | Metadata                      |
+----+------------+-------------------+------------+-------------------------------+
| 3  | enterprise | Ezone-IND         | 'compute1' | 'availability_zone=Ezone-IND' |
+----+------------+-------------------+------------+-------------------------------+

Now you can see the Host Compute1 is under Ezone-IND.

How to Know Interface is Connected to network - Linux

If you are managing the Linux server remotely, and your Data Center technician informed you that he already connected the Cable to an interface, but in case of Blade servers he cannot identify he connected to which interface (Not only him, nobody). Most of the cases there will be more than 3 interfaces, in this case identify which interface is connected you can use the following command.

[root@compute ~]# ethtool em1
Settings for em1:
    Supported ports: [ TP FIBRE ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 2
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
[root@compute ~]#

Here is the output, and you can see the parameter Link detected: no. Which mean its not connected.

[root@compute ~]# ethtool p3p1
Settings for p3p1:
    Supported ports: [ FIBRE ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseX/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseX/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  1000baseT/Full
                                         10000baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000000 (0)
                  
    Link detected: yes
[root@compute ~]# 



In the second attempt but with another interface you can see the Link detected: yes. Change the interfaces names accordingly. In my case those are em1 and p3p1