CommunicationError: Error communicating with http://XX.XX.XX.XXX:9292 [Errno 111] ECONNREFUSED

Some times, when you trying to deploy a machine, you may ended up with the following error. In my case, the infrastructure is as follows.

One Controller Node, Two compute Nodes, One cinder and One Neutron node. All are Centos 6.6 with Icehouse Release.

2013-10-30 12:34:03.574 15487 TRACE nova.api.openstack CommunicationError: Error communicating with http://192.168.72.100:9292 [Errno 111] ECONNREFUSED

Even your glance_host configuration parameter in the /etc/cinder/cinder.conf is right, you will face this issue.  To resolve this issue, add the following in the /etc/nova/nova.conf file of Controller node.

image_service=nova.image.glance.GlanceImageService

glance_api_servers=192.168.72.145:9292  #IPis the Glance Host

Restart the Nova scheduler and API services in Controller node.

You are good to Go !!!

VirtualInterfaceCreateException: Virtual Interface creation failed : Openstack Compute Error

Some times, you may face this issue in your KVM hypervisor.  This is because libvirt wait for neutron to confirm plugging before boot. This makes the libvirt driver use the instance event mechanism to wait for neutron to confirm that VIF plugging is complete before  actually starting the VM. Which will put the vm status to error and in the compute.log you can see the following error.

TRACE nova.compute.manager [instance: 2a9464d0-0fed-423d-b855-a56f7a5f85a9] VirtualInterfaceCreateException: Virtual Interface creation failed

Inorder to prevent this, you have to add the following parametres in the nova.conf of the compute node.

vif_plugging_is_fatal: false
vif_plugging_timeout: 0

And restart the compute Node.

/etc/init.d/openstack-nova-compute restart

The error will disappear.

Tested with Openstack Icehouse, KVM having Kernel 3.18 and a separate node for Neutron.

Over-Committing And Ballooning in KVM

Over-Committing the resource is a vital part in Virtualization. Idea of over-committment is to allocate more memory to guest machines than the host really has. Suppose a 2GB host is running two guest operating system who is having 2GB memory, in this case we can say its over committed.

In Ballooning process, Host have a right to take the free memory from Guest machines if its there. Consider the above said example, where suppose one virtual machine is using 1GB and other 1GB is free, host can ask the guest to return the free memory. Guest will give it back and host can allocate somewhere else. And if the reduced guest wants to run an application that consumes more than the 1G it currently has, it has to grow again. 

virtio_balloon - This is the driver who open a communication pathway between Host and the Guest virtual machines. Hypervisor (Host) uses this communication pathway to over-committ the resource from Guest machines. If you are not loaded the virtio_balloon driver, you are not able to do the following.

1. The guest operating system ignores memory requests from the hypervisor.
2. The hypervisor cannot over-commit the memory.

The main problem is, this ballooning process is not automatic. it's entirely manual. Someone (or some tool) is supposed to be watching the pressure in the host and guest and then operate the balloon accordingly. In 2013 October KVM forum, Redhat introduced Automatic Ballooning. You can find the presented slides here, and its available with from Kernel 3.10 onwards. I would say, its still experimental and need more feedback.

Adding Aggregater in Openstack

People may have little bit of confusion about Host Aggregates and Availabaility Zones in Nova. Here I am trying to explain how these things are worked together. All configuration options aer based on Icehouse release of Openstack.

A host aggregate is a grouping of hosts with associated metadata. A host can be in more than one host aggregate.  The concept of host aggregates is only exposed to cloud administrators. A host aggregate may be visible to users in the form of an availability zone. When you create a host aggregate, you have the option of providing an availability zone name.

The following will create a Availability zone called Ezone-IND with a Name of enterprise.

[root@controller ~]# nova aggregate-create enterprise Ezone-IND
+----+------------+-------------------+-------+-------------------------------+
| Id | Name       | Availability Zone | Hosts | Metadata                      |
+----+------------+-------------------+-------+-------------------------------+
| 3  | enterprise | Ezone-IND         |       | 'availability_zone=Ezone-IND' |
+----+------------+-------------------+-------+-------------------------------+

 List the available hypervisor list to add under this zone.

[root@controller ~]# nova hypervisor-list
+----+----------------------+
| ID | Hypervisor hostname  |
+----+----------------------+
| 1  | compute              |
| 4  | domain-c15(Cluster1) |
| 5  | compute1             |
+----+----------------------+

Now, we are going to add hypervisor "compute1" under Ezone-IND

[root@controller ~]# nova aggregate-add-host 3 compute1
Host compute1 has been successfully added for aggregate 3
+----+------------+-------------------+------------+-------------------------------+
| Id | Name       | Availability Zone | Hosts      | Metadata                      |
+----+------------+-------------------+------------+-------------------------------+
| 3  | enterprise | Ezone-IND         | 'compute1' | 'availability_zone=Ezone-IND' |
+----+------------+-------------------+------------+-------------------------------+

Now you can see the Host Compute1 is under Ezone-IND.

How to Know Interface is Connected to network - Linux

If you are managing the Linux server remotely, and your Data Center technician informed you that he already connected the Cable to an interface, but in case of Blade servers he cannot identify he connected to which interface (Not only him, nobody). Most of the cases there will be more than 3 interfaces, in this case identify which interface is connected you can use the following command.

[root@compute ~]# ethtool em1
Settings for em1:
    Supported ports: [ TP FIBRE ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 2
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
[root@compute ~]#

Here is the output, and you can see the parameter Link detected: no. Which mean its not connected.

[root@compute ~]# ethtool p3p1
Settings for p3p1:
    Supported ports: [ FIBRE ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseX/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseX/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  1000baseT/Full
                                         10000baseT/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000000 (0)
                  
    Link detected: yes
[root@compute ~]# 



In the second attempt but with another interface you can see the Link detected: yes. Change the interfaces names accordingly. In my case those are em1 and p3p1