Can not able to instantiate VM in compute node in multinode setup

[parulagrawal14]

Hi,

In multinode set up with controller and compute node, we are not able to instantiate VM in Compute node. VM instantiation is successful in controller node. Kindly help us in resolving this issue.

[raviprasad239]

what error you are getting ?

[parulagrawal14]

Hi Ravi,

My controller node is up and I could able to instantiate VM. But when i attach a compute node and try to instantiate a VM, error comes in dashboard as "Unable to instantiate VM Error [Unknown]".

I have used master branch for both compute and controller node.

[raviprasad239]

can you show local.conf file content for controller and compute node ?

[parulagrawal14]

==================================================================
Contents of local_compute.conf

[[local|localrc]]

ADMIN_PASSWORD=nomoresecret
DATABASE_PASSWORD=stackdb
RABBIT_PASSWORD=stackqueue
SERVICE_PASSWORD=$ADMIN_PASSWORD

MULTI_HOST=1
CONTRAIL_BRANCH=R3.0
HOST_IP=10.1.0.228
SERVICE_HOST=10.1.0.226
CONFIG_IP=10.1.0.226
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
KEYSTONE_AUTH_HOST=$SERVICE_HOST
KEYSTONE_SERVICE_HOST=$SERVICE_HOST

ENABLED_SERVICES=n-cpu,n-net,n-api-meta,c-vol
disable_service api-srv disco svc-mon schema control collector analytic-api query-engine dns named ui-jobs ui-webs
enable_service vrouter

NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5

SWIFT_REPLICAS=1

SWIFT_DATA_DIR=$DEST/data
enable_plugin contrail https://github.com/zioc/contrail-devstack-plugin.git

==================================================================
Contents of local_controller.conf

[[local|localrc]]

ADMIN_PASSWORD=nomoresecret
DATABASE_PASSWORD=stackdb
RABBIT_PASSWORD=stackqueue
SERVICE_PASSWORD=$ADMIN_PASSWORD

HOST_IP=10.1.0.226
SERVICE_HOST=10.1.0.226

MULTI_HOST=1
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2

NOVNC_BRANCH=v0.6.0

SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data
enable_plugin contrail https://github.com/zioc/contrail-devstack-plugin.git

[raviprasad239]

can u please re-try with following changes in (lib/neutron-legacy):

function is_neutron_enabled {
return 0
[[ ,${ENABLED_SERVICES} =~ ,"q-" ]] && return 0
return 1
}

[parulagrawal14]

Hi Ravi,

I tried with the above changes but vrouter is not coming up. Below error is thrown

2017-01-19 Thu 00:07:27:397.289 CST compute [Thread 139897626900224, Pid 24550]: Current receive sock buffer size is 262144
2017-01-19 Thu 00:07:27:397.441 CST compute [Thread 139897622701824, Pid 24550]: KsyncTxQueue CPU pinning policy <>. KsyncTxQueuen not pinned to CPU
2017-01-19 Thu 00:07:27:397.453 CST compute [Thread 139897626900224, Pid 24550]: Vrouter family is 26
contrail-vrouter-agent: controller/src/vnsw/agent/vrouter/ksync/ksync_flow_memory.cc:145: void KSyncFlowMemory::InitFlowMem(): Assertion `flow_table_size_ != 0' failed.
vrouter failed to start

[parulagrawal14]

Hi Ravi,

After changing lib/neutron-legacy file I just did ./unstack.sh and ./stack.sh. Do i need to do ./clean.sh also

[raviprasad239]

I have same configuration(local.conf) as yours and it works well for me. you can try with "clean.sh".
Anyways which kernel vs. you are using ?

[parulagrawal14]

Kernel version used:

Linux compute 3.13.0-45-generic #74-Ubuntu SMP Tue Jan 13 19:36:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[parulagrawal14]

Hi Ravi,

After reboot of machine I could able to start vrouter. But instance creation is failing

failed network setup after 1 attempt(s)^[[00m
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00mTraceback (most recent call last):
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1570, in _allocate_network_async
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m bind_host_id=bind_host_id)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/opt/stack/nova/nova/network/api.py", line 49, in wrapped
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m return func(self, context, *args, **kwargs)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/opt/stack/nova/nova/network/base_api.py", line 77, in wrapper
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m res = f(self, context, *args, **kwargs)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/opt/stack/nova/nova/network/api.py", line 283, in allocate_for_instance
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/opt/stack/nova/nova/network/rpcapi.py", line 163, in allocate_for_instance
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m macs=jsonutils.to_primitive(macs))
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m retry=self.retry)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m timeout=timeout, retry=retry)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m result = self._waiter.wait(msg_id, timeout)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m message = self.waiters.get(msg_id, timeout=timeout)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m 'to message ID %s' % msg_id)
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00mMessagingTimeout: Timed out waiting for a reply to message ID 36ac31ba1dd348f7a9469e7f57584244
^[[01;31m2017-01-19 01:58:05.640 TRACE nova.compute.manager ^[[01;35m^[[00m
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in call
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/utils.py", line 1159, in context_wrapper
return func(*args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 1587, in _allocate_network_async
six.reraise(*exc_info)
File "/opt/stack/nova/nova/compute/manager.py", line 1570, in _allocate_network_async
bind_host_id=bind_host_id)
File "/opt/stack/nova/nova/network/api.py", line 49, in wrapped
return func(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/network/base_api.py", line 77, in wrapper
res = f(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/network/api.py", line 283, in allocate_for_instance
nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
File "/opt/stack/nova/nova/network/rpcapi.py", line 163, in allocate_for_instance
macs=jsonutils.to_primitive(macs))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
retry=self.retry)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
timeout=timeout, retry=retry)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
timeout=timeout, retry=retry)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
retry=retry)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
result = self._waiter.wait(msg_id, timeout)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
message = self.waiters.get(msg_id, timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
'to message ID %s' % msg_id)
MessagingTimeout: Timed out waiting for a reply to message ID 36ac31ba1dd348f7a9469e7f57584244

I am using master branch for both compute and controller node and in compute node only vrouter is started.

[raviprasad239]

But from your local.conf looks like you are using R3.0 contrail branch in compute(however i have same in my compute node and it works). And I hope you do not have n/w issue (controller and compute are reachable from each other). You can check log of controller and find out if they are able to communicate. Also you can check contrail GUI for status of vRouters. If default management port in your compute node is not 'eth0' then you can use "VHOST_INTERFACE_NAME=em1" to change it (Generally ubuntu will have interface name like em1,em2 etc.)

[parulagrawal14]

I have commented that line "CONTRAIL_BRANCH=R3.0" in conf file of compute node. So i am using master branch for controller and compute. Recently few check-ins has happened because of which compilation issue was coming with release 3.0.

IN master branch also do i need to change lib/neutron-legacy file changes.

[raviprasad239]

I had used lib/neutron-legacy file changes in R3.0 branch. I have not tried this with master branch.

[parulagrawal14]

Anyone has tried multinode setup with master branch in both compute and controller node.

[ethuleau]

I tried but not recently