xen and libvirt daemons not started

Hello,

in Opensuse 12.2 I had a xen VM that I used for testing purpose. I started it without problem from YaST.
I’m now on Opensuse 12.3 and I try to restart the VM. But seems that a lot of things has changed and I cannot start teh xen and libvirt daemons
My system and xen rpm installed

Uname -a
Linux hpprol.pce23.net 3.11.0-27.g0a1c41f-xen #1 SMP Thu Sep 5 10:20:59 UTC 2013 (0a1c41f) x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa xen*
xen-tools-4.3.0_10-7.1.x86_64
xen-doc-pdf-4.2.2_06-1.16.1.x86_64
xen-xend-tools-4.3.0_10-7.1.x86_64
xen-4.3.0_10-7.1.x86_64

The network bridge is present

 ifconfig -a
br0       Link encap:Ethernet  HWaddr 00:21:5A:4D:FE:50  
          inet addr:192.168.1.130  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1827321 errors:0 dropped:0 overruns:0 frame:0
          TX packets:479423 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:211494387 (201.6 Mb)  TX bytes:3405825139 (3248.0 Mb)
eth0      Link encap:Ethernet  HWaddr 00:21:5A:4D:FE:50  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1830541 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3041939 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:275606072 (262.8 Mb)  TX bytes:3557489465 (3392.6 Mb)
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1155323 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1155323 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9127609739 (8704.7 Mb)  TX bytes:9127609739 (8704.7 Mb)
vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5935 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19286 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:523004 (510.7 Kb)  TX bytes:5619195 (5.3 Mb)

When I start the XEN daemon in a console there is no output but the xend start command seems present two times

hpprol:/etc/xen/scripts # /usr/sbin/xend start 
hpprol:/etc/xen/scripts # ps -ef | grep xen
root        37     2  0 Sep16 ?        00:00:00 [xenwatch]
root        38     2  0 Sep16 ?        00:00:00 [xenbus]
root       819     2  0 Sep16 ?        00:00:00 [xen_pciback_wor]
root       980     1  0 Sep16 ?        00:00:03 /usr/sbin/xenstored --pid-file /var/run/xenstored.pid
root      1078     1  0 Sep16 ?        00:00:00 /usr/sbin/xenconsoled --pid-file=/var/run/xenconsoled.pid
root      1301     1  0 Sep16 ?        00:00:05 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize -monitor /dev/null -serial /dev/null -parallel /dev/null -pidfile /var/run/qemu-dom0.pid
root     21698     1  0 Sep17 ?        00:00:04 /usr/lib/xen/bin/qemu-dm -d 1 -serial pty -domain-name Samba -videoram 4 -k fr -vnc 127.0.0.1:0 -vncunused -M xenpv
root     23903     1  0 Sep17 ?        00:00:00 /usr/sbin/xenwatchdogd 30 15
root     25133     1  0 16:24 ?        00:00:00 /usr/bin/python /usr/sbin/xend start
root     25134 25133  8 16:24 ?        00:00:00 /usr/bin/python /usr/sbin/xend start

if I execute /usr/sbin/xend stop the start processes are removed
in the log I have

Sep 18 16:45:41 hpprol.pce23.net systemd[1]: Starting Xend - Starts and stops the Xen management daemon...
Sep 18 16:45:41 hpprol.pce23.net xend[26915]: usage: /usr/sbin/xend {start|stop|reload|restart}
Sep 18 16:45:41 hpprol.pce23.net systemd[1]: xend.service: control process exited, code=exited status=1
Sep 18 16:45:41 hpprol.pce23.net systemd[1]: Failed to start Xend - Starts and stops the Xen management daemon.
Sep 18 16:45:41 hpprol.pce23.net systemd[1]: Unit xend.service entered failed state

the libvrtd daemon also gives an error

Sep 18 17:04:51 hpprol.pce23.net systemd[1]: Started Virtualization daemon.
Sep 18 17:04:52 hpprol.pce23.net libvirtd[28634]: libvirt version: 1.1.2
Sep 18 17:04:52 hpprol.pce23.net libvirtd[28634]: invalid argument: Failed to parse user 'qemu'
Sep 18 17:04:52 hpprol.pce23.net libvirtd[28634]: Initialization of QEMU state driver failed: invalid argument: Failed to parse user 'qemu'
Sep 18 17:04:52 hpprol.pce23.net libvirtd[28634]: Driver state initialization failed

Seems that an user “qemu” must be defined or that an user must be set in some configuration file

If I try starting the xend via Yast system service I receive an error message “/bin/systemctl start xend.service returned 1 (Error not specified):”

Trying to configure a VM within YaST generate the XML data in /etc/xen/vm but cannot start the VM probably because the xen daemon is not running

cat Samba.xml
<domain type='xen'>
  <name>Samba</name>
  <description>samba serveur</description>
  <uuid>e2f217ba-138c-bda6-d1c1-3c51cb24c1ef</uuid>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <vcpu>1</vcpu>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <clock sync='utc'/>
  <keymap>fr</keymap>
  <bootloader>/usr/bin/pygrub</bootloader>
  <bootloader_args>--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen</bootloader_args>
  <os type='opensuse12'>
    <type>linux</type>
    <cmdline> </cmdline>
  </os>
  <devices>
        <disk type='block' device='disk'>
        <driver name='phy'/>
        <source dev='/dev/cciss/c0d5'/>
        <target dev='xvda'/>
      </disk>
        <disk type='block' device='cdrom'>
        <driver name='phy'/>
        <source dev='/dev/sr0'/>
        <target dev='xvdb'/>
        <readonly/>
      </disk>
        <interface type='bridge'>
          <source bridge='br0'/>
          <mac address='00:16:3e:33:e8:b7'/>
          <script path='/etc/xen/scripts/vif-bridge'/>
        </interface>
    
      <graphics type='vnc'/>
  </devices>
</domain>

Any idea what I can do to start the xen and libvirtd daemons
Many thanks in advance
Philippe

You might take a look at some other posts which have described troubleshooting commands specific to 12.3 (in particular the expanded implementation of systemd).

Thumbnail essential background on systemd: systemd Units replace a variety of various objects including services, functions, initializations, configurations and more. By implementing as Units, the theory is that you now have a single way of reading and analyzing every object instead of having to learn where/how objects exist and unique ways of working with each. Oftentimes if an init script exists (eg in 12.2) then a Unit may simply invoke that init script if it still exists.

eg.
The following will search for systemd Units with “name” in the name. the “-a” switch will search all units no matter their state or configuration.

systemctl -a | grep *name*/CODE]

The following displays the Unit status, a few relevant lines from the system log and more. If you're reading the proper service Units, your error will likely be described in moderate detail here.

systemctl status unitname



Although you can still read /var/log/messages directly with tail, head or whatever you wish, you now also have a new tool "journalctl" which strives to incorporate log data from multiple sources. View the help and man pages for use

journalctl --help



After running those commands, if you're still not sure what is happening, post those commands you ran with their results (in particular the "status" commands) and people will be able to help analyze.

HTH,
TSU

Thanks for your answer,

Here the result for libvirt daemon

hpprol:/var/log # /bin/systemctl start libvirtd.service 
hpprol:/var/log # /bin/systemctl status libvirtd.service 
libvirtd.service - Virtualization daemon
          Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
          Active: inactive (dead) since Thu, 2013-09-19 20:34:38 CEST; 9s ago
         Process: 11005 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/libvirtd.service

Sep 19 20:34:37 hpprol.pce23.net systemd[1]: Starting Virtualization daemon...
Sep 19 20:34:37 hpprol.pce23.net systemd[1]: Started Virtualization daemon.
Sep 19 20:34:38 hpprol.pce23.net libvirtd[11005]: libvirt version: 1.1.2
Sep 19 20:34:38 hpprol.pce23.net libvirtd[11005]: invalid argument: Failed to parse user 'qemu'
Sep 19 20:34:38 hpprol.pce23.net libvirtd[11005]: Initialization of QEMU state driver failed: invalid argument: Failed to parse user 'qemu'
Sep 19 20:34:38 hpprol.pce23.net libvirtd[11005]: Driver state initialization failed

I created an user qemu (user system in group libvirt, bin), reload the commands and received a slighty differend error

Sep 19 20:51:07 hpprol.pce23.net libvirtd[11583]: invalid argument: Failed to parse group 'qemu'

So I created the group qemu and set this group as primary group of user qemu, then restarted the commands

hpprol:/etc # /bin/systemctl start libvirtd.service 
hpprol:/etc # /bin/systemctl status libvirtd.service 
libvirtd.service - Virtualization daemon
          Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
          Active: active (running) since Thu, 2013-09-19 20:55:16 CEST; 2s ago
        Main PID: 11739 (libvirtd)
          CGroup: name=systemd:/system/libvirtd.service
                  └ 11739 /usr/sbin/libvirtd --listen

Sep 19 20:55:16 hpprol.pce23.net systemd[1]: Starting Virtualization daemon...
Sep 19 20:55:16 hpprol.pce23.net systemd[1]: Started Virtualization daemon.
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: libvirt version: 1.1.2
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: Configured security driver "none" disables default policy to create confined guests
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: Failed to query host NUMA topology, disabling NUMA capabilities
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: Failed to get host CPU
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: Failed to query host NUMA topology, disabling NUMA capabilities
Sep 19 20:55:17 hpprol.pce23.net libvirtd[11739]: Failed to query host NUMA topology, disabling NUMA capabilities
hpprol:/var/log # ps -ef | grep libvirt
root     11739     1  0 20:55 ?        00:00:00 /usr/sbin/libvirtd --listen

Now libvirt is started :slight_smile: . I loaded then the xen daemon but there no success.
result for xen darmon

hpprol:/var/log # /bin/systemctl start xend.service 
Job for xend.service failed. See 'systemctl status xend.service' and 'journalctl -n' for details.
hpprol:/var/log # /bin/systemctl status xend.service 
xend.service - Xend - Starts and stops the Xen management daemon
          Loaded: loaded (/usr/lib/systemd/system/xend.service; disabled)
          Active: failed (Result: exit-code) since Thu, 2013-09-19 20:45:00 CEST; 20s ago
         Process: 11452 ExecStart=/usr/sbin/xend (code=exited, status=1/FAILURE)
         Process: 11450 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/xend.service

Sep 19 20:45:00 hpprol.pce23.net xend[11452]: usage: /usr/sbin/xend {start|stop|reload|restart}
Sep 19 20:45:00 hpprol.pce23.net systemd[1]: Failed to start Xend - Starts and stops the Xen management daemon.
Sep 19 20:45:00 hpprol.pce23.net systemd[1]: Unit xend.service entered failed state

The Xen daemon gives an error but the description is not meaningfull for me.

In virt_manager I can now create and open a connection
Trying to create a VM gives these errors

Error polling connection 'xen:///': internal error: client socket is closed

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 294, in _tickconn.tick()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1311, in tickself._tick(noStatsUpdate)
  File "/usr/share/virt-manager/virtManager/connection.py", line 1320, in _tickself.hostinfo = self.vmm.getInfo()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3630, in getInfo if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed

Regards
Philippe

From your error, I suspect your CPU hardware virtualization is either not available or disabled.

You can display all the cpu info including flags with the following, look for vmx if you have an Intel processor or svm if you have an AMD processor

cat /proc/cpuinfo

Or, you can grep for the strings
If you have an Intel processor

  grep --color vmx /proc/cpuinfo

If you have an AMD processor

  grep --color svm /proc/cpuinfo

Also, check to make sure it’s enabled in your BIOS.

If your hardware virtualization checks out, then I suspect that your xen may not have been installed correctly with full virtualization (try re-installing and make sure it’s 64-bit). One way to check whether your xen is being installed correctly anecdotally is to install virtualization using the YAST applet (in virtualization). If the virtualization install applet doesn’t exist, some have reported that it’s likely because 64-bit hardware virtualization isn’t detected.

TSU

Hello,

I checked in the BIOS ==> processor options and the intel virtualization is “Enabled”.
The processor is a intel xeon quad core.

all xen software are X86-64

xen-tools-4.3.0_10-7.1.x86_64
xen-xend-tools-4.3.0_10-7.1.x86_64
xen-4.3.0_10-7.1.x86_64
xen-libs-4.3.0_10-7.1.x86_64

I found this error in the /var/log/xen/xend-debug.log

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/Hald.py", line 55, in shutdown
    os.kill(self.pid, signal.SIGINT)
OSError: [Errno 3] No such process
Xend started at Fri Sep 20 19:24:18 2013.
Exception in thread UnixHttpServer:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/xen/web/httpserver.py", line 312, in run
    (sock, addr) = self.accept()
  File "/usr/lib64/python2.7/site-packages/xen/web/httpserver.py", line 331, in accept
    return self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 22] Invalid argument

Regards
Philippe

Assuming that “hald” references some kind of hardware abstraction layer, I still think something isn’t recognized or installed properly.

Did you run

grep --color vmx /proc/cpuinfo

Also, just to verify which repos you’re pulling from…

zypper lr -d

Just out of curiousity,
Especially if you used a DVD to upgrade to 12.3 instead of an online method like zypper dup, did you update? It’s a critical thing to do immediately after any new install or upgrade (especially when using old packages), you need to get the most recent everything installed. After that, you can decide whether to continue to update (generally my advice) or patch (which will keep your system more stable but still secure, not getting all improvements).

zypper up

TSU

Hello,

here the zypper lr -d

hpprol:/var/log/xen # zypper lr -d
#  | Alias                            | Name                         | Enabled | Refresh | Priority | Type     | URI                                                                                | Service
---+----------------------------------+------------------------------+---------+---------+----------+----------+------------------------------------------------------------------------------------+--------
 1 | Tumbleweed                       | Tumbleweed                   | Yes     | Yes     |   90     | rpm-md   | http://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/           |        
 2 | Updates-for-openSUSE-11.4-11.4-0 | Updates for openSUSE current | Yes     | Yes     |   99     | rpm-md   | http://download.opensuse.org/update/openSUSE-current/                              |        
 3 | download.opensuse.org-games_1    | openSUSE BuildService - Jeux | Yes     | Yes     |   99     | rpm-md   | http://download.opensuse.org/repositories/games/openSUSE_12.3/                     |        
 4 | opensuse-guide.org-repo          | libdvdcss repository         | No      | No      |   99     | rpm-md   | http://opensuse-guide.org/repo/11.4/                                               |        
 5 | packman                          | packman_tumbleweed           | Yes     | Yes     |   89     | rpm-md   | http://packman.inode.at/suse/openSUSE_Tumbleweed                                   |        
 6 | packman.inode.at-suse            | Packman Repository           | Yes     | Yes     |   91     | rpm-md   | http://packman.inode.at/suse/12.3/                                                 |        
 7 | repo-non-oss                     | openSUSE-Curent  Oss         | Yes     | Yes     |   99     | yast2    | http://download.opensuse.org/distribution/openSUSE-current/repo/oss/               |        
 8 | repo-oss                         | openSUSE-Current Non-Oss     | Yes     | Yes     |   99     | yast2    | http://download.opensuse.org/distribution/openSUSE-current/repo/non-oss/           |        
 9 | repo-update-non-oss              | openSUSE-12.3-Update-Non-Oss | Yes     | Yes     |   99     | rpm-md   | http://download.opensuse.org/update/12.3-non-oss/                                  |        
10 | rpm_install                      | rpm_install                  | No      | No      |  100     | plaindir | dir:///local/download64/rpm_install                                                |        
11 | tumbleweed_oo                    | tumbleweed_oo                | Yes     | Yes     |   90     | rpm-md   | http://download.opensuse.org/repositories/LibreOffice:/Stable/openSUSE_Tumbleweed/ | 

zypper up gives “Nothing to do”
Here the cat /proc/cpuinfo

hpprol:/var/log/xen # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz
stepping        : 6
cpu MHz         : 2333.450
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall lm constant_tsc rep_good nopl aperfmperf pni est ssse3 cx16 sse4_1 hypervisor lahf_lm dtherm
bogomips        : 4779.37
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

and same for the 3 other cores.
I don.t see the vmx flag but the Intel specs say

Intel® Virtualization Technology (VT-x) ‡                                          Yes 

see ARK | Intel® Xeon® Processor E5410 (12M Cache, 2.33 GHz, 1333 MHz FSB)
This hardware worked with xen in opensuse 12.0, 12.1 and 12.2
Regards
Philippe

  1. The missing vmx flag is probably the indication something is disabled, probably at the BIOS or kernel level.
  2. I see you have the Tumbleweed repo enabled. run the following against all your xen packages and verify none come from Tumbleweed or if any do, which ones
zypper info *packagename* 
  1. I find it hard to believe possible, but I wonder if your xen kernel might have been compiled incorrectly to support only 32-bit although it says 64-bit.
  2. I’m not up on Intel CPU architectures, but when was this CPU purchased? Although I have not run into any problems on Linux in general, I’ve recently become aware of problems running versions of MS Hyper-V on different versions of Windows Server, and I wonder with the technology exchange between Xen and MS which made the two work more similarly whether some problems could be similar for both(would have to be researched). Am wondering for instance if you install KVM into your non-Xen kernel whether (a) your vmx flag appears as it should (b) whether KVM starts and supports 64-bit guests

You’re running into things which are surprising…

TSU

Hello,

  1. checked all the bios options : only one option in the processor tab controls the intel virtualisation and it is enabled. System is a HP Proliant ML350 G6 with one processor XEON quadcore and 10 GB memory (4 years old)

  2. The kernel and all xen related packages are coming from Tumbleweed

  3. I never compile the kernel or any module

  4. I’ll have a look to KVM. I never tested it.

I see that a new kernel and new xen packages are now available in Tumbleweed: I’ll update it and retest

Regards
Philippe

Just repeating something that has been said often before,
Tumbleweed can toss a few unexpecte wrenches into the mix.
Always test in non-Tumbleweed before enabling Tumbleweed.
Tumbleweed enables “rolling” updates which can be less reliable and stable than the regular repos, so use at your own risk. It’s cool to use the leading edge and everything released through Tumbleweed is deemed stable, but still might not have been sufficiently tested rigorously.

TSU

You have solved or finded the solutions for this. I am facing this one. Cant find a way to solve it.??

If the problem is solved means Give your suggestions to solve this error.

regards,
cooldharma06.

Hello,

no the problem is still present in Opensuse 13.1.
I didn’t find any solution. Without more detail in the log it is difficult to find the cause .

Regards
Philippe

Hello,

with the last xen kernel from tumbleweed (13.1) I can again start the VM
kernel: 3.12.3-2.gf37dca6-xen
xen: xen-4.3.0_14-1.3.x86_64

Other remark: only libvirtd is needed the xen daemon has been deprecated see Migration Guide To Xen4.1+ - Xen

Regards
Philippe