Results 1 to 8 of 8

Thread: how to install correct debuginfo for systemtap

  1. #1

    Default how to install correct debuginfo for systemtap

    I tried to use systemtap but it complains about:

    Code:
    semantic error: missing x86_64 kernel/module debuginfo under '/lib/modules/2.6.34.7-0.7-desktop/build'...
    Fair enough I did not have such file/directory.

    I looked a bit in the web and the question has certainly been asked before. However, I did not find a complete/perfect answer that works for my OpenSUSE 11.3 x86_64 desktop kernel (just installed from the repo, nothing fancy).

    Here's what I did:

    1. enabled the predefined debuginfo repo
    2. installed kernel-desktop-debuginfo
    3. additionally installed kernel-desktop-devel (because the symbolic link /lib/modules/2.6.34.7-0.7-desktop/build was still missing)

    Now the symbolic link exists, but the error message has not changed.

    If I look at the installed versions, it looks a bit weird:

    Code:
    > zypper se -s -i kernel
    Loading repository data...
    Reading installed packages...
    
    S | Name                     | Type    | Version        | Arch   | Repository                                                       
    --+--------------------------+---------+----------------+--------+------------------------------------------------------------------
    i | Kernel                   | patch   | 3038           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3709           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3396           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3315           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3173           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3105           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-desktop           | package | 2.6.34.7-0.7.1 | x86_64 | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-desktop-debuginfo | package | 2.6.34-12.3    | x86_64 | openSUSE-11.3-Debug                                              
    i | kernel-desktop-devel     | package | 2.6.34.7-0.7.1 | x86_64 | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-devel             | package | 2.6.34.7-0.7.1 | noarch | Updates for openSUSE 11.3 11.3-1.82
    So how comes the debuginfo version does not match? Does that cause the problem?
    Do I need another debug repo than the one that is predefined in 11.3? I thought I shouldn't because the kernel comes also from a default location.

  2. #2

    Default Re: how to install correct debuginfo for systemtap

    "So how comes the debuginfo version does not match? Does that cause the problem?"

    Yes, that is the problem. Try finding the exact version match under:
    Index of /debug/update/11.3/rpm/x86_64
    instead of just:
    Index of /debug/distribution/11.3/repo/oss/suse/x86_64
    probably this is the file you need:
    http://download.opensuse.org/debug/u...7.1.x86_64.rpm

  3. #3

    Default Re: how to install correct debuginfo for systemtap

    Quote Originally Posted by fche View Post
    "So how comes the debuginfo version does not match? Does that cause the problem?"

    Yes, that is the problem. Try finding the exact version match under:
    Index of /debug/update/11.3/rpm/x86_64
    Thanks for your reply. I didn't want to install just a single rpm from a random location, because that will no longer work when the next update becomes available.

    But from your link I was able to define a new repo location at Index of /debug/update/11.3 and after updating from there the version of the debuginfo matched to that of the kernel.

    Code:
    zypper se -i -s kernel
    Loading repository data...
    Reading installed packages...
    
    S | Name                     | Type    | Version        | Arch   | Repository                                                       
    --+--------------------------+---------+----------------+--------+------------------------------------------------------------------
    i | Kernel                   | patch   | 3038           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | dbg113-Kernel            | patch   | 3038           | noarch | Update 11.3 debug                                                
    i | dbg113-kernel            | patch   | 3709           | noarch | Update 11.3 debug                                                
    i | dbg113-kernel            | patch   | 3396           | noarch | Update 11.3 debug                                                
    i | dbg113-kernel            | patch   | 3315           | noarch | Update 11.3 debug                                                
    i | dbg113-kernel            | patch   | 3173           | noarch | Update 11.3 debug                                                
    i | dbg113-kernel            | patch   | 3105           | noarch | Update 11.3 debug                                                
    i | kernel                   | patch   | 3709           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3396           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3315           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3173           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel                   | patch   | 3105           | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-desktop           | package | 2.6.34.7-0.7.1 | x86_64 | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-desktop-debuginfo | package | 2.6.34.7-0.7.1 | x86_64 | Update 11.3 debug                                                
    i | kernel-desktop-devel     | package | 2.6.34.7-0.7.1 | x86_64 | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-devel             | package | 2.6.34.7-0.7.1 | noarch | Updates for openSUSE 11.3 11.3-1.82                              
    i | kernel-firmware          | package | 20100617-2.2   | noarch | http://download.opensuse.org/repositories/openSUSE:/11.3/standard
    i | kernel-firmware          | package | 20100617-2.2   | noarch | openSUSE-11.3-Oss                                                
    i | kernel-firmware          | package | 20100617-2.2   | noarch | openSUSE-11.3 11.3-1.82
    Unfortunately this did not make any difference, the error message is unchanged.

    Code:
    semantic error: missing x86_64 kernel/module debuginfo under '/lib/modules/2.6.34.7-0.7-desktop/build' while resolving probe point kernel.function("do_execve").call
    semantic error: no match while resolving probe point syscall.execve
    I tried with strace to find out what missing file is really causing the problem, but I don't understand the whole thing well enough to come to a conclusion.

    Code:
    $ grep ENOENT stap-strace.5987 
    access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
    open("/usr/lib64/elfutils/tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
    stat("/usr/lib64/elfutils/tls/x86_64", 0x7ffff0506af0) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/elfutils/tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
    stat("/usr/lib64/elfutils/tls", 0x7ffff0506af0) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/elfutils/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
    stat("/usr/lib64/elfutils/x86_64", 0x7ffff0506af0) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/elfutils/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib64/elfutils/libbz2.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34.7-0.7-desktop/x86_64", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34.7-0.7-desktop", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34.7/x86_64", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34.7", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34/x86_64", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6.34", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6/x86_64", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/usr/share/systemtap/tapset/2.6", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/debug.a", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/debug.a", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/.debug/debug.a", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/debug/lib/modules/2.6.34.7-0.7-desktop/debug.a", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/build/debug.a", O_RDONLY) = -1 ENOENT (No such file or directory)
    lstat("/lib/modules/2.6.34.7-0.7-desktop/debug.a", 0x7ffff0502890) = -1 ENOENT (No such file or directory)
    open("/boot/vmlinux-2.6.34.7-0.7-desktop", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/vmlinux-2.6.34.7-0.7-desktop", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/.debug/vmlinux-2.6.34.7-0.7-desktop", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/debug/boot/vmlinux-2.6.34.7-0.7-desktop", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/build/vmlinux-2.6.34.7-0.7-desktop", O_RDONLY) = -1 ENOENT (No such file or directory)
    lstat("/boot/vmlinux-2.6.34.7-0.7-desktop", 0x7ffff0502830) = -1 ENOENT (No such file or directory)
    open("/boot/vmlinux-2.6.34.7-0.7-desktop.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/.debug/vmlinux-2.6.34.7-0.7-desktop.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/debug/boot/vmlinux-2.6.34.7-0.7-desktop.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/build/vmlinux-2.6.34.7-0.7-desktop.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    lstat("/boot/vmlinux-2.6.34.7-0.7-desktop", 0x7ffff0502830) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/.debug/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/debug/lib/modules/2.6.34.7-0.7-desktop/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/build/vmlinux", O_RDONLY) = -1 ENOENT (No such file or directory)
    lstat("/lib/modules/2.6.34.7-0.7-desktop/vmlinux", 0x7ffff0502850) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/vmlinux.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/.debug/vmlinux.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/usr/lib/debug/lib/modules/2.6.34.7-0.7-desktop/vmlinux.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib/modules/2.6.34.7-0.7-desktop/build/vmlinux.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
    lstat("/lib/modules/2.6.34.7-0.7-desktop/vmlinux", 0x7ffff0502850) = -1 ENOENT (No such file or directory)
    The error message is written in the very end after many successful file accesses. So that did not reveal anything.

    Code:
    $ tail -30  stap-strace.5987 
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/i2o/i2o_proc.ko", {st_mode=S_IFREG|0644, st_size=68072, ...}) = 0
    open("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptsas.ko", {st_mode=S_IFREG|0644, st_size=81680, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptlan.ko", {st_mode=S_IFREG|0644, st_size=29368, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptspi.ko", {st_mode=S_IFREG|0644, st_size=34872, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptfc.ko", {st_mode=S_IFREG|0644, st_size=33552, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptscsih.ko", {st_mode=S_IFREG|0644, st_size=53080, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptctl.ko", {st_mode=S_IFREG|0644, st_size=52856, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/kernel/drivers/message/fusion/mptbase.ko", {st_mode=S_IFREG|0644, st_size=118832, ...}) = 0
    open("/lib/modules/2.6.34.7-0.7-desktop/vdso", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    stat("/lib/modules/2.6.34.7-0.7-desktop/vdso/vdso32-sysenter.so", {st_mode=S_IFREG|0755, st_size=1796, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/vdso/vdso.so", {st_mode=S_IFREG|0755, st_size=3704, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/vdso/vdso32-syscall.so", {st_mode=S_IFREG|0755, st_size=1788, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/vdso/vdso32-int80.so", {st_mode=S_IFREG|0755, st_size=1776, ...}) = 0
    open("/lib/modules/2.6.34.7-0.7-desktop/weak-updates", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/systemtap", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
    open("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates/vboxdrv.ko", {st_mode=S_IFREG|0644, st_size=1915584, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates/nvidia.ko", {st_mode=S_IFREG|0644, st_size=14699050, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates/vboxnetadp.ko", {st_mode=S_IFREG|0644, st_size=12800, ...}) = 0
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/updates/vboxnetflt.ko", {st_mode=S_IFREG|0644, st_size=32616, ...}) = 0
    open("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/systemtap", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    stat("/lib/modules/2.6.34.7-0.7-desktop/weak-updates/systemtap/preloadtrace.ko", {st_mode=S_IFREG|0644, st_size=125880, ...}) = 0
    write(2, "semantic error: missing x86_64 k"..., 165) = 165
    write(2, "semantic error: no match while r"..., 68) = 68
    So all hints are still welcome. Has anybody managed to use Systemtap under OpenSUSE 11.3? Command "stap" comes installed by default, so I would expect it to be usable in some way.

  4. #4

    Default Re: how to install correct debuginfo for systemtap

    The problem appears to be that the more recent kernel-desktop-debuginfo
    files don't in fact contain the kernel debuginfo file (/usr/lib/debug/boot/vmlinux*).

    Please open a bug against the kernel, as this is a regression against e.g.
    2.6.34-12-desktop.

    In the mean time, some debuginfo-independent functionality is available,
    such as kernel.trace("...") and kprobe.function("...") probes.

  5. #5

    Default Re: how to install correct debuginfo for systemtap

    Quote Originally Posted by fche View Post
    The problem appears to be that the more recent kernel-desktop-debuginfo
    files don't in fact contain the kernel debuginfo file (/usr/lib/debug/boot/vmlinux*).
    Another problem is that /boot/vmlinux-FOO is compressed (/boot/vmlinux-FOO.gz),
    and current systemtap/elfutils don't check for that in this context. (The accept that
    for modules only.) So even if the kernel-*-debuginfo RPM build was fixed, one'd
    need to gunzip /boot/vmlinux* for now.

  6. #6

    Default Re: how to install correct debuginfo for systemtap

    Quote Originally Posted by fche View Post
    Please open a bug against the kernel, as this is a regression against e.g.
    2.6.34-12-desktop.
    Done. https://bugzilla.novell.com/show_bug.cgi?id=671559

    Quote Originally Posted by fche View Post
    Another problem is that /boot/vmlinux-FOO is compressed (/boot/vmlinux-FOO.gz),
    and current systemtap/elfutils don't check for that in this context. (The accept that
    for modules only.) So even if the kernel-*-debuginfo RPM build was fixed, one'd
    need to gunzip /boot/vmlinux* for now.
    gunzipped. Yes, now the error message changes to


    Code:
    parsed 'do_execve', func 'do_execve'
    pattern 'kernel' matches module 'kernel'
    focused on module 'kernel = [0xffffffff81000000-0xffffffff81db6000, bias 0x0] file /boot/vmlinux-2.6.34.7-0.7-desktop ELF machine |x86_64 (code 62)
    focused on module 'kernel'
    WARNING: cannot find module kernel debuginfo: No DWARF information found
    semantic error: no match while resolving probe point kernel.function("do_execve").call
    semantic error: no match while resolving probe point syscall.execve
    Which looks like an expected result. Before using vmlinuz it couldn't even find out where execve is in the kernel. Now with vmlinux available it can do that, but it would still need debug info to proceed further. But no vmlinux*.debug (whatever the exact filename would be) is installed anywhere.

    Code:
    > rpm -ql kernel-desktop-debuginfo | grep -c vmli
    0

  7. #7

    Default Re: how to install correct debuginfo for systemtap

    Quote Originally Posted by fche View Post
    In the mean time, some debuginfo-independent functionality is available,
    such as kernel.trace("...") and kprobe.function("...") probes.
    Yes, these probes work OK.

  8. #8

    Default Re: how to install correct debuginfo for systemtap

    Quote Originally Posted by geuder View Post
    gunzipped. Yes, now the error message changes to


    Code:
    parsed 'do_execve', func 'do_execve'
    pattern 'kernel' matches module 'kernel'
    focused on module 'kernel = [0xffffffff81000000-0xffffffff81db6000, bias 0x0] file /boot/vmlinux-2.6.34.7-0.7-desktop ELF machine |x86_64 (code 62)
    focused on module 'kernel'
    WARNING: cannot find module kernel debuginfo: No DWARF information found
    semantic error: no match while resolving probe point kernel.function("do_execve").call
    semantic error: no match while resolving probe point syscall.execve
    I believe you would need the appropriate kernel-*-devel-debuginfo package:

    Code:
    > rpm -qf /usr/lib/debug/boot/vmlinux-3.4.11-2.16-desktop.debug
    kernel-desktop-devel-debuginfo-3.4.11-2.16.1.x86_64

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •