Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: SystemTap fails to find kernel tracepoints

  1. #1

    Question SystemTap fails to find kernel tracepoints

    I'm trying to learn how to use SystemTap, but I currently fail at even running the examples provided by the systemtap-docs package:

    Code:
    # stap /usr/share/doc/packages/systemtap/examples/process/schedtimes.stp
    semantic error: while resolving probe point: identifier 'kernel' at /usr/share/doc/packages/systemtap/examples/process/schedtimes.stp:34:7
            source: probe kernel.trace("sched_switch") {
                          ^
    
    semantic error: no match
    semantic error: while resolving probe point: identifier 'kernel' at :105:7
            source: probe kernel.trace("sched_wakeup") {
                          ^
    
    WARNING: cannot find module /tmp/stapmft4Pc/typequery_kmod_1/typequery_kmod_1.ko debuginfo: Unsupported relocation type
    WARNING: cannot find module /root/.systemtap/cache/99/typequery_99c4493878436ab6976588e4e37d9622_708.ko debuginfo: Unsupported relocation type
    Pass 2: analysis failed.  Try again with another '--vp 01' option.
    Indeed, stap seems to indicate that there are no kernel tracepoints defined on my system:

    Code:
    # stap -L 'kernel.trace("*")'
    returns nothing.

    Generally, stap seems to be installed correctly on my openSUSE 12.2, as a simple check script runs as expected:

    Code:
    # stap -e 'probe vfs.read { printf("vfs.read\n"); exit() }'
    WARNING: cannot find module nfs debuginfo: Unsupported relocation type
    WARNING: cannot find module sunrpc debuginfo: Unsupported relocation type
    vfs.read
    I can see the trace events in debugfs:

    Code:
    # ls -l /sys/kernel/debug/tracing/events/sched/sched_wakeup
    total 0
    -rw-r--r-- 1 root root 0 Jan 23 14:59 enable
    -rw-r--r-- 1 root root 0 Jan 23 14:59 filter
    -r--r--r-- 1 root root 0 Jan 23 14:59 format
    -r--r--r-- 1 root root 0 Jan 23 14:59 id
    My Google-fu fails me. Does anyone here have an idea what I could do to successfully use kernel tracepoints in SystemTap?

    Thanks in advance!

  2. #2
    dd NNTP User

    Default Re: SystemTap fails to find kernel tracepoints

    On 01/23/2013 03:16 PM, braunseb wrote:
    > Does anyone here have an idea what I could do to
    > successfully use kernel tracepoints in SystemTap?


    what version of systemtap?
    from where did you fetch it and, how did you install it?

    did you also install (for example) kernel-debuginfo,
    kernel-debuginfo-common, and kernel-devel which exactly matches the
    running kernel ?

    --
    dd
    openSUSE®, the "German Engineered Automobile" of operating systems!


  3. #3

    Default Re: SystemTap fails to find kernel tracepoints

    Hi, and thanks for answering!

    Quote Originally Posted by dd View Post
    what version of systemtap?
    from where did you fetch it and, how did you install it?
    I'm running 2.0.59, which I got from OBS (devel:tools repository) and installed via zypper.

    Quote Originally Posted by dd View Post
    did you also install (for example) kernel-debuginfo,
    kernel-debuginfo-common, and kernel-devel which exactly matches the
    running kernel ?
    I installed everything I could think of:
    Code:
    > sudo zypper se -i -s kernel systemtap
    Loading repository data...
    Reading installed packages...
    
    S | Name                           | Type    | Version           | Arch   | Repository                
    --+--------------------------------+---------+-------------------+--------+---------------------------
    i | kernel-debug                   | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-debug-devel             | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-default-debuginfo       | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-default-devel           | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-desktop                 | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-desktop-debuginfo       | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-desktop-debugsource     | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-desktop-devel           | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-desktop-devel-debuginfo | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-devel                   | package | 3.4.11-2.16.1     | noarch | openSUSE-12.2-Update      
    i | kernel-firmware                | package | 20120719git-2.9.1 | noarch | openSUSE-12.2-Update      
    i | kernel-source                  | package | 3.4.11-2.16.1     | noarch | openSUSE-12.2-Update      
    i | kernel-syms                    | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-trace                   | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-trace-debuginfo         | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-trace-debugsource       | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-trace-devel             | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | kernel-trace-devel-debuginfo   | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update-Debug
    i | kernel-xen-devel               | package | 3.4.11-2.16.1     | x86_64 | openSUSE-12.2-Update      
    i | systemtap                      | package | 2.0-59.2          | x86_64 | devel:tools               
    i | systemtap-docs                 | package | 2.0-60.4          | noarch | devel:tools               
    i | systemtap-runtime              | package | 2.0-59.2          | x86_64 | devel:tools               
    i | systemtap-sdt-devel            | package | 2.0-59.2          | x86_64 | devel:tools               
    i | systemtap-server               | package | 2.0-59.2          | x86_64 | devel:tools
    The versions should match, too:
    Code:
    > uname -r
    3.4.11-2.16-trace
    software.opensuse.org does not know about kernel-debuginfo-common, and I can't find it with zypper se. What else am I missing?
    Last edited by braunseb; 23-Jan-2013 at 08:27. Reason: formatting

  4. #4
    dd NNTP User

    Default Re: SystemTap fails to find kernel tracepoints

    On 01/23/2013 04:26 PM, braunseb wrote:
    >
    > I'm running 2.0.59, which I got from OBS (devel:tools repository) and
    > installed via zypper.


    do as you wish, but i believe if you were to install the version
    available for 12.2 it might run okay...

    http://software.opensuse.org/package/systemtap shows that to be 1.7, i'd
    _guess_ (i don't have a 12.2 to try) it is in oss or non-oss and should
    'just work' if installed with YaST Software Management or "zypper in"
    from the standard repo (OBS is a different kettle of fish)

    of course, i'd begin by uninstalling the version from OBS.


    > software.opensuse.org does not know about kernel-debuginfo-common, and
    > I can't find it with zypper se. What else am I missing?


    hmmmmmm....i don't know, i googled to
    [/url]http://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup[/url]
    and found some notes that are probably far too specific to Fedora (that
    happens sometimes, sorry)..

    oh! try picking up missing specific hints here (and note the use of
    google's "site specifier" to force the search inside the openSUSE
    universe, which i should'a done the first time):
    https://www.google.com/search?q=site...ling+systemtap
    https://www.google.com/search?q=site....org+systemtap
    https://www.google.com/search?q=systemtap+suse


    --
    dd
    openSUSE®, the "German Engineered Automobile" of operating systems!
    http://goo.gl/PUjnL

  5. #5

    Default Re: SystemTap fails to find kernel tracepoints

    Please run stap-report, and stap -vvv [other options], and send their outputs to systemtap@sourceware.org.

  6. #6

    Default Re: SystemTap fails to find kernel tracepoints

    Quote Originally Posted by dd View Post
    On 01/23/2013 04:26 PM, braunseb wrote:
    >
    > I'm running 2.0.59, which I got from OBS (devel:tools repository) and
    > installed via zypper.


    do as you wish, but i believe if you were to install the version
    available for 12.2 it might run okay...

    software.opensuse.org: shows that to be 1.7, i'd
    _guess_ (i don't have a 12.2 to try) it is in oss or non-oss and should
    'just work' if installed with YaST Software Management or "zypper in"
    from the standard repo (OBS is a different kettle of fish)

    of course, i'd begin by uninstalling the version from OBS.
    Just tried downgrading to 1.7, and confirmed that it generally works:
    Code:
    > zypper se -s -i systemtap
    Loading repository data...
    Reading installed packages...
    
    S | Name              | Type    | Version   | Arch   | Repository          
    --+-------------------+---------+-----------+--------+---------------------
    i | systemtap         | package | 1.7-3.4.1 | x86_64 | openSUSE-12.2-Update
    i | systemtap-runtime | package | 1.7-3.4.1 | x86_64 | openSUSE-12.2-Update
    
    > stap --version
    Systemtap translator/driver (version 1.7/0.153 non-git sources)
    Copyright (C) 2005-2012 Red Hat, Inc. and others
    This is free software; see the source for copying conditions.
    enabled features: LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
    
    > sudo stap -e 'probe vfs.read { printf("Works.\n"); exit() }'
    WARNING: cannot find module nfs debuginfo: Unsupported relocation type
    WARNING: cannot find module sunrpc debuginfo: Unsupported relocation type
    Works.
    But still no luck with tracepoints:

    Code:
    > sudo stap -L 'kernel.trace("*")'
    >

    oh! try picking up missing specific hints here (and note the use of
    google's "site specifier" to force the search inside the openSUSE
    universe, which i should'a done the first time):
    https://www.google.com/search?q=site...ling+systemtap
    https://www.google.com/search?q=site....org+systemtap
    https://www.google.com/search?q=systemtap+suse
    I actually tried that before posting and found a lot of examples where someone was able to get systemtap to run for other probes or the canonical vfs.read example.
    And indeed systemtap works for a bunch of other probes I tried. I also tried https://www.google.com/search?q=site...ernel.trace%22, but that did not yield any information pertaining to my problem.

    When I try running stap -vvvv, it provides a hint:
    Code:
    > sudo stap -vvvv -L 'kernel.trace("*")'
    Systemtap translator/driver (version 1.7/0.153 non-git sources)
    Copyright (C) 2005-2012 Red Hat, Inc. and others
    This is free software; see the source for copying conditions.
    enabled features: LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
    Created temporary directory "/tmp/stap9cJQKh"
    Session arch: x86_64 release: 3.4.11-2.16-trace
    Parsed kernel "/lib/modules/3.4.11-2.16-trace/build/.config", containing 4484 tuples
    Parsed kernel /lib/modules/3.4.11-2.16-trace/build/Module.symvers, which contained 5890 vmlinux exports
    Searched: " /usr/share/systemtap/tapset/x86_64/*.stp ", found: 7, processed: 7
    Searched: " /usr/share/systemtap/tapset/*.stp ", found: 77, processed: 77
    Pass 1: parsed user script and 84 library script(s) using 76904virt/27072res/2540shr kb, in 100usr/20sys/125real ms.
    Attempting to extract kernel debuginfo build ID from /lib/modules/3.4.11-2.16-trace/build/vmlinux.id
    Attempting to extract kernel debuginfo build ID from /sys/kernel/notes
    Located kernel source tree (DW_AT_comp_dir) at '/usr/src/debug//////////////kernel-trace-3.4.11/linux-obj'
    Checking tracepoint glob /lib/modules/3.4.11-2.16-trace/build/include/trace/events/*.h
    Checking tracepoint glob /lib/modules/3.4.11-2.16-trace/build/include/trace/*.h
    Checking tracepoint glob /lib/modules/3.4.11-2.16-trace/build/arch/x86/kvm/*trace.h
    Checking tracepoint glob /lib/modules/3.4.11-2.16-trace/build/fs/xfs/linux-*/xfs_tr*.h
    Checking tracepoint glob /usr/src/debug//////////////kernel-trace-3.4.11/linux-obj/include/trace/events/*.h
    Checking tracepoint glob /usr/src/debug//////////////kernel-trace-3.4.11/linux-obj/include/trace/*.h
    Checking tracepoint glob /usr/src/debug//////////////kernel-trace-3.4.11/linux-obj/arch/x86/kvm/*trace.h
    Checking tracepoint glob /usr/src/debug//////////////kernel-trace-3.4.11/linux-obj/fs/xfs/linux-*/xfs_tr*.h
    Pass 2: getting a tracepoint query for 0 headers: 
    blacklist regexps:
    blfn: ^(atomic_notifier_call_chain|default_do_nmi|__die|die_nmi|do_debug|do_general_protection|do_int3|do_IRQ|do_page_fault|do_sparc64_fault|do_trap|dummy_nmi_callback|flush_icache_range|ia64_bad_break|ia64_do_page_fault|ia64_fault|io_check_error|mem_parity_error|nmi_watchdog_tick|notifier_call_chain|oops_begin|oops_end|program_check_exception|single_step_exception|sync_regs|unhandled_fault|unknown_nmi_error|xen_[gs]et_debugreg|xen_irq_.*|xen_.*_fl_direct.*|check_events|xen_adjust_exception_frame|xen_iret.*|xen_sysret64.*|test_ti_thread_flag.*|inat_get_opcode_attribute|system_call_after_swapgs|.*raw_.*_lock.*|.*raw_.*_unlock.*|.*raw_.*_trylock.*|.*read_lock.*|.*read_unlock.*|.*read_trylock.*|.*write_lock.*|.*write_unlock.*|.*write_trylock.*|.*write_seqlock.*|.*write_sequnlock.*|.*spin_lock.*|.*spin_unlock.*|.*spin_trylock.*|.*spin_is_locked.*|rwsem_.*lock.*|.*mutex_.*lock.*|raw_.*|atomic_.*|atomic64_.*|get_bh|put_bh|.*apic.*|.*APIC.*|.*softirq.*|.*IRQ.*|.*_intr.*|__delay|.*kernel_text.*|get_current|current_.*|.*exception_tables.*|.*setup_rt_frame.*|.*preempt_count.*|preempt_schedule|__switch_to|special_mapping_.*|.*_pte_.*)$
    blfn_ret: ^(do_exit|sys_exit|sys_exit_group)$
    blfile: ^(kernel/kprobes\.c|arch/.*/kernel/kprobes\.c|.*/include/asm/io\.h|.*/include/asm/io_64\.h|.*/include/asm/bitops\.h|drivers/ide/ide-iops\.c|arch/.*/kernel/paravirt\.c|.*/include/asm/paravirt\.h|fs/seq_file\.c)$
    blsection: ^(\.init\.|\.exit\.|\.devinit\.|\.devexit\.|\.cpuinit\.|\.cpuexit\.|\.meminit\.|\.memexit\.)
    pattern '*' matches module 'kernel'
    semantic error: no match while resolving probe point kernel.trace("*")
    deleting module_cache
    tracepoint_builder releasing dwflpp
    Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 204828virt/28504res/2880shr kb, in 10usr/60sys/72real ms.
    Running rm -rf /tmp/stap9cJQKh
    Spawn waitpid result (0x0): 0
    and it looks to me like the lines with header globs show that there is a problem finding the kernel include files (they are in /usr/src/debug/kernel-trace-3.4.11/linux-3.4 on my machine).
    Symlinking them to the location where SystemTap looks for them does yield a change in that stap actually compiles a kernel module, but the error condition remains:
    Code:
    > sudo stap -vvvv -L 'kernel.trace("*")'
    SNIP lots of compilation messages
    semantic error: no match while resolving probe point kernel.trace("*")
    SNIP
    ... full output at stap -vvvv -L 'kernel.trace("*")' - Pastebin.com
    stap-report's output is at stap-report - Pastebin.com, for the record.

  7. #7
    dd NNTP User

    Default Re: SystemTap fails to find kernel tracepoints

    On 01/24/2013 09:16 AM, braunseb wrote:
    >
    > > sudo stap -e 'probe vfs.read { printf("Works.\n"); exit() }'
    > > sudo stap -L 'kernel.trace("*")'
    > > sudo stap -vvvv -L 'kernel.trace("*")'
    > > sudo stap -vvvv -L 'kernel.trace("*")'


    it may not be helpful in this case, but i think it important for you to
    know that 'sudo [someExecutable]' is _not_ the same as being root and
    running [someExecutable]...

    so, just try

    Code:
    su -c [someExecutable]
    like:
    su -c stap [-e | -L | -vvv | whatever]
    as said, it might make a difference, it might not...it depends because
    root's PATH is not the same as your PATH using sudo....and if you must
    have root's PATH to be successful you will fail if using sudo (in
    openSUSE...Debian--i have no idea.)

    more info:
    http://tinyurl.com/593e4c
    http://tinyurl.com/ydbwssh
    http://tinyurl.com/4nsaqst
    http://tinyurl.com/665h5ek
    http://www.linfo.org/root.html

    --
    dd
    openSUSE®, the "German Engineered Automobile" of operating systems!
    http://tinyurl.com/DD-Caveat


  8. #8

    Default Re: SystemTap fails to find kernel tracepoints

    Quote Originally Posted by dd View Post
    On 01/24/2013 09:16 AM, braunseb wrote:
    >
    > > sudo stap -e 'probe vfs.read { printf("Works.\n"); exit() }'
    > > sudo stap -L 'kernel.trace("*")'
    > > sudo stap -vvvv -L 'kernel.trace("*")'
    > > sudo stap -vvvv -L 'kernel.trace("*")'


    it may not be helpful in this case, but i think it important for you to
    know that 'sudo [someExecutable]' is _not_ the same as being root and
    running [someExecutable]...

    so, just try

    Code:
    su -c [someExecutable]
    like:
    su -c stap [-e | -L | -vvv | whatever]
    as said, it might make a difference, it might not...it depends because
    root's PATH is not the same as your PATH using sudo....and if you must
    have root's PATH to be successful you will fail if using sudo (in
    openSUSE...Debian--i have no idea.)
    I tend to avoid su -c, because su -c accepts a single argument, forcing you to add an additional layer of quotes, so actually
    Code:
    su -c stap -L 'kernel.trace("*")'
    wouldn't work, but
    Code:
    su -c "stap -L 'kernel.trace(\"*\")'"
    would (be sure to get the quotes right, though).

    In this case, I don't think it's a problem with the PATH. Just to be sure, I ran it again from a root login session, but no change. Also, stap is supposed to run equally well if the calling user is a member of the stapdev or stapsys groups, so you (theoretically) should not need to be root anyway.

  9. #9

    Default Re: SystemTap fails to find kernel tracepoints

    Okay, I managed to figure it out:

    SystemTap looks for tracepoints by compiling a fake kernel module and looking at the debug information that is generated, using the elfutils libraries libelf1 and libdw1. These in turn depend on the backends that are packaged in libebl1 to actually resolve the relocations in the debug info, so if libebl1 is not installed, the whole process fails. After installing libebl1, systemtap does find the tracepoints.

    I guess I'll file a bug against systemtap.

  10. #10
    dd NNTP User

    Default Re: SystemTap fails to find kernel tracepoints

    On 01/24/2013 12:26 PM, braunseb wrote:
    > After installing libebl1, systemtap does find the tracepoints.



    happy you found the problem....just wondering: did you find it because
    it finally popped up in an error message, or ???


    > I guess I'll file a bug against systemtap.


    i'd _guess_ it best to file that in our bugzilla, here
    http://tinyurl.com/nzhq7j

    because it might be an openSUSE packaging problem (failed to include all
    dependencies)..

    once you have bug number/URL please return that info to this thread (so
    any who google in with the same problem can look to the bug to find work
    around and/or follow the progress of the resolution)

    tia

    --
    dd

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •