zorbuz
March 23, 2009, 12:02am
#1
Couple days ago, samba suddenly started to act weird.
For example, file copy used to be steady 50-60MB/s from samba -> windows workstation, now it starts fast and drops down to 10MB/s or less
Even mp3’s dont play properly from that share anymore. Couple seconds its ok, then playback freezes, then it plays again and freezes…
Checked disk usage during mp3 playback with iotop and it says:
PID USER DISK READ DISK WRITE SWAPIN IO> COMMAND
9474 root 31.84 K/s 0 B/s 0.00 % 98.99 % smbd -D -s /etc/samba/smb.conf
9470 root 0 B/s 207.17 K/s 0.00 % 0.11 % [kjournald]
Something is quite messed up when 31kb/s disk read hogs 99% of the IO.
samba-3.2.7-11.2.1
fully updated openSuse 11.1
Xen kernel 2.6.27.19-3.2-xen (its also an VM host)
samba share is 1.75TB LVM volume, with 80% used (2 disks, 1TB and 750GB).
fsck said volume is just fine.
Machine has 8GB of RAM (swap usage 0%)
Anyone have any pointers how to start diagnosing this?
zorbuz
March 23, 2009, 1:10am
#2
Little update:
It does not misbehave all the time. For exmple, now everything seems to be just fine:
PID USER DISK READ DISK WRITE SWAPIN IO> COMMAND
766 root 0 B/s 0 B/s 0.00 % 1.78 % [kjournald]
12704 root 0 B/s 12.75 K/s 0.00 % 0.02 % qemu-dm ...
9470 root 0 B/s 0 B/s 0.00 % 0.01 % [kjournald]
11049 root 25.50 K/s 0 B/s 0.00 % 0.00 % smbd -D -s /etc/samba/smb.conf
12705 root 0 B/s 1.59 K/s 0.00 % 0.00 % qemu-dm ...
SMB using 0% IO with 25.5K/s read which is just like it should.
user
March 23, 2009, 6:35pm
#3
I had a similar problem. Mine ended up as the NIC is going bad. Just curious as to whether you are having the same problem, post the output of ifconfig. Mine was showing thousands of errors within 10-15 mins. of playback of .mp3’s or file transfers.
zorbuz
March 24, 2009, 7:49am
#4
During file copy and mp3 playback (both froze pretty much completely after a while).
This time noticed kjournald hogging 99% of the IO at some point in iotop (then it disappeared and samba started to misbehave).
ifconfig:
br0 Link encap:Ethernet HWaddr 00:19:66:8B:A5:C2
inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fec0::c:219:66ff:fe8b:a5c2/64 Scope:Site
inet6 addr: 2002:5b9c:251c:c:219:66ff:fe8b:a5c2/64 Scope:Global
inet6 addr: fe80::219:66ff:fe8b:a5c2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1180409 errors:0 dropped:0 overruns:0 frame:0
TX packets:2029496 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:843943791 (804.8 Mb) TX bytes:2508078814 (2391.8 Mb)
eth0 Link encap:Ethernet HWaddr 00:19:66:8B:A5:C2
inet6 addr: fe80::219:66ff:fe8b:a5c2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1183478 errors:0 dropped:0 overruns:0 frame:0
TX packets:2041664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:862723379 (822.7 Mb) TX bytes:2510738431 (2394.4 Mb)
Interrupt:253 Base address:0xe000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1463 errors:0 dropped:0 overruns:0 frame:0
TX packets:1463 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:77704 (75.8 Kb) TX bytes:77704 (75.8 Kb)
tap1.0 Link encap:Ethernet HWaddr 6E:9F:E4:66:B4:B4
inet6 addr: fe80::6c9f:e4ff:fe66:b4b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32187 errors:0 dropped:0 overruns:0 frame:0
TX packets:58326 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:6369018 (6.0 Mb) TX bytes:10861676 (10.3 Mb)
tap2.0 Link encap:Ethernet HWaddr AA:7B:DE:6E:43:24
inet6 addr: fe80::a87b:deff:fe6e:4324/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26101 errors:0 dropped:0 overruns:0 frame:0
TX packets:49818 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:3920834 (3.7 Mb) TX bytes:11717170 (11.1 Mb)
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:29515 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:29441 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
iotop:
13244 root 50.96 K/s 0 B/s 0.00 % 99.99 % smbd -D -s /etc/samba/smb.conf
766 root 0 B/s 0.80 K/s 0.00 % 1.08 % [kjournald]
12705 root 0 B/s 38.22 K/s 0.00 % 0.08 % qemu-dm ...
20575 root 25.48 K/s 0 B/s 0.00 % 0.00 % smbd -D -s /etc/samba/smb.conf
user
March 24, 2009, 8:14pm
#5
On Tue March 24 2009 01:56 am, zorbuz wrote:
>
zorbuz;
This could perhaps be the inotify bug. Check your Samba logs:
http://www.mail-archive.com/samba%40lists.samba.org/msg98069.html
See also:https://bugzilla.novell.com/show_bug.cgi?id=463372
P. V.
zorbuz
March 25, 2009, 6:48am
#6
>>> This could perhaps be the inotify bug.
Yeah, logs were full of inotify errors. And disabling it in [global] did the trick.
Thank you