11.3 Kernel updates: please slow down

[twelveeighty]

We now have had two kernel updates in as many weeks, following a rather disastrous one a number of weeks ago that impacted MANY users in a seriously bad way.

My question is: just like KDE updates are not pushed out through Online updates, but go through Stable/Factory, etc. repos, should we not advocate the same thing for kernel updates? Because it is very evident that testing on these kernel updates leave a lot to be desired.

I fully understand that building and testing is a community responsibility. So let’s take a breather and make sure the community can fully test these kernel updates before they are pushed out through Online Update? Yes, it will slow down the roll-out of security patches, but I’d rather wait 2 weeks for a security patch than having a non-booting machine because of a bug in a kernel update.

[malcolmlewis]

@twelveeighty

We now have had two kernel updates in as many weeks, following a rather
disastrous one a number of weeks ago that impacted MANY users in a
seriously bad way.

My question is: just like KDE updates are not pushed out through Online
updates, but go through Stable/Factory, etc. repos, should we not
advocate the same thing for kernel updates? Because it is very evident
that testing on these kernel updates leave a lot to be desired.

I fully understand that building and testing is a community
responsibility. So let’s take a breather and make sure the community can
fully test these kernel updates before they are pushed out through
Online Update? Yes, it will slow down the roll-out of security patches,
but I’d rather wait 2 weeks for a security patch than having a
non-booting machine because of a bug in a kernel update.

Hi
If your running ATI, it’s ATI’s driver…not kernel updates.

KDE updates from anywhere other than the standard update channel are
NOT supported officially

I have multiple machines running openSUSE and SLED and have no
issues with the kernel updates, I run nvidia and intel video all are
working fine, so I’m a happy camper.

If you have problems and can’t get a solution here, IRC or mailing
lists, I suggest the best place is to raise a bug if the updates are
causing you a problem.

–
Cheers Malcolm °¿° (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.32.19-0.3-default
up 5:38, 2 users, load average: 0.05, 0.08, 0.08
GPU GeForce 8600 GTS Silent - Driver Version: 256.53

[twelveeighty]

@malcolmlewis

KDE updates from anywhere other than the standard update channel are
NOT supported officially

But I guess that’s my point: the lack of reliability of the last set of kernel updates leads me to believe the kernel updates have the same “level” of support as the KDE repos: they destroy users’ installations in frighting numbers. With the KDE repos, everyone knows (or should know) the risk involved. With the Online Update, most users will expect a higher level of guarantee their machines are not worse off than before the update.

For me personally, I was not affected, since I’m conservative with kernel updates: I check the forums for a couple of days before applying them. However, if the general recommendation is: do NOT apply Online Updates without checking the forums first, we may want to post a “sticky” somewhere or put that warning in the Online Update Yast module itself.

I have seen various posts of frustrated users turning to a different Linux distro because of these kernel issues - that can’t possibly be in line with the openSUSE mission statement.

[malcolmlewis]

@twelveeighty

malcolmlewis;2229446 Wrote:
>
> KDE updates from anywhere other than the standard update channel are
> NOT supported officially
>
I have seen various posts of frustrated users turning to a different
Linux distro because of these kernel issues - that can’t possibly be in
line with the openSUSE mission statement.

Hi
But, there in lies the problem if you go off the beaten path so to
speak (as anything not in oss, non-oss and updates), which includes the
ATI, Nvidia proprietary drivers (which taint the kernel), how is it the
release maintainers fault?

Maybe a read of this may help (In fact the whole thread )
http://lkml.org/lkml/2006/12/14/63

–
Cheers Malcolm °¿° (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.32.19-0.3-default
up 9:01, 2 users, load average: 0.27, 0.14, 0.06
GPU GeForce 8600 GTS Silent - Driver Version: 256.53

[oldcpu]

@twelveeighty

But I guess that’s my point: the lack of reliability of the last set of kernel updates leads me to believe the kernel updates have the same “level” of support as the KDE repos: they destroy users’ installations in frighting numbers.

This is IMHO a fundamental problem with Linux, and it is a concern not just with openSUSE, but all Linux distributions. Some Linux distributions handle this a bit better than others, but no Linux distribution is immune to the risk that can come from a kernel update. I believe this risk is tied into the manner in which proprietary (and open source) drivers are implemented in Linux.

Because the Kernel typically comes with kernel modules, where such modules can include hardware drivers, there is always a risk that a kernel update will break a driver. There is a VAST amount of different PC hardware available, and the open source community who create the kernel patches/updates upstream, are not able to test their updated code against all the different hardware available.

Typically (but not always) open Source drivers are not impacted by a kernel update as much as proprietary drivers. The classic proprietary driver breakage with a new kernel is the ATI graphic driver, because in comparison to nVidia, ATI are much slower than their nVidia competitors in producing Linux drivers. This last ATI graphic driver breakage with the 2.6.34.7 updated kernel (with important security patches) is especially frustrating, as it was only a week earlier that ATI ‘finally’ produced a decent proprietary driver (the Catalyst 10.9) for the 2.6.33/2.6.34 kernels, … only to now see that new Catalyst 10.9 broken on openSUSE by an upstream kernel change applied in the 2.6.34.7 kernel.

When friend’s ask me about what are the major disadvantages of Linux, I have to concede that kernel updates are one of the disadvantages. In my case, I have multiple PCs and I usually update my ‘sandbox’ PC first with a new kernel, followed by my ‘backup’ PC. Only at that point (if happy) will I update my main PC. Of course not everyone has this luxury of multiple PCs, … and even multiple PCs (with different hardware) is no guarantee against the possibility of a kernel breakage.

In fact, one of my laptops is still running the 2.6.27 kernel (with openSUSE-11.1) because all kernels after 2.6.27 broke the Intel 855GM graphic hardware on this laptop, and despite a couple years having gone by, and numerous bug reports across multiple distributions and bug reports on the kernel itself, a fix still has not made it upstream to Linus git tree.

I went through a phase (when I had less PCs) of still keeping the old kernel, when I installed the new kernel. I do not do that as much now, as it requires a bit extra work, and I tend to be lazy.

I definitely agree caution is of paramount importance when accepting a kernel update, and any friend who starts with Linux, always receives a warning from me NOT to accept a kernel update until at least a week or more has gone by after the update, … and even then to be prepared in advance for the possibility of a proprietary driver breakage (where typically graphic card and wireless card breakage are the most damaging to a user’s PC’s functionality).

[twelveeighty]

@malcolmlewis

… if you go off the beaten path so to
speak (as anything not in oss, non-oss and updates), which includes the
ATI, Nvidia proprietary drivers (which taint the kernel), how is it the
release maintainers fault?

Yes, this is true - and I’ll grant you that on two out of the three kernel updates in 11.3, but, the “USB” problem on kernel update #1 had nothing to do with non-standard drivers, as far as I know, and, more importantly, could have been caught with an extra week of testing before releasing.

I guess I’m more inclined with @oldcpu’s “advice to a friend” - why would we not extend this friendship to everyone and add the feature in Yast Online Update to have a confirmation on kernel updates with a big red warning stating that this may hose the machine?

[malcolmlewis]

@twelveeighty

Yes, this is true - and I’ll grant you that on two out of the three kernel updates in 11.3, but, the “USB” problem on kernel update #1 had nothing to do with non-standard drivers, as far as I know, and, more importantly, could have been caught with an extra week of testing before releasing.

Hi
Yes, that USB one was a pain, at least all I had to do was unplug the USB and plug it back in, but still a pain…

Any update could break a perfectly good system, not just kernel updates People need to read what the update is for and analyse themselves as to whether to update or not. They are they ones who know what is installed, additional repositories added etc… (On IRC, I’ve see a user paste info with over 130 repositories added and wonders why they are having problems with 11.3 not working).

[Chananko]

Well, I appear to be hosed by the latest kernel update. One thing here is my Ubuntu system on my multi-boot always leaves the last kernel in place. Sort of a failsafe just in case. You can’t beat that. I can then remove the old kernel and grub automatigcally updates. Why in the world can’t openSUSE incorporate this as well?

[knurpht]

@Chananko

Well, I appear to be hosed by the latest kernel update. One thing here is my Ubuntu system on my multi-boot always leaves the last kernel in place. Sort of a failsafe just in case. You can’t beat that. I can then remove the old kernel and grub automatigcally updates. Why in the world can’t openSUSE incorporate this as well?

Ubuntu keeps all kernels. I often see loads of kernels because people do not dare to remove them.
openSUSE has an option to keep all installed kernels. In Ubuntu that option is on, in openSUSE it’s off. You can set it.

[SixDegrees]

@Knurpht

Ubuntu keeps all kernels. I often see loads of kernels because people do not dare to remove them.
openSUSE has an option to keep all installed kernels. In Ubuntu that option is on, in openSUSE it’s off. You can set it.

Would you care to say how? This used to be the default behavior, and it should be on by default. I don’t have ATI proprietary drivers - or anything other than a fresh 11.3 installation - yet the last two kernel updates managed to hose over my system to the point where Linux needed to be reinstalled.

At a minimum, some sort of rollback mechanism absolutely MUST be provided. As things now stand, I simply refuse to accept any kernel updates, which really isn’t much of a solution.

So if you wouldn’t mind letting us know just where this option can be toggled, at least we’d have some way to recover from these flawed releases.

[oldcpu]

@SixDegrees

I don’t have ATI proprietary drivers - or anything other than a fresh 11.3 installation - yet the last two kernel updates managed to hose over my system to the point where Linux needed to be reinstalled.

You can also simply download the kernel rpm to your hard drive, and install via:

rpm -ivh  <rpm1> <rpm2> .... 

NOTE that is NOT "rpm -Uvh <rpm1> <rpm2> … "

Do you have a complex partitioning with many multiple partitions ? If the partitioning is simple, and there is no proprietary driver, what you describe is IMHO uncommon.

[SixDegrees]

Nope - my setup is a dirt-simple, straight out of the box dual boot system with nothing special added. When I accepted the first kernel update, I was greeted with a black screen when X started. The second kernel update wiped out the grub loader to the point where Windows was the only choice available. Although there were multiple posts on both of these problems here and elsewhere, there weren’t any solutions. Reinstallation - and careful avoidance of anything that even smelled like a kernel update since - fixed things.

Keeping the old kernel used to be the default on openSuse, as it is on so many other systems. It needs to be made the default again. If there’s a way to change the current behavior so old kernels are retained and offered as options on the boot screen, I’d like to hear about it.

[oldcpu]

@SixDegrees

Nope - my setup is a dirt-simple, straight out of the box dual boot system with nothing special added. When I accepted the first kernel update, I was greeted with a black screen when X started. The second kernel update wiped out the grub loader to the point where Windows was the only choice available. Although there were multiple posts on both of these problems here and elsewhere, there weren’t any solutions. Reinstallation - and careful avoidance of anything that even smelled like a kernel update since - fixed things.

Keeping the old kernel used to be the default on openSuse, as it is on so many other systems. It needs to be made the default again. If there’s a way to change the current behavior so old kernels are retained and offered as options on the boot screen, I’d like to hear about it.

After installing the new kernel BUT before rebooting, did you 1st check to see if the rpm that installs the kernel did a reasonable job of the /boot/grub/menu.lst file?

I always recommend users backup their /boot/grub/menu.lst file, and then after the kernel is installed, BUT before rebooting, compare the backed up version to the new version, to ensure the new version makes sence.

Also, note if you had a special boot code (such as ‘nomodeset’) in your original grub boot menu (ie menu.lst file) there is a strong possibility it may be wiped by a kernel update when it re-writes the menu.lst, so do not forget in such a case to apply the parameter again. Its all simple stuff, but I suspect some users forget these simple precautions.

[SixDegrees]

Nope, no special boot options, either.

I’ve been using openSuse for many years, and I have never had any occasion to check my menu.lst file for anything other than tweaking. I have also never had a kernel update, or an update of any kind, wipe out booting.

I really shouldn’t have to worry about such things, either. That’s the sort of thing that ought to be tested before release - something as serious as a bad overwrite of the boot code seems like it would be near the top of the priority list of things to check. The all-too-familiar open source mantra - fix it yourself - gets tiresome, especially when applied to things that have worked flawlessly for years and that any reasonable distributor would test for prior to release.

As noted several times now, the simple retention of old kernels presented as an option at boot time would have at least provided a recovery path, and a working system if not an updated one.

So: is it true, as an earlier poster suggested, that kernel retention can somehow be reactivated? As things stand now, I cannot accept any kernel updates from openSuse at all, given past experience - I just don’t have time to go through a reinstallation for something that ought to be simple. If the upgrade fails again, I need a quick way to revert to my old, working version.

I noted, also, that the installation DVD no longer provides a useful repair option other than dumping the user into a text console. In the past, this utility actually performed a repair, with little or no user intervention.

Maybe I should just buy Suse. I used to pay for the openSuse boxed sets, and if openSuse is going to become increasingly unstable and unsupported perhaps the for-profit version receives more attention.

[oldcpu]

@SixDegrees

That’s the sort of thing that ought to be tested before release - something as serious as a bad overwrite of the boot code seems like it would be near the top of the priority list of things to check.

Of course these things are tested before release.

BUT the testers do not have an example of every configuration possible, nor every hard drive, nor every partitioning scheme there is. They also do not have unlimited funds to go purchase every hard drive, nor every partitioning scheme there is, nor to setup every possible configuration. Who is going to pay for such an effort? So unfortunately there will always be someone’s boot that is crippled, while it ‘just works’ for the remainder of us. And in the majority of cases these failed boots (after a kernel update) could have been avoided if the user did some checks BEFORE rebooting after the kernel update.

YES, in an ideal world where we ALL paid a hundred euros or more per year for a maintenance contract for our SuSE version, then we should expect more testing and less occasional problems. Which makes me wonder, how many users on this forum paid for their openSUSE ? … Note there is SLED if one wants to pay for more stability and a longer support contract.

I provided an indication how to keep multiple kernels using the rpm command.

If you wish to setup yast/zypper to do this, then read here: Keeping the current kernel when doing a kernel update through yast

@SixDegrees

Maybe I should just buy Suse. I used to pay for the openSuse boxed sets, and if openSuse is going to become increasingly unstable and unsupported perhaps the for-profit version receives more attention.

Note buying openSUSE won’t provide you a long contract to last kernel updates. To get that sort of longer term support you will need SLED.

[robin_listas]

On 2010-10-29 23:36, SixDegrees wrote:

> Keeping the old kernel used to be the default on openSuse,

Was it? i don’t remember that.

> as it is on
> so many other systems. It needs to be made the default again. If there’s
> a way to change the current behavior so old kernels are retained and
> offered as options on the boot screen, I’d like to hear about it.

It has been explained several times in the forums, I wrote myself the recipe one or two days ago in
this same forum. Just search for it

Hint: zypp.conf

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

[tcubed_bus]

I have ATI graphics cards in my Linux boxes, and I have a bunch of different SuSE installations bootable in my main workstation. Having been burned, like you, with kernel “upgrades” with scary bug fix lists that rendered my system unusable, I developed a foolproof strategy for remaining whole. Before commanding YAST to do any kernel modification:

• boot Puppy linux,
• dd SuSE’s / partition to a gzip’ed image file on a different partition
• boot SuSE
• do the kernel change
• perform the ATI driver removal/reinstallation (the “hard” way)
• reboot SuSE, enjoy the improvements

In the event of any disaster, you can always reboot Puppy and restore the partition image.
HTH