Why /boot/vmlinuz differs from what i get after recompling the kernel source

Hi,
could somebody explain me, please the difference between the official default kernel and the one I get after compiling the kernel source:


-rw-r--r-- 1 root root  7331952 15. maj 02:26 vmlinuz-4.12.14-lp151.28.10-default
-rw-r--r-- 1 root root  7338256 30. lip 14:25 vmlinuz-4.12.14-lp151.28.10-orion

I copied /boot/config-4.12.14-lp151.28.10-default to /usr/src/linux/.config, started “make bzImage” and moved the resulted image to /boot with the extension -orion. I wonder why the compilat is not identical with the -default.


kernel-default-**4.12.14**-lp151.28.10.1.x86_64
kernel-source-**4.12.14**-lp151.28.10.1.noarch

The question is insofar important that I cannot fully start Linux with my -orion-kernel: It breaks down with problems to mount sysroot, or X-Windows breaks down. The procedure I know and I apply is:


make bzImage
copy bzImage, System.map to /boot
make modules
make modules_install
mkinitrd 

I did nothing with symvers*, sysctl* and *hmac files. Is this correct ?

What would be the point of compiling, if you get the same thing?

It’s probably 20 years since I last compiled a kernel, and that was for slackware. So I’m not the most knowledgable about this.

I would expect that if you compile the kernel, you would get something closer to the vanilla kernel. Some of the features from newer kernels are backported to the default kernel, to support newer hardware.

Also keep in mind that if you compile the kernel twice, you will get different results – because the compile time is built into the compiled kernel.

Hi
So what is the end game for recompiling the kernel? I normally use /proc/config.gz for the config file. The other option is to use OBS for kernel builds.

To get the starting point identical with the one provided with the distribution. Surely I’d like to change it later on.

if you compile the kernel twice, you will get different results – because the compile time is built into the compiled kernel.
You are joking.

Hi
AFAIK it’s being worked on via reproducible builds…
https://reproducible-builds.org/
https://lists.opensuse.org/opensuse-factory/2019-07/msg00364.html

I see this from 2011 on deterministic builds: https://lwn.net/Articles/437864/

There are quite a lot of steps performed by build system after make. If you want identical result, you need to replicate them all.

I copied /boot/config-4.12.14-lp151.28.10-default to /usr/src/linux/.config, started “make bzImage”

congratulations, you just ruined your kernel source installation. You should never build kernel inside of /usr/src.

Oh, this is a simple misunderstanding. They speak of “various timestamps embedded in the object files”, but an usual C-language compilat (an object file) does not include any timestemp. What would be the reason for it ? A timestemp is written into a filesystem with each created or changed file, but not into a file-contents ! Under Linux (UNIX) each file has three timestemps: a creation time, last change and last open for reading. You change them while accessing the file, but the file contents does not change by reading of it !

The author of the excerb, you citied shoud learn Linux filesystem first, before any compilation, otherwise he mixes it.

If you don’t belive, compile any C-cource several times and see the checksum of each product.

Comming back to my question, we obviously get a kernel source and a kernel ready for use created from a different source. I wonder why ? The question could answere the person, who is responsible for suse-linux package creation.

Comming back to my question, we obviously get a kernel source and kernel ready for use generated from a different source. I wonder why ? The main advantage of open source is that we see and we can exactly follow what this software does.

Yes, and what are they ? Where are they written down ?

Why not ? What do you mean by “ruined” ? The standard -default kernel with compiled vmlinuz work well outside of /usr/src. They don’t need this directory, do they ?

OBS sources, project and package configuration in OBS, kernel SRPMs specs, sources in kernel SRPMs.

The standard -default kernel with compiled vmlinuz work well outside of /usr/src.

I said kernel source, not kernel binary.

Oh, I understand. The precision of your answer explains almost everything.

Yet as long as one cannot follow step by step what a compiled Linux kernel is doing SuSE linux remains no open source !

Hi
You can find everything about how the kernel for openSUSE (not SuSE anymore…) is built on the openSUSE Build Service;

https://build.opensuse.org/package/show/Kernel%3AopenSUSE-15.1/kernel-default
https://build.opensuse.org/package/show/openSUSE%3ALeap%3A15.1%3AUpdate/patchinfo.10626

Every part of the building;
https://build.opensuse.org/public/build/Kernel:openSUSE-15.1/standard/x86_64/kernel-default/_log

Anybody with an internet connection and an OBS account can branch, submit patches, fix do what ever they like to the kernel and install on their systems, submit to the project etc… if that’s not the meaning of open source…well <shrug>.

Hi Malcolm, thank you very much for your answer. Yet this is not “open source”, what you describe. “Open source” means only that the sources of a concrete system with a given release number are opened for reading enought to follow step by step what such a system is doing. This is not a project open for participation for everybody.

And now why I claim that OpenSUSE LEAP 15.1 release of the kernel
vmlinuz-4.12.14-lp151.28.10-default is not an open source ? Because I cannot follow step by step on source C-languege code what it is doing ! Why ? Because if I take the source code from the newly installed package kernel-default ver 4.12.14-lp151.28.10-default and compile only the kernel from it I get a file, which is different from the above vmlinuz-file

 

-rw-r--r-- 1 root root  7340144 15. lip 13:25 vmlinuz-4.12.14-lp151.28.10-default
-rw-r--r-- 1 root root  7338256 30. lip 14:25 vmlinuz-4.12.14-lp151.28.10-orion

-orion is simply my name I gave to the compilat bzImage from arch directory after the compilation with “make bzImage” .

I am thankful for your links, but none of them provided an infromation which made me capable to compile the given sources in the way they gave a result exactly the same I found in “default-kernel” package.

The larger and the more complex a build process becomes, the more illusory might it get to expect bit-identical build results. Consider, for example:

  • version differences of compiler and linker

  • version differences of other build tools
    like GNU-Make and uncounted contributing perl and python and shell scripts contained in /usr/src/linux/scripts/

  • different patch history
    (I couldn’t notice that half of the above log consisted of patches not applied to the locally downloaded kernel sources)

  • varying order
    of compile and link runs dependent on wildcard operators (*.c, *.o) which may not list in lexical order, but based on varying filesystems and inode-/tree-based directory orderings (see »ls -U« as in »unsorted« and contemplate the implications of this behavior; especially extended runs of patch utilities on the kernel tree may scramble these orderings differently each time)

  • some kind of security-based randomization
    like address-space or ELF-header randomization (I know, those two are runtime-only, but may be some of those also can influence a complex build process)

  • a multitude of other possible differences, possibly undocumented, between your and SuSE’s build host

Just for peace of mind though, I too would like to be able to perfectly reproduce the SuSE kernel build, mind you.

While in general you have several points IMHO, I repeat Malcom’s remark to the OP that there is no SuSE. There is SUSE and there is openSUSE.

Last SuSE kernel was built in year 2003 and at that time it quite likely was closed source (at least, as far as build tools is concerned); and as you explained yourself getting proper environment to reproduce build would be a challenge after 15 years. If you mean SUSE or openSUSE then you cannot perfectly reproduce kernel build outside of (open)SUSE build system for the simple reason - kernel is signed by SUSE key.

Oh, excuse me, to be not exact. I surely mean only contemporary software from opensuse.org.
Hi UNIX111, thank you for the answer. With all software being up to date I should have exactly the same environment, than the one, where official kernel builds are compiled. That means no compiler nor linker, nor other tools differences . All of them must be in the same version as in the original build environment. Patches are the same and their order must not have an influence on the compiled result. The same regards the same the order of compilation.

Regarding “a multitude of other differences between my and SuSE’s build host” such an influence makes me worry - it breaks the rule.

I did not succede to recover the difference with tools like “strings”, but both kernels behave in a different way.

Which method did you use to build your own kernel?

  • There’s information about building a SUSE kernel without using OBS is here: <https://www.suse.com/c/compiling-de-linux-kernel-suse-way/>.
  • In the Kernel source there’s a ReadMe – ‘/usr/src/linux-4.12.14-lp151.28.7/README.SUSE’ – “COMPILING YOUR OWN KERNEL” …
  • The openSUSE kernel is provided by SUSE employees – it’s the SUSE SLE kernel …

Wrong. It is maintained by SUSE employees (mostly) but it is built as part of normal openSUSE build process.

You both are correct, of course; old habits died hard. I’m looking at about half a meter of S.u.S.E. books that came in nice boxes with the install CDs way back when there was a S.u.S.E. GmbH. The »u« is an abbreviation for the German »und«. It’s still extra effort for me to write that versal »U« in »openSUSE«. At work, we use SLES/SLED, so the »U« doesn’t come up. That’s also why I like »Leap« and not »TUmbleweed«. Just kidding, it’s all about stability.

Maybe, while your tools are up to date, the tools on that SUSE build server weren’t. I can imagine that kernel specialists like Hubert Mantel may have their personal and experience-based preferences as to which versions of make, binutils or ld (the GNU linker). They may not use the newest tools, but older versions proven to be more speedy, stable and bug-free.

How can you be so sure? Considering the SUSE log protocols a set of patches applied immediately before the kernel build; however, your and my local builds probably don’t involve the involvement of any patches. Folks like us just grab the default kernel-source RPMs or Tarballs and build kernels going on from that.

Based on my experience with filesystems, I disagree. Create a directory, copy files into it, then delete some from it, then add some other files. Make an exact copy of the directory (with »cp -a« or »rsync -av«). Do a »ls -f« on both and notice the difference. Or: write a short C program using readdir(3) and the »dirent« struct — this mechanism is probably also used by gcc and all accompanying build tools. You will find that the »on-disc« order of files in a folder is generally unpredictable and will change while you do work within that folder. It would be inefficient for filesystem drivers to handle it any other way.

This unpredictability is enough to get different results after patching, compiling and linking kernels based on thousands of source files and hundreds of intermediate object files — wholly dependent on the physical order in which tar or rpm unpack archives and in which filesystems like ext4, btrfs, xfs and zfs happen to hash, extend, shrink or serialize their directory contents.

It makes me a little worried as well, now that I’m thinking more about it, climbing down the proverbial rabbit hole. But — what is »the rule« you think all that complexity of a build environment would break?

My understanding up to this point was that the repositories are signed by certificates, and also the packages in those repositories.
But if it is true that the kernel binaries themselves are uniquely signed, then of course the question any bit-by-bit exact recompiles becomes moot.