**Summary
** The latest version of VMware Player (VMware-Player-15.1.0-13591040.x86_64.bundle) as downloaded from the VMware web site, does not work on TW. It will install just fine, but when VMware starts, it needs to compile and install some kernel modules, which fails for kernel versions 5.1 and later, i.e. for the current Tumbleweed. I found this to be due to two changes in the newer kernels, one is a change of a data type, the other the dropping of a macro definition. By modifying the source code for VMware’s kernel module vmmon, one can fix both.
**Analysis
**
1) vm_fault_t
In the driver.c file for vmmon, the following initialisation appears on line 108:
static struct vm_operations_struct vmuser_mops = {
.fault = LinuxDriverFault
};
which assigns a pointer to the static function LinuxDriverFault() to the member fault of struct vm_operations_struct. The function LinuxDriverFault()is declared at lines 100 as
static int LinuxDriverFault(struct vm_fault *fault);
Digging for vm_operations_struct in the kernel header files, one finds with some help from the Elixir website, that up to kernel version 4.16.xx the header file mm.h indeed defined the fault member of vm_operations_struct as a “pointer to a function returning int”.
This changed with version 4.17, and at line 421 of mm.h we have now:
struct vm_operations_struct {
*//<snip>...*
vm_fault_t (*fault)(struct vm_fault *vmf);
*//<snip>...*
};
Also in 4.17, a new typedef defined vm_fault_t to be an int. Nothing had thereby really changed and *fault still returned an int.
But, starting with version 5.1, the header file mm_types.h (line 628) now defines:
typedef __bitwise unsigned int vm_fault_t;
This means a real change of type, from int to unsigned int (also, the __bitwise attribute now restricts operations on the data type to bitwise operators).
Since “pointer to a function returning int” is incompatible with “pointer to a function returning unsigned int”, the change to the kernel headers breaks VMware’s code and it won’t compile in the strict environment of kernel module compilation.
2) get_ds()
In the hostif.c file for vmmon, this call to get_ds() appears three times:
set_fs(get_ds());
Previously, the kernel header file asm-generic/uaccess.h had “#define get_ds() (KERNEL_DS)”.
Starting with version 5.1, an authority no less than Linus Torwalds broke VMware’s code. He personally removed that definition, as he writes:
get rid of legacy ‘get_ds()’ function
Every in-kernel use of this function defined it to KERNEL_DS (either as an actual define, or as an inline function). It’s an entirely historical artifact, and long long long ago used to actually read the segment selector valueof ‘%ds’ on x86.
Which in the kernel is always KERNEL_DS.
…] Inspired by a patch from Jann Horn …] I then just took it to the logical extreme and removed all the remaining gunk. …]
Hence the calls in hostif.c fail.
Fixes
To make VMware Player work, first install the bundle. Then manually expand the vmmon archive and go to the source code directory:
# cd /tmp
# tar xvf /usr/lib/vmware/modules/source/vmmon.tar
# cd vmmon-only/linux
Save a copy of driver.c as driver.c.old and edit the original file as follows:
Change lines 99-103 from
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 11, 0)
static int LinuxDriverFault(struct vm_fault *fault);
#else
static int LinuxDriverFault(struct vm_area_struct *vma, struct vm_fault *fault);
#endif
to
static
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 17, 0)
vm_fault_t
#else
int
#endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 11, 0)
LinuxDriverFault(struct vm_fault *fault);
#else
LinuxDriverFault(struct vm_area_struct *vma, struct vm_fault *fault);
#endif
and, having fixed the declaration, also fix the definition by changing line 598 from
static int
to
static
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 17, 0)
vm_fault_t
#else
int
#endif
This takes care of the problem with vm_fault_t.
To fix the problem with get_ds(), just locally revert Linus’s change: Save a copy of the file hostif_priv.h as hostif_priv.h.old and add to the very end of the original file (i.e. after the last #endif) the lines
#ifndef get_ds
#define get_ds() (KERNEL_DS)
#endif // ifndef get_ds
Then create a new archive:
# cd /tmp
# tar cvf vmmon.tar vmmon-only/
and replace the one distributed by VMware with yours:
# cd /usr/lib/vmware/modules/source
# mv vmmon.tar vmmon.tar.old
# mv /tmp/vmmon.tar .
Firing up VMware Player should now work (at least it does for me!).