Porting UIDs and GIDs from old installation to a new upgrade installation

English Install/Boot/Login
1

Hello - I am new to this particular forum but think it is the appropriate one to ask my question. If not please tell me which forum would be more appropriate.

I keep running into a problem each time I upgrade to a new OpenSuSE release. Currently I am in the process of upgrading systems from 15.0 and 15.1 to 15.2. When I do a new installation, I create a new partition for / and do most of the installation to that new partition. Some file systems, particularly /home, /srv, /mail and /websites are brought in as mount points from the previous file system setup so as to preserve data and configuration information in the new version of OpenSuSE. The trouble I keep running into is that UIDs and GIDs can either get dropped or assigned a new value for a particular name. This can lead to a lot of hours trying to debug problems and it makes it nearly impossible to switch back and forth between the old version of OpenSuSE and the new version when I want to boot up one or the other. I can’t simply change the underlying value of these IDs, nor can I go about doing chown everywhere because I simply don’t know all the places where a particular UID or GID was used.

So my question is, is there any way to export all the name/UID and name/GID pairs from the older version of OpenSuSE and then import all those name/value pairs before the actual installation of a new system commences? I don’t see anyway to accomplish this in YaST. In this way the new system would have the name/value pairs already in place when files are installed, or already in place because a particular file system was being mounted and the installation process could keep the UIDs and GIDs in sync with what the previous installation was using.

If there is a tool that does this I would greatly appreciate hearing about it. If not, I just need a set of steps that I could follow so as to effectively keep the UIDs and GIDs in sync when upgrading. Not knowing how to do this is costing me a great deal of time debugging and neither Google nor the OpenSuSE Wiki seem to be providing me with anything helpful.

Thanks in advance… Marc…

2 
"If you install openSUSE Leap on a machine with one or more existing Linux    installations, YaST allows you to import user data such as user names    and passwords. Select Import User Data from a Previous    Installation and then Choose Users for    import.!"

https://doc.opensuse.org/documentation/leap/startup/html/book-opensuse-startup/cha-install.html#sec-yast-install-user

3

What I know is that when an new installation is done on the same / file system, the installer, somewhere in the beginning, finds out there is an old installation there. During the phase where normally the root account end one other user account are created, there is a checkbox saying something “shall I take over users from the old installation?”. When that is checked a new screen appears that shows a list of those “old” accounts" and offers the opportunity to choose from it which accounts one wants to take over to the new installation. That worked very satisfying for me. As expected, after the installation and the mounting of the still existing /home file system, all is fine as it was.

Now I have to be a bit vague. I said above “when an new installation is done on the same / file system”. I am not sure that that restriction is true, because I often do the same as you, installing on a / file system on another partition to save the old installation as fall back and to make multi-boot between old and new possible. And I have the strong Idea that the installer in that case also finds the old / file system and thus offers that “take over the users” feature.

My advice is to check hat. After all, as long as you do not really kick off the installation at he end of the configuration screens, nothing will be done to the old installation and you can experiment a lot.

I will now go a bit off topic. But why are you doing a fresh installation for 15.0 > 15.1 and/or from 15.1 > 15.2 The “switch repos and do zypper dup” way of working does a fine job and has the additional advantage that you problem will never surface.

=======

As a last, minor, remark. Please look at the all the pages on the different web-sites of this distribution. Everywhere the name of the product is spelled openSUSE. As said, it is only minor, but I assume that you also do not like it that your name is misspelled when you are addressed (I know about people who burned letters they got from officialdom like the tax office, because it misspelled their name: “this is not for me”). So please try to use the correct upper and lower case for openSUSE.

Edit: Karl is a bit faster then I am. My excuse is that I created much more text to say basically the same (and he filled it up by repeating what you already said).

4

I’ll mostly agree with others who have replied.

My normal practice is to do what you describe. So I have both Leap 15.1 and Leap 15.2 in different partitions. When I move to Leap 15.3 (currently at alpha testing), that will replace 15.1. That way, I can switch between 15.2 and 15.3 while testing.

In my case, I will install 15.3 while it is at the beta stage, perhaps about 1 month before it is released.

So back to UIDs and GIDs. There are system UIDs (values <1000) and similarly system GIDs. And there are values set for users that I have defined (values 1000 and up). For the system values, I just take what I get. Some of those change between releases, but that has not caused any problems. On the switch to 15.2, I may have used a “chown” for user “qemu” – or maybe I didn’t bother.

For user ID values, I just import users during install. There’s an option to do that. It imports users, but not groups. And groups that I have added, I will have to re-add. I use “vipw -g” for that.

5

I like to add a warning about the system ones (good to mention them @nrickert). Yes, they do change from time to time and that can give troubles. E.g. I have a web-site running that uses user wwwrun and the UID once changed. As these users are not handled with the procedure outlined above, they request separate attention.

6

Thanks everyone for your replies, and Hank your last post hit the nail on the head! I know about importing regular users during the installation process and use that feature every time I install a new version of openSuSE. (Guess I should have mentioned that.) It is the system users that I have a LOT of troubles with, especially since I run a lot of services on my systems! Not only do I have the normal sets of system users, but some applications/services install their own new system users. (Bacula and Apache James for example) and it is all these system users such as wwwrun, named, dhcpd, tomcat, bacula etc that cause me a lot of headaches when their underlying ID value changes. And it makes it all but impossible to switch back and forth between an old version of openSuSE and a new version when shared mounted file systems are used between the two version. Think of Apache Tomcat service for example, both the UID and the GID that it requires are named tomcat:tomcat and the system user tomcat must have ownership of the docbase for the websites the tomcat service is handling. If the websites are mounted on a separate file system, that is then used on both the old version of openSuSE and the new version, and the UID and GID values for tomcat:tomcat are different under each version, how does one go about switching between the two systems and keep the ability of running the tomcat service on each?

Sometimes I have managed to work around this problem by going in and changing the underlying value of the UID or GID on one of the systems so that it is the same as the value used on the other system and then doing a chown on all the places I know about where needed. BUT that doesn’t always work! Sometimes I don’t remember/know where all the places where a particular UID or GID was used, that needs to be updated, or worse, trying to change the value of a UID or GID conflicts with another UID/GID that got that value assigned to a different system user or system group name. Then I have no idea where I need to make changes of ownership. Then I have to come up with a unique UID/GID value for that particular user or group on both versions of openSuSE.

Another important aspect of managing system users across different versions of openSuSE is that one must also be sure to set the memberships of which groups a particular system user (and regular user for that matter) needs to be a member of. I often have to do several iterations of booting back and forth between the two different versions of openSuSE in order to manually port all this information across from the older version to the newer version. (Wish YaST’s User/Group management tool had some way to print this info out!) It would be nice to also have this handled in a more automated fashion as well.

So I am not saying it is impossible to manage UIDs and GIDs across two different versions of openSuSE, but it is **** difficult to handle this manually. And keep in mind that I have to KNOW where all the places that a particular UID or GID is used, and make sure that I do the chown commands everywhere needed. This is almost impossible to do and often leads to an extraordinarily amount of debugging just to find out that I missed changing the UID or GID somewhere. (Error messages concerning permissions are often obtuse and misleading as well) Having a tool or a procedure to automate the setting up of system users, so that their underlying UID’s and GID’s are kept consistent when upgrading openSuSE would sure be helpful! IMHO of course!

If you are only running one or two services, such as Apache2, then managing the UID’s and GID’s by hand is not terribly difficult, I agree, but when running lots of services, using shared file systems across 2 or more versions of openSuSE, like I do, then managing all the different UIDs and GIDs becomes nearly impossible! I noted a few of the replies just brushed this problem aside, saying they either ignored the issue or managed it by hand. I have been trying to manage it by hand and my complaint is that that approach is costing me many hours of debugging problems that arise due to permission issues that often lead back to this issue of managing UID’s and GID’s across different versions of openSuSE.

7

A few observations after glancing through your post. I can only say: yes, that is life of a system manager running a shop of some size. :wink:

I would say that the starting point here is to identify those users you use (because there are many in /etc/passwd that are never used), preferable when you start using software that uses them. And then on a new system, before even starting all those extras like Apache, first change their UIDs to the old values. Of course carefully checking if there is no double usage. But I agree that it involves much “manual” work (like making notes, checking, changing, checking again)

When there is much to do there I probably would experiment with simply copying the set of system users from old to new. Doing some tests firsts if any files are owned by users you think are there “only for the case of …”

find / -uid <n>

I also agree that using the same UIDs (and GIDs) for the same users across all sysems you manage (and that have or are potential having closer relations) is very good, Sometimes, thinki ng of NFS, even a must. There are of course toold for that, like NIS.

And about which users are added to other groups then their primary one, just read /etc/group. But that is configured using usernames, not UIDs and thus works always.

And again, when you do the on-line upgrade, none of these should be in your way.

8

find supports querying uids and gids. Thus in case of changes between versions changing uids/gids automatically is straight forward.

9

Thanks again Henk for taking the time to reply, Much appreciate it!. Your suggestion to wait until the basic system has been installed and then add/change the UID’s and GID’s doesn’t really work either. The problem is that by the time the basic system is set up, it will already have added a lot of system users and groups, many of which will glom onto UID and GID values that are needed for a different system user or group from the previous installation of openSuSE.

Hmm Both you and karlmistelberger mention using the find command and that certainly seems like a good possibility. Gotta admit I hadn’t thought about using find to search for UID’s and GID’s, I generally only use it to find files by name ;). Seems like a pipe of find, grep, and chown might be put together to accomplish what I need (though still tedious). Can’t claim that I am a script kiddie, but will fiddle around with it.:\

/etc/group has been my friend also… Wish there was an equivalent file for UID’s

Hmm, Not sure I agree with you unless you are talking about doing an in-place upgrade over a previous installation? But that is not what I want to do. But this thought does conjure up another idea, perhaps I could somehow copy the old instillation into the new partition where I plan to install/“upgrade” the new version of openSuSE? Seems like this would also require making changes to the GRUB boot manager so that the clone is/appears as a legitimate installation. Then perhaps I could do the installation “upgrade” of the new version of openSuSE, over the top of the “old version” of openSuSE and still retain all the old UIDs/GID’s/name association? I am not sure how to do this but seems like a possible path…

I still feel that have the ability to export/import old UIDs and GIDs as part of the installation process of a new version of openSuSE would be more safe, reliable, and a good way to resolve this issue. It would have to be done before any files/directories are created/moved into the new partition. I don’t know enough about the openSuSE installation process, nor are my script writing abilities strong enough to attempt trying to come up with a solution such as this on my own, and would need the help of some openSuSE gurus to accomplish it.

Nice to have a couple of new paths to explore in trying to find an easier solution! Marc…

10

Sorry I’m late to this thread,
But I see that Henk asked but no answer was provided why you’re creating new installs at each version.
Why won’t you simply clone your 15.0 to a new partition and then upgrade one of the two to 15.1, then do the same thing… clone your 15.1 to a new partition and then upgrade one of the two to 15.2?

I think that would accomplish the same result in maybe less than 3 hrs you’re otherwise spending days and days manually exporting and importing, likely accompanied by hours of additional troubleshooting anything that didn’t work as expected.

TSU

11

BTW -
If I were in your position planning for this kind of an upgrade,
I’d also strongly consider converting your physical installation into virtualization of your choice.
You’d enjoy the benefits of maintaining a virtualized machine (Backups and restores, portability from one physical machine to another, storing a fully running backup which can be activated in seconds if needed, more efficient use of resources like over-provisioning, etc)
And
Your upgrade path would be very simple.
Locate the P2V tool or method to do the conversion.
Then, upgrade “offline” on your laboratory virtual network.
When fully upgraded and tested, “flip the switch” taking down the original machine while activating the new machine. You could probably even do the switchover in the middle of the day if you plan things right (or wait until off hours if you don’t want to take chances). The point is that it’s easy peasy to upgrade virtual machines and swap them in and out of Production.

TSU

12

Yes, the “online upgrade” is a form of in-place upgrade. Just all the packages are replaced by their new version. Dpending on your network speed, takes less then an hour and ready. I only recommend (and is often obly supported for) small steps like 15.2 > 15.2, no skipping in bewteens.

I normally always installed on a new system partition and often skipped versions to cling to the old one. Since about 15.0 (but maybe even 42,3) I did the online updates.

Of course I do now not have the “old” one to multi-boot with, but like @tsu2 suggests you can always first copy it to a partition (or even to another disk) as a fallback before doing the upgrade.

And the new feature of having $releasever in the URIs of your repositories make it even easier. No editing of repo definitions required anymore.

13

Although as Henk suggests the online upgrade using “zypper dup” is probaby easiest, I’d expect the offline upgrade should probably work, too (I haven’t actually tried since I’m used to running the online upgrade).

It’s likely the the DVD will probe the partitions, discover every partition with an openSUSE install on it, and then prompt you to select which one you want to upgrade.

TSU

14

Like you I assume that the installer Upgrade function from the ISO does likewise. But then you still have to do a “normal” update to get all the patches and recommended updates that were published after release. Doing it directly might be cleverer.

15

I need to be able to keep the old system running while I am integrating and upgrading the “new” services on the new system. I do short reboots, (with advance warnings given to users) to test out the new system each time I make changes, then reboot back to the old system while I continue to work on the upgrade. But one of the problems I keep running up against is mismatched UIDs and GIDs between the two versions, on shared mounted file systems. Hence this thread and questions…

I said I thought this might be a possible route in my last posting, so “great minds are thinking alike!” :wink: Trouble is I don’t have a clue how to integrate the cloned partition into GRUB and the boot manager so that the cloned version is also recognized as a possible system to upgrade. Google identifies a few possible locations where instructions are given, but so far I have found that these sites it found are outdated and don’t seem to apply anymore. I already tried this approach once, following instructions on How to Migrate the Root Filesystem to a New Disk | Support | SUSE and quickly ran into difficulties because the steps could not be followed. I will keep looking around for better/more current instructions (would appreciate any links you know about). From what I can tell, integrating a new cloned partition into the boot manager/loader seems pretty complicated.

Another issue is that a cloned version has an fstab file that is incorrect and must be manually changed prior to trying to upgrade it, particularly for the root partition. The YaST partitioner does not provide the UUID information for the new partition unfortunately. What’s worse is that I compared the fstab files between the old version of openSuSE and the new version I have currently installed. Every UUID for every mounted file system was different between the two (even though I am mounting the identical file partitions!) so I am currently at a total loss on how to set up the fstab file for a cloned version and am beginning to feel that trying to clone my old system into a new partition before updating it is going to be too difficult!:\

No argument, if only I knew how to do this. I have just started exploring this path as I mentioned. Much appreciate your taking some time to join in and lend me a helping hand!

Thanks again, Marc.

16

The SUSE knowledge base article says:

“The range of environments that can impact the necessary steps to migrate a root filesystem makes it near impossible to cover every case. Some environments could require tweaks in the steps needed to make this migration a success. As always in administration, have backups ready and proceed with caution.”

Actually with UEFI there are only a few items to check. ‘blkid’ displays UUIDs. Adjust /etc/fstab and /etc/default/grub accordingly. Reinstall grub: Grub – EFI – Btrfs | Karl Mistelberger

17

The reason is, the UIDs and GIDs for this part of the (Linux) File System Hierarchy (FHS) are part of the System UID/GID range of values …

  • And, the associated pseudo-users get regenerated by the installation procedures at each new installation …

The Linux Standards definition is here: <https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/usernames.html&gt;.
The “User ID Ranges” are defined on the next page – the UID values 100 to 499 are “reserved for dynamic allocation by system administrators and post install scripts” …

Yes, it’s a bit of a pain but, it can usually be resolved by the use of “vipw” and “vigr” – whether or not a system script could safely handle this edit process is a moot point …
[HR][/HR]Question on a side-issue:

  • Why are the “Web Sites” not located under the top-level “/srv/” directory?
18

Yes, the “bit of pain” is what I am trying to resolve and I think it is actually fairly difficult to solve in such a way that one can use some of the same file system mount points on two different versions of openSuSE and keep both systems runable. The solution requires more than simply changing the underlying UID/GID values in a database somewhere. If one wants to change a UID/GID value then one has to traverse the entire set of mounted filesystems and find/change the directories and file’s UID/GID values that are effected by these changes also and do a chown on them. I did not know about the vipw and vigr commands, thanks for the tip, and could be helpful.

I am currently attempting to put together a script that will accomplish porting UID/GID names and values from one version of openSuSE to another version. So far I have gotten a script going that finds and reports the full path of all files/directories with a particular UID, but that is only a start. Next I am going to write a script to iterate through one (older) password file to retrieve the name/UID and name/GID pairs, replace the corresponding name/UID and name/GID in the other (newer) password file, then use the script I now have to find all files/dirs with the newer UID/GID and do a chown on them in order to update the file/dir UID and GID with the corrected (normalized) UID/GID from the older password file. (This may be hard to grok, it is hard to explain but you should be able to understand where I am heading.)

This would be so much easier to accomplish if the openSuSE installation process would allow the user to import an older password, group, gshadow, and shadow files before any directories are created and any files are installed. It has been suggested to clone my older version of openSuSE, and do an upgrade on it. That effectively would accomplish exactly what I am after, i.e. the UID/GID values would not change between the older and newer versions of openSuSE. But there are other issues involving fstab and boot loading that seem even harder to solve on a cloned system, so I am going back to my original plan and write a somewhat complex script to redefine the UID/GID values on a newly installed system so that they are the same as my previously installed system.
[HR][/HR]

The web sites are located on an NTFS file system. These are dual boot systems and sometimes the system must run under Windows for awhile. We keep the web service running despite what operating system we are running under. There are both remote and local data gathering and other operations that only run under Windows yet affect what is stored in these websites. Much of the time however we are running under openSuSE, and unlike the Windows IIS server, Apache and Tomcat don’t care about whether the web sites are on an NTFS or some other file system type. So NTFS is the common denominator, hence the mount under /websites and not under /srv. While it might be possible to mount /srv as an NTFS partition I chose not to explore that possibility just to keep things as simple as possible.