Try:
First mount your root partition (includes use of cryptsetup) at “/mnt”
Then:
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt
mount -a ### mount everything else needed
shim-install
exit ### exit from chroot
reboot
This is for a UEFI install. For non-uefi grub install, you have to use “grub2-install “location” where “location” is the device on which to install grub – typically the MBR (”/dev/sda" or similar) or the device for “/boot”.
Note the use of “–rbind” for “/sys”. That’s needed for EFI access. And you have to use EFI booting for the rescue system for this to work.
Your mistake seems to be that you missed those “–bind” and “–rbind” mounts. And if you are using “btrfs” for the root file system, then you need that “mount -a” to mount the “btrfs” subvolumes.