Stylus position significantly off

on a tablet/notebook with touchscreen, the stylus position is completely off, meaning that touching the screen with the stylus causes a mouse pointer at a drastically different position.
touching the screen with a finger however works fine.
how can the stylus position be calibrated without affecting the finger touch?

the display is rotated to landscape.
it looks like the stylus position is not off, but that the stylus operates on the unrotated screen.

Adjusting rotation of screen works better with Wayland.
For X11 you need manual settings.

Which desktop environment? X11 session or Wayland?

If not sure post the results from inxi -G.

For the input devices run sudo libinput list-devices. If the command not found, install libinput-tools first.

AFAIU, the pen is handled as a tablet/pen device and has its own coordinate mapping. Depending on the hardware it may be supported by the wacom driver. Check if the wacom driver is involved with lsmod | grep wacom

must be

sudo libinput list-devices

Yep, typo. (Fixed)

inxi -G
Graphics:
  Device-1: Intel Raptor Lake-P [Iris Xe Graphics] driver: i915 v: kernel
  Display: x11 server: X.Org v: 1.21.1.11 with: Xwayland v: 24.1.1 driver:
    X: loaded: modesetting unloaded: fbdev,vesa dri: iris gpu: i915
    resolution: 2560x1600~60Hz
  API: OpenGL v: 4.6 Mesa 23.3.4 renderer: Mesa Intel Graphics (RPL-U)
sudo libinput list-devices
...
Device:           HIMX1234:00 4858:121A
Kernel:           /dev/input/event9
Group:            6
Seat:             seat0, default
Size:             164x273mm
Capabilities:     touch 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           HIMX1234:00 4858:121A Stylus
Kernel:           /dev/input/event10
Group:            6
Seat:             seat0, default
Size:             167x267mm
Capabilities:     tablet 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   none
Rotation:         n/a
...
lsmod | grep wacom

gives no result

xinput list-props  'HIMX1234:00 4858:121A Stylus'

Device 'HIMX1234:00 4858:121A Stylus':
        Device Enabled (179):   1
        Coordinate Transformation Matrix (181): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        libinput Calibration Matrix (378):      1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        libinput Calibration Matrix Default (379):      1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
        libinput Send Events Modes Available (319):     1, 0
        libinput Send Events Mode Enabled (320):        0, 0
        libinput Send Events Mode Enabled Default (321):        0, 0
        Device Node (322):      "/dev/input/event10"
        Device Product ID (323):        18520, 4634

I tried

xinput set-prop  'HIMX1234:00 4858:121A Stylus' 'Coordinate Transformation Matrix'   0 1 0 -1 0 1 0 0 1

but it results in

X Error of failed request:  BadMatch (invalid parameter attributes)
  Major opcode of failed request:  131 (XInputExtension)
  Minor opcode of failed request:  57 ()
  Serial number of failed request:  20
  Current serial number in output stream:  21
xinput set-prop   'HIMX1234:00 4858:121A Stylus Pen (0x8020132c)' 'Coordinate Transformation Matrix'   0 1 0 -1 0 1 0 0 1

seems to have made it

That’s good (just a rotation issue). You can make it permanent if desired. As you are still using X11 desktop session, then create an Xorg file eg
sudo nano /etc/X11/xorg.conf.d/90-himx-stylus.conf
with

Section "InputClass"
    Identifier "HIMX Stylus Rotation Fix"
    MatchProduct "HIMX1234:00 4858:121A Stylus Pen"
    Option "TransformationMatrix" "0 1 0 -1 0 1 0 0 1"
EndSection

The product name must match exactly (including capitalization and spaces) to take effect. It should work the next time the Xorg is (re)started.