I do have the key in the keyring :
# keyctl list @u
1 key in keyring:
678611289: --alswrv 0 0 user: cryptsetup
# keyctl print 678611289
keyctl_read_alloc: Permission denied
The key is there, I cannot print it, but it is there. My boot is quite fast (SSD and good cpu).
So, I think the issue might be related to PAM.