OpenSwan Client sends "Malformed Packet" of type ESP

Hi forum,

I found similar but not equal problems in this forum so I post this.:slight_smile:

The OpenSwan-Daemon is sending “malformed packages” of type “esp” to a CISCO-router - which the CISCO never gets. That happens when I start openl2tpd to create the tunnel of the VPN.

Log of WireShark:


(Phase 1 Main Mode and Quick Mode logs are OK!)
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef)
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef) 
[Malformed Packet]
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef) 
[Malformed Packet]
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef) 
[Malformed Packet]
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef) 
[Malformed Packet]
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef)
client-ip      cisco-ip           ESP        ESP(SPI=0x87fbfeef) 
[Malformed Packet]

The PSK used is absolutely correct. I changed it and ipsec failed with
errors

“Informational Exchange message is invalid because it has a Message
ID of 0” from startup log
and
“inserting event EVENT_CRYPTO_FAILED, timeout in 300 seconds for#1”
from messages-log with “plutodebug=all” set
and
“MESZ: %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from
failed its sanity check or is malformed” from CISCO-log.

So this can’t be the problem.

The SPI’s are equal as far as both outputs from “setkey -DP” and “setkey
-D” compared to the CISCO’s “show crypto ipsec sa” output are the same.

The question is : Why is ipsec sending one good ESP packet followed by
malformed packages and why are they malformed?


------------IPSEC.CONF------------
# basic configuration
config setup
nat_traversal=yes
# exclude networks used on server side by adding %v4:!a.b.c.0/24
virtual_private=%v4:10.255.255.1/32,%v4:192.168.0.0/24
# OE is now off by default. Uncomment and change to on, to enable.
OE=off
# which IPsec stack to use. netkey,klips,mast,auto or none
protostack=netkey
fragicmp=yes # only for KLIPS - disable PMTU
nhelpers=0


# Add connections here

conn L2TPPSKCLIENT
#
# ----------------------------------------------------------
# Use a Preshared Key.
authby=secret
pfs=no # default is yes
rekey=yes
keyingtries=3
keyexchange=ike
type=transport
#
# Specify type of encryption for ISAKAMP SA (IPsec Phase 1)
# Cipher= 3des, Hash = sha, DH-Group = 2
ike=3des-sha1-modp1024
# Specify type of encryption for IPSEC SA (IPsec Phase 2)
# Cipher= 3des, Hash = sha, DH-Group = 2
phase2=esp
phase2alg=3des-sha1
#
# Specifiy liftime of ike and key management
# Note: Should match values on remote end
ikelifetime=3600s
salifetime=600s
#
# Keep connection alive through DPD (Dead Peer Detection)
dpddelay=30
dpdtimeout=120
dpdaction=clear
left=%defaultroute
leftid=87.0.0.0
leftprotoport=17/1701
#
# ----------------------------------------------------------
# The remote server.
#
# Connect to the server at this IP address.
right=217.0.0.0
rightprotoport=17/1701

auto=add


## Disabling OE -- I think this is the old notation
conn block
auto=ignore

conn private
auto=ignore

conn private-or-clear
auto=ignore

conn clear-or-private
auto=ignore

conn clear
auto=ignore

conn packetdefault
auto=ignore
---------------------------------------------------

Thanks for your help.

Markus

some more information on this problem.:wink:

It’s really remarkable, because there is no indication of errors anywhere.

Problem:

  • I can set up a ipsec (transport) connection - packets are encapsulated in ESP when they are transported.
  • I can set up a openl2tp connection - when ipsec IS NOT STARTED!
  • The ipsec keys and all following handshake on port 500 (isakmp) function properly.
  • Ipsec communicates on port 1701 for both sides (CISCO <—> openswan).
  • The SPI’s are equal at the time of communication for inbound and outbound on both systems.
  • Setkey -D(P) showing the right routing information (as I can see).
217.0.0.0[any] 87.0.0.0[any] udp
fwd prio high + 1073739744 ipsec
esp/transport//unique#16385
created: Jan 21 11:08:23 2009 lastused:
lifetime: 0(s) validtime: 0(s)
spid=4202 seq=12 pid=32726
refcnt=1
87.0.0.0[any] 217.0.0.0[any] udp
out prio high + 1073739744 ipsec
esp/transport//unique#16385
created: Jan 21 11:23:16 2009 lastused:
lifetime: 0(s) validtime: 0(s)
spid=4209 seq=0 pid=32726
refcnt=1
  • CISCO shows only sendig packages to client on port 1701
  • Client sends 5-6 packages and CISCO does the same, but I don’t see any incoming on port UDP 1701 ON BOTH SYSTMES. So ESP packages are lost in space.

Conclusion:

  1. ESP packets are encrypted correctly, …because the SPI’s are equal and none of both systems complain that.Except the wireshark of the client, which says some packets are malformed, but CISCO has no send or receive errors and has the right count of decrypt/encrypt packets.
  2. ESP packets are send correctly, …because port on both sides is 1701 on any send message.
  3. The tunnel with l2tp - without ipsec - function properly, so ipsec must be the problem.

Questions:

  • A) There are my ESP packets and how can I find them?
  • B) Is there a way to look into ESP packets except of tcpdump (which I can’t compile with crypto).
  • C) Could it be that the NETKEY module of the kernel is the problem and can I trace it’s output somehow?

Please help!

Markus