IPsec negotiation failure

I have set up IPsec IPv4 transport between two internet servers using racoon with certificate authentication (transport, not tunnel, because these two machines don’t have networks behind them).

Things sometimes work. Sometimes however I will “ping” or “telnet REMOTE 25” and nothing will happen. The packets just seem to be dropped. If I try it from the other machine, then things start working in both directions for a while. Therefore it seems that there is negotiation failure of some sort. tcpdump on the target of the initial ping shows ESP packets arriving, but no response going back. I am using “exchange_mode aggressive, main;” Should I try not using aggressive? Both machines have port 500 open in their firewalls.

The relevant portion of racoon.conf is
remote REMOTEIPv4ADDR
{
exchange_mode aggressive, main;
ca_type x509 “CA.pem”;
my_identifier asn1dn;
peers_identifier asn1dn;
certificate_type x509 “LOCAL.pem” “LOCAL.key”;
verify_identifier on;
verify_cert on;
proposal {
encryption_algorithm aes;
hash_algorithm sha1;
authentication_method rsasig;
dh_group 2;
}
}

sainfo anonymous
{
pfs_group modp3072;
lifetime time 1 hour;
encryption_algorithm aes;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

Please post ouput between CODE tags, if not, it’s hardly readable.

Sorry. I am not familiar with this forum system.

A is running openSUSE 12.1
kernel-default-3.1.10-1.13.1.x86_64
ipsec-tools-0.7.3-19.1.3.x86_64
B is running openSUSE 11.4
kernel-default-2.6.37.6-0.11.1.x86_64
ipsec-tools-0.7.3-9.1.x86_64

Here’s host A:


% sudo tcpdump -i eth0 host B.B.B.B
root's password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:08:13.066710 IP A.com.openvpn > B.com.46263: UDP, length 69
10:08:13.076694 IP B.com.46263 > A.com.openvpn: UDP, length 69
10:08:26.888573 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x2): ESP(spi=0x0acd7254,seq=0x2), length 116
10:08:27.895785 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x3): ESP(spi=0x0acd7254,seq=0x3), length 116
10:08:28.091630 IP A.com.openvpn > B.com.46263: UDP, length 69
10:08:28.101425 IP B.com.46263 > A.com.openvpn: UDP, length 69
10:08:28.952222 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x4): ESP(spi=0x0acd7254,seq=0x4), length 116
10:08:29.959684 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x5): ESP(spi=0x0acd7254,seq=0x5), length 116
10:08:30.967896 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x6): ESP(spi=0x0acd7254,seq=0x6), length 116
10:08:31.975858 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x7): ESP(spi=0x0acd7254,seq=0x7), length 116
10:08:33.027797 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x8): ESP(spi=0x0acd7254,seq=0x8), length 116
10:08:34.048002 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0x9): ESP(spi=0x0acd7254,seq=0x9), length 116
10:08:35.055714 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0xa): ESP(spi=0x0acd7254,seq=0xa), length 116
10:08:36.063927 IP B.com > A.com: AH(spi=0x0fa2d2a2,seq=0xb): ESP(spi=0x0acd7254,seq=0xb), length 116
^C
14 packets captured
23 packets received by filter
0 packets dropped by kernel

Here’s host B:


% ping A.A.A.A
PING A.A.A.A (A.A.A.A) 56(84) bytes of data.
^C
--- A.A.A.A ping statistics ---
15 packets transmitted, 0 received, 100% packet loss, time 14193ms

Exit 1

Now reverse the direction. Starting on Host A makes it work:


% ping B.B.B.B
PING B.B.B.B (B.B.B.B) 56(84) bytes of data.
64 bytes from B.B.B.B: icmp_seq=1 ttl=59 time=9.81 ms
64 bytes from B.B.B.B: icmp_seq=2 ttl=59 time=9.90 ms
64 bytes from B.B.B.B: icmp_seq=3 ttl=59 time=9.87 ms
^C
--- B.B.B.B ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.814/9.862/9.904/0.120 ms

Here is A’s /etc/racoon/racoon.conf:


path certificate "/etc/racoon/cert";

# "log" specifies logging level.  It is followed by either "notify", "debug" or "debug2".
#log debug;

# "padding" defines some padding parameters.  You should not touch these.
padding
{
    maximum_length 20;    # maximum padding length.
    randomize off;        # enable randomize length.
    strict_check off;    # enable strict check.
    exclusive_tail off;    # extract last one octet.
}

# if no listen directive is specified, racoon will listen on all
# available interface addresses.
listen
{
    isakmp A.A.A.A [500];
}

# Specify various default timers.
timer
{
    # These value can be changed per remote node.
    counter 5;        # maximum trying count to send.
    interval 20 sec;    # maximum interval to resend.
    persend 1;        # the number of packets per send.

    # maximum time to wait for completing each phase.
    phase1 30 sec;
    phase2 15 sec;
}

remote B.B.B.B
{
    exchange_mode aggressive, main;
    ca_type x509 "CA.pem";
    my_identifier asn1dn;
    peers_identifier asn1dn;
    certificate_type x509 "A.com.pem" "A.com.key";
    verify_identifier on;
    verify_cert on;
    proposal {
        encryption_algorithm aes;
        hash_algorithm sha1;
        authentication_method rsasig;
        dh_group 2;
    }
}

sainfo anonymous
{
    pfs_group modp3072;
    lifetime time 1 hour;
    encryption_algorithm aes;
    authentication_algorithm hmac_sha1;
    compression_algorithm deflate;
}

Host B’s racoon.conf is identical except for swapping A and B.

Here is A’s setkey.conf:


#!/usr/sbin/setkey -f

# Flush the SAD and SPD
flush;
spdflush;

# Security policy for B.com
spdadd A.A.A.A B.B.B.B any -P out ipsec
           esp/transport//require
           ah/transport//require;
spdadd B.B.B.B A.A.A.A any -P in ipsec
           esp/transport//require
           ah/transport//require;