Ajitabh Pandey's Soul & Syntax

Exploring systems, souls, and stories – one post at a time

Tag: SNAT

  • Using SNAT for Highly Available Services

    Problem

    Often network based services are restricted to a particular source IP address. A common example is SNMP. A good system/network administrator will restrict access to the SNMP daemon from a particular host, usually a central management server. Sometimes these central management servers are HA pair. Under these circumstances, a service address can be used for the active node. This service address has access to the desired networking resource. Heartbeat usually will start this service IP address as a resource on the active node. This will result in the Active node taking over the IP address, which enables the node to listen on that IP address for incoming requests. But this still does not solve the problem of active node attempting to access a network resource, because all packets originating from this node will bear the primary IP address of this node and not the secondary address(es) or aliased address(es).

    Solution

    For such cases, SNAT (Source Network Address Translation) can be useful. Using SNAT we can ask the kernel to change the source IP addresses on all outgoing packets. But the IP address which we want on our packets must be present either as a primary or secondary or aliased IP address. This can be checked as:

    # ip addr show bond0
    6: bond0:  mtu 1500 qdisc noqueue
        link/ether 00:18:fe:89:df:d8 brd ff:ff:ff:ff:ff:ff
        inet 192.168.1.3/16 brd 172.16.255.255 scope global bond0
        inet 192.168.1.2/16 brd 172.16.255.255 scope global secondary bond0:0
        inet 192.168.1.1/16 brd 172.16.255.255 scope global secondary bond0:1
        inet6 fe80::218:feff:fe89:dfd8/64 scope link
           valid_lft forever preferred_lft forever

    or

    # ifconfig bond0
    bond0     Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              inet addr:192.168.1.3  Bcast:172.16.255.255  Mask:255.255.0.0
              inet6 addr: fe80::218:feff:fe89:dfd8/64 Scope:Link
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
              RX packets:53589964 errors:0 dropped:0 overruns:0 frame:0
              TX packets:25857501 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:40502210697 (37.7 GiB)  TX bytes:4148482317 (3.8 GiB)

    Instead of specifying a interface, all interfaces can also be viewed using:

    # ip addr show
    1: lo:  mtu 16436 qdisc noqueue
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0:  mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
        link/ether 00:18:fe:89:df:d8 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::218:feff:fe89:dfd8/64 scope link
           valid_lft forever preferred_lft forever
    3: eth1:  mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
        link/ether 00:18:fe:89:df:d8 brd ff:ff:ff:ff:ff:ff
    4: sit0:  mtu 1480 qdisc noop
        link/sit 0.0.0.0 brd 0.0.0.0
    6: bond0:  mtu 1500 qdisc noqueue
        link/ether 00:18:fe:89:df:d8 brd ff:ff:ff:ff:ff:ff
        inet 192.168.1.3/16 brd 172.16.255.255 scope global bond0
        inet 192.168.1.2/16 brd 172.16.255.255 scope global secondary bond0:0
        inet 192.168.1.1/16 brd 172.16.255.255 scope global secondary bond0:1
        inet6 fe80::218:feff:fe89:dfd8/64 scope link
           valid_lft forever preferred_lft forever

    or

    # ifconfig
    bond0     Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              inet addr:192.168.1.3  Bcast:172.16.255.255  Mask:255.255.0.0
              inet6 addr: fe80::218:feff:fe89:dfd8/64 Scope:Link
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
              RX packets:53587551 errors:0 dropped:0 overruns:0 frame:0
              TX packets:25855600 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:40501872867 (37.7 GiB)  TX bytes:4148267377 (3.8 GiB)
    
    bond0:0   Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              inet addr:192.168.1.2  Bcast:172.16.255.255  Mask:255.255.0.0
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
    
    bond0:1   Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              inet addr:192.168.1.1  Bcast:172.16.255.255  Mask:255.255.0.0
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
    
    eth0      Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              inet6 addr: fe80::218:feff:fe89:dfd8/64 Scope:Link
              UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
              RX packets:53587551 errors:0 dropped:0 overruns:0 frame:0
              TX packets:25855600 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:40501872867 (37.7 GiB)  TX bytes:4148267377 (3.8 GiB)
              Interrupt:185
    
    eth1      Link encap:Ethernet  HWaddr 00:18:FE:89:DF:D8
              UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
              Interrupt:193
    
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:536101 errors:0 dropped:0 overruns:0 frame:0
              TX packets:536101 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:59243777 (56.4 MiB)  TX bytes:59243777 (56.4 MiB)

    My NICs are bonded and hence bond0 is the interface I use.

    Setting Up SNAT

    In Linux IPTables can be used to setup SNAT.
    To change the source IP address of all packets going out of the box to anywhere, following rule can be used:

    $ sudo /sbin/iptables -t nat -A POSTROUTING -o bond0 -j SNAT --to-source 192.168.1.1

    The result can be seen as follows:

    $ sudo /sbin/iptables -t nat -L
    Chain PREROUTING (policy ACCEPT)
    target     prot opt source               destination
    
    Chain POSTROUTING (policy ACCEPT)
    target     prot opt source               destination
    SNAT       all  --  anywhere             anywhere            to:192.168.1.1
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination

    I normally restrict SNAT to selected services and destination IP addresses only. The following three IPTables command respectively translates the source address for all packets destined for 10.199.65.191 to 192.168.1.1, only ICMP packets destined for 192.168.2.4 and all packets destined for network 192.168.1.0/24:

    $ sudo /sbin/iptables -t nat -A POSTROUTING -d 10.199.65.191 -o bond0 -j SNAT --to-source 192.168.1.1
    $ sudo /sbin/iptables -t nat -A POSTROUTING -d 192.168.2.4 -p ICMP -o bond0 -j SNAT --to-source 192.168.1.1
    $ sudo /sbin/iptables -t nat -A POSTROUTING -d 192.168.1.0/24 -o bond0 -j SNAT --to-source 192.168.1.1

    The result of all these commands can be seen as:

    $ sudo /sbin/iptables -t nat -L
    Chain PREROUTING (policy ACCEPT)
    target     prot opt source               destination
    
    Chain POSTROUTING (policy ACCEPT)
    target     prot opt source               destination
    SNAT       all  --  anywhere             anywhere            to:192.168.1.1
    SNAT       all  --  anywhere             10.199.65.191       to:192.168.1.1
    SNAT       icmp --  anywhere             192.168.2.4         to:192.168.1.1
    SNAT       all  --  anywhere             192.168.1..0/24     to:192.168.1.1
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination

    Setting Heartbeat and IPTables for SNAT

    The /etc/ha.d/haresources file in heartbeat can be set to accept the desired IP address as a resource and associate it with a script which can start/stop/restart these IPTables rules.

    $ sudo vi /etc/ha.d/haresources
    node01 192.168.1.1 iptables

    Red Hat and Fedora has such a script and which is located in /etc/init.d/iptables. This script reads a file /etc/sysconfig/iptables, which contains various rules in iptables-save format. I had created a similar script for Debian and derivatives distributions which reads the rules from /etc/iptables file. The script is given below:

    #! /bin/sh
    # Script      - iptables
    # Description - Read IPTables rule from a file in iptables-save format.
    # Author      - Ajitabh Pandey 
    #
    PATH=/usr/sbin:/usr/bin:/sbin:/bin
    DESC="IPTables Configuration Script"
    NAME=iptables
    DAEMON=/sbin/$NAME
    SCRIPTNAME=/etc/init.d/$NAME
    
    # Exit if the package is not installed
    [ -x "$DAEMON" ] || exit 0
    
    # Load the VERBOSE setting and other rcS variables
    [ -f /etc/default/rcS ] && . /etc/default/rcS
    
    if [ ! -e /etc/iptables ]
    then
            echo "no valid iptables config file found!"
            exit 1
    fi
    
    case "$1" in
      start)
            echo "Starting $DESC:" "$NAME"
            /sbin/iptables-restore /etc/iptables
            ;;
      stop)
            echo "Stopping $DESC:" "$NAME"
            $DAEMON -F -t nat
            $DAEMON -F
            ;;
      restart|force-reload)
            echo "Restarting $DESC:" "$NAME"
            $DAEMON -F -t nat
            $DAEMON -F
            /sbin/iptables-restore /etc/iptables
            ;;
      *)
            echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
            exit 3
            ;;
    esac

    Following is a sample iptables rules file, in iptables-save format.

    *nat
    :PREROUTING ACCEPT [53:8294]
    :POSTROUTING ACCEPT [55:11107]
    :OUTPUT ACCEPT [55:11107]
    
    # Allow all ICMP packets to be SNATed
    -A POSTROUTING  -p ICMP -o bond0 -j SNAT --to-source 192.168.0.1
    
    # Allow packets destined for SNMP port (161) on local network to be SNATed
    -A POSTROUTING -d 192.168.0.0/24 -p tcp -m tcp --dport snmp -o bond0 -j SNAT --to-source 192.168.0.1
    -A POSTROUTING -d 192.168.0.0/24 -p udp -m udp --dport snmp -o bond0 -j SNAT --to-source 192.168.0.1
    
    # These are for the time servers on internet
    -A POSTROUTING -p tcp -m tcp --dport ntp -o bond0 -j SNAT --to-source 192.168.0.1
    COMMIT
    
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [144:12748]
    COMMIT