Here is an interesting problem I encountered the other day, as I was trying to assist a friend to connect to office networks located in two different boroughs of the great City of New York.
He had an IP network in Manhattan, that used private, non-routable IP numbers. On this network sat an application server, which for a variety of reasons was either extremely difficult or impossible to reconfigure. This application server was capable of communicating with other machines on this private IP network, but not with anything else. To make things worse, the application server also utilized UDP broadcast packets to transmit information to client computers.
My friend also had a branch office in Brooklyn. This is where he had created a set of workstations that he had hoped to connect to his Manhattan server. Short of using software like Symantec's PC-Anywhere, and setting up a mirror workstation in Manhattan for each computer used in Brooklyn, is there another solution?
There is indeed... but it involves a convoluted setup making use of two Linux servers as gateways, IP tunnelling, IP masquerading, and a small, proprietary program that takes care of the retransmission of UDP broadcast packets.
System Configuration
My friend's network configuration is illustrated by the diagram below.
His Manhattan network uses the private IP number range 192.168.1.0-192.168.1.255 (or, in modern notation, 192.168.1.0/24). He is connected to the Internet using a router supplied by his Internet Service provider; the ISP also gave him a set of routable IP numbers in the range of 172.16.1.0-172.16.1.31* (i.e., 172.16.1.0/27), of which the Internet router uses the address 172.16.1.30. He bridges these two IP subnets using a Linux machine that has two network cards; one connects the machine to the private network using the IP number 192.168.1.126, while the other connects the machine to the Internet router using the IP number 172.16.1.12.
In Brooklyn, my friend set up his private network using the IP number range 192.168.2.0/24. His ISP provided here the IP number range 172.17.2.0/28; the Internet router's address is 172.17.2.14. He set up a Linux machine here as well, with two network cards; one connects to the private network using the IP number 192.168.2.126, while the other connects to the Internet router using the IP number 172.17.2.10.
*The keen observer may have noticed that the subnet 172.16.0.0/12 is also reserved for private networks. I am using these IP numbers for demonstration purposes only; in the actual situation, my friend used the IP number ranges assigned to his networks by his Internet Service Provider.
Requirements
With this physical configuration in place, my friend needed to set up the Linux machines to perform the following functions:
- Provide a "tunnel" between the two private networks that have non-routable IP numbers
Ordinarily, there is no way for my friend's Brooklyn machines to send packets addressed to the Manhattan private network or vice versa. That is because any packet with a destination address such as 192.168.1.1 or 192.168.2.3 will be rejected by the Internet; the public Internet doesn't know how to forward these packets, there is no route to these IP numbers. IP tunnelling is a method that allows such an IP packet to be encapsulated within another IP packet, this one with a valid destination address; specifically, the external address of the Linux machine on the target network. This Linux machine, also equipped with IP tunnelling, will be able to extract the original packet from the encapsulated one and route it to the true destination host. - "Pretend" that packets sent to the Manhattan application server from Brooklyn actually originated in Manhattan
The Manhattan application server can only talk to machines whose IP number is within the 192.168.1.0/24 range. IP masquerading, normally used as a security/firewall feature, allows the Manhattan Linux machine to alter packets coming from Brooklyn to make them appear locally originated. It will also ensure (at least for connection-oriented TCP sockets and ICMP echo replies used in responses to the ping command) that replies from the Manhattan application server are rerouted to the true originating computer in Brooklyn. - Forward UDP broadcasts from Manhattan to Brooklyn
Because the Manhattan application server uses UDP broadcasts to disseminate some information to workstations, it is necessary to write a custom program that listens to such broadcasts in Manhattan, forwards them to its counterpart in Brooklyn, which in turn rebroadcasts them on the Brooklyn private network.
Basic Network Configuration
Before advanced networking features, such as IP tunnelling, can be used, they must be enabled in the Linux kernel. Typically, this requires that the Linux kernel be recompiled with the appropriate options. Here is just a quick reminder of the commands used to recompile the kernel (remember, you must be root to do this):
# cd /usr/src/linux # make config compile-options are entered here # make dep # make clean # make bzlilo
In addition to IP tunnelling, in Manhattan we must also enable IP firewall functions, and specifically IP masquerading. In both Manhattan and Brooklyn we must also enable IP forwarding in order to ensure that the Linux machines function as routers. Note that in order for IP forwarding to work, it may also be necessary to have a command similar to the following somewhere in the system startup files:
echo 1 >/proc/sys/net/ipv4/ip_forward
On both Linux machines, it is also necessary to enable both network cards, with properly assigned IP numbers. Different Linux distributions use different system startup files, so it is not possible to provide detailed instructions that are not distribution-specific. The bottom line is that the startup files must contain the necessary ifconfig commands, as described below, to configure the two network interfaces.
In Manhattan, the commands will be this (assuming that eth0 is the interface that connects the Linux machine to the private network, and eth1 is the one that connects it to the Internet router):
ifconfig eth0 192.168.1.126 broadcast 192.168.1.0 netmask 255.255.255.0 ifconfig eth1 172.16.1.12 broadcast 172.16.1.31 netmask 255.255.255.224
On earlier Linux versions, it was also necessary to issue route commands to add the two subnets to the kernel routing table. Beginning with kernel 2.2.0, this is no longer necessary; routes are implied, that is, entries are added to the routing table automatically when an interface is activated using ifconfig. We still need one route command, the one that specifies the ISP's router as the default gateway to the Internet (i.e., the route to which all packets that cannot be routed locally will be sent):
route add default gw 172.16.1.30
In Brooklyn, the configuration is similar, but the numbers are reversed. First, the Ethernet cards and the default route are set up thus:
ifconfig eth0 192.168.2.126 broadcast 192.168.2.0 netmask 255.255.255.0 ifconfig eth1 172.17.2.10 broadcast 172.17.2.15 netmask 255.255.255.240 route add default gw 172.17.2.14
With both Linux machines up and running, the Internet connection can be tested by pinging one from the other. E.g., in Manhattan:
$ ping 172.17.2.10 PING 172.17.2.10 (172.17.2.10): 56 data bytes 64 bytes from 172.17.2.10: icmp_seq=0 ttl=128 time=128.5 ms 64 bytes from 172.17.2.10: icmp_seq=1 ttl=128 time=128.5 ms 64 bytes from 172.17.2.10: icmp_seq=2 ttl=128 time=148.4 ms --- 172.17.2.10 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 128.5/135.1/148.4 ms
IP Tunnelling Configuration
IP tunnels (also called IP-IP, or IPIP tunnelling, or IPIP encapsulation) are also enabled using the ifconfig utility. However, there's a trick; it is not enough to simply say something like ifconfig tunl0 and get on with it. Instead, a set of three commands is necessary.
In Manhattan:
ifconfig tunl0 192.168.1.126 netmask 255.255.255.255 pointopoint 172.17.2.10 route add -net 192.168.2.0 netmask 255.255.255.0 gw 172.17.2.10 tunl0 route del -host 172.17.2.10
The first of these three commands established the tunl0 interface as a tunnel to the remote machine 172.17.2.10 (which, as you may notice, is the "public" address of the Brooklyn Linux machine.) Note the use of the pointopoint keyword; without this keyword, the interface will not work properly.
The second command adds the entire Brooklyn private network, 192.168.2.0/24, to the routing table of the Manhattan Linux machine. The combination of these two commands means, in plain English: if you encounter a data packet with a destination address on the Brooklyn private network, wrap it inside another packet and send it to the public address of the Brooklyn Linux machine, which will then remove the wrapping and forward the data packet to its true destination on the private network there.
So what about the third line, which deletes a route? Remember the stuff about implied routes? The problem is that the ifconfig tunl0 command not only configures the tunl0 interface, but also adds an incorrect entry to the kernel routing table, one that says that packets sent to 172.17.2.10 must be routed through tunl0. This is not correct; 172.17.2.10 is the other endpoint of the tunnel alright, but packets with this destination address must go through the default gateway, i.e., the Internet router supplied by the ISP. Hence, this implied route must be removed.
In Brooklyn, a similar set of three lines is necessary:
ifconfig tunl0 192.168.2.126 netmask 255.255.255.255 pointopoint 172.16.1.12 route add -net 192.168.1.0 netmask 255.255.255.0 gw 172.16.1.12 tunl0 route del -host 172.16.1.12
Once again, the first line establishes a tunnel from the Brooklyn Linux machine to the public address of the machine in Manhattan; the second line establishes a route from Brooklyn to the Manhattan private network; and the third line deletes a superfluous implied route.
If all went well, it should now be possible to ping the Brooklyn machine's private address from Manhattan, or vice versa. In Manhattan:
$ ping 192.168.2.126 PING 192.168.2.126 (192.168.2.126): 56 data bytes 64 bytes from 192.168.2.126: icmp_seq=0 ttl=248 time=52.4 ms 64 bytes from 192.168.2.126: icmp_seq=1 ttl=248 time=52.6 ms 64 bytes from 192.168.2.126: icmp_seq=2 ttl=248 time=56.4 ms --- 192.168.2.126 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 52.4/53.8/56.4 ms
Workstation Setup
In order for the Brooklyn workstations to see the Manhattan private network, it is necessary to add an entry to each and every workstation's routing table. On most versions of Windows, this can be accomplished from the command line:
C:\>ROUTE ADD 192.168.1.0 MASK 255.255.255.0 192.168.2.126
This line tells any Windows machine in Brooklyn that packets with a destination address on the 192.168.1.0/24 network (that is, on the Manhattan private network) must be routed through the Linux machine in Brooklyn; this Linux machine will then forward the data via the IP tunnel that was configured above.
Since workstations in Manhattan do not need to "see" the private network in Brooklyn, it is not necessary to enter a similar command on Manhattan machines.
At this point, it should be possible to ping the private address of the Manhattan Linux machine from any Brooklyn workstation:
C:\>PING 192.168.1.100 Pinging 192.168.1.100 with 32 bytes of data: Reply from 192.168.1.100: bytes=32 time=78ms TTL=248 Reply from 192.168.1.100: bytes=32 time=47ms TTL=248 Reply from 192.168.1.100: bytes=32 time=62ms TTL=248 Reply from 192.168.1.100: bytes=32 time=47ms TTL=248
IP Masquerading
In order for the Brooklyn workstations to see the Manhattan private network, it was necessary to alter their routing table. In order for the Manhattan application server to see the Brooklyn private network (and the workstations on it), it would be necessary to alter the application server's routing table, except that we cannot; the whole point of this exercise is to avoid any changes whatsoever to this application server's configuration. So rather than telling the application server where to find the Brooklyn private network, we instead use the Linux machine's masquerading capabilities to pretend that the Brooklyn network's machines are, in fact, on the local network in Manhattan.
For this, we require only one command* on the Manhattan Linux machine (preferably added, along with the other network configuration commands, to the system startup files):
ipchains -A forward -s 192.168.2.0/24 -j MASQ
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -j SNAT --to-source 192.168.1.126
This commands tells the Linux kernel that any IP packets that it forwards from the 192.168.2.0/24 originating network, it must masquerade to make it appear as though the packets are coming from itself. In other words, on all these packets, the originating address (e.g., 192.168.2.3) will be replaced by this Linux machine's own IP address; if the packet goes out on eth0 (which is the interface that connects the Manhattan Linux machine to the application server there), this address will be 192.168.1.126. The application server in Manhattan will be able to respond to connection requests coming from this address, and it will send its responses to the Linux machine which, in turn, will take care of the reverse translation, properly returning these responses to the true originating host in Brooklyn.
After the ipchainsiptables command has been properly entered, it should be possible to ping the Manhattan application server from any Brooklyn workstation:
C:\>PING 192.168.1.126 Pinging 192.168.1.126 with 32 bytes of data: Reply from 192.168.1.126: bytes=32 time=47ms TTL=248 Reply from 192.168.1.126: bytes=32 time=47ms TTL=248 Reply from 192.168.1.126: bytes=32 time=62ms TTL=248 Reply from 192.168.1.126: bytes=32 time=47ms TTL=248
Thanks to IP masquerading, the Manhattan application server will never know that these ping requests originated from outside the Manhattan private network.
Surprisingly, we may find that although we're able to ping the Manhattan application server from any Brooklyn workstation, we're not able to ping the Manhattan application server from the Brooklyn Linux machine. That is because the Brooklyn Linux machine will send its ping requests to any external destination using its eth1 IP address, 172.17.2.10, as the originating address. This address is not masqueraded by the Manhattan Linux machine. This problem is easily rectified by adding the following command on the Manhattan machine:
ipchains -A forward -s 172.17.2.10 -j MASQ
iptables -t nat -A POSTROUTING -s 172.17.2.10 -j SNAT --to-source 192.168.1.126
This command tells the Manhattan Linux machine that any packets that arrive from the Brooklyn Linux machine must also be masqueraded, to make them appear as though they originated locally. However, this command is not really necessary (Brooklyn workstations will connect to the Manhattan application server fine, with or without this command) and may in fact cause problems later on, if other types of IP connections are established between the Brooklyn and Manhattan Linux machines.
UDP Broadcasts
At this point, only one problem remains: the routing of UDP broadcast packets from Manhattan to Brooklyn. Because UDP broadcast packets are manifestly non-routable (otherwise, a suitably formulated UDP packet could be broadcast to the entire world, either by accident or as a result of malicious intent) this problem cannot be solved using standard tools. The simplest solution that I found was to write a small program, a Linux demon, that listens to broadcast packets on a specific UDP port, and forwards these packets to a specific destination address (which may be the address of an individual host, or a broadcast address.)
Usage of this program, udp-proxy, is very simple. The syntax is as follows:
udp-proxy [-d] port-number ip-address
The optional -d command-line switch tells the program to run in debug mode; instead of detaching itself from the terminal and going into the background, it will run in the foreground and print diagnostic messages.
In Manhattan, the following line is added to the system startup files:
udp-proxy 6799 172.17.2.10
This tells udp-proxy to forward any packets it receives on port 6799 to 172.17.2.10:6799; that is, the same UDP port on the Brooklyn Linux machine.
In Brooklyn, the following line is used:
udp-proxy 6799 192.168.2.255
That is, packets received on port 6799 (packets sent by the copy of udp-proxy running in Manhattan) are forwarded to the broadcast address of the Brooklyn private network.
With these commands, the setup is now complete. My friend was able to make full use of his Brooklyn workstations, connecting to his Manhattan application server via a tunnel through the public Internet.
Conclusion
When I wrote my book, Linux: A Network Solution for your Office, I argued that while Linux may not replace Windows on the desktop anytime soon, it is a superior operating system when it comes to the Internet needs of small offices. The present case is a perfect example; using the full power of the IP networking capabilities built into Linux, it was possible to implement an unusual network configuration that would have been impossible to build using Windows NT or many other network operating systems, and thus save tens of thousands of dollars (literally!) that would otherwise have had to be spent on new application servers, dedicated firewall gateways, and additional software licenses.
Appendix: udp-proxy Source Code
Here is the source code for the udp-proxy utility:
#include <stdio.h>
#include <netdb.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <syslog.h>
void main(int argc, char *argv[])
{
struct sockaddr_in saSRC, saDST, saRCV;
char cBuf[1<<16];
int sSRC, sDST;
struct hostent *ph;
int nLen;
int bDebug = 0;
int on = 1;
pid_t pid;
int nAS;
unsigned long aRCV, aDST;
int i;
char *pszApp = *argv++;
if (argc > 1 && !strcmp(*argv, "-d"))
{
printf("Debug mode.\n");
bDebug = 1;
argc--;
argv++;
}
if (argc != 3)
{
printf("Usage: %s [-d] port-number ip-address\n", pszApp);
exit(1);
}
ph = gethostbyname(argv[1]);
if (ph == NULL)
{
printf("Invalid address\n");
exit(1);
}
saDST.sin_family = AF_INET;
saDST.sin_port = htons(atoi(argv[0]));
saDST.sin_addr.s_addr = *((unsigned long *)ph->h_addr);
saSRC.sin_family = AF_INET;
saSRC.sin_port = saDST.sin_port;
saSRC.sin_addr.s_addr = 0;
sSRC = socket(AF_INET, SOCK_DGRAM, 0);
sDST = socket(AF_INET, SOCK_DGRAM, 0);
if (bind(sSRC, (struct sockaddr *)&saSRC, sizeof(saSRC)))
{
printf("Unable to bind to socket\n");
exit(1);
}
setsockopt(sDST, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));
if (!bDebug)
{
close(0);
close(1);
close(2);
pid = fork();
if (pid < 0) syslog(LOG_ERR, "Could not go into background.");
if (pid > 0) exit(0);
}
aDST = htonl(saDST.sin_addr.s_addr);
while (1)
{
nAS = sizeof(saRCV);
nLen = recvfrom(sSRC, cBuf, sizeof(cBuf), 0, (struct sockaddr *)&saRCV, &nAS);
// Imperfect method for filtering loopback of broadcast packets;
// it may also filter packets from certain hosts on the local
// network, but for our purposes, that's irrelevant.
aRCV = htonl(saRCV.sin_addr.s_addr);
for (i = 0; i < 32; i++)
{
if (!(aDST & (1 << i))) break;
aRCV |= (1 << i);
}
if (nLen > 0 && aRCV != aDST)
{
if (bDebug)
printf("Relaying a packet of length %d from %d.%d.%d.%d.\n",
nLen,
((unsigned char *)&saRCV.sin_addr.s_addr)[0],
((unsigned char *)&saRCV.sin_addr.s_addr)[1],
((unsigned char *)&saRCV.sin_addr.s_addr)[2],
((unsigned char *)&saRCV.sin_addr.s_addr)[3]);
sendto(sDST, cBuf, nLen, 0, (struct sockaddr *)&saDST, sizeof(saDST));
}
}
}
*Long after this article was originally written, the Linux ipchains architecture was replaced with a more comprehensive system called iptables. I recently (June 2011) updated this article to reflect these changes.
