What is Outbound SMTP Filtering?
SMTP (Simple Mail Transfer Protocol) is the protocol used to send email on the Internet. All of the emails you receive are transmitted using SMTP, including all of the spam. Filtering software is often deployed that intercepts inbound SMTP connections, and prevents unwanted emails from reaching your inbox. It is also becoming more common for network operators to deploy outbound SMTP filtering. Outbound filtering protects the reputation of the sending network, making it more likely for legitimate messages to be delivered. This blog post is not going to cover why outbound filtering is important, but you can find more detail in our white paper here.
What is Transparent Filtering?
MailChannels Traffic Control is an SMTP proxy. That means it is positioned in between the SMTP client (the host that is sending the message), and the SMTP server. When the client sends a message to the server, the connection is redirected to the Traffic Control host. Now Traffic Control can inspect the message and take appropriate action, for example by blocking a spam message.
The setup pictured above is straightforward to implement, if the network operator has control of the SMTP client configuration. In that case, the client can be configured to use the Traffic Control host as a 'smart host', and to direct all SMTP connections through it. In some cases, a network operator may not be able to mandate configuration changes to all SMTP clients on their network. For example, some Internet service providers allow their clients to send mail directly to the Internet. It is usually impractical for them to reconfigure every SMTP client. In this case, the best way to implement outbound filtering may be to use policy-based routing. Policy-based routing allows the network operator to redirect traffic based on the TCP connection properties. For outbound SMTP filtering, routers should be configured to route all packets heading out of the network, with a destination port of 25, to the Traffic Control host. In addition, all the return packets, headed into the network, with a source port of 25, need to be routed to Traffic Control.
To support this configuration, Traffic Control needs to be run in "transparent" mode. In transparent mode, Traffic Control accepts connections on port 25 for any IP address, even IP addresses that are not associated with the Traffic Control host. Once it accepts the connection, Traffic Control connects to the SMTP server that the upstream client was connecting to before policy-based routing was applied. When it makes this downstream connection, it binds to the IP address of the original SMTP client. For example, suppose the SMTP client is at 192.168.0.1, and it tries to send mail to 10.0.0.1. The initial connection request is diverted to Traffic Control using policy-based routing. Traffic Control makes a connection to 10.0.0.1, and the connection appears to be from 192.168.0.1. When Traffic Control is in transparent mode, the client and server appear to be connected directly to each other, even though the connection is being proxied.
Testing Transparent Filtering
Here at MailChannels, we frequently have to develop and test new features for use with transparent filtering. Deploying into an environment with policy-based routing enabled is not always practical during development, it's much easier to let each developer have their own separate environment. Fortunately, it's possible to set up a complete transparent filtering environment that can run on a single host, using virtual machines. For this blog post, we're going to use Vagrant to manage the virtual machines. You can learn more about Vagrant at vagrantup.com. All of our VMs are going to be based on Ubuntu 18.04, but it should be straightforward to modify this procedure to work with any Linux distribution. The first thing we need to do is set up the client and server virtual machines. We can describe these two machines in a Vagrantfile with the following contents:
Vagrantfile
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.define "client" do |node|
node.vm.network "private_network", ip: "10.42.100.2"
node.vm.provision "file", source: "util.sh", destination: "/tmp/util.sh"
node.vm.provision "shell", path: "client.sh"
end
config.vm.define "server" do |node|
node.vm.network "private_network", ip: "10.42.102.2"
node.vm.provision "file", source: "util.sh", destination: "/tmp/util.sh"
node.vm.provision "file", source: "smtp-sink.service", destination: "/tmp/smtp-sink.service"
node.vm.provision "shell", path: "server.sh"
end
endThe Vagrantfile configures two virtual machines, named ‘client’ and ‘server’. The ‘client’ VM represents a mail client inside the customer network, it will send email to the Internet. The ‘server’ VM represents the receiving mail server, somewhere outside the customer network.
This Vagrantfile depends on a few other files. The first is util.sh, where we will put some common functions that are used on most VMs:
util.sh
# set the hostname for this VM
function sethost {
hostname $1
echo "127.0.0.1 $1" >> /etc/hosts
}
Next, we have client.sh, which is used to provision the 'client' VM:
client.sh
#!/bin/bash
set -v
source /tmp/util.sh
sethost "client"
apt-get update
debconf-set-selections <<< "postfix postfix/mailname string sender.vntp.mailchannels.net"
debconf-set-selections <<< "postfix postfix/main_mailer_type string 'no configuration'"
apt-get install -y postfix
systemctl stop postfix
systemctl disable postfixSo far, all client.sh does is set the hostname, install the 'postfix' Ubuntu package, and stop the postfix service. We're not going to use postfix as a full fledged MTA (Message Transfer Agent) for this test, but the postfix package includes some testing tools that will be useful later. The last file we need is server.sh, which is used to provision the 'server' VM:
server.sh
#!/bin/bash
set -v
source /tmp/util.sh
sethost "server"
apt-get update
debconf-set-selections <<< "postfix postfix/mailname string receiver.example.com"
debconf-set-selections <<< "postfix postfix/main_mailer_type string 'Internet Site'"
apt-get install -y postfix
systemctl stop postfix
systemctl disable postfix
mv /tmp/smtp-sink.service /etc/systemd/system/
systemctl daemon-reload
systemctl start smtp-sinkLike the client.sh script, server.sh installs the 'postfix' package. Once again, we're not going to use the full Postfix MTA for these tests, but we are going to use the 'smtp-sink' tool that is included in the postfix package. smtp-sink acts like an SMTP server, but simply discards any messages sent to it. This is perfect for simple filtering tests, where we're not modifying the message. The server.sh script configures a systemd service that automatically runs an instance of smtp-sink, as described in the smtp-sink.service file:
smtp-sink.service
[Unit]
Description=Dummy SMTP server for testing
After=network.target
[Service]
ExecStart=/usr/sbin/smtp-sink -u vagrant 25 4096
Type=simple
PIDFile=/var/run/smtp-sink.pid
[Install]
WantedBy=default.targetOnce we have all the files in place, we can start vagrant and test our setup. Place all the files in a single directory, and run vagrant up.
This will produce a test setup as shown in the diagram below.
In this configuration, the client is sending directly to the server, without passing through a filter. This is similar to a customer network that has not deployed transparent filtering. We can verify the setup by sending a test message from the client machine. Log in to the client (vagrant ssh client), and then run smtp-source -v 10.42.102.2. You should see a successful SMTP transaction, like this:
smtp-source: name_mask: all
smtp-source: smtp_stream_setup: maxtime=300 enable_deadline=0
smtp-source: vstream_tweak_tcp: TCP_MAXSEG 1460
smtp-source: <<< 220 smtp-sink ESMTP
smtp-source: HELO ubuntu-bionic
smtp-source: <<< 250 smtp-sink
smtp-source: MAIL FROM:<foo@ubuntu-bionic>
smtp-source: <<< 250 2.1.0 Ok
smtp-source: RCPT TO:<foo@ubuntu-bionic>
smtp-source: <<< 250 2.1.5 Ok
smtp-source: DATA
smtp-source: <<< 354 End data with <CR><LF>.<CR><LF>
smtp-source: .
smtp-source: <<< 250 2.0.0 Ok
smtp-source: QUIT
smtp-source: <<< 221 ByeThe above output shows smtp-source’s view of the traffic. We can get another view of the traffic using tcpdump. Open another ssh connection to the client, and run sudo tcpdump -i any -en tcp port 25, then run smtp-source -v 10.42.102.2 again in the first session. You should see output that looks like this:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
22:52:28.509690 Out 02:8e:72:66:23:f8 ethertype IPv4 (0x0800), length 76: 10.0.2.15.59938 > 10.42.102.2.25: Flags [S], seq 55517135, win 29200, options [mss 1460,sackOK,TS val 616708843 ecr 0,nop,wscale 7], length 0
22:52:28.510398 In 52:54:00:12:35:02 ethertype IPv4 (0x0800), length 62: 10.42.102.2.25 > 10.0.2.15.59938: Flags [S.], seq 55360001, ack 55517136, win 65535, options [mss 1460], length 0
22:52:28.510434 Out 02:8e:72:66:23:f8 ethertype IPv4 (0x0800), length 56: 10.0.2.15.59938 > 10.42.102.2.25: Flags [.], ack 1, win 29200, length 0
. . .This shows you the individual IP packets that make up the SMTP transaction. It’s not that useful right now, but it can be useful later, when we’re troubleshooting more complicated setups.
Now that we have a basic setup working, we need to add a router, so we can do the policy-based routing that is necessary for transparent filtering. The Linux kernel has all the routing capabilities we need, so we can just create another virtual machine based on Ubuntu. We can create the VM by adding the following lines to the Vagrantfile, before the final end:
sender_router_ip = "10.42.100.100"
receiver_router_ip = "10.42.102.100"
filter_router_ip = "10.42.103.100"
config.vm.define "router" do |node|
node.vm.network "private_network", ip: sender_router_ip
node.vm.network "private_network", ip: receiver_router_ip
node.vm.network "private_network", ip: filter_router_ip
node.vm.provision "file", source: "util.sh", destination: "/tmp/util.sh"
node.vm.provision "shell", path: "router.sh"
endThe router has three interfaces, because it forwards packets between three networks:
10.42.100.0/24:The 'sender' network, i.e. the internal ISP network10.42.102.0/24:The 'receiver' network, i.e. the public Internet10.42.103.0/24:The 'filter' network, where the mail filters operate
For now, we’re just going to configure the router to forward traffic from the client to the server. The router is configured using the router.sh script:
router.sh
#!/bin/bash
set -v
source /tmp/util.sh
sethost "router"
sysctl -w net.ipv4.ip_forward=1
At this stage, the router configuration is simple. It just enables IP forwarding in the kernel, which is what allows it to act as a router.
Now, we just need to configure the client and server to make use of the router. First, we add the following line to client.sh:
ip route add 10.42.102.0/24 nexthop via 10.42.100.100 dev enp0s8We add a similar line to server.sh:
ip route add 10.42.100.0/24 nexthop via 10.42.102.100 dev enp0s8
These commands add new routes, which tell the kernel to send any packets with a destination in the receiver (or sender) network to the router. The router should forward the packets to the appropriate interface, based on the destination network. The routes needed to accomplish this are set up automatically when we configure the interfaces on the router.
Once all the files are modified, we run vagrant destroy and vagrant up to rebuild the test environment. The setup now looks like this:
To make sure it works, we can ssh into the ‘router’ and run the same tcpdump command we ran earlier: sudo tcpdump -i any -en tcp port 25. Then we log in to the ‘client’ in another terminal and send a test message: smtp-source -v 10.42.102.2. The message should be delivered successfully, and we should see the network traffic pass through the router on the tcpdump output.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:45:09.454976 In 08:00:27:7f:51:22 ethertype IPv4 (0x0800), length 76: 10.42.100.2.46082 > 10.42.102.2.25: Flags [S], seq 2844839854, win 29200, options [mss 1460,sackOK,TS val 1433615845 ecr 0,nop,wsc
ale 7], length 0
00:45:09.455010 Out 08:00:27:e4:0b:bf ethertype IPv4 (0x0800), length 76: 10.42.100.2.46082 > 10.42.102.2.25: Flags [S], seq 2844839854, win 29200, options [mss 1460,sackOK,TS val 1433615845 ecr 0,nop,wscale 7], length 0
00:45:09.455677 In 08:00:27:1f:90:46 ethertype IPv4 (0x0800), length 76: 10.42.102.2.25 > 10.42.100.2.46082: Flags [S.], seq 81285228, ack 2844839855, win 28960, options [mss 1460,sackOK,TS val 2084613560 ecr 1433615845,nop,wscale 7], length 0
00:45:09.455690 Out 08:00:27:63:79:53 ethertype IPv4 (0x0800), length 76: 10.42.102.2.25 > 10.42.100.2.46082: Flags [S.], seq 81285228, ack 2844839855, win 28960, options [mss 1460,sackOK,TS val 2084613560 ecr 1433615845,nop,wscale 7], length 0
00:45:09.456178 In 08:00:27:7f:51:22 ethertype IPv4 (0x0800), length 68: 10.42.100.2.46082 > 10.42.102.2.25: Flags [.], ack 1, win 229, options [nop,nop,TS val 1433615847 ecr 2084613560], length 0
00:45:09.456195 Out 08:00:27:e4:0b:bf ethertype IPv4 (0x0800), length 68: 10.42.100.2.46082 > 10.42.102.2.25: Flags [.], ack 1, win 229, options [nop,nop,TS val 1433615847 ecr 2084613560], length 0
00:45:09.456722 In 08:00:27:1f:90:46 ethertype IPv4 (0x0800), length 89: 10.42.102.2.25 >
. . .In this trace, you can see two copies of each packet, one coming in to the router, and one heading out.
Now that a router is set up, the last thing to do is to set up the filter. We’d normally install our Traffic Control package at this point, but we can demonstrate transparent filtering using the haproxy open-source package. First, we’ll add another machine to our Vagrantfile to host the filter, by adding the following lines:
config.vm.define "filter" do |node|
node.vm.network "private_network", ip: "10.42.103.2"
node.vm.provision "file", source: "util.sh", destination: "/tmp/util.sh"
node.vm.provision "file", source: "haproxy.cfg", destination: "/tmp/haproxy.cfg"
node.vm.provision "shell", path: "filter.sh"
endThe filter VM needs two files for its configuration: haproxy.cfg and filter.sh.
haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 5000
timeout client 50000
timeout server 50000
frontend smtp
option tcplog
bind *:2501 transparent
default_backend smtp_transparent
backend smtp_transparent
option transparent
source 0.0.0.0 usesrc clientipThis configuration causes haproxy to mimic a default Traffic Control installation. It listens on port 2501, and it will connect to the server using the client’s IP address. This won’t work until we add some iptables rules to redirect connections to port 25 to port 2501, which we do in filter.sh:
filter.sh
#!/bin/bash
set -v
source /tmp/util.sh
sethost "filter"
apt-get update
apt-get install -y haproxy hatop
cp /tmp/haproxy.cfg /etc/haproxy/haproxy.cfg
systemctl restart haproxy
iptables -t mangle -N mailchannels
iptables -t mangle -A PREROUTING -j mailchannels
iptables -t mangle -A mailchannels -p tcp -m conntrack --ctstate NEW,ESTABLISHED -m socket -j MARK --set-mark 1
iptables -t mangle -A mailchannels -p tcp -m conntrack --ctstate NEW,ESTABLISHED -m socket -j ACCEPT
iptables -t mangle -A mailchannels -p tcp -m conntrack --ctstate NEW,ESTABLISHED -m addrtype ! --dst-type LOCAL -m tcp --dport 25 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 2501
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
ip rout delete default
ip route add default via 10.42.103.100 dev enp0s8A full explanation of these iptables rules is beyond the scope of this post. In our normal test setup, it’s not necessary to configure them separately, they are installed automatically with Traffic Control.
In addition to the iptables rules, this script installs and configures haproxy, and configures the route table so that all outbound packets are sent via the router VM.
With the filter machine set up, we just need to configure the router, to redirect all port 25 traffic through the filter. We do that by adding the following to router.sh:
for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 2 > $i; done
iptables -t mangle -N mailchannels
iptables -t mangle -A PREROUTING -j mailchannels
# mark packets from client, so they get routed to the filter
iptables -t mangle -i enp0s8 -A mailchannels -p tcp --dport 25 -j MARK --set-mark 1
# mark packets from server, so they get routed to the filter
iptables -t mangle -i enp0s9 -A mailchannels -p tcp --sport 25 -j MARK --set-mark 1
# mark packets from the filter so they get routed to the server
iptables -t mangle -i enp0s10 -A mailchannels -p tcp --dport 25 -j MARK --set-mark 3
ip rule add fwmark 1 lookup 100
ip route add 0.0.0.0/0 table 100 nexthop via 10.42.103.2 dev enp0s10
ip rule add fwmark 3 lookup 101
ip route add 0.0.0.0/0 table 101 nexthop via 10.42.102.2 dev enp0s9This is the core change that causes all port 25 traffic to be routed to the filter. The first line disables return path filtering in the kernel. Normally, a host will reject packets which arrive at an interface which wouldn’t be chosen as the route to the source address. This won’t work for the router VM, since it will have to forward packets from the filter that appear to come from the client and server hosts.
Next, we create a set of iptables rules to ‘mark’ the relevant packets. iptables can not affect the routing of packets directly, and the routing tools included in this version of Ubuntu can not inspect TCP port information. This means we have to do this in several stages. First, we use iptables to ‘mark’ the packets we’re concerned about. Next, we use ip rule to select which routing table to use. Finally, the packet is routed to the appropriate interface: towards the filter, for SMTP packets coming from the client and server, and to the server, for packets coming from the filter going to port 25.
We now have full transparent filtering setup:
Now, we’re finally ready to start transparent filtering. We’ll run vagrant destroy and vagrant up again, then vagrant ssh client to log in to the client VM. On the client VM, run smtp-source -v 10.42.102.2. You should see a successful delivery, just like the first time:
smtp-source: name_mask: all
smtp-source: smtp_stream_setup: maxtime=300 enable_deadline=0
smtp-source: vstream_tweak_tcp: TCP_MAXSEG 1460
smtp-source: <<< 220 smtp-sink ESMTP
smtp-source: HELO ubuntu-bionic
smtp-source: <<< 250 smtp-sink
smtp-source: MAIL FROM:<foo@ubuntu-bionic>
smtp-source: <<< 250 2.1.0 Ok
smtp-source: RCPT TO:<foo@ubuntu-bionic>
smtp-source: <<< 250 2.1.5 Ok
smtp-source: DATA
smtp-source: <<< 354 End data with <CR><LF>.<CR><LF>
smtp-source: .
smtp-source: <<< 250 2.0.0 Ok
smtp-source: QUIT
smtp-source: <<< 221 Bye
Now, let’s log in to the filter (vagrant ssh filter), and inspect /var/log/haproxy.log. There should be an entry for the connection we just made to deliver the test message, that looks like this:
Jul 23 18:02:51 filter haproxy[2967]: 10.42.100.2:38528 [23/Jul/2020:18:02:51.668] smtp smtp_transparent/<NOSRV> 1/1/48 124 CD 1/1/0/0/0 0/0This is proof that the connection was routed through our transparent proxy.
Finally, we’ll log in to the receiving server (vagrant ssh server) and run a packet trace: sudo tcpdump -i any -n tcp port 25. When we send another test message from the client, we should have a connection to the server that appears to be directly from the client, but we know it’s actually being routed through the filter:
18:29:10.240700 IP 10.42.100.2.53516 > 10.42.102.2.25: Flags [S], seq 2880543568, win 29200, options [mss 1460,sackOK,TS val 1197128222 ecr 0,nop,wscale 7], length 0
18:29:10.240756 IP 10.42.102.2.25 > 10.42.100.2.53516: Flags [S.], seq 3259943175, ack 2880543569, win 28960, options [mss 1460,sackOK,TS val 1463294663 ecr 1197128222,nop,wscale 7], length 0
18:29:10.241493 IP 10.42.100.2.53516 > 10.42.102.2.25: Flags [.], ack 1, win 229, options [nop,nop,TS val 1197128223 ecr 1463294663], length 0And that’s it! We now have a fully working transparent filtering setup that we can use for development and testing. It all runs in virtual machines which easily run on a single development box. We can quickly add any debugging tools to the virtual machine images, and it’s easy to completely reset the test environment using vagrant destroy and vagrant up.
This environment simulates the most common on-premises deployment that we do at MailChannels, where all the traffic for a connection is sent over the same router. It can also be modified to test more complex deployments, such as asymmetric routing, where outbound and return packets may take different paths. Such deployments may be covered in future blog posts.
