Topology
This configuration uses a number of different FIBs. I often pair FIB IDs with VLAN IDs to keep things consistent, but to be clear I use VLANs specifically
for layer 2 segmentation (when switch ports apply, networks are also separated by VLANs.) Clients on the LAN and other air gapped networks use either HTTP_PROXY
and HTTPS_PROXY environment variables to specify http://192.88.99.0:3128 as the proxy server to use to access the internet or they specify it in the browser
settings (firefox proxy settings.)
FIB vs VLAN
FIB on FreeBSD is a multiple routing table mechanism used to control layer 3 routing policy, while VLAN is a mechanism for layer 2 ethernet segmentation. For FIBs, layer 3 routing policy is essentially dictated by the fact that each FIB is it's own separate routing table.
Routing uses Longest Prefix Match (LPM)
Consider these two prefixes:
10.0.0.0/810.1.0.0/18
Of these two routes, 10.1.0.0/18 is a longer prefix than 10.0.0.0/8 and therefore a reject route such as:
10.0.0.0/8 link#3 URS lo0
will not prevent these gateway route from working:
10.1.0.0/18 192.0.0.15 UGS epair2a
10.1.64.0/18 192.0.0.35 UGS epair12a
but it would still reject, for destination 10.0.0.0/24 if there isn't a route for 10.0.0.0/24.
Minimal need for firewall rules
Because so much of what I need to restrict is done by the software itself and by policy routing, we don't need really anything in pf except for NAT:
table <resvd_networks> { 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16
172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24
224.0.0.0/4 233.252.0.0/24 240.0.0.0/4 255.255.255.255/32 }
nat on wlan0 inet from 192.0.0.30/32 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 203.0.113.53/32 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 192.88.99.2/32 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 10.1.0.0/18 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 10.1.64.0/18 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 10.242.0.0/24 to !<resvd_networks> -> wlan0
nat on wlan0 inet from 192.0.0.25/32 to !<resvd_networks> -> wlan0
block drop in log quick on { gif1, wlan0, wg0 } from urpf-failed to any label URPF
Core FIB
All of the epair interfaces denoted from the routing table for FIB 0 have peers in other FIBS. All epair interfaces are assigned a /31 (exactly 2 addresses) from
192.0.0.0/24. These are used for providing hops between FIBs.
Reject routes
➜ ~ netstat -nr | grep UR
0.0.0.0/8 link#3 URS lo0
10.0.0.0/8 link#3 URS lo0
100.64.0.0/10 link#3 URS lo0
127.0.0.0/8 link#3 URS lo0
169.254.0.0/16 link#3 URS lo0
172.16.0.0/12 link#3 URS lo0
192.0.0.0/24 link#3 URS lo0
192.88.99.0/24 link#3 URS lo0
192.168.0.0/16 link#3 URS lo0
198.18.0.0/15 link#3 URS lo0
198.51.100.0/24 link#3 URS lo0
203.0.113.0/24 link#3 URS lo0
223.252.0.0/24 link#3 URS lo0
240.0.0.0/4 link#3 URS lo0
::/96 link#3 URS lo0
::ffff:0.0.0.0/96 link#3 URS lo0
::ffff:0:0:0/96 link#3 URS lo0
64:ff9b::/96 link#3 URS lo0
64:ff9b:1::/48 link#3 URS lo0
100::/64 link#3 URS lo0
2001::/32 link#3 URS lo0
2001:20::/28 link#3 URS lo0
2001:db8::/32 link#3 URS lo0
2002::/16 link#3 URS lo0
3fff::/20 link#3 URS lo0
5f00::/16 link#3 URS lo0
fc00::/7 link#3 URS lo0
fe80::%lo0/10 link#3 URS lo0
ff02::/16 link#3 URS lo0
Gateway Routes
- the peer for
epair10ais in FIB 255,epair10bso a different routing table than the one we're currently looking at. 192.88.99.0is the IP address for the ingress squid proxy192.88.99.2is the IP address for the egress squid proxy
➜ ~ netstat -nr | grep UG
default 192.0.0.31 UGS epair10a
10.1.0.0/18 192.0.0.15 UGS epair2a
10.1.64.0/18 192.0.0.35 UGS epair12a
10.255.255.0/24 192.0.0.33 UGS epair11a
192.0.2.2 192.0.0.41 UGHS epair22a
192.88.99.0 192.0.0.23 UGHS epair6a
192.88.99.2 192.0.0.21 UGHS epair5a
192.168.32.128/25 192.0.0.11 UGS epair0a
192.168.64.128/25 192.0.0.13 UGS epair1a
192.168.72.128/25 192.0.0.17 UGS epair3a
198.51.100.0 192.0.0.37 UGHS epair19a
203.0.113.53 192.0.0.19 UGHS epair4a
203.0.113.89 192.0.0.39 UGHS epair21a
203.0.113.123 192.0.0.25 UGHS epair7a
2000::/3 fcfc:48::192:0:0:29 UGS epair9a
2001:470:e845::23 fcfc:180::192:0:0:37 UGHS epair19a
2001:470:e845::53 fcfc:53::192:0:0:19 UGHS epair4a
2001:470:e845::80 fcfc:180::192:0:0:37 UGHS epair19a
2001:470:e845::443 fcfc:180::192:0:0:37 UGHS epair19a
2001:470:e845::3128 fcfc:80::192:0:0:21 UGHS epair5a
2001:470:e845::80:443 fcfc:180::192:0:0:37 UGHS epair19a
2001:470:e845::555:1212 fcfc:30::192:0:0:17 UGHS epair3a
2001:470:e845:f::/64 fcfc:30::192:0:0:17 UGS epair3a
fcff::53 fcfc:53::192:0:0:19 UGHS epair4a
fcff::123 fcfc:123::192:0:0:25 UGHS epair7a
fcff::3128 fcfc:81::192:0:0:23 UGHS epair6a
fcff:16::/32 fcfc:16::192:0:0:11 UGS epair0a
fcff:18::/32 fcfc:18::192:0:0:13 UGS epair1a
fcff:20::/32 fcfc:20::192:0:0:35 UGS epair12a
fcff:25::/32 fcfc:25::192:0:0:15 UGS epair2a
fcff:30::/32 fcfc:30::192:0:0:17 UGS epair3a
fcff:48::192:88:99:255 fcfc:48::192:0:0:29 UGHS epair9a
fcff:80::192:88:99:2 fcfc:80::192:0:0:21 UGHS epair5a
fcff:180::198:51:100:0 fcfc:180::192:0:0:37 UGHS epair19a
fdb8:3260:d80f:a056::/64 fcfc:255::192:0:0:31 UGS epair10a
LAN FIB
This is the router for the LAN; it routes traffic from the LAN to the core router, it doesn't have any specific reject routes
because there is no default gateway. This of course means that we have to provide static routes for every address we want to be reachable
from the LAN, notably 192.88.99.0 which our clients use as an HTTP proxy to access the internet indirectly.
Gateway Routes
45.61.188.232is a host that LAN clients use UDP to connect to, so this host is reachable by LAN clients
➜ ~ netstat -F 25 -nr | grep UG
45.61.188.232 192.0.0.14 UGHS epair2b
192.88.99.0 192.0.0.14 UGHS epair2b
192.168.64.128/25 192.0.0.14 UGS epair2b
203.0.113.53 192.0.0.14 UGHS epair2b
203.0.113.123 192.0.0.14 UGHS epair2b
fcff::53 fcfc:25::192:0:0:14 UGHS epair2b
fcff::123 fcfc:25::192:0:0:14 UGHS epair2b
fcff::3128 fcfc:25::192:0:0:14 UGHS epair2b
fcff:18::/32 fcfc:25::192:0:0:14 UGS epair2b
Squid Ingres FIB/container
Gateway Routes
From this you can basically tell that the ingress squid server doesn't have direct access to the internet, either. It has a route to 192.88.99.2 and the
reason for this will become clear in the egress squid section. For ingress, we configure 192.88.99.2 (the egress squid server) as a cache peer, requests
are never handled directly by the ingress server and are to be forwarded thru cache peers only.
➜ ~ netstat -F 81 -nr | grep UG
10.1.0.0/18 192.0.0.22 UGS epair6b
10.1.64.0/18 192.0.0.22 UGS epair6b
10.255.255.0/24 192.0.0.22 UGS epair6b
192.0.2.2 192.0.0.22 UGHS epair6b
192.88.99.2 192.0.0.22 UGHS epair6b
192.168.1.0/24 192.0.0.22 UGS epair6b
192.168.32.128/25 192.0.0.22 UGS epair6b
192.168.64.128/25 192.0.0.22 UGS epair6b
192.168.72.128/25 192.0.0.22 UGS epair6b
203.0.113.53 192.0.0.22 UGHS epair6b
squid.conf
- There's quite a bit of extra filtering being provided at this stage:
root@proxy-ingr:~ # cat /usr/local/etc/squid/weirdos.txt | wc -l
19
root@proxy-ingr:~ # cat /usr/local/etc/squid/ad_domains.txt | wc -l
42536
- Most of the acls apply to the live servers FIB, which is also air gapped however they are even more restricted to only have access to things I have specified in ACLs.
http_port 192.88.99.0:3128
snmp_port 3401
forwarded_for off
dns_nameservers 203.0.113.53
visible_hostname proxy-ingr.netcrave.local
coredump_dir /var/squid/cache
access_log syslog:local2 squid
cache_log syslog:local2 squid
pinger_enable off
acl snmp_server src 10.255.255.254/32
acl local_rtr src 192.0.0.0/24
acl snmp_com snmp_community public
acl ads dstdomain "/usr/local/etc/squid/ad_domains.txt"
acl weirdos dstdomain "/usr/local/etc/squid/weirdos.txt"
acl pypi_pkg dstdomain pypi.org files.pythonhosted.org
acl opensuse_pkg dstdomain download.opensuse.org \
mirror.sfo12.us.leaseweb.net \
mirror.umd.edu \
mirror.xenyth.net \
slc-mirror.opensuse.org \
rsync.opensuse.org \
codecs.opensuse.org
acl ubuntu_pkg dstdomain security.ubuntu.com \
archive.ubuntu.com
acl debian_pkg dstdomain deb.debian.org
acl void_pkg dstdomain repo-default.voidlinux.org
acl freebsd_pkg dstdomain pkg.FreeBSD.org \
pkgmir.geo.freebsd.org \
ftp.freebsd.org \
git.freebsd.org
acl docker_hub dstdomain auth.docker.io hub.docker.io
acl letsencrypt dstdomain acme-v02.api.letsencrypt.org
acl github_api dstdomain api.github_api.com \
codeload.github.com \
github.com \
objects.githubusercontent.com \
api.github.com \
pipelinesghubeus14.actions.githubusercontent.com \
tokenghub.actions.githubusercontent.com \
broker.actions.githubusercontent.com \
run-actions-2-azure-eastus.actions.githubusercontent.com \
run-actions-1-azure-eastus.actions.githubusercontent.com \
run-actions-3-azure-eastus.actions.githubusercontent.com \
launch.actions.githubusercontent.com \
results-receiver.actions.githubusercontent.com \
.blob.core.windows.net
acl netcrave dstdomain .netcrave.network .netcrave.io .netcrave.chat
acl dns src 203.0.113.53/32
acl ntp src 203.0.113.123/32
acl rev_web src 198.51.100.0/32
acl management src 10.255.255.0/24
acl wave src 192.168.32.128/25
acl tailscale src 100.64.0.0/10
acl link_local src 169.254.0.0/16
acl dmz src 192.168.1.0/24
acl lan src 10.1.0.0/18
acl wifi src 10.1.64.0/18
acl live_servers src 192.168.72.128/25
acl traefik src 192.168.72.250/32
acl docker src 192.168.72.250/32
acl home_servers src 192.168.64.128/25
acl link_local_ipv6 src fe80::/10
acl Safe_ports port 80
acl Safe_ports port 21
acl HTTPS port 443
acl resvd4 dst 0.0.0.0/8 \
10.0.0.0/8 \
100.64.0.0/10 \
127.0.0.0/8 \
169.254.0.0/16 \
172.16.0.0/12 \
192.0.0.0/24 \
192.0.2.0/24 \
192.88.99.0/24 \
192.168.0.0/16 \
198.18.0.0/15 \
198.51.100.0/24 \
203.0.113.0/24 \
224.0.0.0/4 \
240.0.0.0/4
acl internet6 dst 2000::/3
snmp_access allow snmp_com snmp_server
snmp_access allow snmp_com local_rtr
snmp_access deny all
http_access deny ads
http_access deny weirdos
http_access allow CONNECT !Safe_ports lan
http_access allow lan
http_access allow CONNECT HTTPS wifi
http_access allow wifi
http_access allow CONNECT !Safe_ports docker
http_access allow docker
http_access allow Safe_ports docker
http_access allow CONNECT Safe_ports management freebsd_pkg
http_access allow CONNECT HTTPS management freebsd_pkg
http_access allow management freebsd_pkg
http_access allow CONNECT HTTPS live_servers freebsd_pkg
http_access allow live_servers freebsd_pkg
http_access allow CONNECT HTTPS live_servers void_pkg
http_access allow live_servers void_pkg
http_access allow CONNECT HTTPS live_servers debian_pkg
http_access allow live_servers debian_pkg
http_access allow CONNECT HTTPS live_servers ubuntu_pkg
http_access allow live_servers ubuntu_pkg
http_access allow CONNECT HTTPS live_servers pypi_pkg
http_access allow live_servers pypi_pkg
http_access allow CONNECT HTTPS live_servers opensuse_pkg
http_access allow live_servers opensuse_pkg
http_access allow CONNECT HTTPS home_servers opensuse_pkg
http_access allow home_servers opensuse_pkg
http_access allow CONNECT HTTPS live_servers docker_hub
http_access allow live_servers docker_hub
http_access allow CONNECT HTTPS live_servers github_api
http_access allow live_servers github_api
http_access allow CONNECT HTTPS wave opensuse_pkg
http_access allow wave opensuse_pkg
http_access allow CONNECT HTTPS wave freebsd_pkg
http_access allow wave freebsd_pkg
http_access allow CONNECT HTTPS ntp freebsd_pkg
http_access allow ntp freebsd_pkg
http_access allow CONNECT HTTPS dns freebsd_pkg
http_access allow dns freebsd_pkg
http_access allow CONNECT HTTPS rev_web freebsd_pkg
http_access allow rev_web freebsd_pkg
http_access allow CONNECT HTTPS docker github_api
http_access allow docker github_api
http_access allow CONNECT HTTPS docker ubuntu_pkg
http_access allow docker ubuntu_pkg
http_access allow CONNECT HTTPS traefik letsencrypt
http_access allow traefik letsencrypt
http_access allow CONNECT HTTPS traefik netcrave
http_access allow traefik netcrave
http_access deny all
cache_peer 192.0.2.2 parent \
3128 0 \
proxy-only no-query \
no-digest no-delay \
name=kama \
connect_timeout=128s
cache_peer 192.88.99.2 parent \
3128 0 \
proxy-only no-query \
no-digest no-delay \
name=general default \
connect_timeout=128s
cache_peer_access general allow docker github_api
cache_peer_access general allow docker ubuntu_pkg
cache_peer_access kama allow docker !resvd4
cache_peer_access kama allow docker internet6
cache_peer_access kama deny all
cache_peer_access general deny docker
never_direct allow all
always_direct deny all
cache deny all
Squid Egress FIB/container
The egress Squid proxy has access to the internet as well as a route back to the ingress proxy.
Reject routes
-
The LAN clients already don't have routes to the egress proxy server, but just in case we can ensure that the egress server wouldn't respond to them even if they could.
-
Again, LPM means that we have a short reject
192.88.99.0/24and a route for192.88.99.0/32which is a longer prefix than the former, therefore it takes precedence over the shorter route. Again for example,192.88.99.3/32would be rejected as would everything up to192.88.99.255which is the last address of the/24.
➜ ~ netstat -F 80 -nr | grep UR
10.0.0.0/8 link#3 URS lo0
192.168.0.0/16 link#3 URS lo0
192.88.99.0/24 link#3 URS lo0
::/96 link#3 URS lo0
::ffff:0.0.0.0/96 link#3 URS lo0
fe80::%lo0/10 link#3 URS lo0
ff02::/16 link#3 URS lo0
Gateway Routes
➜ ~ netstat -F 80 -nr | grep UG
default 192.0.0.20 UGS epair5b
192.88.99.0 192.0.0.20 UGHS epair5b
203.0.113.53 192.0.0.20 UGHS epair5b
2000::/3 fcfc:80::192:0:0:20 UGS epair5b
fcff::53 fcfc:80::192:0:0:20 UGHS epair5b
fcff:48::192:88:99:255 fcfc:80::192:0:0:20 UGHS epair5b
squid.conf
access_log syslog:local2 squid
cache_log syslog:local2 squid
forwarded_for off
pinger_enable off
dns_nameservers 203.0.113.53
visible_hostname proxy-egr.sttl.wa.us.clandestine.network
nonhierarchical_direct off
acl localnet src 192.88.99.0/24
tcp_outgoing_address 192.88.99.2
tcp_outgoing_address 2001:470:e845::3128
http_port 192.88.99.2:3128 name=general
http_port [fcff:80::192:88:99:2]:3128 name=general
acl general_acl myportname general
always_direct allow general_acl
http_access allow localnet
http_access deny all
cache deny all
coredump_dir /var/squid/cache
Final thoughts
This setup has been rock solid for me for over a year. I feel like I could probably add more filtering of content, but I just haven't bothered. This setup only works for TCP I'm not really sure what they're going to do about QUIC, maybe QUIC+TCP would work since the proxy could determine to use UDP from that? I Feel like this could be more restricted, not less considering how hostile the internet is these days. But this at least feels like a good start, and everything is logged; logs are aggregated to another server on it's own FIB/VLAN. I seldom notice any side effects of the filtering that is already being done and the ones that I do notice are usually certain websites that are just trying to coerce me to allowing ads to use their site.
A very effective setup
Doesn't cost me anything, it's already doing a lot and it's setup to do more.
Copilot and Cursor
These prefer to use HTTP2 and/or QUIC, but they will also work with this setup. Copilot I believe does so with respect to HTTP_PROXY environment variables but
if it doesnt, then it has settings for it; same with Cursor--Cursor definitely has settings for it.
Proxychains
If you have something that doesn't really support HTTP_PROXY correctly, you can use proxychains with it:
[sq@msi ~/paigeadelethompson]$ cat /usr/local/etc/proxychains.conf | grep -v "#" | grep "."
strict_chain
quiet_mode
remote_dns_subnet 224
tcp_read_time_out 8000
tcp_connect_time_out 2000
[ProxyList]
http 192.88.99.0 3128
then proxychains <command>
TCP/IP tuning
This does come with it's own set of problems related to the limitation of TCP connections that the two squid servers can handle. these settings probably
should be revisited but at least soacceptqueue will probably be about the same in any case:
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
vfs.zfs.min_auto_ashift=12
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
vfs.nfs.enable_uidtostring=1
net.inet.ip.source_address_validation=0
net.inet6.ip6.source_address_validation=0
kern.ipc.soacceptqueue=4096
net.inet.icmp.icmplim=0
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.icmp_may_rst=1
net.inet.ip.check_interface=1 # verify packet arrives on correct interface (default 0)
net.inet.ip.portrange.randomized=1 # randomize outgoing upper ports (default 1)
net.inet.ip.process_options=0 # IP options in the incoming packets will be ignored (default 1)
net.inet.ip.random_id=1 # assign a random IP_ID to each packet leaving the system (default 0)
net.inet.ip.redirect=0 # do not send IP redirects (default 1)
# these two options may cause problems for jails:
net.inet.ip.accept_sourceroute=0 # drop source routed packets since they can not be trusted (default 0)
net.inet.ip.sourceroute=0 # if source routed packets are accepted the route data is ignored (default 0)
net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent to IP broadcast addresses (default 0)
net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address Mask Request packets (default 0)
net.inet.icmp.maskrepl=0 # replies are not sent for ICMP address mask requests (default 0)
net.inet.icmp.log_redirect=1 # do not log redirected ICMP packet attempts (default 0)
net.inet.icmp.drop_redirect=1 # no redirected ICMP packets (default 0)
net.inet.icmp.icmplim_output=1 # show "Limiting open port RST response" messages (default 1)
net.inet.tcp.always_keepalive=1 # tcp keep alive detection for dead peers, can be spoofed (default 1)
net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on initial connection (default 0)
net.inet.tcp.msl=15000 # 15s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
net.inet.tcp.rfc3042=0 # disable limited transmit mechanism which can slow burst transmissions (default 1)
net.inet.tcp.sack.enable=1 # TCP Selective Acknowledgments are needed for high throughput (default 1)
net.inet.udp.blackhole=1 # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2 # drop tcp packets destined for closed ports (default 0)
net.inet6.icmp6.nodeinfo=0 # disable nodeinfo replies
net.inet6.ip6.use_tempaddr=1 # enable privacy extensions
net.inet6.ip6.prefer_tempaddr=1 #
net.inet6.icmp6.rediraccept=0 # disable redirects
net.inet.tcp.fastopen.server_enable=1 # TCP_RFC7413