An interesting case was reported to us last November by one of our clients. They were getting a DNS leak observation from one Beacon device that was no longer online. Even though the Beacon was taken out of the network, shut down, and was no longer running, they still had a DNS leak showing up in the Beacon Home observation and their dashboard, once every six days.

While we designed SensorFu Beacon to continuously verify network isolation by testing a variety of escape methods, Beacon needs to be online in order for it to work. That doesn’t mean we don’t get surprises every now and then.

A few years back one of our clients’ security teams started to receive Beacon alerts from one of their business-critical remote sites from a retired device. The culprit in that case was a device not being decommissioned and erased properly before being used somewhere else. While that made sense to us, this current situation was a bit more puzzling. A Beacon Ghost Leak? This time Beacon was offline so the source of the observation had to be something other than Beacon itself. Apparently, we’d discovered a feature we didn’t know we had.

“We hadn’t anticipated this, but once we stepped back and considered the DNS caching behavior, it made complete sense. Beacon didn’t create the mystery; it simply revealed it.” –Lauri Jämsä

We learned from our client that they were using a public DNS server whose resolver had cached the escape and was periodically refreshing it. Because public DNS servers maintain cached entries with a certain amount of time-to-live (TTL), whether minutes or days, this particular server was configured to automatically refresh the cached entry just before it expired, even if no one had actively requested it. As a result, the DNS server was periodically asking Beacon Home about the DNS entry which essentially retriggered the leak.

Minor as it may seem, this could still pose a potential risk. We don’t know for certain if the client was expecting communication with a public DNS server, but in general, any outbound traffic, to a public DNS server or somewhere else is probably not the best idea unless it is planned and monitored.

Our client did manage to resolve the periodic observation by deleting Beacon from Home. Something we do recommend to our clients if Beacon is decomissioned. This won’t solve the problem completely as the underlying DNS cache refresh event observation could still occur, but by deleting Beacon from Home there will be no visible observation.

While this type of caching behaviour is somewhat unusual as it’s expected that when the cache expires the DNS server drops the query until someone asks again, we have seen similar cases in the past. This however is the first time we got confirmation that this is how the DNS server was working. DNS continues to be one of the most common leak paths from isolated networks, reminding us that DNS control cannot be left implicit when isolation truly matters.

Ghost Leak: Beacon is already dead and gone, but keeps calling back to Home. was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

For years SSAB has been systematically developing the cybersecurity of industrial systems. As part of these efforts the isolation of networks has also been significantly improved.

Isolation of critical network segments such as industrial control systems is one of the most important cybersecurity policies and best practices. Policies and best practices have always existed, but have evolved over time.

In an increasingly networked world, it has been understood that network isolation in industrial systems is paramount because the system update and life cycles are very long. Often the support for devices has already ended long before the systems can be renewed.

SensorFu Beacon is a solution that allows the functionality of network isolation to be monitored and improved continuously and automatically. Network isolation is also an essential part of SSAB’s company-wide safety ideology.

“Deploying Beacon was extremely easy. Actual deploying took approximately five minutes including the coffee break,” says Pertti Valtonen, OT Service Manager at SSAB.

“Configuring and downloading the beacons from the management interface was smooth, and there was no need to read the instructions at all. SSAB had deployed beacons as both virtual and physical devices. Some minor delays were found when the IT solution supplier did the virtual machine deployment to an HCI environment. SensorFu’s support has also been very well available.”

“After deploying the Beacons, results were found immediately. We caught an old firewall rule that allowed traffic to pass from one network segment to another. After finding this vulnerability the corrections were made quickly.”

The problems found with SensorFu Beacon were not surprising as environments that have been in place for so long have unfortunately deteriorated slightly. There’s been turbulence in the processes and over time old firewall rule openings that were necessary at the time may have remained in the environment.

“When the observations were made, the corrective actions were easy, and thanks to the findings, the isolation of several networks was improved at once. After discovery, it was easy to show what the issue was, and the necessity of a firewall rule was easily decided,” says Valtonen.

According to SSAB’s experience, SensorFu Beacon is a good solution for continuous monitoring of network isolation, based on the findings of which network isolation has been improved. Deploying beacons was easy and they did not cause any unexpected issues in industrial systems, but of course, the implementation had to be done carefully and widely communicated.

“At first, we were cautious about introducing new software into the production network segments, but SensorFu’s references and descriptions of the software’s operation convinced us of the system’s safety. We also tested the operation of SensorFu Beacon initially in a less critical environment.”

From Assumed to Assured — SSAB Validates ICS Segmentation with Beacon was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

At SensorFu, we are committed to serving our customers with seamless deployment solutions and are always eager to collaborate with companies that share our vision. That’s why we’ve teamed up with Eximprod, a Romanian company offering a comprehensive range of equipment, solutions, and services tailored to the energy sector. Through this partnership, Eximprod can offer SensorFu Beacon to their clients.

Eximprod, who specializes in smart grid edge computing solutions, was tasked to deploy 100 SensorFu Beacons and one SensorFu Home to continuously test a European DSO’s network isolation from 100 secondary substations, located geographically in various and often hard-to-reach locations.

What is SensorFu Beacon and how does it work.

SensorFu Beacon was designed to continuously try to find new network leak paths from isolated networks or network segments using two components: Beacon (sensor) and Beacon Home (home server). It is an easy-to-install solution that does not need a cybersecurity specialist to do. It can be used as an application, a virtual machine, or with a device.

Beacons are placed in isolated data networks, from which the beacons try to connect to Beacon Home, using various methods. Beacon Home can be run on the public internet, hosted by us or on-prem, hosted by the customer.

If a beacon manages to connect to Home, it means that there is a gap in the network isolation, and Beacon Home alerts you about the successful connection. Once the gap is known, it can be repaired. Beacon does not require a management connection or any other changes to firewall or network settings to function.

Deployment process with a real-world case.

Since Eximprod already had experience in running third-party applications in Cisco IOx environment, we were there just to offer support if needed. Eximprod had developed their own software RTU product, the ES200 vRTU and deployed it to multiple substations using Cisco IOx devices. They understand that getting new hardware deployed and managed at multiple sites can sometimes be difficult due to company policies, logistics and space constraints. Having the hardware and software orchestration solution on site already, deploying Beacons inside the same hardware was the rational thing to do.

The deployment was supported with

IoT Hardware Platform

• Cisco IR1101 (aarch64)
• Cisco IR809 (amd64)

IoT Software

• Cisco IOx, ES200 vRTU
• SensorFu Beacon

Communications & Security

• Next-gen encryption
• WWAN backup
• Firewall (ACL)

The Beacon Home was installed on-premise in the client’s data center (Debian 12 on a dedicated server) by Eximprod specialists, within the DMZ. Within the first few minutes of active deployment, Beacons managed to escape the network isolation and call Home.

The value given to the client.

By having an immediate alert that there are leaks within their secondary substation network isolation, the client now has the capability to close these leaks before they can be used by malicious actors. As well as trust that they will know if any other leaks occur in the future.

“SensorFu’s Beacon is a game-changer for OT network security”, states the client.

“We were struggling to proactively identify potential vulnerabilities in our highly segmented environment. Your solution provides the critical visibility we need to ensure our network remains truly isolated. The ability to detect and address network leaks before they become a serious issue is invaluable to our operations.”

If you are interested in learning about how SensorFu Beacon can assist you in detecting network isolation leaks, contact us at contact@sensorfu.com

Rapid deployment of Beacons to 100 substations via Eximprod’s ES200 vRTU technology — with… was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In today’s rapidly evolving landscape of technology, networks form the backbone of modern systems. Every second is a race, as malicious actors relentlessly probe for vulnerabilities, seeking their next weak link. But what if we could turn the tables, forcing attackers to question their assumptions and strategies?

That’s precisely what our project sets out to achieve. Developed as part of the University of Oulu’s Software Project course for SensorFu, our approach draws inspiration from Tom Liston’s LaBrea tarpit technique, which traps malicious scans in a “sticky” environment. Building on this idea, we’ve crafted a bold defense strategy that not only slows scans but actively disrupts and deceives attackers. By confusing and overwhelming intruders with false data, we transform defense into offense, leaving them unsure of what’s real.

This project is about more than just protection — it’s about flipping the script. Instead of evading threats, we force attackers to confront an unpredictable and invisible adversary. Welcome to a new era of cybersecurity, where confusion becomes our weapon, and the scanner becomes the prey.

How does scanning work?

Network scanning is a common reconnaissance tactic used by attackers to gather crucial information about a system. But how does it work? At its core, scanning involves identifying devices, discovering open ports, and detecting services or operating systems. The process typically begins with the scanner sending Address Resolution Protocol (ARP) requests to every device on the network. If a device exists at a specific IP address, it responds with its Media Access Control (MAC) address, revealing its presence.

Once active devices are identified, the scanner shifts its focus to finding open ports. For TCP this involves sending SYN packets and waiting for a SYN-ACK response. In some cases, the scanner may send a final ACK packet to establish a connection, but this generates detectable noise. To avoid detection, attackers often use a technique known as a half-open scan, skipping the final ACK. Solutions like Nmap, Masscan, and Nessus utilize methods such as ARP requests and TCP half-open scans, enabling attackers to probe networks for vulnerabilities.

Detecting these scans amidst normal network traffic is challenging. Sophisticated attackers employ low-and-slow scanning techniques, probing only a few ports at a time to remain undetected. These evasive strategies explain why traditional defenses often struggle to identify and block scanning activity.

Network scanning isn’t limited to reconnaissance. In many cyber-attacks, after exploiting a known vulnerability to gain initial access to the internal network, attackers often conduct scans to map the environment and identify connected systems. By uncovering high-value targets, such as sensitive databases or critical servers, they can effectively plan their next steps. These scans demonstrate how attackers continuously adapt to evade detection and maximize the impact of their campaigns.

Payback time

But what if you could turn this challenge into an opportunity? If you suspect your network is being scanned, it’s time to fight back. That’s where our solution comes in — a solution designed specifically for internal networks, one that doesn’t just defend but creates chaos for attackers.

Here’s how it works: When scanners send Address Resolution Protocol (ARP) requests to map the network, our solution intercepts unanswered ARP requests. Most scanners send three requests per IP address. Our solution observes the first two requests to check if a device exists at that IP. If no device responds, it sends an ARP reply to the third request, creating the illusion of a real device at that address. In effect, we populate the network with imaginary machines, tricking scanners into believing they’ve struck gold.

But it doesn’t stop there. When scanners attempt to identify open TCP ports by sending SYN packets, our solution introduces a second layer of disruption. It delays SYN-ACK responses, sending them after a set time. While the delay for a single port may seem minor, the impact compounds when attackers scan thousands of ports across a network teeming with virtual devices. The result? Overwhelming false positives, wasted time, and mounting frustration as their scans yield no actionable data.

In the first test, we scanned the IP range 172.19.0.0/24 using the command nmap 172.19.0.0/24 in a Docker environment without our software running. The scan completed in just two seconds, identifying three active hosts. This result provided a benchmark for evaluating our solution’s impact.

With our solution deployed, the same scan produced dramatically different results:

• Detected Hosts: Nmap identified 256 active hosts, all imaginary devices created by our software.
• Scan Duration: The scan took over 7,500 seconds (more than 2 hours) to complete.

Even after the scan finished, the attacker would need significant time to analyze and verify the results. This delay gives network administrators a critical window to identify and patch vulnerabilities before attackers can act.

We also tested using the nmap -sS 172.19.0.0/24 command, which scans the 1,000 most common ports. Without our software, the scan completed in approximately two seconds, again identifying three active hosts. With our solution, Nmap detected 256 active hosts and required over 7,700 seconds to finish.

Testing was also conducted in real network environments. For example, the nmap -A <target ip> command was tested on a single IP address. The command is used for a thorough scan to gather extensive information about the target. The scan took over six hours to complete for just one host — and the best part is, the host was created by us specifically to slow and deceive!

In comparison same command ran on IP which have actual device with two TCP ports open took roughly about 10 seconds.

This solution isn’t a replacement for firewalls, intrusion detection systems (IDS), or other traditional defenses. Instead, it works alongside them, adding a novel layer of protection. By transforming the attacker’s tactics into an opportunity for disruption, we tip the scales in favor of defenders. In a network fortified with our solution, scanners become mired in false data, delays, and confusion — buying valuable time for administrators to safeguard critical systems.

Summary

Our solution introduces a practical and innovative approach to network defense, transforming the way organizations safeguard their systems. By deploying our solution, you gain a unique advantage: the ability to deceive and disrupt malicious actors while buying valuable time to respond to threats.

Imagine a network with 254 IP addresses, where only 10 have active devices. Using our solution, you can populate the remaining IP addresses with “imaginary” machines, creating a network teeming with seemingly active devices. This tactic forces attackers to waste time scanning and verifying false targets while struggling to distinguish real devices from decoys. Meanwhile, your team leverages this disruption to strengthen defenses and stay ahead of potential intrusions.

Unlike traditional defense mechanisms that often require extensive infrastructure, our solution is lightweight, efficient, and easy to deploy. A single machine is all it takes to protect your entire network. This makes it an ideal solution for large organizations, small businesses, and even individuals seeking to secure their systems without high costs or complexity.

In summary, our solution complements existing cybersecurity measures by adding a dynamic layer of protection that delays, disrupts, and confuses attackers. Lightweight yet powerful, it empowers you to take control of your network security with minimal effort. Deploy our solution today and experience the future of proactive defense!

• GitHub - sensorfu/ants
• LaBrea-Intro History

Toni Perälä

https://www.linkedin.com/in/toni-per%C3%A4l%C3%A4-03908b223/

Tuukka Reinikka

https://www.linkedin.com/in/tuukkareinikka/

Ville Ailunka

https://www.linkedin.com/in/ville-ailunka-45601a2a5/

Scanners Beware: Welcome to the Network from Hell was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of our latest escape methods is the capability send Domain Name System (DNS) queries via a broadcast ethernet packet. We call this the Broadcast DNS escape. Our hypothesis was that DNS server or cache could pick those up from the network and happily redirect it to another network.

And our hypothesis has been proven right. We present you two stories of Broadcast DNS finding real issues in real networks.

Active Directory, Hidden Server

Our client updated Beacons on their site to the latest version which added our new Broadcast DNS test. After the upgrade we were alerted that the Beacon had leaked from the site using this new escape.

Beacon was deployed in a production network — a network to monitor and control industrial processes — and the DNS resolver was properly segmented in the DMZ area. The DNS servers on their site were on a different IP subnet than the Beacon. Beacon had not leaked with unicast DNS before as the firewalls were blocking the traffic between the Beacon and the DNS servers. So how could the DNS leak out via broadcast?

One possibility was that the isolated network containing the Beacon and the network containing the DNS resolvers were accidentally connected together. This might happen due to incorrect cabling or VLAN settings. Networks connected together at ethernet level could forward ethernet frames within those unintentionally connected networks allowing the Beacon to reach the local DNS resolver via broadcast. This means that the networks are not isolated and the segmentation has failed.

Other possibility is that the network has an unknown DNS resolver which accepts the broadcasted DNS query and is allowed to forward it towards the upstream resolver.

After investigating the finding it was discovered that the network actually had an old and supposedly decommissioned Microsoft Active Directory (AD) server still powered on in the network. That AD server also acted as a DNS server which happily processed the broadcasted DNS packets and forwarded them to the upstream DNS infrastructure. Firewall also had a rule to allow DNS traffic out from that particular DNS server.

Interconnected Networks

Another Broadcast DNS escape case soon followed. The company has been using Beacon for over a year now in a production site, deployed in production and IT networks. After upgrading the Beacons to the latest version an unexpected broadcast DNS escape was observed from the Beacon deployed in the isolated production network. In this case the logical hypotheses were the same; unknown DNS resolver in the production network or that the switched network is somehow connected to another network which has a DNS resolver processing and resolving the packet.

After notifying the customer, they quickly investigated the issue and found indications that there were two networks accidentally connected together. Two networks being connected together, the broadcast packets sent by the Beacon in the isolated production network reached the IT network containing the DNS resolver which then forwarded the query to the upstream DNS resolvers all the way to the Beacon Home. The switch ports connecting the networks were identified and the networks disconnected after which the Beacon stopped calling Home.

Afterthought

This kind of broadcast packets (be it DNS, ICMP or something else) take advantage of a weakness in TCP/IP network stacks. The ethernet layer takes the broadcast packet and allows it to flow up in the stack to the next network layer. Then the next layer inspects their part of the packet and acts accordingly, completely missing the fact that it was a broadcast packet and maybe not intended for them to process.

Escaping isolated networks using Broadcast DNS was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bidirectional testing has been on our to-do list for a long time. The feature request was opened on our issue tracker already in April 2017 with a ticket number 73. As with every engineering task there were quite a few “trivial” intermediate steps.

Now we are happy to announce the bidirectional testing was released as part of Beacon 4!

The ticket #73: Triple-Ping for UDP

The ticket from 2017 was brief.

The ticket (by yours truly) in full:

1. Ping from Beacon is received by Receiver. Now Receiver knows that Beacon can send data one-way.
2. Pingback from Receiver to Beacon. With this Beacon knows it can receive data. Or Receiver might get ICMP (or other) error.
3. Pingback from Beacon to Receiver. Now Receiver knows that two-way UDP communication works between Beacon and Receiver.

The trivial intermediate steps

Our original design included only one-way testing and so our cryptographic keys were only deployed for one-way communication: from Beacon to Home. During the development phase of Beacon 3 we added a second pair of keys for supporting two-way communication: from Home to Beacon. While we dive into the code we also found ways to simplify our payload structure by removing some old and unnecessary cruft which made all our escape packets smaller thus causing less traffic for networks. We are still using the XSalsa20Poly1305 authenticated encryption cipher for our encryption between Beacon and Home.

The cryptographic keys are always unique to a Beacon config: One keypair for Beacon to Home communication and another keypair to Home to Beacon communication.

The design and implementation of bidirectional testing

After cryptography primitives were in place we had a good look into design and implementation of the bidirectional testing itself. The original plan was still sound and we just sprinkled it with details. The three messages used for “Triple-Ping” are named Ping, Pong and Ding. Ping contains most of the information since it is also used in unidirectional case: It contains beacon’s name, sequence number and description of the escape: transport type (TCP, UDP, IP, etc.), escape technique (TCP connection, UDP datagram, DNS query, broadcast message, etc.) source and destination IP addresses, maybe related port or protocol number and hop count (used to calculate how many hops the packet traveled).

Pong and Ding messages only contain two values: A sequence number from the Ping message and an identification number to identify this particular bidirectional testing for every other test. No other additional information is needed. And both ways are encrypted with public key encryption.

Every bidirectional test starts with a single Ping message. If Home receives this Ping message it is logged and Alert is sent immediately. Then Home replies with a Pong message towards Beacon. If Beacon can receive Pong it sends Ding back to Home and if Home can receive Ding message it will send another Alert immediately, this time indicating a bidirectional test was possible to escape.

Launch and feedback

The bidirectional testing was released with Beacon 4 and the feedback has been positive. By updating your Beacons to the latest release you’ll get this and other new goodies also!

Bidirectional escape testing was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

We were told a story which piqued our curiosity. Our customer’s security team started to get a flood of Beacon alerts from one of their business critical remote sites. Beacon continuously monitors network isolation and it has been running there already around a year without leakages, but all the sudden all different escapes were able to get through. They contacted the operators of that remote site just to find out that there is no such device on the network. It caused plenty of confusion and doubt about the documentation but one thing was for sure. This Beacon should not be able to call home and the device where it should be running does not exist. It was a mystery.

Discussion continued between the security team and the remote site operators and after a while they found out that the system which was calling home was recently replaced by a new device which explained why the device could not be found. But why is Beacon calling home? It means that the old device is now running somewhere else and this should not happen. All removed devices should be securely erased before retirement. A 3rd party service provider who was responsible for the device replacement was contacted after the incident and bit after they got the message Beacon went offline.

When production equipment, computers and other devices are retired from operation, the normal procedure is that the software they used to run should be wiped and/or the equipment destroyed in a controlled way so that there is no chance for the precious assets or important information to leak outside. In this case, it seems that something went wrong with the process. It happened that someone accidentally connected a piece of retired equipment to the public internet and the Beacon immediately called Home.

Sometimes retired equipment is left around as spare parts to be reused later at the site, or for various reasons dumped into electric waste without properly erasing them. It also can happen that equipment can end up for sale on online marketplaces without data erase. We have read some unfortunate examples all the way from military secrets to personal secrets being exposed and found from the used devices sold on Ebay.

Thankfully this case was caught by the Beacon, the offending device properly decommissioned and our customer’s supply chain made more aware of the process failures in the later stages of the system life-cycle. We believe strongly in continuous monitoring of your security controls, once again it paid off.

Retired Device called Home was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why does DNS leak?

When we deploy new Beacons, usually at least some leaks are observed in the first hours or days. DNS escape is one of those leaks that occur in nearly every new Beacon deployment. DNS is usually considered an essential part of network operation, and its security implications on isolated networks are relatively easy to miss.

DNS converts IP addresses to human readable domain names and vice versa. DNS is a hierarchically organized system with servers for different subdomains. When a user browses to website www.sensorfu.com, the browser asks the resolver configured in the system for the IP address of the www.sensorfu.com. The resolver contacts the root server and asks for the authoritative .com Top Level Domain (TLD) server. Then the .com TLD server is queried for the IP address of the authoritative DNS server responsible for handling sensorfu.com domain. Next the resolver queries that server to retrieve the IP address of the host www.sensorfu.com and returns the IP address to the browser. The browser continues communicating with www.sensorfu.com with that necessary piece of information. See figure 1 for an illustrated recursive DNS lookup. This makes it possible for humans to remember computer and service names by a canonical name, and machines to reach other machines using names in a possibly dynamic IP environment where the IP addresses of the machines might not be static. [1]

Figure 1: Local recursive DNS server resolving “www.sensorfu.com”’s IP-address

DNS is a fundamental protocol of the Internet. Without DNS, finding services or machines on the Internet would be more or less impossible. Nearly all networks utilize DNS one way or another. That includes isolated Operational Technology (OT) networks as well. Historically OT networks and nodes have not really depended on DNS for their main functionality, which is to enable communication between production equipment and stay running by itself for long periods in a relatively statically and pragmatically configured environment. In the modern days a functional name resolving infrastructure becomes more important and even essential in OT networks as well. Current security policies require centralized user credential management, which usually means AD server which in turn requires DNS by design. Periodical operating system security patches and antivirus definition updates are such cases where computers need to find the server from which the patches are available. These addresses are sometimes a dynamic pool of IP addresses in a cloud infrastructure, and therefore the update procedure needs DNS in order to function reliably.

When we have Internet technology (DNS) that enables global connectivity creeping into isolated OT networks it is easy to get more than you asked for. DNS servers might actually do a too good job bridging networks together when they should not.

How does DNS leak?

But why is DNS leak considered a leak? Since DNS is usually considered an essential part of network operation, its security implications are relatively easy to ignore or disregard. DNS is a common way to smuggle data in and out often under the radar. Malicious actors also know this.

To illustrate this try fetching Wikipedia content (from dgl.cx) via DNS by running this command on a terminal:

dig +short txt 'domain_name_system.wp.dg.cx'

Or in a Windows prompt:

nslookup -q=txt domain_name_system.wp.dg.cx

You should get part of the Wikipedia article about DNS in a TXT record. This is one way to transfer arbitrary data in and out using DNS infrastructure. A malicious actor can also obfuscate the data and use other records, like A or CNAME to hide. This example shows that DNS can be used for other purposes beyond the original name resolution. You don’t need direct access to an external DNS server, it is enough that DNS servers in the chain can relay the query to the next hop. If your network is isolated and blocking arbitrary domain name queries this shouldn’t work. [2]

Figure 2: Beacon sends queries to local and public DNS resolvers to communicate with Home

We find DNS leaks by having our Beacon product trying to resolve external entities under our control through local and public DNS resolvers from within the isolated networks (see Figure 2). If it succeeds to reach the “Home” in the public network, then there is a leak.

How to stop DNS leaks?

Fully functioning DNS infrastructure can be used to initiate arbitrary two way communication. When this is undesirable or outright dangerous then the isolated networks should limit or “firewall” unwanted DNS communication.

Conditional forwarding

Simplest way to limit DNS is known as conditional forwarding, where only queries for whitelisted domains are allowed to be forwarded further to the upstream DNS system and thus to be resolved. Queries that do not match the entries in the whitelist will be discarded. In Windows DNS Manager this is easy to do. Caveat is that this method requires a dedicated DNS server for the isolated infrastructure.

It is also necessary to know the domains to be whitelisted. If not already done, the first thing to do is to enable and check the DNS server logs to see what names are being resolved.

https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn800669(v=ws.11)

This example forwards “*.windowsupdate.microsoft.com” queries to the specified servers for resolution. [3]

1. In Windows DNS Manager, right-click on “Conditional Forwarders” and add the allowed domains.

2. Specify servers where the specific DNS queries should be forwarded.

3. Also disable forwarders and recursive forwarding towards root servers from [Server] “Properties” -> “Advanced” “Disable recursion (also disables forwarders)”

How to check conditional forwarding?

You can try resolving both whitelisted and generic domains from the isolated network, this is one way to do it on Windows:

C:\Users\User>nslookup windowsupdate.microsoft.com
Server: localhost
Address: 127.0.0.1
Non-authoritative answer:
Name: windowsupdate.microsoft.com
Address: 20.72.235.82
C:\Users\User>nslookup google.com
Server: localhost
Address: 127.0.0.1
*** localhost can't find google.com: Query refused

Whitelisted domains should resolve normally and other domains should refuse the query.

Other means to limit DNS

Windows 2016 Server introduced a more granular method to limit queries with DNS policies. This allows definition of resolution policies per subnet and does not necessarily need a dedicated DNS server for isolated networks. More information can be found here: https://learn.microsoft.com/en-us/windows-server/networking/dns/deploy/apply-filters-on-dns-queries [4]

Also, you could try firewalling or further isolating your AD server so that it can’t freely connect and resolve DNS but it might be a painful road when trying not to break things, such as Windows updates.

Wrapping it up

Even if you do a good job in firewalling, air gapping or otherwise isolating your OT networks, the DNS might surprise you. We see a lot of OT networks adopting Windows AD and Windows AD depends on DNS. DNS has been built for the Internet and you should take care that your AD doesn’t act as a gateway between the Internet and your OT network. Configuring your AD for conditional forwarding may be the lowest hanging fruit to regain the isolation you were aiming for.

References

1. “Domain Name System”, Wikipedia, https://en.wikipedia.org/wiki/Domain_Name_System,
2. “Wikipedia over DNS”, dgl.cx, https://dgl.cx/2008/10/wikipedia-summary-dns, accessed
3. “Deploy Windows Server Update Services”, Microsoft, https://learn.microsoft.com/en-us/windows-server/administration/windows-server-update-services/deploy/2-configure-wsus#21-configure-network-connections
4. “Use DNS Policy for Applying Filters on DNS Queries”, Microsoft, https://learn.microsoft.com/en-us/windows-server/networking/dns/deploy/apply-filters-on-dns-queries

DNS in isolated networks, does it leak and how to mitigate when it does was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of our isolated network test environments is using the well known OPNsense firewall. It’s a widely used FreeBSD based open source software solution as a firewall and a router. It is a fork of the pfSense firewall software and it uses the Packet Filter (aka PF) ported from OpenBSD as the firewall engine.

To our surprise we noticed IP protocol number 112 was leaking from our environment. This was interesting because the firewall rules were supposed to deny all outgoing traffic from our environment.

Protocol number 112 is assigned by IANA as a VRRP, which stands for Virtual Router Redundancy Protocol (1). But due to historical circumstances the protocol number is also used by CARP, Common Address Redundancy Protocol, developed by OpenBSD.

As per the OpenBSD FAQ pages (2), the CARP is used by redundant routers or other hosts to share a common IP address:

CARP is the Common Address Redundancy Protocol. Its primary purpose is to allow multiple hosts on the same network segment to share an IP address. CARP is a secure, free alternative to the Virtual Router Redundancy Protocol (VRRP) and the Hot Standby Router Protocol (HSRP).

For historical reasons, the OpenBSD CARP was designed to be different from Cisco VRRP to avoid patent issues. But there is the issue of both protocols sharing the same IP Protocol number and MAC addresses (3).

But why would the traffic leak out? A quick check of the firewall rules shows that the OPNSense indeed allows CARP traffic by default from any to any. The PF rule allowing the CARP traffic is “pass log quick proto carp all keep state”. The rule is generated automatically and added to the Floating rules and is not configurable by GUI.

According to PF source code (4) CARP uses the same IP addresses as VRRP. An IP packet destined to a link-local IPv4 multicast address 224.0.0.18 (or IPv6 address FF02:0:0:0:0:0:0:12) should never be forwarded by a router as the requirement of TTL being set to 255 means that the packet should be dropped after the first forwarding device. So, the default rule allowing CARP from any to any should not be a problem if the participating network devices involved in the CARP cluster followed the protocol correctly.

But what happens when the destination address is not a multicast address, or the TTL is not 255? A hand crafted IP packet with protocol 112 and an unicast destination address is flowing through the OPNSense:

…

11:44:24.599836 carp 10.67.0.52 > 10.255.215.100: CARPv0-unknown 0: [ttl=63!] (ttl 63, id 1, len 20)

…

It appears, as expected, that the OPNsense, and all other routers between the Beacon and the Home happily forward the packet if there is a rule allowing it and the destination is not a multicast address. The user may think that the traffic would be totally blocked, which is not true in this case.

The default rules are automatically generated in OPNSense by the file /usr/local/etc/inc/filter.lib.inc. After editing the correct line to explicitly block the CARP traffic, the leaks stopped.

Even if the user specifically tried to block the CARP traffic in floating or interface rules, the traffic would not be blocked because the default rules are on top of the PF ruleset and have a “quick” keyword by default so they override any later lines in PF ruleset.

The automatically generated rule allowing CARP traffic from any to any should be modified so that the allowed traffic is matched better with the specification. It seems that the CARP traffic should not need to go to unicast address in any case. Destination of the packets should only be inbound to the proper multicast addresses which the CARP interface (if exists) listens to:

…

pass in quick inet proto carp from any to 224.0.0.18

pass in quick inet6 proto carp from any to ff02::12

…

Also if it would be possible to match packets with TTL 255 in PF, it should be more in line with CARP protocol.

Do you know how your network infrastructure handles less usual IP protocols?

Update: OPNSense has now incorporated the changes into https://github.com/opnsense/core/commit/c5c1ed91210fe7665ba79e6542f9f2ab17f021e3

Update 28.3.2022: Fixed TTL description.

Update 12.1.2023: PF/OPNsense relation corrections

1. “Protocol Numbers”. Internet Assigned Numbers Authority (IANA) https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
2. OpenBSD PF — Firewall Redundancy (CARP and pfsync) https://www.openbsd.org/faq/pf/carp.html
3. Interview: Ryan McBride https://web.archive.org/web/20040420111759/http://kerneltrap.org/node/view/2873
4. OpenBSD GitHub https://github.com/openbsd/src/tree/master/sys/netinet

Firewall bypass with CARP in OPNsense Packet Filter was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.

On each of the platforms, we started with the same simple setup. An isolated VPC and subnet without any egress gateways attached to the network and a virtual server with no public IP address running SensorFu Beacon.

On AWS and GCP this setup worked as we expected. Without either public IPs or NAT, there were no leaks from direct connections to the Internet. On Azure however, we saw that Beacon was successfully calling home with multiple protocols and ports. We found out that networks without explicit outbound connectivity have an implicit default outbound access. Adding a security group rule blocking egress Internet access stopped the leaks. This served as a reminder that proper firewalling is always important and, of course, reading the documentation.

With direct routes to the Internet plugged, it was time to address the remaining leak vector. This vector was common to all platforms.

It’s Always DNS

Each cloud platform has metadata servers that, among other things, provide recursive DNS resolution. If left unrestricted, this can be used to tunnel out of an otherwise isolated network. Last time we did these tests, only AWS allowed disabling the DNS service. Luckily things have improved since.

Previously, only AWS allowed you to disable the DNS service and use your own DNS servers. Since then the Route 53 Resolver DNS Firewall has been introduced for more fine grained control.

We were glad to find out that both Azure and GCP have since introduced options to disable or filter the DNS service.

In Azure, the DNS service can be blocked by outbound security group rules. Custom DNS servers can then be configured if needed.

GCP does not allow completely disabling the DNS service, but DNS server policies can be used to forward all queries to your own server where they can be filtered.

With these improvements we could now create truly isolated networks on all three platforms. Still, the differences in default behavior and configuration between the platforms are easy to overlook, highlighting the importance of testing the policies.

Revisiting Isolated Networks in the Cloud was originally published in SensorFu on Medium, where people are continuing the conversation by highlighting and responding to this story.