When we started integrating observability tools, the chaos felt impossible to untangle.

• The number of errors was more than we could handle. Some of them were already known, but we just kind of “let them slide.”

• The number of painfully slow queries was more than we could optimize.

Over time (and with a lot of work), we started to overcome these challenges that once felt way too big. Here are a few tips:

• Don’t be too strict with limits in the beginning. Setting a threshold that’s considered high (bad) by industry standards might actually be better than using what “everyone else” uses. That way, you can keep improving steadily, tackling the most critical issues first. Over time, you can lower the threshold and push quality even higher.

• Go way beyond the basics. In addition to classic metrics like machine resources and response times, define business-oriented metrics. That ensures your system is getting better for your users, not just on paper.

• Keep a close eye on signs of regression. Document and set alerts for the goals your team defines, so you can catch regressions as soon as they pop up and avoid letting things pile up.

Bringing this over to security, I see a lot of companies that genuinely don’t know where to start because there’s just so much to fix. If you’ve got an open wound, it’s easy to know what to patch first. But a lot of times, prioritization gets fuzzy because it feels like there’s no way to solve everything in time.

Set a threshold (not too easy, not impossible), and keep fixing things bit by bit—eventually, the tide will turn!

By the way, what do you all use to make sure your products are continuously improving?

For those who don’t know, RSS allows you to follow other content sources by subscribing to a feed. It was mainly used for blogs, but as of today it can be used with other kinds of feeds like podcasts, security vulnerability feeds, Mastodon accounts and many more.

The idea is receiving all the content in one single application, being able to consume text offline, just from the sources you want, in chronological order, with few distractions without other intermediaries between you and the feed you follow.

Reddit provides an RSS feed for every subreddit, which is really cool. However, following subreddits as RSS feeds sucks. There is always too much noise, duplicated posts, things that you would not even notice accessing the website depending on the filters you apply. If you follow many subreddit it becomes unmanageable.

Last Saturday, I decided to fix this because I am on a quest to centralize as much as I can in my RSS reader. Fortunately, Reddit still provides an API that can allow us to fetch the content of a subreddit with filters applied. So I just needed to transform the response from the API into an RSS feed.

A quick search revealed some GitHub projects that could be used for that, but they were too much for what I needed. I am always skeptical of using tools that are too generalist and have lots of dependencies, but I also didn’t want this to be another unfinished project, so I had to draw a line. Thinking about the issue for a while, I decided it needed to:

• Be really simple and have few dependencies
• Consume Reddit API and spit out the RSS (Atom) feed to be subscribed to
• Filter posts by a given number of upvotes for each subreddit
• Provide links to the link shared, if any, and to the post/comments page
• Show the amount of comments
• Show images

An here is the result: reddit-feed-generator

After the setup I quickly noticed I needed a couple of things more to be happy:

• Run the server on my Raspberry Pi (alongside with other utilities). So I would not have the process running on my machine all the time and it would look like I’m fetching from the web.
• Provide an OPML file that can be imported by RSS reader so I don’t have to import every feed manually. Ok, this was not necessary but whatever.

This project excites me because it empowers individuals to reduce reliance on large websites. Computers and the Internet are incredible tools, and the ability to tailor my experience to get more of what I want and less of what I don’t is liberating.

The same method could be applied to other websites that you want to consume. The core idea is to ingest a source (via APIs, web scraping, or existing RSS feeds), filter and modify the content, and produce a customized RSS feed. I don’t expect people to use this code, but I hope it inspires someone to get your hands dirty. Build your own algorithm!

Other cool stuff I’ve found along the way

• hnrss provides custom, realtime RSS feeds for Hacker News
• AboutRSS is a collection of links related to RSS

I’m not talking about UI testing here, so assume everything I’m saying is about server-side JS. If you’re happy with Jest, don’t switch! I’m not trying to convince anyone. I’m just sharing my experience because I bet there are people out there in the same situation as I was.

The decision to go with Jest wasn’t well thought out. At the time, I wasn’t heavily involved in software development, so I opted for the well-known, fully-fledged test framework. It seemed promising initially, with its great API.

However, after hundreds of tests, things started to break a lot. Memory leaks began to surface, the number of hacky flags multiplied, and visiting Jest’s issues tab became routine.

The problem

Jests loads modules over and over again, leading to a memory leak. It is designed in a way that makes those problems arise very likely, specially in backend projects that are getting big. I’m not going to try to describe the kind of issues here, but you can find them pretty easy there. I’m just going to leave some leads for the poor users that are still fighting this battle.

1. jest —logHeapUsage
   As I’ve mentioned before, memory leaks will be your best friend, so you’d better watch your heap usage to spot sudden growth.

2. jest —maxWorkers=50%
   Some benchmarks show that running with this configuration might make your tests run up to 20% faster. However, you have to test it in your project; some folks said it made things worse.

3. jest —runInBand
   This command runs all tests serially in the current process instead of creating a worker pool of child processes. They say it’s useful for debugging, but oddly enough, some people reported that it can actually improve performance.

4. jest —changedSince
   If you run a workflow for each PR update, your tests will run a LOT. So, if you have limited compute minutes or a constrained CI pipeline, this flag can significantly reduce the time your PR workflows take.

5. jest-slow-test-reporter
   You can use this reporter to view which tests are the slowest in your project and start tackling them first.

6. Exposing Node.js gargabe collector
   In some cases, running Node with the flag --expose-gc seems to handle memory leaks better.

Not good enough

Some of these strategies significantly reduced the running time during this period. However, the process of learning and implementing them came at the expense of delivery time, which is ultimately more critical.

The tests were so sluggish that I resorted to running them only on the module we were currently developing, then solely on the changed modules in the PR, and finally, all tests were run only when merging to the main branch. Unfortunately, this approach resulted in delays in identifying bugs.

The tests were so time-consuming that I found myself hesitating to write tests for certain features, worrying about the additional build process time they would cause. At this point, I realized it was time to make the switch.

Switching to Mocha

I used Mocha a decade ago and it was awesome. So, I thought transitioning back to it would be smooth sailing. Over the past years, I’ve been seeing people throwing out the idea of switching from Jest to Mocha, and always I found it funny. I remember there were a lot of guides and people talking about migrating from Mocha to Jest. Like me, most of people would assume the newer tool would have a better or at least similar performance.

The migration was much easier, than expected. A couple of replace cases and and less than an hour refactoring some code. The tricker part was the mock engine, which isn’t included in Mocha.

I could have used Sinon.js for that, but I really like the idea of one day not relying on any testing libraries. I even considered using only the new Node.js built-in test runner, but it’s not quite there yet for me. So, I decided to stick with just the built-in MockTracker.

Trying things out was mind-blowing. Single tests that were taking 3 seconds with Jest would run in less than 200ms with Mocha. It shouldn’t be a surprise – what I was running shouldn’t take so long, but I had gotten used to that slowness. In the end, our test running time reduced from more than 12 minutes to less than 40 seconds.

The speed of Mocha helped us uncover hidden bugs that would occasionally fail tests because they only occurred under very specific conditions – conditions that were unlikely to be met in Jest due to its slower performance.

Conclusion

I still use Jest on some smaller repositories that I maintain, and I’m not mad enough to migrate them until they become a problem. However, for future projects, I’ll definitely opt for Mocha or the Node.js test runner.

The thing is, even if there’s a way to optimize Jest and run thousands of tests in a reasonable time, there’s something wrong when simply switching the testing framework results in significantly faster performance. Do you agree? Do you have any similar experiences? I’d love to hear about them.

You can indeed make money producing content online. It is also true that social media platforms can boost your reach. However, the majority of the population is just consuming and creating poor content that will never lead to a financial reward.

I would consider myself in the above situation. I’m not in a fair trade with social media. Not that I ever felt I could be. But the fear of missing out makes me play the game without even thinking about it.

Most of the value generated from our actions is turned into revenue for the big techs. In exchange for that, you will be gifted with a subtle dependency on the feedback you could receive if you saw something good enough.

If you can’t escape from it, maybe it’s a good idea to spend some time implementing filters to avoid the consumption of unhealthy content. You need to feed yourself, just not with junky food.

Enumerating subdomains with wildcard records is tricky but not impossible, here are some tips. Also, don’t trust wildcards as a security mechanism for hiding sensitive apps.

The problem

If you did some subdomain brute-force enumeration in the wild you already bumped into a record that resolves for any type of prefix. This is called a wildcard record and it can be configured by inserting a record entry with a label “*”. This record will also resolve for other sublevels unless it is inhibited by another record entry.

Many companies use wildcard records as part of their architecture. A well-known example is Slack which uses it for their workspaces. For example, today I asked my favorite DNS server to resolve the following records and got the same IP address:

shopify.enterprise.slack.com 18.231.0.250
enterprise.slack.com 18.231.0.250
big-name-non-existent.slack.com 18.231.0.250

In this case you might conclude that there is a wildcard record *.slack.com and maybe we should ignore this domain in your subdomain enumeration. However, you could end up missing something like status.slack.com which does not resolve to this address. Instead it has a CNAME pointing to another infrastructure that could be interesting to you.

It is curious how often subdomain enumeration tools mess up or do not handle this kind of behaviour. Many times the wildcard records are just dropped without any further check. The problem is that you might lose some interesting apps by discarding them .

With that in mind, adding a wildcard record can be a tempting strategy to hide your own services like a needle in the haystack. I can’t blame anyone for doing that, but just keep in mind that this is not going to save you for long.

Finding interesting stuff

Thinking about how to make a better reconnaissance one could try to overcome this problem by treating enumeration in wildcard records differently. The response returned by the wildcard could be stored (sorted if it is multiple entries) and every subsequent DNS response would be compared with this one. Everytime we find a new response it would be saved in a map structure.

This would make sure we have at least one subdomain that points to that new location that we found. However, the world ain’t all sunshine and rainbows and we could obviously have a different application sitting on a machine that will only show up when we set a specific Host header in the HTTP request.

Therefore, this is just something you could use to have more places to look for security vulnerabilities. There are many other more edgy cases (for example when including CNAME) that can happen when trying to find assets using DNS. I hope I can dig into that more in future posts.

Do you have any tips for finding apps on records with wildcard?

Firstly, make sure you have extracted the zip file. Using the file command we can verify that the firmware was compiled for Linux MIPS.

$ file fw_NPLUG_1_0_0_14.bin
fw_NPLUG_1_0_0_14.bin: u-boot legacy uImage, Linux Kernel Image, Linux/MIPS, OS
Kernel Image (lzma), 1731916 bytes, Wed Oct 12 17:28:28 2016, Load Address:
0x80000000, Entry Point: 0x802CB000, Header CRC: 0xEC8AD091, Data CRC:
0x3A3E5EAC

There is not much you can do with this file as it is. In order to extract useful stuff from it we need to find the offset of known file format signatures inside the binary. Once you know the offset you could extract the chunks using the Linux dd tool.

However, it is not necessary to do that manually, instead, we will use a tool called Binwalk. This tool was created by Craig Heffner and it helps a lot in the process of extracting file systems from firmware files.

Using Binwalk

Running Binwalk against the firmware without specifying any options would give us the following output:

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             uImage header, header size: 64 bytes, header CRC: 0xEC8AD091, created: 2016-10-12 17:28:28, image size: 1731916 bytes, Data Address: 0x80000000, Entry Point: 0x802CB000, data CRC: 0x3A3E5EAC, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
64            0x40            LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 3829956 bytes

That means the binary has a header and some compressed data starting at offset 0x40. This assumption might be true or not, the tool is just trying to interpret the bytes in the file. Using the xxd tool we can examine the first bytes from the binary which refer to the header:

$ xxd -l 64 fw_NPLUG_1_0_0_14.bin
00000000: 2705 1956 ec8a d091 57fe 72bc 001a 6d4c  '..V....W.r...mL
00000010: 8000 0000 802c b000 3a3e 5eac 0505 0203  …..,..:>^.....
00000020: 4c69 6e75 7820 4b65 726e 656c 2049 6d61  Linux Kernel Ima
00000030: 6765 0000 0000 0000 0000 0000 0000 0000  ge..............

Binwalk assumed the following chunk was related to some LZMA compressed data because of the signature 5d 00 00 00 02. This signature might vary depending on the compression level used, but apparently it always start with 5d 00 00. Here is the start of the compressed data:

[...]
00000040: 5d00 0000 02c4 703a 0000 0000 0000 006f  ].....p:.......o
00000050: fdff ffa3 b7ff 473e 4815 7239 6151 b892  …...G>H.r9aQ..
00000060: 28e6 a386 07f9 eee4 1e82 d32f c53a 3c01  (........../.:<.
00000070: 4bb1 7ec9 8a8a 4d2f a30d d97f a6e3 8c23  K.~...M/.......#
00000080: 1153 e059 18c5 758a e277 f886 f31f 9269  .S.Y..u..w.....i
00000090: 516e f9a2 84d4 b228 94f8 6af5 1bf9 141e  Qn.....(..j.....
[...]

The -e (--extract) flag, will extract the files identified during a signature scan and the -M (--matryoshka) flag will perform this extraction recursively. There is also another really useful option in Binwalk which allows an entropy analysis of the file. This can be helpful when Binwalk doesn’t work properly on the target file and you have to do a manual analysis.

For example, taking a step back, if we used Binwalk extraction only once, we would find a file named 40 (this number refers to its offset in hexadecimal). Using Binwalk with the -E (--entropy) flag in this file will pop the following line chart (generated with pyqtgraph).

The Y axis is the entropy and the X axis refers to the offset in that file. With this graph we can infer that there are some readable stuff in that file but there are some other regions with high entropy. A high entropy in a file probably means compressed or encrypted data, but it could also be just garbage.

The signature scan for this file would give us the following output:

$ binwalk 40
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
2658360       0x289038        Linux kernel version "2.6.21 (root@linux-qxix)
(gcc version 3.4.2) #1086 Thu Oct 13 01:28:23 CST 2016"
2659360       0x289420        CRC32 polynomial table, little endian
2686144       0x28FCC0        SHA256 hash constants, little endian
2728378       0x29A1BA        Unix path: /mru/rcvseq/sendseq/lns debug
reorderto
2736880       0x29C2F0        Unix path: /etc/Wireless/RT2860AP/RT2860AP.dat
2737968       0x29C730        XML document, version: "1.0"
2870768       0x2BCDF0        CRC32 polynomial table, little endian
3059712       0x2EB000        LZMA compressed data, properties: 0x5D,
dictionary size: 1048576 bytes, uncompressed size: 2943488 bytes

Performing another extraction with Binwalk in the file 40 will output another file named 2EB000. We can generate the entropy analysis line chart for the new file too.

This time we can see a lot of regions in the file with data that is probably readable. Not surprisingly, we have found the file system of the firmware. Of course, we could have simply extracted everything with just one command like the following demonstration:

$ binwalk -eM fw_NPLUG_1_0_0_14.bin
$ cd fw_NPLUG_1_0_0_14.bin.extracted/_40.extracted/_2EB000.extracted/cpio-root/
$ tree
.
├── bin
│   ├── ash -> busybox
│   ├── ate
│   ├── ated
│   ├── busybox
│   ├── cat -> busybox
│   ├── chmod -> busybox
│   ├── cp -> busybox
│   ├── date -> busybox
│   ├── default.cfg
│   ├── dnrd
│   ├── echo -> busybox
│   ├── hostname -> busybox
│   ├── httpd
│   ├── inadyn
│   ├── iptables
│   ├── iwconfig
│   ├── iwpriv
│   ├── kill -> busybox
│   ├── ls -> busybox
│   ├── miniupnpd
│   ├── mkdir -> busybox
│   ├── mount -> busybox
│   ├── msg
│   ├── mtd_write
│   ├── netctrl
│   ├── nvram_clear -> ralink_init
│   ├── nvram_commit -> ralink_init
│   ├── nvram_get -> ralink_init
│   ├── nvram_set -> ralink_init
│   ├── nvram_show -> ralink_init
│   ├── ping -> busybox
│   ├── pppd
│   ├── ps -> busybox
│   ├── ralink_init
│   ├── rm -> busybox
│   ├── sh -> busybox
│   ├── sntp
│   ├── switch
│   ├── tenda_chck_conn
│   ├── tenda_deamon
│   ├── tquser
│   ├── udhcpc
│   └── udhcpd
├── dev
│   └── pts
├── etc
│   └── fstab
├── etc_ro
│   ├── inittab
│   ├── linuxigd
│   ├── miniupnpd.conf
│   ├── motd
│   ├── ppp
│   │   ├── 3g
│   │   ├── ip-up
│   │   ├── peers
│   │   └── plugins
│   │       └── rp-pppoe.so
│   ├── rcS
│   ├── usb
│   ├── web
│   │   ├── blank.htm
│   │   ├── css
│   │   ├── direct_reboot.asp
│   │   ├── direct_reboot_wizard.asp
│   │   ├── error.asp
│   │   ├── firewall_clientfilter.asp
│   │   ├── firewall_clientsetting.asp
│   │   ├── firewall_mac.asp
│   │   ├── firewall_macsetting.asp
│   │   ├── firewall_packagefilter.asp
│   │   ├── firewall_urlfilter.asp
│   │   ├── firewall_urlsetting.asp
│   │   ├── images
│   │   │   ├── bg_button.gif
│   │   │   ├── bg_button_over.gif
│   │   │   ├── bt_disable2.png
│   │   │   ├── bt_disable.png
│   │   │   ├── bt_in_long.PNG
│   │   │   ├── bt_long.PNG
│   │   │   ├── bt_mouse-in2.png
│   │   │   ├── bt_mouse-in.png
│   │   │   ├── bt_nomal_login.gif
│   │   │   ├── bt_normal2.png
│   │   │   ├── bt_normal.png
│   │   │   ├── bt_over_login.gif
│   │   │   ├── bt_press2.png
│   │   │   ├── bt_press.png
│   │   │   ├── load_bg.png
│   │   │   ├── loading.gif
│   │   │   ├── login_bg.jpg
│   │   │   ├── login_left.gif
│   │   │   ├── login_right.gif
│   │   │   ├── login_top.gif
│   │   │   ├── menubg.gif
│   │   │   ├── menu_bg.jpg
│   │   │   ├── nav_normal.gif
│   │   │   ├── ok.png
│   │   │   ├── product.png
│   │   │   ├── top.png
│   │   │   └── wan.png
│   │   ├── ipMacBind.asp
│   │   ├── js
│   │   │   ├── encode.js
│   │   │   ├── menus_list.js
│   │   │   └── system_tool.js
│   │   ├── lan.asp
│   │   ├── lan_dhcp_clients.asp
│   │   ├── lan_dhcps.asp
│   │   ├── login.asp
│   │   ├── mac_clone.asp
│   │   ├── main.htm
│   │   ├── mode_select.asp
│   │   ├── nat_dmz.asp
│   │   ├── nat_virtualportseg.asp
│   │   ├── net_tc.asp
│   │   ├── password_error.asp
│   │   ├── password_succeed.asp
│   │   ├── public
│   │   │   ├── base.css
│   │   │   ├── common.css
│   │   │   ├── gozila.js
│   │   │   ├── j.js
│   │   │   ├── menu.js
│   │   │   ├── page_div.css
│   │   │   ├── page_div.js
│   │   │   ├── style.css
│   │   │   └── table.js
│   │   ├── reboot.asp
│   │   ├── routing_static.asp
│   │   ├── routing_table.asp
│   │   ├── system_backup.asp
│   │   ├── system_hostname.asp
│   │   ├── system_log.asp
│   │   ├── system_password.asp
│   │   ├── system_reboot.asp
│   │   ├── system_remote.asp
│   │   ├── system_restore.asp
│   │   ├── system_status.asp
│   │   ├── system_upgrade
│   │   ├── upgrading.asp
│   │   ├── upnp_config.asp
│   │   ├── wan_connected.asp
│   │   ├── wan_dns.asp
│   │   ├── wan_portParam.asp
│   │   ├── wireless_basic.html
│   │   ├── wireless_filter.html
│   │   ├── wireless_scan.asp
│   │   ├── wireless_security.html
│   │   ├── wireless_state.html
│   │   └── wizard.asp
│   ├── Wireless
│   │   ├── iNIC
│   │   ├── RT2860AP
│   │   │   ├── RT2860_default_novlan
│   │   │   └── RT2860_default_vlan
│   │   └── RT61AP
│   ├── wlan
│   │   └── RT5350_AP_1T1R_V1_0.bin
│   └── xml
├── home
├── init -> bin/busybox
├── lib
│   ├── ipsec
│   ├── ld-uClibc-0.9.28.so
│   ├── ld-uClibc.so.0 -> ld-uClibc-0.9.28.so
│   ├── libcrypt-0.9.28.so
│   ├── libcrypt.so -> libcrypt-0.9.28.so
│   ├── libcrypt.so.0 -> libcrypt-0.9.28.so
│   ├── libc.so -> libuClibc-0.9.28.so
│   ├── libc.so.0 -> libuClibc-0.9.28.so
│   ├── libdl-0.9.28.so
│   ├── libdl.so -> libdl-0.9.28.so
│   ├── libdl.so.0 -> libdl-0.9.28.so
│   ├── libiw.so.29
│   ├── libnvram-0.9.28.so
│   ├── libnvram.so -> libnvram-0.9.28.so
│   ├── libnvram.so.0 -> libnvram-0.9.28.so
│   ├── libpthread-0.9.28.so
│   ├── libpthread.so -> libpthread-0.9.28.so
│   ├── libpthread.so.0 -> libpthread-0.9.28.so
│   ├── libresolv-0.9.28.so
│   ├── libresolv.so -> libresolv-0.9.28.so
│   ├── libresolv.so.0 -> libresolv-0.9.28.so
│   ├── libtenda_common.so
│   ├── libuClibc-0.9.28.so
│   ├── libutil-0.9.28.so
│   ├── libutil.so -> libutil-0.9.28.so
│   ├── libutil.so.0 -> libutil-0.9.28.so
│   └── modules
│       ├── 2.6.21
│       │   └── kernel
│       │       └── net
│       │           └── netfilter
│       │               └── xt_webstr.ko
│       ├── ipt_httpredirect.ko
│       └── xt_turbo_qos.ko
├── media
├── mnt
├── proc
├── sbin
│   ├── arp -> ../bin/busybox
│   ├── halt -> ../bin/busybox
│   ├── ifconfig -> ../bin/busybox
│   ├── init -> ../bin/busybox
│   ├── insmod -> ../bin/busybox
│   ├── lsmod -> ../bin/busybox
│   ├── poweroff -> ../bin/busybox
│   ├── pppoe.sh
│   ├── reboot -> ../bin/busybox
│   ├── rmmod -> ../bin/busybox
│   ├── route -> ../bin/busybox
│   └── vconfig -> ../bin/busybox
├── sys
├── tmp
├── usr
│   ├── bin
│   │   ├── expr -> ../../bin/busybox
│   │   ├── killall -> ../../bin/busybox
│   │   └── top -> ../../bin/busybox
│   ├── codepages
│   └── sbin
│       ├── brctl -> ../../bin/busybox
│       └── telnetd -> ../../bin/busybox
└── var

41 directories, 185 files

Conclusion

In this post we learned how to extract the file system from a firmware file. Sometimes this task can be much harder than what was demonstrated here. A firmware file can be constructed in many different ways which makes it difficult to analyze a bunch of them.

In the next part I am planning to go through the file system trying to find interesting stuff, stay tuned!

References

• Binwalk
• Notes on the LZMA format
• /dev/ttys0

SPOILER ALERT: This is post will give the solutions for the first set of exercises, if you are planning to do it, stop reading this now and try them by yourself. However, if you are still unsure if this is for you or not, maybe try skim reading through this post to see if you get interested.

While resolving the first set I had mixed feelings of love and hatred. Sometimes I was able to get the idea but ended up making a programming mistake that would keep me stuck for while in one challenge or another. They stated in their website that this is a “relatively easy” set of challenges with the except of one. I can’t tell you about the further sets but one challenge in this set took me 2 days to get it right.

I decided to write my solutions in Node.js and this was a good way to learn about some quirks of the language that I was not used to face in a common daily programming session. You might also want to choose a new programming language to kill two birds with one stone.

Challenge 1: Convert hex to base64

There is no mystery in here. If you were dealing with client-side JavaScript you would be looking for atob and btoa, but in Node.js you will want to be familiar with the Buffer API. Needless to say, you shouldn’t include any third-party libraries to solve the challenges.

var result = Buffer
  .from('49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d','hex')
  .toString('base64')

Challenge 2: Fixed XOR

This challenge will introduce you to XOR operation for cryptographic purposes. The simple XOR cipher is a type of additive cipher that was used in the past to encrypt text, but it is clearly not safe since the ciphertext could be decrypted using character frequency analysis. On the other hand, the XOR operation is still frequently used in more complex ciphers, due to its simplicity and performance.

The challenge asks for a function that takes two equal-length buffers and produces their XOR combination. A solution for this exercise would be to loop through one of the buffers doing a XOR with each correspondent byte from the other buffer. The code that can do this will be shown further in this post.

Challenge 3: Single-byte XOR cipher

Now that we already know how to cipher and decipher messages using XOR, things are going to become interesting. The challenge presents a hex encoded string that was XOR’d against a single character and require us to discover this key and decrypt the message. This can be achieved by brute-forcing the key and analysing the character frequency of the output messages. The output with the best score is probably the message decrypted.

/**
 * Break a single-byte XOR cipher and returns the message with the best score
 * @param {String} hex encoded ciphertext
 * @return {Object} key, msg and score
 */
function breakSingleByte (ciphertext) {
  var plainText, tempBuffer
  var results = []

  for (let keyChar = 0, len = 256; keyChar < len; keyChar++) {

    tempBuffer = cipher(
      Buffer.from(ciphertext, 'hex'),
      Buffer.from(String.fromCharCode(keyChar))
    )

    plainText = tempBuffer.toString()

    if (!/[^\x00-\x7E]/g.test(plainText))
      results.push({
        key: keyChar,
        msg: plainText,
        score: parseFloat(Util.calculateFrequency(plainText).toFixed(6))
      })
  }

  return results.reduce((x, y) => (x && x.score > y.score) ? x : y, 0)
}

To calculate the score of a string according to the frequency of english letters you can find the percentual values on the Internet and sum them. Don’t forget to add the space character to your list, it took me some time to notice that.

/**
* Computes the score of a string according to the frequency of letters
* @param {String} string
* @return {Number} score
*/
function calculateFrequency (string) {

  /* Source: http://www.oxfordmathcenter.com/drupal7/node/353 */
  const charFrequency = {
    'e': 0.12702, 't': 0.09056, 'a': 0.08167, 'o': 0.07507, 'i': 0.06966,
    'n': 0.06749, 's': 0.06327, 'h': 0.06094, 'r': 0.05987, 'd': 0.04253,
    'l': 0.04025, 'c': 0.02782, 'u': 0.02758, 'm': 0.02406, 'w': 0.02360,
    'f': 0.02228, 'g': 0.02015, 'y': 0.01974, 'p': 0.01929, 'b': 0.01492,
    'v': 0.00978, 'k': 0.00772, 'j': 0.00153, 'x': 0.00150, 'q': 0.00095,
    'z': 0.00074, ' ': 1
  }

  return string.toLowerCase().split('')
    .reduce((sum, value) => sum + (charFrequency[value] || 0), 0)
}

If everything works fine, we will get the decrypted message:

Cooking MC\'s like a pound of bacon

Challenge 4: Detect single-character XOR

This time we were given a file containing a lot of encrypted strings. The goal was to find the string that was encrypted with single-character XOR, but we know from the previous exercise that decrypting it would be really easy too. We can use the function from the previous challenge to brute-force each string as a single-byte XOR cipher and find out the one with the best score.

var decrypted = [], result

strings.forEach((string, index) => {
  result = XOR.breakSingleByte(string)

  if (result) {
    result.string = string
    result.line   = index + 1
    decrypted.push(result)
  }
})

result = decrypted.reduce((x, y) => x.score > y.score ? x : y)

If everything works fine, we will get the decrypted message:

Now that the party is jumping\n

Challenge 5: Implement repeating-key XOR

This challenge is similar to the Challenge #2, but this time the message length is bigger than the key length, so we will use the Vigenère cipher. In this case, the key needs to be repeated to cipher the whole message. Maybe this challenge could be positioned after the second one.

In order to reuse code, some of the functions were created in a separated file. Since XOR is a commonly used operation, I decided to create a function that will XOR two buffers repeating the smallest one which would be the key. This function could be used with fixed or repeating-key XOR.

/**
 * XOR two buffers repeating the smallest
 * @param {Buffer} msg
 * @param {Buffer} key
 * @return {Buffer} result
 */
function cipher (msg, key) {

  if (key.length > msg.length)
    [msg, key] = [key, msg]

  var result = Buffer.alloc(msg.length)

  for (var i = 0, j = 0, len = msg.length; i < len; i++, j++)
    result[i] = msg[i] ^ key[j % key.length]

  return result
}

Ciphering the message with the above function and converting the result to a hex string will give the expected output.

Challenge 6: Break repeating-key XOR

“It is officially on, now.” This one took me a while to make it work right. They give you a file containing a message that is encrypted with repeating-xor and encoded in base64. This challenge is when you put together everything you learned until now. Fortunately, they give you the steps to do it.

In order to get this one right, you will need to use something called the Hamming distance, which is the number of different position between two symbols. In this case, we need to calculate the number of differing bits between the two strings.

/**
* Calculates the Hamming distance between two strings
* @param {String} string1
* @param {String} string2
* @return {Number} distance 
*/
function calculateDistance (string1, string2) {
  var distance = 0
  var first, second

  string1 = Buffer.from(string1 || '0', 'ascii')
  string2 = Buffer.from(string2 || '0', 'ascii')

  // compose the binary representation of the strings 
  for (let i = 0, len = string1.length; i < len; i++)
    first += dec2bin(string1[i])

  for (let i = 0, len = string2.length; i < len; i++)
    second += dec2bin(string2[i])

  // count the number of different bits
  for (let i = 0, len = first.length; i < len; i++)
    if (first[i] ^ second[i])
      distance++

  return distance
}

/**
* Converts a decimal number to a 8 char binary representation
* @param {Number} decimal number
* @return {String} binary representation
*/
function dec2bin (dec) {
  var binaryNum = (dec >>> 0).toString(2);
  while (binaryNum.length < 8)
    binaryNum = '0' + binaryNum
  return binaryNum
}

The first thing you need to do is to figure out the size of the key. Unless you have a crystal ball at home, you can start guessing. They suggest values from 2 to 40.

/**
* Test all key sizes from 2 to 40 and returns the best one
* @param {String} msg
* @return {Number} keysize
*/
function findKeySize (msg) {
  var results = []

  for (let keysize = 2, len = 40; keysize <= len; keysize++)
    results.push({
      keysize : keysize,
      score   : testKeySize(msg, keysize)
    })

  return results.reduce((x, y) => x.score < y.score ? x : y).keysize
}

If you are wondering what kind of test is that, no worries, they explain you how to do it. It basically consists of getting the first two chunks of KEYSIZE bytes from the ciphertext, computing the Hamming distance between them, repeating this process a couple of times and normalizing the result. The KEYSIZE which gives the least distance between the chunks of bytes in the ciphertext is probably the right one.

/**
* Given a message and the size of a key, returns the average hamming distance
* of between chunks of that message
* @param {String} msg
* @param {Number} keysize
* @return {Number} distance
*/
function testKeySize (msg, keysize) {
  var first, second
  const TESTS = 50 // arbitrary
  var score   = 0

  for (let i = 0, len = TESTS; i < len; i++) {
    first  = msg.slice(keysize * i, keysize * (i + 1))
    second = msg.slice(keysize * (i + 2), keysize * (i + 3))
    score += Util.calculateDistance(first, second)
  }

  return score / (TESTS * keysize)
}

Once you have the KEYSIZE, you can use the following strategy to solve the challenge:

1. Divide the ciphertext into blocks of KEYSIZE length;
2. Transpose the blocks: make a block that is the first byte of every block, and a block that is the second byte of every block, and so on;
3. Solve each block as it was single-character XOR;
4. For each block, the single-byte XOR key that produces the best output is the XOR key byte for that block;
5. Put them together and you have the key;
6. Decipher the message with the key you obtained.

var key, keysize, blocks, msg, result
var keyScores = []

msg     = Buffer.from(fs.readFileSync('../res/6.txt', 'utf-8'), 'base64')
keysize = XOR.findKeySize(msg)
blocks  = Util.divideInBlocks(msg, keysize)
blocks  = Util.transposeBlocks(blocks, keysize)
key     = findKey(blocks)
result  = XOR.cipher(msg, key).toString()

Using this key against the file will give you the lyrics of this song.

Challenge 7: AES in ECB mode

After solving the previous challenge the last two seemed like a walk in the park. This is an introduction to the block cipher AES. A block cipher is a deterministic algorithm which operates on fixed-length groups of bits. Examples of block ciphers are DES, RC5, Blowfish, AES. The following code will do the job of decrypting a base64 encoded ciphertext in Node.js:

/**
* Decrypts a AES ciphertext with a given key and algorithm
* @param {String} ciphertext
* @param {String} key
* @param {String} algorithm
* @return {String} msg
*/
function decrypt (ciphertext, key, alg) {
  var decipher = crypto.createDecipheriv(alg, key, null)
  var out = decipher.update(ciphertext, 'base64', 'utf-8')
  return out += decipher.final('utf-8')
}

If everything works fine, we will get the same lyrics of the previous challenge.

Challenge 8: Detect AES in ECB mode

This challenges requires you to find which one of the ciphertexts in a file was encrytpted with AES in ECB mode. The ECB mode is simplest mode of encryption in which the message is divided into blocks and they are independently encrypted. The problem with this mode is that a block is always encrypted into the same ciphertext block. This is exactly the weakness you need to abuse in order to detect which ciphertext was encrypted in ECB mode, you will count the repetitions.

ciphertexts.forEach((ciphertext, index) => {
  var cipherTextBlocks = Util.divideInBlocks(ciphertext, 16)

  cipherTextBlocks.forEach((block, i) => {
    results.push({
      line: index + 1,
      ciphertext: ciphertext,
      repetitions: Util.countRepetitions(block, cipherTextBlocks)
    })
  })
})

Conclusion

The complete source code of the solutions you can find on my Github. If you solved this first set of exercises you have the knowledge of performing the following tasks:

• Convert between data base64, hex, binary
• Detect and decrypt single-character XOR ciphers
• Break repeating-key XOR ciphers
• Detect and decipher AES in ECB mode

I am planning on doing the other sets soon. I will probably release the solutions on Github and make my comments about them here. Stay tuned!

References

• The cryptopals crypto challenges

For the first part, we are going to understand the basics and learn how to collect information about our target. If you have any questions, please leave a comment below so I can improve this article.

What is a firmware?

The term “firmware” was firstly used in a magazine, published in 1967, and it was referring to the CPU microcode existent between the hardware and the software. Today, the term is more broadly used to refer to the BIOS, bootloaders or embedded management systems.

There are many different kinds of firmwares files. It could be a complete operation system with custom drivers and user applications, or it could contain only the user applications. Sometimes, it could be composed only of partial updates for an application, for example, the front-end of an embedded web server.

Why should we do this?

SOHO routers are the front door of your home network, therefore, a security issue in this device might compromise all the connected clients. On the other hand, SOHO routers are known for having security issues that are widely exploited by malwares. In addition, the source code of the firmware is usually not public. Unless someone does a reverse engineering on it, we won’t be able to know what is actually running inside it.

Another motivation is learning. Reversing SOHO routers is a good way to study about common low-level vulnerabilities and reversing due to its simplicity compared to modern systems.

Information gathering

For our case study we are going to use an old wireless repeater named NPLUG, previously offered by the Brazilian company Intelbras. Unfortunately, I couldn’t find this product on their website anymore, but this is actually good for everyone. :)

Firstly, it is good to gather as much information about the device as you can. This allows us to get a general idea of how it works and uncover some weakness without the need to look inside. An easy way to do that is looking for the FCC ID number of the device. The FCC ID is the registration number of that device in Federal Communications Commission of United States. Using this number you can find more information like user manuals, specifications and internal pictures.

Once you have that number you can search for it using sites like FCC ID.io or the official FCC page. Depending on where are your device distributed you might not find a FCC ID number on it, but another certification number instead.

In Brazil, Anatel is the agency responsible for registering and regulating electronic devices. Taking a look at the back of our device, between other information, we can find a number near the Anatel logo. This is the Anatel’s certification number and, fortunately, FCC ID.io has a database for that too.

I couldn’t find much information (beyond what was on the vendor’s website) about the device using this number. However, if you are into embedded devices you might have noticed similarities between the NPLUG and other wireless repeaters. I had the same feeling and while looking at similar devices I discovered a “cousin” of NPLUG, which is the Tenda A5s.

Searching for “fcc id tenda a5s”, I was able to find the FCC ID.io page for the Tenda product and within it I could internal pictures of the device. Looking at the information found I realized that both devices are almost the same. They were just sold with a different name, by a different company, in a different country. This is something that was previously pointed out by other security researchers. The code reuse between router manufacturers has propagated not only the same features but also the same security holes between devices.

Firmware acquisition

The vendor website usually provides a download section where you can get manuals and the firmware file. If this is not the case you might get lucky searching for it in your search engine. For this series we are going to use the firmware of the NPLUG wireless repeater which can be acquired here.

When none of the above options is available you need to extract the firmware image from the hardware using one of the techniques that will be briefly described on the following posts.

Conclusion

This is all for now. I hope I had increased your appetite for what is coming next. Make sure you subscribe to the RSS feed and follow me on Twitter so you can see when the Part 2 is released.

References

• Zaddach, J., & Costin, A. (2013). Embedded devices security and firmware reverse engineering. Black-Hat USA.
• FCC ID Search Form
• FCC ID.io