Ways to Improve Your Development Process - Fuzzing
My last post was me jotting down a few ideas that I've had involving ways to improve development processes. The easiest way to do that is to get a baseline of continuous testing and measuring your code to improve the quality of your code. So all of that gets us to what I consider the baseline that you should consider the minimum for most projects and work environments. The main question at this point is how do we improve upon that?
For now let's stick to testing improvements. Any time we can reduce the number of bugs in our code, that means less time spent after launch at 2AM dealing with a dead service. In later posts I'll talk about automation, tools, etc. that can help with delivery speed a lot more but for now we're going to stick to testing. In this instance we're going to look at:
Fuzz Testing #
Fuzz testing was originally thought up in a graduate Advanced Operating Systems class in 1988 that was being taught by Barton Miller at University of Wisconsin as a class project. The results were eventually published in 1990. The concept was to test a Unix utility meant to automatically generate random files and parameters as inputs. By doing this with a large number of random inputs and in quick succession they were able to find ways to crash the software. At the time they were able to crash about 30% of the utilities they tested. These were previously undiscovered issues in the software that they were able to debug to determine the cause and categorize each failure.
Since then there have been a number of advances, tons of study, software built, as well as interesting use cases for them. The obvious one is simply to make software more reliable, however fuzz testing has traditionally found a home in security circles as reliability and security tend to go hand in hand. Also note that fuzz testing can happen at multiple levels: the application, a library, a class, or a method. We just need something with an entry point.
Generally speaking, it's very good at finding bugs in your software. For instance the paper linked was able to find issues in the IRIX kernel within a couple of minutes of testing and this was back in 1996. You may be thinking that the old tests aren't very valid but more recently it was used to find a number of bugs in Linux, MacOS, and FreeBSD. Specifically 12% of the apps in Linux were found to fail, 16% for MacOS, and 19% for FreeBSD. And those numbers were an increase from the same test they ran years ago.
We can set this up like any other test to run automatically for us. Nothing needs to be done except look for the exceptions to be thrown.
It's a slow process. Depending on what you're testing you can run the thing for weeks without hitting just the right set of inputs to trigger a failure. This is especially true the higher up the method => class => library => application tree you go.
It does not test correctness of your software. It isn't going to check that your app creates a correctly formatted xml file for instance. Instead it tests reliability. You're looking for the bugs when a user's kid hits the keyboard 14 times and it causes the app to crash sort of bugs.
There are a number of directions that people have gone to improve the slow issue as well as increasing the number of issues that the fuzzer finds. For instance some take the approach of trying to maximize code coverage to improve the number of issues found. Some use machine learning to direct the search process. All of them are interesting but if you want a very detailed run down on the various approaches and concepts, I recommend the Fuzzing Book. Instead we'll go over a very basic example.
Let's assume you have a method that takes in a string and an integer that indicates the number of characters in the string. The method is supposed to return the string. Now let's assume that you want to write some tests for your code.
Assert.Equals("car", MyMethod("car", 3));
Assert.Equals("test", MyMethod("test", 4));
Assert.Equals("testy mctestface", MyMethod("testy mctestface", 16));
In the above the code works and the test doesn't fail. Yay, we passed our test. We're done right? Well, if you've ever dealt with C/C++ you already know the answer is hell no. What happens if the string is null? What happens if we enter a number less than the number of characters in the string? What happens if we enter a number greater than the number of characters in the string? What happens if the number is negative? What about the \0 character in the string?
You'll notice that our search space is growing a lot and I'm lazy and don't want to add all those test cases by hand. In this instance we could actually use the concepts found in fuzzing to generate all of those values and more that we didn't think of to test our code for correctness as well as stability. But even in instances where we can't guarantee the outcome of the method as easily based on the input, we could still ensure that our code doesn't fail. Bulletproof code that we can reuse is always a good thing. So how do we go about doing this automatically with little to no setup on our part? Thankfully there are tools for that.
Fuzzing Tools #
This is more of a quick run down than a deep dive and aimed at showing you some options out there. For this I'll be talking about the .Net space as that's the world that I come from. Anyway, the main option for fuzzing an application or library is SharpFuzz. SharpFuzz is an AFL-based fuzzer. It works well, very easy to get going, and AFL is one of the most popular fuzzers out there and SharpFuzz is a wrapper around AFL.
For API endpoint testing, there are a number of options like RAFT, APIFuzzer, among others. The number of options here goes up as not only are there a ton of open source projects but a number of paid services. And since your API is just a web end point, language or platform isn't an issue. Just find a tool that you like. Personally I like OWASP ZAP just because it's free and easy to get going with if you want to do fuzzing.
In the example above, we're down at the unit testing/individual method levels of fuzzing and sadly I couldn't find that many options out there when I was originally searching for them. IntelliTest could be considered a fuzzer. It does take an iterative approach to try to increase code coverage by generating different inputs for a method. The caveat is that the values it generates aren't truly random but I'd still say that it falls under the general umbrella. That said if you don't have Enterprise edition of Visual Studio then you don't have access. Great tool if you have it. Other than that searching around only popped up a number of old/unsupported libraries more or less. However since I had a need a couple years ago, I created a plugin for xUnit called TestFountain. Just add the Nuget package and then add the attribute to your unit test:
public void MyUnitTest(string value1, [Range(1, 100)]int value1Length)
Assert.Equals(value1, MyMethod(value1, value1Length));
This would be the equivalent of the tests above using the plugin that I created. That will generate 100 random input values based on the range that you set for the input and will attempt to take a max of 1 second. Anything over that and it will stop after the next run. Also there may be other options out there but I couldn't find them when I needed them.
More recently I've taken this small plugin and added a lot more features that I've been looking for and created the Mecha library. Going forward that will be where I will add features and improvements as I find a need for them. I've been testing it with a couple of my other libraries and already found bugs (null reference, index out of bounds, etc.) so I'm happy with the results thus far. Best part for me is that I can test an entire class with one line of code, because once again I'm lazy:
While I created something for my own needs, in reality all you need to do is create random data for a method that you want to test to get the basic benefits of fuzzing. You could use Bogus, GenFu, or any number of libraries for generating values. Heck want to try a machine learning setup for input generation, I've done it with ML.Net and it works fairly well. But generally any random value generator will get you started. In my case, once again, I use my own but there are about 20 different ones out there to choose from. Just generate values and see if it throws an error. Because with fuzzing, you're not generally looking for code correctness, you're looking for what breaks your code so you can defend against it appropriately.
That's the basic concept in a nutshell. So with all of that we have our first easy upgrade to our testing environment. Once you add it to your inventory you should be able to find a number of new bugs that you had never found before. Next time I'll talk about fault tolerance.
Items in the Series #
Sign up for More Posts
I have no idea why you'd put yourself through that but if you'd like more posts from me then how about using my RSS feed? Take a look.