The recent HackMiami contest produced some interesting results regarding "point-and-shoot" pen testing, but we feel like IMPACT Pro still stands up to any product in a true penetration comparison.

When reviewing any product test or review, I think it’s important to consider what the reviewer is trying to accomplish. The recent HackMiami contest was a test involving a simple, straightforward first level remote server exploitation and report using only the product’s “point and shoot” capabilities.  Clearly, the test was trying to see what n00bz (hence the URL) could do with these products with a first dance in the lab.

While the contest was an interesting exercise for fun (and based on the Twitter buzz it generated, clearly an area of interest for people in the security testing space), it was at best a limited view into what penetration testing really entails.

To appreciate the full context of the findings to get the big picture of where the products actually line up, one has to look at what happened, and how these products would be used in the real world. In this exercise only the basic remote exploiting and the first level information-grab elements were incorporated, which doesn’t really begin to evaluate the need of professional penetration testers.

The reviewers tested eight boxes at low patch levels and with weak passwords, targeting them with the simplest wizard-driven elements of the products. Metasploit was given credit for many 'shells' as it exploited null or weak passwords, which IMPACT Pro finds but does not exploit unless you explicitly choose to do so. So, if the testers had chosen to attempt take the next step to exploit these shells using IMPACT Pro, they could have.

When performing a real penetration test there are, quite simply, many key functions which the other products simply don’t have. The reason is that the other products are more purely exploiting tools, whereas IMPACT Pro is designed to be a more complete penetration testing solution.

Some of the other areas IMPACT Pro can test, which are not available in other products are:

- Web application testing – there is no bigger vector out there today.
- Client side (end point/user/phishing/spear phising) – advanced attacks have shifted to users.
- Pivoting – or stringing together multiple vulnerabilities to form an attack path, as attackers do.
- Cross vector testing, i.e. pivoting from a web or client app – again, this is how real attacks work.
- Wireless – a rapidly emerging attack vector that hasn’t been given as much attention.

Using these capabilities, IMPACT Pro can be used to test for vulnerability to real world threats, such as replicating the Gonzales (Heartland, TJX) and Aurora (Google) attacks, which is not possible with other products. To be fair, several of these IMPACT Pro features were presented in the report, but not all, and their availability seemingly wasn’t weighted very heavily in the testers’ scores.

In terms of presenting testing results data, IMPACT Pro’s sophisticated reports such as Trends, Delta, Attack Path and Executive summary reports are also very important and not available in the other products.  The 'looting' function of Metasploit Express – which grabs screen shots (also available in IMPACT Pro) and puts them in a report – is a great demo, but in day-to-day security testing it doesn't really add useful information other than an in-your-face kind of proof you were there (if the screen isn't taken from a screen saver).

By comparison, the truly relevant information – such as tested users, groups and services – provided by IMPACT Pro's information gathering, as well as the deep set of post exploitation capabilities available in our product, are not available in the others tested.

At the end of the day, in a real-world penetration test, IMPACT Pro is easier to use, more automated, faster, with more exploits and more reporting – and when you’re doing a real pen test you need a full suite of tests to see if you're vulnerable. The reality is that the other tools in the HackMiami contest just don't have them, so these other tests could not be contested, and of course in that sense there’s “no contest.”

In general, it appears the real difference in the score handed down in HackMiami was simply that IMPACT Pro cost more than the other products, which, as acknowledged in the reports, is because it provides a full set of penetration testing capabilities rather than just point-and-shoot exploiting.

And for the record, we love that compared to a few years ago there is now greater choice for different use scenarios in this space. It will make more people take notice of pen testing and its’ incredible value, and push them to learn more about the various solutions out there including IMPACT Pro.

We even like to see comparisons like this, as it gives us a perfect opportunity to point out how we’re the best product on the market. In this case the comparison was just about a quick Conga with the products, which is fun, but there’s a lot more music than that to review.

Can you feel the rhythm (is going to get you)? Tonight?

And yes, I’m old.

-Fred Pinkett, VP of Product Management