Penetration tests have become a common practice in the information Security industry during the past decade. However it is still a very immature practice in term of professionalism, methodology and quality. Automating the penetration test practice will bring it to a new level of quality and trustworthiness. But attempts to do so will face interesting technical challenges. This is perhaps a new challenge to the IS industry for the next years. In our talk we attempt to clarify and define the penetration test practice as it is now. When they proceed to identify current flaws and conclude that automating the practice might solve many of them. Finally we describe the technical difficulties we face in doing so and a possible way to address them.