The Need for Measuring Software Security

By Markus Schumacher and Sebastian Schinzel

Given the complex nature of modern software solutions, software testing is a crucial process step in the development cycle. Best practices in software testing are (more or less) standardized and supported with a variety of different tools. As a result, we see complex applications that can be used efficiently without them failing too often. Furthermore, an experienced software tester is able to measure the quality of a software application and compare the results to other software applications.

So far, so good – but we can tell a different story, when it comes to software security. Annual reports of computer security incidents such as The Web Hacking Incidents Database regularly reminds us that cyber criminals keep on exploiting security vulnerabilities and literally get their hands on your business assets processed by your computer systems. Most attacks target to steal sensitive information and attackers mostly achieved this goal by exploiting SQL-Injection vulnerabilities. This type of security vulnerability is known for decades, but still widely present in operational application environments.

Measuring Software Security


The Crux of Security Testing

For quite some time, companies and organizations are performing security testing in order to find security vulnerabilities before the bad guys do. But why is the level of software security still so poor? While software testing focuses on positive tests (example: given a certain input, the software application must compute a defined output), security testing is just the opposite – it mostly relies on negative tests (example: “a user must not be able to access other user‘s data in the database backend”. In order to illicitly access restricted information in a database, attackers often exploit SQL-Injection vulnerabilities. These vulnerabilities emerge when a developer appends user input to SQL queries without performing proper input validation and output encoding. As an example, let‘s assume that the following SQL query statement is build to check authentication credentials against a database of user IDs and passwords.

String query = “SELECT fld_userId
FROM tbl_users WHERE”
+ “fld_userId=’ ” + userId
+ ” ‘AND password=’ ”
+ fld_password
+ ” ‘ “

If userId is a variable controlled by the attacker, the attacker is able to logon as any user without having the matching password by issuing the string 1′ OR ‘1’=’1 as userId. Let‘s look at the resulting SQL query

SELECT fld_userId
FROM tbl_users
WHERE fld_userId=’1′ OR ‘1’=’1′ AND fld_password=’ ‘

This SQL query will log on the attacker as the user with user ID 1, no matter if the valid password was issued by the attacker or not. One way for developers to prevent SQL injection vulnerabilities is to SQL-encode all user input before appending it to SQL queries. Imagine that a developer translates SQLencoding into ―prefix all occurrences of an apostrophe in user input with a back slash‖. This version of SQL-encoding transforms the user input Marc O’Neil into Marc O\’Neil, thus preventing the above SQL-Injection attack shown above. What if the developer uses the following statement to build a SQL query?

String query = “SELECT fld_userId
FROM tbl_users WHERE “
+ “fld_userId=” + sqlEncode(userId)
+ “ AND fld_password=‟” + password
+ “ ‟ ”

Note that userId is SQL-encoded before it is appended to the SQL string. Unfortunately, userId is not enclosed by apostrophes because the user‘s ID is expected to be numeric and not a string. If the user enters the string 1 OR 1=1;– into the username field, the resulting SQL query looks like this:

userId=1 OR 1=1;– AND

This query will successfully authenticate the attacker as user with ID 1 no matter if a valid password was issued or not. The attacker was able to perform an SQL-injection attack against the application even though the application‘s developer used SQL-encoding.

This is just one of many examples of the hidden pitfalls that developers face (assuming that they know about it at all) in their every day work. The complexity of these vulnerabilities makes it difficult for security testers to estimate the security of the software even after a security assessment. Is the application secure, because the security tester does not find vulnerabilities any more? The answer to this question is crucial for the customer and statements such as “almost secure” or “not that secure” are not sufficient. Finding no security vulnerabilities does not mean that the software is secure. It merely means that this tester did not find any security vulnerabilities. Another security tester with higher or different skill may still find many very critical security vulnerabilities. Thus, to make a more precise statement about the security of an application, we define and measure security metrics.

Measuring Software Security

Software quality metrics are used to measure software quality and to compare the quality of applications among each other. This metric-based measurement can also be used to assess the security of an application. To determine fitting metrics, the Goal/Question/Metric paradigm can be used.

As an example, take the above requirement “A user must not be able to access other user’s data in the database backend” as a goal. As this goal is too generic, it is split into several sub-goals such as “Non-validated or non-encoded user input must not be copied into SQL query strings.” or better: “No user input should be copied into SQL query strings.” The last goal is realistic, because for most database systems developers can use prepared statements, which eliminate SQL-injection attacks.

A metric can be formally defined in terms of one of four functions (understand, evaluate, control, predict), the attribute of the entity being measured and the goal for the measurement based on the following metrics objective template:

To [understand|evaluate|control|predict] the [entity attribute] in order to [goal(s)]

As a result, one metric to estimate the probability that a software application contains security vulnerabilities is

“To evaluate the number of possible SQL-injection vulnerabilities in order to assure that a user is not able to access other user’s data in the database backend.”

Using this metric-based approach a security tester is able to measure the security of an application and present the results in a way the customer understands.



  • Software Security Metrics – ISQI Certified Professional for Secure Software Engineering
  • Andrew Jaquith, Security Metrics – Replacing Fear, Uncertainty, and Doubt, 2007
  • Linda Westfall, 12 Steps to Useful Software Metrics
  • NIST Special Publication 800-55, Security Metrics Guide for Information Technology Systems, 2003
  • Bernhard Orth: Einführung in die Theorie des Messens, 1995
  • ISO 9126 Software Engineering Product Quality


Further articles of the testing experience magazine

Adopt Your Local Professor: The Need for Industry and Academia to Work Together
By Patricia A. McQuaid

Putting the ‘Analysis’ in a Test Analyst
By Mike Smith

Leave a Reply

Your email address will not be published. Required fields are marked *