First and foremost, it is simply not feasible to identify every possible scenario or testing attribute because at some point this effort will be subject to diminishing returns.  This is generally known as the fallacy of exhaustive testing.  It’s also akin to the 80/20 rule.  The objective of testing is typically to identify and test as many important features and functionalities as possible, not necessarily all of them.

Another reason is that the participants of the testing effort, especially in terms of usability testing, are typically not representative of the population of end users as a whole.  Everyone is different, and as such it is not possible to test a software application in a way that covers all possible user behaviors.  Thus, much like with the fallacy of exhaustive testing, the goal is to identify the majority or most important classifications of users, not necessarily all of them.

Additionally, the environment within which the testing occurs is usually fabricated.  Most times, testing occurs in a lab or testing environment that has been created for this specific purpose.  Thus, it may or may not be completely representative of the actual environments within which end-users will use the software application.

The bottom line is that, although software testing seeks to ensure the highest probability of success, there is no way to guarantee with 100% certainty that the software will be glitch free in the ‘real world.’  That said, software testing – when conducted accurately, during the correct phase of the software development lifecycle, and as part of a user-oriented methodology – can dramatically reduce the risk of deploying faulty software into the market.  Simply put, in nearly all cases, it is better to test than not to test.

First on the list is a glitch that could potentially impact users of Skype.  A security analyst recently identified a glitch in Skype that could allow unauthorized access to a user account by inputting a “string of code” into the mobile phone field associated with that account.  When the user subsequently logs in, the hacker can then run a script to gain access to the user’s Skype ID and password, thereby facilitating the unauthorized access.

Next, Medvet Laboratories, one of the biggest DNA test firms in Australia, had a glitch that resulted in the uploading of the private info of approximately 800 people onto the company website.  The glitch was in play for about 3-4 days before it was discovered, and since that time all the private information has been removed from the public’s view.  Although the data breach seems to be limited to just names and addresses, it is likely that this incident will open the company up to legal scrutiny.

Finally, you can’t even get away from software glitches in space!  Recently, the crew of the Atlantis space shuttle was rudely awakened during the night when an alarm erroneously went off.  There’s no word on why exactly this alarm went off, but the crew did spend some time uploading software onto another of the vessel’s main computers before they went back into their slumber, if that’s any indication.

So there you go – more examples of why software testing is absolutely, positively critical in today’s world.  As they say, an ounce of prevention is worth a pound of cure!

Thus, the combination of potentially fluid requirements, interdependency, and diverse distribution mandates that the test cases be written in a way that accommodates these attributes.  As such, there are a few best practices that should be followed whenever a test case is being written.

First and perhaps most obvious is the fact that the test cases should be written in a clear way, without any ambiguity. In other words, they should be in the correct sequence, always clearly map to expected results, indicate interdependencies, and be organized based on testing category and/or related parts of the application.  Doing this will not only make the test cases easier to follow, but it will make it much easier to revise/update them.

Second, after all the initial test cases are written, go back to the beginning and review each one from the perspective of the tester. Make sure that no conflicts exist, that all references and mapping schemes are accurate, and that they are clearly written. Evaluate the test cases as if you are doing a dry run.

Finally, always write test cases with the end user in mind. End users must always be the focal point because they are the ones who will actually use the software application.  Thus, pay special attention to those test cases that are most critical to the end-user’s actual usage of the application.

The bottom line is that software testing test cases can dramatically impact the overall success of any testing process, so it is very important to follow best practices when writing them.  Spending a little more time and effort upfront to get these right will pay huge dividends down the road in terms of a much more effective and efficient testing effort.

Companies have a variety of reasons for not automating their software testing processes.   The 2 key barriers seem to be cost / complexity and skill deficiency, and both of these are interrelated.  The cost & complexity factor is self-explanatory.  Because it is relatively complicated, the costs associated with the automation tool itself, the integration / deployment work, and licensing fees can truly add up.

In terms of skill, automation tools are typically script-based and require programming expertise. Companies often do not have employees with the necessary skills, so they must either train current employees or hire from the outside.  This represents another barrier because these organizations tend to focus on the costs of acquiring said skills instead of the benefits of bringing the skills in-house.

Despite these typical barriers, companies would be well-served to consider both sides of the equation when determining whether or not to automate.  The barriers mentioned above focus on the cost side of the equation with little regard to the benefits of improved efficiency and productivity, increased speed-to-market, and more thorough testing coverage.  It is only after considering the costs as well as the benefits that a truly informed decision can be made.

The bottom line is that automation is an investment, and like any capital outlay it takes time to recoup that investment.  Although software testing organizations should conduct both manual and automated testing, the ROI associated with automation is almost always positive…assuming that both sides of the equation are factored into the decision making process.

It is typically executed before system testing but after unit testing, essentially grouping together individual unit-tested modules as the testing inputs or entry criteria.  Test cases are created to explicitly test the interfaces between the various modules or components. The objective is to ensure that the interaction of multiple dependant components satisfies the requirements surrounding performance, reliability, and functionality.  The process is said to be complete when all the interfaces function as expected.

Integration testing can be conducted using a big bang, top down, or bottom up approach.  In the big bang method, most or all of the modules are grouped together to create the complete system.  This has the benefit of saving time in the testing process, but relies on the accurate recording of test cases and results.  In the top down approach, the highest level components are integrated and tested first, thereby allowing the top level rules and logic to be validated early in the overall process.  This helps to reduce the need for drivers, but the necessity of stubs tends to complicate matters.  The bottom up approach is the opposite of this; specifically, the lowest level components are integrated and tested first.  The benefit is that the need for stubs is reduced, but on the downside this approach relies on drivers which can complicate the process.

In summary, integration testing is useful for testing complex systems with components or modules that interface together at various levels.  The process of validating these interfaces and dependencies helps ensure that defects are identified and fixed before the program or application is released into the market.

Therefore, it is important to come up with some standardized categories that can aid in the development of the business and functional requirements.  This helps to provide an overall framework that can provide focus and streamline efforts.  As such, listed below are 3 categories of requirements that might help develop such a framework.  It’s obviously not a comprehensive list, but it is a good starting point.

First, error messages should be incorporated into the requirements.  Things such as when they will be displayed, to whom, and how should all be tested.  You will also want to make sure the error messages are written in an understandable way, as developers sometimes are not the best writers.

Second, load testing must be incorporated, both in terms of bandwidth as well as volume of users.  Specifically, you will need to test what happens if too much load is introduced into the system, what error messages are displayed, whether or not the system should have an auto shut-off mechanism, what happens to transactions that are already in the queue, etc.

Third, a category for permissions is useful, particularly if the company handles sensitive or personally-identifiable data.  There should be clear, testable requirements around who can access various parts of the system, who can administer the permissions, what happens when access is denied, etc.

There are dozens of other categories that could potentially be used to develop a standardized requirements-development framework, depending on your company’s industry and product line.  Use the 3 suggestions described here as a starting point, and then expand the list in a way that synergizes with your company’s products, processes and competences.

So this begs the question: what exactly is quality? Can we come up with some sort of standardized definition? Well, before we can answer that question, let’s first identify some of the more common viewpoints of this.

Some people will say that quality correlates with the reputation of the brand. Big companies with well-known brands are often perceived to deliver higher-quality products. After all, with access to all those resources surely they can ensure the highest quality possible, right? The problem with this definition is that it is subjective and will depend on each individual’s perceptions, expectations, and preconceived notions regarding the brand. Since this is not quantifiable, it is not measurable and therefore is a poor definition of quality in the context of software testing.

Another view holds that quality relates to perfection. In other words, a software application with no identified bugs is often said to be of quality. Unfortunately, although this sounds nice, a zero-defect application is generally not something we find in the real world. Usually, you wouldn’t be testing 100% of the code because there are diminishing returns in doing this. Some people refer to this as the “fallacy of exhaustive testing.”

There are other views as well, but hopefully you get the point. The bottom line is that any definition of quality should be specific, measurable, and realistic. Therefore, in my view at least, the best definition of quality is that the product meets the requirements around the product’s attributes, features, and functionality. This seems like the best viewpoint because it defines specific indicators which are testable and measurable. You test a requirement, and the software either executes it properly or it doesn’t. It really doesn’t get much clearer than that!

Although a full blown database test plan is beyond the scope of this post, we can at least touch on the highpoints so you can get a general idea of what is involved. First, make sure that the data is mapped accurately. This will help test whether or not the database performs as expected when the application interacts with it. For example, if the end user deletes some data from the user interface, you will want to make sure that these data points are subsequently deleted from the associated database table.

You will also want to validate data integrity. This means that the most current data values are shared across the board. In other words, each shared system must show the most recent value for each data point whenever updated. This is important because different modules of a software program might utilize the same data in different ways.

Additionally, unless the database is designed simply to store records, you will want to test the functionality of any business logic or rules that reside at the database level. Finally, you will want to test the database transaction properties – atomicity, consistency, isolation, and durability.

In order to execute the database test, it is worth mentioning that the tester should have explicit knowledge of the database tool/language (such as SQL), DML (database manipulation language), the internal structure of the database, and AUT. Then, the tester will simply perform operations from the user interface and validate the output via SQL queries (or by observing the actual database tables if the tester is not good with SQL or if the application is relatively simplistic). Note that the more complex the database, the less likely that you’ll be able to manage all the SQL queries, and as such assistance from the development team may be necessary.

The bottom line is that database testing is becoming a necessity in today’s high-tech world. Just make sure the tester in charge knows what he or she is doing, including adequate knowledge of SQL, AUT and the internal workings of the database’s structure.

Here’s what happened.  The company made a code update which resulted in the introduction of a major authentication bug.  Specifically, for almost 4 hours the defect caused the authentication mechanism to fail, which essentially allowed anyone to access anyone else’s account without the proper log-in credentials!

If you’ve never heard of DropBox, it is a tool that allows you to “drop” your computer files into an icon on your desktop so that they can be stored / backed-up on a remote server.  Once backed up, the user can log-in to his/her account to access or share the files as he/she sees fit.  The company claims to have about 25 million users, and it is not a stretch to assume that a good percentage of them keep personally identifiable information and other sensitive data stored in their accounts.  Yikes!

Although ‘only’ up to 1% of all user accounts were impacted, 1% of 25 million is still a whopping 250,000 people!  I sure hope the company’s public relations’ staff is ready to engage in some heavy-duty damage control!

Although the issue has been resolved, you have to wonder how vulnerable a service like DropBox is to future security breaches, whether caused by a bug or a hack.  In this case, it is obvious that the company’s software testing efforts epically failed, and hence it is reasonable to assume that something like this will happen again sooner or later.  I sincerely hope that DropBox is now committed to better software testing, because if something like this does ever happen again due to their negligence, they’ll be looking at a mass exodus of users and a rapid erosion of their brand and reputation.  Let’s hope that they now “get it”!

In an attempt to try and shed light on this burning question, let’s take a look at the different viewpoints.  The main benefit of having a college education is that it clearly helps with analytical thinking.  It provides the tools and skills to analyze problems at a deeper level, to think systematically, and to see the big picture.  Obviously, these are skills that are critical in the world of software testing.

Additionally, attending college likely helps develop the social skills needed to engage in effective communication.  Because most full-time university students are essentially thrown into a foreign environment all by themselves, they are forced to learn how to interact and build relationships where none previously existed.  This social “sink or swim” process forces the student to learn, practice, and refine their communication skills over time.

On the flip side, many people believe that having a college education, although helpful, is not a substitute for an individual’s natural work-ethic, intelligence and analytical skills.  Simply put, the thought is that some people are simply “hard wired” to perform better, whether they go to college or not.  Another thing to consider is that having a mix of college educated and non-college educated software testers helps foster diverse viewpoints amongst the team.  This increases the team’s ability to analyze a situation from all angles and subsequently can be expected to facilitate better overall team performance.

So, what’s the verdict?  In my opinion, if software testing is your career – one where you aspire to climb the company ranks – then having a college degree can only help.  Whether right or wrong, this will give you an extra tool in your tool chest compared to your non-college educated peers.  Additionally, having a college degree myself, I can attest to the fact that it does indeed enhance analytical and problem-solving skills, both of which are important attributes of good testers.  But if you are not career-oriented and are just looking for a job, then a college degree probably is not necessary.  However, earning at least your ISEB Foundation / ISTQB Foundation software testing certification can be beneficial because it could help you get your foot in the proverbial door.