Risk based testing

What is risk ?

Risk is an uncertain event or condition that, if it occurs, has an effect on at least one objective.

or

The probability of something happening multiplied by the resulting cost or benefit if it does. (This concept is more properly known as the ‘Expectation Value’ or ‘Risk Factor’ and is used to compare levels of risk)

With respect to software testing there are mainly two types of risks.
– Product Risk
– Project Risk

Product risk – These types of risk generally effect quality of product delivered. These types of risks directly effect test object.

Where as Project risk effect overall success of a project. For example, potential staffing shortage that could result in delay of completion of project can be classified as project risk.

In Risk Based testing we use the risk identified with the level of risks identified for each risk item to help our testing.

Risks can be used to guide testing in different ways like:

1.  Based on the risks identified we can identify test objects which require more effort and accordingly allocate more time to high risk items.

2. Test reporting based on Residual Risks. Like which tests have not yet run or have been skipped? Which tests we have run ? What passed and what failed.

Example:

Just like you buy insurance when you are worried about potential risks like accident or premature death. Similarly we should test areas/objects that are worrisome and ignore the ones which are not having any risks.

Benefits of Risk based testing

  • As testers we often face time pressure and seldom find enough time to run the test we want to. Mostly all testings are generally time boxed. Risk based testing provides a way to prioritize and triage test at any point of time.
  • Risk based testing provides a smart way to choose finite number of tests from an infinite set of tests.
  • Often when there are time pressure, risk based testing provides a way to drop test intelligently while also providing a way to discuss with project stake holders the risk inherent in doing so.
  • Risk based testing helps in determining acceptable level of residual risk rather than relying on inadequate metrics like bug and test counts.

 

The Economies of Software Testing

What do you mean by Economies of Software testing?
Why software defects are costly to find and fix?
How early defect detection can save much cost in fixing defects?

There is a definite economic impact of software testing. One economic impact is from the cost of defects and the another is the way we perform testing. Cost of defect is a very real and very tangible cost. The second point the way we perform testing will not consider it now.

Where defects originate?

defect_originate

One of the most commonly understood facts about defects is that most defects originate in the requirements definition phase of a project. The next runner-up is the design phase.

Some problems in getting accurate, clear, and testable requirements are:

  • Many people do not have a solid requirements gathering process
  • Few people have been trained in or understand the dynamics of requirements
  • Projects, people, and the world around us change very quickly
  • The English language is ambiguous and even what we consider clear language can be interpreted differently by different people.

 Where testing resources are used?

In the previous section we saw that most defects originate in requirements and design, but most of the testing effort occurs in a traditional “testing” phase toward the end of the project. The problem with the big bang approach to testing is that defects are not found until toward the end of the project. This is the most costly and risky time to fix defects. Some complex defects may even be impossible to fix.

 Relative cost of fixing defects

One of the known facts about software defects is that the longer they go undetected, the more expensive they are to fix. Although research differs on the exact ratios, the general rule is 1:10:100. That is, if a defect costs one unit (hour, dollar, etc.) to fix in requirements and design, it costs 10 units to fix in testing (system/acceptance) and over 100 times to fix in production. The cosThis cost of defects doesn’t even take into account the impact cost of defects. These t of fixing in production may be even higher than 100 times. Costs could be attributed to lost revenue, reimbursements, fraud, lost customers, bad public relations, and litigation.

cost_of_defect_fixing

Conclusion

  • Most defects are created in the early stages of a project
  • Most defects are found in the later stages of a project
  • It costs 10 to 100 times as much to fix a defect in the later phases of a project.

So, what does all of this mean? The main conclusion is that most people perform testing too late in the process. These people wonder why testing is so expensive and why their projects are often over budget.

If you really want to make your testing more efficient and reduce the overall cost of testing and defects, test early in the project and continue testing throughout the project.

Security testing – revisited

This blog (security testing revisited) is intended to summarize  few of my post related to security testing :-

  1. Ethical Hacking – part 1 – 

    Before going into details of how to ethical hacking we should be clear of what it is. What is ethical hacking? ….

  2. Ethical Hacking – part 2

    In this part we will go a step further in to ethical hacking and discuss how to do Penetration testing. Warning: Proceed only after permission from network owner else it will be treated as hacking….

  3. What is Cross-Site Scripting?

    What is Cross-Site Scripting (XSS)? Cross-Site Scripting (XSS) attacks are a type of injection problem, in which malicious scripts are injected into the otherwise trusted web sites….

  4. SQL Injection

    SQL injection is a technique often used to attack data driven applications. This is done by including portions of SQL statements in an entry field in an attempt to get the website to pass a newly formed SQL command to the database. ….

  5. Security Testing tips

    What is Security testing? Security testing is a process to determine that an information system protects data and maintains functionality as intended. It is the process that determines that confidential data stays confidential and users can perform only those tasks that they are authorized to perform….

     

Selenium Ruby Binding – how ?

Good part of selenium is that the selenium tests can be written in multiple programming languages like c#, Java, Perl, PHP, Ruby etc. I normally hear saying that “This Java project, so we can write tests in Java as well”.  The only advantage I see with this is that you can get help from development team incase you are stuck somewhere. But if you are good at c# or any other language it should not be an hindrance. Test framework will be different that coding framework unless you want to integrate it.

I covered Java and C# in most of my previous posts. Here I will be using Ruby to write the Selenium tests.

 A simple Ruby Test

require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox
driver.navigate.to "http://www.google.com"
element = driver.find_element(:name, 'q')
element.send_keys "Hello Selenium WebDriver!"
element.submit

require “selenium-webdriver”  – This is similar to import in Java and using in c#

The above example is without using any Test framework. In Ruby you can use Test Framework as you do with C# ( NUNIT test framework) and Java( JUNIT or TestNG framework).

In Ruby you can use minitest or BDD framework  RSpec or Cucumber

Structure of RSpec test

load File.dirname(__FILE__) + '/test_helper.rb' 
#'describe' marks the a test group
describe "Selenium Ruby Tests" do 
include TestHelper
 # 'it' marks the start of a test case, ends with the matching 'end'
it "Start Chrome" do
 browser = Selenium::WebDriver.for(:chrome) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end
it "Start FireFox" do
browser = Selenium::WebDriver.for(:firefox) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end
it "Start IE" do
 browser = Selenium::WebDriver.for(:ie) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end 
end

For more information on RSPec click here .

Test Pattern – Pair Testing

Pair Testing

Pair testing is another example of test pattern.

Context

  1. Pair testing is a way of approaching a test design process by having two people test the same time and place continuously.
  2. The dynamics of pairing enables the generation of more and different ideas than either tester is likely to produce on his own.
  3. It is an effective complement to individual testing.

Examples

  1. Composer/ lyricist pair
  2. Pilot and Co-pilot in a aeroplane

Solution

Two tester working together produce tests, over a period of time continuously exchanging ideas. Assuming that the conditions exists that enables test design, successful pair testing requires three specific conditions

  1. At least one tester is available who can be trusted to test without supervision
  2. Another tester is available who can join the first tester for a session of test design.
  3. The two testers are otherwise capable of working together

Forces

  1. Idea Exchange – The process of explaining and questioning helps pollinate new ideas. This is true even when one of the testers is much less knowledgeable than the other one.
  2. Attention flow – The core dynamic of pair testing is the flow of attention. Pair testing requires that the testers synchronize their pace of work. They continuously share ideas and direct themselves to various problems.
  3. Test strategy – If each tester specializes in a different sub-systems, then as a pair they may be especially effective at system testing that examines the interaction among those sub-systems.

Test Patterns – Scenario Testing

Scenario Testing

Scenario testing is one of the example of Test Patterns

Context

  • The objective of scenario testing is to prove that the program will fail when asked to do real work ( significant tasks) by an experienced user. A failure at this level is a validation failure ( a failure to meet the stated or implicit program requirements.)
  •  “All” of the features have been tested in isolation. More precisely, all of the features that will be called within this scenario have been tested on their own and as fas we can tell, none of them has an error that will block  this scenario test.)
  • The tester must have sufficient knowledge of the domain( eg: accounting, if this is an accounting application) and many of the ways in which skilled users will use the program.

Forces

  • Time trade off
  • There is often much time pressure to develop “realistic” tests quickly.
  • Because complex tests are expensive. You can not develop many of them. What level of coverage will you get from these tets? What level should you expect.

Solution

Ideally scenario test has four attributes

Attributes
Attributes
  1. The test is realistic
  2. It is complex
  3. It is easy to tell whether the program passed of failed the test
  4. Serious failure if the program will not pass a given scenario

The key message of this pattern is that you should keep this four attributes in mind when you design a scenario test and try hard to achieve them.

Next we will cover another example of test pattern.

 

 

 

 

How to convert xml to json format?

To Convert XML to JSON format

Consider a situation where in you have data in xml file and want to use it as a input to some other system where in it takes data in the form of JSON. You would require to convert XML to json format.

Below is an example of converting XML file into json file format.

#! /usr/bin/env ruby -rubygems

require 'rubygems'
require 'active_support/all'
require 'json'
json_str = Hash.from_xml(File.open(ARGV[0]).read).to_json
File.open(ARGV[1], 'w+').write json_str

To run the command,

xml-to-json.rb .file1.xml .file_out.json

 

 

Applying Patterns to Software Testing

What is Patterns ?

  • a regular or repetitive form, order or arrangement
  • a model that is considered to be worthy of imitation

Why Patterns ?

Patterns are a way of helping people who design things. Patterns accomplish at least three things

  1. They provide vocabulary for problem solvers.
  2. They focus attention on the forces behind the problem. It allows designers to better understand when and why a solution applies.
  3. They encourage iterative thinking. Each solution creates a new context in which new problems can be solved.

Why Test Patterns?

Testers lack useful vocabulary and are hampered by rigid “one size fits all” methodologies, and face many problems whose solutions are not described in the literature.

Some of the test patterns are

  1. Pair -testing
  2. Architecture- Achilles- Heels
  3. Architecture – reverse engineering
  4. Scenario testing

More on Test Patterns in my next post.

 

 

Difference between __CSVRead and __Stringfromfile function

__CSVRead() function

The CSVRead function returns a string from a CSV file.
When a filename is first encountered, the file is opened and read into an internal array. If a blank line is detected, this is treated as end of file – this allows trailing comments to be used.
All subsequent references to the same file name use the same internal array.

Each thread has its own internal pointer to its current row in the file array. When a thread first refers to the file it will be allocated the next free row in the array, so each thread will access a different row from all other threads.

eg: You need to test login to an app with different user login credentials for different threads.

__Stringfromfile()

The StringFromFile function can be used to read strings from a text file. Each time it is called it reads the next line from the file. All threads share the same instance, so different threads will get data from different lines. When the end of the file is reached, it will start reading again from the beginning, unless the maximum loop count has been reached.

Note: If there are multiple references to the function in a test script, each will open the file independently, even if the file names are the same.  So it is advisable to use different file name across same scripts otherwise the output will be unpredictable.
Eg: You need to add different products to a shopping cart for the same user. In this case you will be using CSVRead to read user credentials and Stringfromfile to read product info that need to be added to each user.
So in a nutshell, Difference between __CSVRead and __Stringfromfile function is that Stringfromfile should be used when different data are required in a loop for same thread and CSVRead should be used when different data are required for different threads(users).

How to uses SYSMON to Monitor Performance counter?

Monitor performance counter using SYSMON

The following example uses VBScript to add counters whose values are retrieved from the local computer, modifies some of the SYSMON properties that control how the monitor is displayed, and processes the OnCounterAdd event. The example uses the wildcard character (*) to add all instances of the process counter.

Save the bleow code in a HTML page and open in browser.

<HTML> <BODY BGCOLOR=#C0C0C0> <SCRIPT LANGUAGE="VBScript">
Sub Monitor_OnCounterAdded(index) Monitor.Counters.Item(1).Width = 8 End Sub </Script> 
<OBJECT CLASSID="clsid:C4D2D8E0-D1DD-11CE-940F-008029004347" ID="Monitor" HEIGHT=80% WIDTH=100%> 
</OBJECT>
<SCRIPT LANGUAGE="VBScript">
Sub Window_OnLoad On Error Resume Next Monitor.ShowValueBar = False Monitor.ShowHorizontalGrid = True Monitor.Counters.Add("\Process(*)\% Processor Time") Monitor.DisplayType=sysmonLineGraph Monitor.GraphTitle="System Performance Overview" End Sub 
</SCRIPT> 
</BODY> 
</HTML>

Note: You need to enable ActiveX in your browser, to run the html file.