The Economies of Software Testing

What do you mean by Economies of Software testing?
Why software defects are costly to find and fix?
How early defect detection can save much cost in fixing defects?

There is a definite economic impact of software testing. One economic impact is from the cost of defects and the another is the way we perform testing. Cost of defect is a very real and very tangible cost. The second point the way we perform testing will not consider it now.

Where defects originate?

defect_originate

One of the most commonly understood facts about defects is that most defects originate in the requirements definition phase of a project. The next runner-up is the design phase.

Some problems in getting accurate, clear, and testable requirements are:

  • Many people do not have a solid requirements gathering process
  • Few people have been trained in or understand the dynamics of requirements
  • Projects, people, and the world around us change very quickly
  • The English language is ambiguous and even what we consider clear language can be interpreted differently by different people.

 Where testing resources are used?

In the previous section we saw that most defects originate in requirements and design, but most of the testing effort occurs in a traditional “testing” phase toward the end of the project. The problem with the big bang approach to testing is that defects are not found until toward the end of the project. This is the most costly and risky time to fix defects. Some complex defects may even be impossible to fix.

 Relative cost of fixing defects

One of the known facts about software defects is that the longer they go undetected, the more expensive they are to fix. Although research differs on the exact ratios, the general rule is 1:10:100. That is, if a defect costs one unit (hour, dollar, etc.) to fix in requirements and design, it costs 10 units to fix in testing (system/acceptance) and over 100 times to fix in production. The cosThis cost of defects doesn’t even take into account the impact cost of defects. These t of fixing in production may be even higher than 100 times. Costs could be attributed to lost revenue, reimbursements, fraud, lost customers, bad public relations, and litigation.

cost_of_defect_fixing

Conclusion

  • Most defects are created in the early stages of a project
  • Most defects are found in the later stages of a project
  • It costs 10 to 100 times as much to fix a defect in the later phases of a project.

So, what does all of this mean? The main conclusion is that most people perform testing too late in the process. These people wonder why testing is so expensive and why their projects are often over budget.

If you really want to make your testing more efficient and reduce the overall cost of testing and defects, test early in the project and continue testing throughout the project.

Security testing – revisited

This blog (security testing revisited) is intended to summarize  few of my post related to security testing :-

  1. Ethical Hacking – part 1 – 

    Before going into details of how to ethical hacking we should be clear of what it is. What is ethical hacking? ….

  2. Ethical Hacking – part 2

    In this part we will go a step further in to ethical hacking and discuss how to do Penetration testing. Warning: Proceed only after permission from network owner else it will be treated as hacking….

  3. What is Cross-Site Scripting?

    What is Cross-Site Scripting (XSS)? Cross-Site Scripting (XSS) attacks are a type of injection problem, in which malicious scripts are injected into the otherwise trusted web sites….

  4. SQL Injection

    SQL injection is a technique often used to attack data driven applications. This is done by including portions of SQL statements in an entry field in an attempt to get the website to pass a newly formed SQL command to the database. ….

  5. Security Testing tips

    What is Security testing? Security testing is a process to determine that an information system protects data and maintains functionality as intended. It is the process that determines that confidential data stays confidential and users can perform only those tasks that they are authorized to perform….

     

Selenium Ruby Binding – how ?

Good part of selenium is that the selenium tests can be written in multiple programming languages like c#, Java, Perl, PHP, Ruby etc. I normally hear saying that “This Java project, so we can write tests in Java as well”.  The only advantage I see with this is that you can get help from development team incase you are stuck somewhere. But if you are good at c# or any other language it should not be an hindrance. Test framework will be different that coding framework unless you want to integrate it.

I covered Java and C# in most of my previous posts. Here I will be using Ruby to write the Selenium tests.

 A simple Ruby Test

require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox
driver.navigate.to "http://www.google.com"
element = driver.find_element(:name, 'q')
element.send_keys "Hello Selenium WebDriver!"
element.submit

require “selenium-webdriver”  – This is similar to import in Java and using in c#

The above example is without using any Test framework. In Ruby you can use Test Framework as you do with C# ( NUNIT test framework) and Java( JUNIT or TestNG framework).

In Ruby you can use minitest or BDD framework  RSpec or Cucumber

Structure of RSpec test

load File.dirname(__FILE__) + '/test_helper.rb' 
#'describe' marks the a test group
describe "Selenium Ruby Tests" do 
include TestHelper
 # 'it' marks the start of a test case, ends with the matching 'end'
it "Start Chrome" do
 browser = Selenium::WebDriver.for(:chrome) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end
it "Start FireFox" do
browser = Selenium::WebDriver.for(:firefox) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end
it "Start IE" do
 browser = Selenium::WebDriver.for(:ie) browser.navigate.to(site_url)
 sleep 1
 browser.quit
end 
end

For more information on RSPec click here .

Test Pattern – Pair Testing

Pair Testing

Pair testing is another example of test pattern.

Context

  1. Pair testing is a way of approaching a test design process by having two people test the same time and place continuously.
  2. The dynamics of pairing enables the generation of more and different ideas than either tester is likely to produce on his own.
  3. It is an effective complement to individual testing.

Examples

  1. Composer/ lyricist pair
  2. Pilot and Co-pilot in a aeroplane

Solution

Two tester working together produce tests, over a period of time continuously exchanging ideas. Assuming that the conditions exists that enables test design, successful pair testing requires three specific conditions

  1. At least one tester is available who can be trusted to test without supervision
  2. Another tester is available who can join the first tester for a session of test design.
  3. The two testers are otherwise capable of working together

Forces

  1. Idea Exchange – The process of explaining and questioning helps pollinate new ideas. This is true even when one of the testers is much less knowledgeable than the other one.
  2. Attention flow – The core dynamic of pair testing is the flow of attention. Pair testing requires that the testers synchronize their pace of work. They continuously share ideas and direct themselves to various problems.
  3. Test strategy – If each tester specializes in a different sub-systems, then as a pair they may be especially effective at system testing that examines the interaction among those sub-systems.

Test Patterns – Scenario Testing

Scenario Testing

Scenario testing is one of the example of Test Patterns

Context

  • The objective of scenario testing is to prove that the program will fail when asked to do real work ( significant tasks) by an experienced user. A failure at this level is a validation failure ( a failure to meet the stated or implicit program requirements.)
  •  “All” of the features have been tested in isolation. More precisely, all of the features that will be called within this scenario have been tested on their own and as fas we can tell, none of them has an error that will block  this scenario test.)
  • The tester must have sufficient knowledge of the domain( eg: accounting, if this is an accounting application) and many of the ways in which skilled users will use the program.

Forces

  • Time trade off
  • There is often much time pressure to develop “realistic” tests quickly.
  • Because complex tests are expensive. You can not develop many of them. What level of coverage will you get from these tets? What level should you expect.

Solution

Ideally scenario test has four attributes

Attributes
Attributes
  1. The test is realistic
  2. It is complex
  3. It is easy to tell whether the program passed of failed the test
  4. Serious failure if the program will not pass a given scenario

The key message of this pattern is that you should keep this four attributes in mind when you design a scenario test and try hard to achieve them.

Next we will cover another example of test pattern.

 

 

 

 

How to convert xml to json format?

To Convert XML to JSON format

Consider a situation where in you have data in xml file and want to use it as a input to some other system where in it takes data in the form of JSON. You would require to convert XML to json format.

Below is an example of converting XML file into json file format.

#! /usr/bin/env ruby -rubygems

require 'rubygems'
require 'active_support/all'
require 'json'
json_str = Hash.from_xml(File.open(ARGV[0]).read).to_json
File.open(ARGV[1], 'w+').write json_str

To run the command,

xml-to-json.rb .file1.xml .file_out.json

 

 

Applying Patterns to Software Testing

What is Patterns ?

  • a regular or repetitive form, order or arrangement
  • a model that is considered to be worthy of imitation

Why Patterns ?

Patterns are a way of helping people who design things. Patterns accomplish at least three things

  1. They provide vocabulary for problem solvers.
  2. They focus attention on the forces behind the problem. It allows designers to better understand when and why a solution applies.
  3. They encourage iterative thinking. Each solution creates a new context in which new problems can be solved.

Why Test Patterns?

Testers lack useful vocabulary and are hampered by rigid “one size fits all” methodologies, and face many problems whose solutions are not described in the literature.

Some of the test patterns are

  1. Pair -testing
  2. Architecture- Achilles- Heels
  3. Architecture – reverse engineering
  4. Scenario testing

More on Test Patterns in my next post.

 

 

Difference between __CSVRead and __Stringfromfile function

__CSVRead() function

The CSVRead function returns a string from a CSV file.
When a filename is first encountered, the file is opened and read into an internal array. If a blank line is detected, this is treated as end of file – this allows trailing comments to be used.
All subsequent references to the same file name use the same internal array.

Each thread has its own internal pointer to its current row in the file array. When a thread first refers to the file it will be allocated the next free row in the array, so each thread will access a different row from all other threads.

eg: You need to test login to an app with different user login credentials for different threads.

__Stringfromfile()

The StringFromFile function can be used to read strings from a text file. Each time it is called it reads the next line from the file. All threads share the same instance, so different threads will get data from different lines. When the end of the file is reached, it will start reading again from the beginning, unless the maximum loop count has been reached.

Note: If there are multiple references to the function in a test script, each will open the file independently, even if the file names are the same.  So it is advisable to use different file name across same scripts otherwise the output will be unpredictable.
Eg: You need to add different products to a shopping cart for the same user. In this case you will be using CSVRead to read user credentials and Stringfromfile to read product info that need to be added to each user.
So in a nutshell, Difference between __CSVRead and __Stringfromfile function is that Stringfromfile should be used when different data are required in a loop for same thread and CSVRead should be used when different data are required for different threads(users).

How to uses SYSMON to Monitor Performance counter?

Monitor performance counter using SYSMON

The following example uses VBScript to add counters whose values are retrieved from the local computer, modifies some of the SYSMON properties that control how the monitor is displayed, and processes the OnCounterAdd event. The example uses the wildcard character (*) to add all instances of the process counter.

Save the bleow code in a HTML page and open in browser.

<HTML> <BODY BGCOLOR=#C0C0C0> <SCRIPT LANGUAGE="VBScript">
Sub Monitor_OnCounterAdded(index) Monitor.Counters.Item(1).Width = 8 End Sub </Script> 
<OBJECT CLASSID="clsid:C4D2D8E0-D1DD-11CE-940F-008029004347" ID="Monitor" HEIGHT=80% WIDTH=100%> 
</OBJECT>
<SCRIPT LANGUAGE="VBScript">
Sub Window_OnLoad On Error Resume Next Monitor.ShowValueBar = False Monitor.ShowHorizontalGrid = True Monitor.Counters.Add("\Process(*)\% Processor Time") Monitor.DisplayType=sysmonLineGraph Monitor.GraphTitle="System Performance Overview" End Sub 
</SCRIPT> 
</BODY> 
</HTML>

Note: You need to enable ActiveX in your browser, to run the html file.

How to Automate UI Testing using Instruments for iOS app

Automate UI Testing using Instruments

An instrument is a powerful tool that can be used to automate UI testing. Instruments can also be used to collect data about the performance and behavior of one or more processes on the system and track that data over time. Although most instruments are geared toward gathering trace data, the User Interface instrument helps automate data collection. With it you can record user events while you gather your trace data. You can use this recording to reliably reproduce the same sequence of events over and over again.

Launching Instruments

Instruments is contained within the Xcode 4 toolset. Download Xcode from the App Store and install it onto your computer. After you have installed Xcode, you are ready to run Instruments. Instruments can be launched in one of three ways.

To run Instruments from Xcode

  1. Open Xcode.
  2. Choose Xcode > Open Developer Tool > Instruments.

Instruments_1

Even though each instrument is different, there is one general workflow when collecting information from your app. This workflow is a four-step process.

  1. Choose a trace template
  2. Direct Instruments to your app
  3. Collect information from about your app
  4. Examine the collected information

For UI automation we will be using  Automation template in Instruments to execute scripts. An important benefit of the Automation instrument is that you can use it with other instruments to perform sophisticated tests such as tracking down memory leaks and isolating causes of performance problems.

Note: The Automation instrument only works with apps that have been code signed with a development provisioning profile. Apps signed with a distribution provisioning profile cannot be automated with the UI Automation programming interface.

Writing an automation test script

In instruments we use JavaScript for writing test scripts. To create a script

  1. Select the Automation trace template.
  2. Click Add > Create.
  3. Double-click New Script to change the name of the script.
  4. In the Detail pane, select Console to enter the code for your script.
  5. Choose a target for your script.
  6. Click the Play button at the bottom of the Console.

tracetemplate

Selecting trace template

Instruments_3

After you create the script, it can be used throughout the development of your app. You can do this by importing your saved script and running it with the Automation instrument.

To import a previously saved script

  1. Select the Automation trace template.
  2. Click Add > Import.
  3. Navigate to your saved script file and click Open

Accessing and Manipulating UI Elements

To perform an action on an element in your app, you explicitly identify that element in terms of the app’s element hierarchy. Each accessible element is inherited from the base element, UIAElement. Every element can contain zero or more other elements. Script can access individual elements by their position within the element hierarchy. However, you can assign a unique name to each element by setting the label attribute and making sure Accessibility is selected in Interface Builder for the control represented by that element.

 

The four properties used in the scripts to access elements are

  • name. Derived from the accessibility label
  • value. The current value of the control, for example, the text in a text field
  • elements. Any child elements contained within the current element, for example, the cells in a table view
  • parent. The element that contains the current element