UI Automation using Appium and WinAppDriver

Introduction to Appium

Appium is an open source cross platform test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. Platforms supported are:

  •  iOS
  • Android
  • Windows
  • FirefoxOS

Step 1: Installing Appium in Windows machine

npm install -g appium  // get appium
npm install wd   // get appium client
appium &         //Start appium

Appium install

Note: Appium will install WindowsAppDriver automatically.

You can explicitly install WindowsAppDriver.exe from 

https://github.com/Microsoft/WinAppDriver/releases

Step 2: Writing your Test code( classic Windows app)

You can use any Selenium supported language and specify the full executable path for the app under test in the app capabilities entry. In my example I will be using C#.

Following is a sample code to create test session for Windows Notepad.

 If you are using Appium as mentioned in Step 1, do not forget to add /wd/hub in ur URL.

You can get the complete sample project at my Github site.

namespace NotepadExample
{
    [TestClass]
    public class NotepadBase
    {
        // Note: append /wd/hub to the URL if you're directing the test at Appium
        protected const string WindowsApplicationDriverUrl = "http://127.0.0.1:4723/wd/hub";
        protected static WindowsDriver<WindowsElement> NotepadSession;

        [ClassInitialize]
        public static void NotepadBaseSetup(TestContext testcontext)
        {
            if (NotepadSession == null)
            {
               //launch the Notepad app
               DesiredCapabilities appCapabilities = new DesiredCapabilities();
               appCapabilities.SetCapability("deviceName", "WindowsPC");
               appCapabilities.SetCapability("app", @"C:\Windows\System32\notepad.exe");
               NotepadSession = new WindowsDriver<WindowsElement>(new Uri(WindowsApplicationDriverUrl), appCapabilities);
            }
        }
        [TestMethod]
        public void TestMethod1()
        {
          NotepadSession.FindElementByClassName("Edit").SendKeys("This is an automated text......");
          Assert.IsNotNull(NotepadSession);
        }
        [ClassCleanup]
        public static void NotepadBaseTearDown()
        {
            if (NotepadSession != null)
            {
                NotepadSession.Dispose();
                NotepadSession = null;
            }
        }
    }
}

Using Appium you can write tests with any WebDriver compatible language such as Java, C#, Nodejs, PHP, Phython, Ruby or Perl With the Selenium WebDriver API. It removes limitation of Selenium by providing support to Windows based desktop applications.

Watch the video for easy reference.

 

What is DevOps

What is DevOps?

DevOps is a term used to refer a set of practices that emphasize the collaboration and communication of software developers and Information technology(IT) professional while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building. Testing and releasing software  can happen at a high velocity and more reliably.

Note that it does not comprises of any one tool neither it excludes any one activities that we do in traditional delivery model.

One of the goal of DevOps is to having an environment where releasing more reliable application can happen more frequently.

To implement DevOps we may need many tools. Many of the tools we already used in one or the other activities.

Under a DevOps model, development and operations teams are no longer “siloed.” Sometimes, these two teams are merged into a single team where the engineers work across the entire application lifecycle, from development and test to deployment to operation. Testing/QA  and security teams also become more tightly integrated with development and operations. They use different set of tool which help them operate and evolve applications quickly and reliably. These tools also help engineers independently accomplish tasks (for example, deploying code or provisioning infrastructure)

Benefits of DevOps

  • Shorter time to Market
  • Speed
  • Rapid Delivery
  • Reliability
  • Scale
  • Improved collaborations
  • Security

Transitioning to DevOps requires a change in culture and mindset. At its simplest, DevOps is about removing the barriers between two traditionally siloed teams, development and operations. In some organizations, there may not even be separate development and operations teams; engineers may do both. With DevOps, the two teams work together to optimize both the productivity of developers and the reliability of operations. They strive to communicate frequently, increase efficiency, and improve the quality of services they provide to customers.

 

What are DevOps essential practices?

  1. Frequent but small updates
  2. Continuous integration
  3. Continuous Deployment/Delivery
  4. Microservices
  5. Infrastructure as a Code
  6. Configuration management
  7. Policy as a code
  8. Monitoring and Logging
  9. Communication and collaboration

 

DevOps tool set

There are no single tools that can achieve all practices. We need different set of tools. Some of the tool categories are listed below.

  1. Code — Code development and review, version control tools, code merging [ eg: GitHub, TFS ]
  2. Build — Continuous integration tools, build status [ eg: TFS, Jenkins]
  3. Test — Test and results determine performance [ eg: TFS test tools, Selenium etc]
  4. Package — Artifact repository, application pre-deployment staging
  5. Release — Change management, release approvals, release automation
  6. Configure — Infrastructure configuration and management, Infrastructure–as–Code tools [Eg: Docker, Puppet, Vagrant ]
  7. Monitor — Applications performance monitoring, end–user experience [ eg: NewRelic, Azure Application Insights]

For more details on DevOps refer the following sites.

Testers – what value add we provide?

As a tester what value add we provide? To answer this we need to answer few questions like
What is a business value of testing?
What is cost of quality?

Running tests by itself has no value add. Testing has value when it connects with some other goals or objective of the organization.
Some of the goals are listed below:

  1. Finding must-fix defects before release. This will reduce long term defect related cost.
  2. Finding less critical defects which have workaround. These workaround can be documented and reduce tech support and helpdesk cost.
  3. Reduce risk by running tests and giving confidence to delivery manager in releasing it to customer. This will give an assurance that the software will also pass the test on customer environments and probability of failure is less.

To measure the quantitative value add and efficiency of testing we need to look into Cost of Quality.

Cost of quality can be understood as cost of poor quality. Cost of quality shows that cost of poor quality is more and that good quality saves money. Cost of quality can be classified as

  1. Cost of prevention – Cost incurred to prevent bugs from happening. Example: training to development team.
  2. Cost of detection – Expense incurred in finding bugs and would include even if we do not find bugs. Examples: Test planning, design and execution etc.
  3. Cost of internal defects/failures – Expense incurred in re-work / bug fixing and expense of re-testing.
  4. Cost of external failures – Expenses we incurred because we did not found and removed all defects before release and there are defect leakages.

Spending effort on external failures are less if we spend more effort in defect prevention, detection and internal failures. It increases confidence that probability of external failures is less.

So it’s very clear that cost of quality is high if there is no internal testing. To phrase it correctly, no proper internal testing as you may argue that developers do testing. By Proper I mean complete end to end testing and not just unit level testing.

We clearly see that there is a need of testers and some value that testers can provide.
Also as a testers we should move out of our comfort zone and think ourselves as an independent advisor to customer and provide trusted advice to customer in terms of quality and quality improvement process.
Feel free to provide your thoughts on this.

 

Multi browser testing using Visual Studio

Multi browser testing using Visual Studio can be achieved by using Selenium and NUnit plugins. Below are step by step details of how it can be achieved.

Step 1

In order to achieve multi browser testing using Visual Studio, we need to install few extensions and plugins. First is NUnit adapter ( used in test execution and NUnit Test framework) and Selenium Web Drivers.

Install Nunit adapter for Visual Studio from Tools-> Extensions and updates..

Step1 extensions and updatesStep1 extensions and updates 2

Similarly install NUnit framework extension.

Step 2

Create an empty Test project in Visual Studio as shown below.

Step2 create project

Select the newly created project and click on “NuGet package manager” and search for Selenium. It will list selenium WebDriver for different browsers. Select them and install. In my example I have selected for browser Edge, Chrome, and Firefox.

Step2 select package managerStep2 select selenium webdrivers 2Step2_installedAfter Installation of Web drivers, they will be automatically added to project references.

Alternative way to add references is by manually downloading the respective Web Drivers and adding references in the project.

Step 3

Now we will start on writing code for launching the browsers. Add a new item of type class file and add following code.

using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Edge;

namespace Sample1
{
    class Sample1_Edge
    {
        private IWebDriver driver;
        [SetUp]
        public void Setup()
        {
            driver = new EdgeDriver();
        }

        [Test]
        public void SampeBrowserLaunch1()
        {
            driver.Navigate().GoToUrl("http://digitalab.org");
        }
        [TearDown]
        public void TearDown()
        {
            driver.Quit();
            driver.Dispose();
        }
    }
}

Note the 3 important sections. Setup, Test and TearDown. Test will repeat for multiple tests.

For other browsers, you need to change the following code.

driver = new EdgeDriver();

to

driver = new FirefoxDriver();
driver = new ChromeDriver();

Step4

Build the code and run the test using Test Explorer.

Step4 Test execution

 

In the next post we will structure this and use xml file to control multiple browser settings.

In this example of Multi browser testing using Visual Studio, we have used VS 2015. But this will also work with version 2013. It should also work with VS 2012 but I have not tested it.

Getting Started with Ruby Cucumber on Windows 10

Ruby can be a great language for testing. With libraries like Capybara for driving web apps and JSON, RestClient, SOAP, and others for interacting with service apps, you’ll find testing in Ruby requires much less code than in C# or Java.

Following instructions can be used to setup Ruby Cucumber on windows 10. Following instruction have been tested on windows 10. Should work on Windows 8.1.

Step1

As a first step download and install Ruby from http://rubyinstaller.org/ . Install the latest release of version 2.2.4 64 bit or 32 bit whichever applicable From <http://rubyinstaller.org/downloads/> . I will be using 32 bit version. The 64-bit versions of Ruby are relatively new on the Windows area and not all the packages have been updated to be compatible with it.

Installing Ruby cucumber on Windows 10

Make sure you check the option to “Add Ruby executables to your PATH.”

Step 2

Also install the Ruby Development Kit from the same place, which will allow Ruby to build native extensions for libraries.

Instructions are available here: https://github.com/oneclick/rubyinstaller/wiki/Development-Kit.

Note: while installing, check if you are installing the right version of DevKit.

C:\devkit> ruby dk.rb review
C:\devkit> ruby dk.rb init
C:\devkit> ruby dk.rb install

Step3: Install Ruby Cucumber Gems

To install Cucumber, first update current gem setup.

C:\Users\pintu>gem update --system:

Next, install the gems you need for cucumber web testing. Following are recommended

C:\Users\you> gem install --no-ri --no-rdoc rspec
C:\Users\you> gem install --no-ri --no-rdoc win32console
C:\Users\you> gem install --no-ri --no-rdoc watir-webdriver
C:\Users\you> gem install --no-ri --no-rdoc cucumber
Run Cucumber
C:\Users\you\Documents> cucumber

If you try to run Cucumber at this stage you will get an error of missing features. Refer the screenshot.

Ruby Cucumber on Windows 10 -2

C:\Users\you\Documents> cucumber --init
C:\Users\you\Documents> cucumber
You are now done.
Next -> Writing  your first test using ruby & cucumber on windows 10

How to configure Visual studio load test Database manually?

To configure Visual studio load test Database manually perform following steps:
  1. Open a Visual Studio Command prompt. Type the following text: On a 32-bit version of Windows type: cd “c:\Program Files\Microsoft Visual Studio 12.0\Common7\IDE” -or- On a 64-bit version of Windows type: cd “c:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE”
  2. In that folder, type the following text:
    • SQL Express: SQLCMD /S localhost\sqlexpress /i loadtestresultsrepository.sql
    • SQL: You can install the database to any existing SQL server. For example, if you had a SQL server named Server1 you would use the following command: SQLCMD /S Server1 -i loadtestresultsrepository.sql

Caution The parameters are case sensitive. You must type uppercase S and lowercase i. SQLCMD /S pintu-home -i loadtestresultsrepository.sql

You might also need to specify a user name and password so that you can connect to Server. If so, you would use the following command: SQLCMD /S Server1 -U -P -i loadtestresultsrepository.sql Caution: The parameters are case sensitive.

configure Visual studio load test Database manually

  1. On the Load Test toolbar, choose Manage Test Controllers. The Manage Test Controllers dialog box is displayed.
  2. In the Load Test Results Connection String, click the browse button (…) to display the Connection Properties dialog box. Note:If you change the connection string for a controller, then you must select the controller.
  3. In Server Name, type localhost\sqlexpress or the name of the server that you used in step 2 such as Server1.
  4. Under Log on to the server, choose Use Windows Authentication.
  5. Under Connect to a database, choose Select or enter a database name. Select LoadTest from the drop-down list box.
  6. Choose OK.
  7. Choose Close in the Administer Test Controller dialog box

 

New features in JMeter 2.13?

JMeter 2.13 is now here for sometime now. There are many new features in JMeter 2.13 . Some of the important and major enhancements are listed below.

Following are the Major enhancements in JMeter 2.13 version.

New Element – New Async BackendListener with Graphite implementation

A new Async BackendListener has been added to allow sending result data to a backend listener. JMeter ships with a GraphiteBackendListenerClient that allows sending results to a Graphite server using Pickle of Plaintext protocols.

New JMeter 2.13

New connect time metric

Starting with this version a new metric called connectTime has been added. It represents the time to establish connection. By default it is not saved to CSV or XML, to have it saved add to user.properties:
jmeter.save.saveservice.connect_time=true

ConnectedTIme

95 and 99 percentile addition on Aggregate Graph and Report

The listeners Aggregate Graph and Aggregate Report previously showed only the 90 percentile (historical behavior), the 95 percentile and the 99 percentile have been added and are customizable. To setup the percentiles value you want, add to user.properties:
aggregate_rpt_pct1=90
aggregate_rpt_pct2=95
aggregate_rpt_pct3=99

New JMeter 2.13

Https test script modification

Now component is able to detect authentication schemes and automatically adds a pre-configured HTTP Authorization Manager with the correct Mechanism

IOT Testing and its Challenges

What is IOT?

As per Wikipedia “The Internet of Things (IoT) is the network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data.”

The Internet of Things allows objects to be sensed and controlled remotely across existing network infrastructure, creating opportunities for more direct integration between the physical world and computer-based systems, and resulting in improved efficiency, accuracy and economic benefit. Each thing is uniquely identifiable through its embedded computing system but is able to interoperate within the existing Internet infrastructure.

Internet of things or IOT covers all those device which are connected to the internet like automobile, washing machine, watch etc. All of these tools were available before. We are simply adding the internet to old things.
The smart watch is a good example of an IoT device. People are using it to track not only distance of run but uses GPS and wireless connection to upload the data to the internet. These information can be used later for analysis. Similarly you can control your home/office lights through internet. Most of you might already be using Smart TV. IOT devices have already started creeping in our day to day activities.

What is IOT testing?

IOT testing involves testing across wide variety of devices with different hardware/software combinations, across different types of network connectivity etc.

Challenges in IOT esting

1. Wide variety of devices – There are thousands or more different internet connected products now. All of them are collecting and sending and displaying data all the time. Testing on all this devices is not feasible for companies developing software’s for it.
2. Connectivity issue – Internet connectivity is what makes this network of things possible, but in many cases that network is intermittent or unreliable. It is important to know that data will be saved and stored correctly if you unexpectedly lose a connection, and is delivered correctly when your connection is restored. You can simulate this by using airplane mode or temporarily disabling the internet connection on the device. Even better than simulation is taking a walk in and out of wifi connections and seeing what really happens when you really lose a connection.

3. Different types and bandwidth of connectivity – There are so much variety of connectivity that testing all possible cases is a challenge

4. Security concerns – With the increase in connectivity the inherent problem of internet will boil down to IOT as well. According to some study, approximately 70 percent of devices in the Internet of Things are vulnerable to security problems — missing data encryption, minimal password requirements, and access to the user interface without passwords.

IOT testing Expectation –

IOT testing is more closely related to how user interact with the devices. Testing scope should also include human experience. “Human Experience” testing has the following components of human interaction with the device.

  1. It should include all physical things, including size, shape and gender of the users.
  2. All sensory reactions including sight, sound, and touch.
  3. Orientation, or the interaction with human movement

Other than that we must also plan for testing in various geographical locations, different weather conditions and contexts.  Finally we must also consider value and test thoroughly in terms of the users perceptions, mind sets, biases and emotions when interacting with the device.

Risk based testing

What is risk ?

Risk is an uncertain event or condition that, if it occurs, has an effect on at least one objective.

or

The probability of something happening multiplied by the resulting cost or benefit if it does. (This concept is more properly known as the ‘Expectation Value’ or ‘Risk Factor’ and is used to compare levels of risk)

With respect to software testing there are mainly two types of risks.
– Product Risk
– Project Risk

Product risk – These types of risk generally effect quality of product delivered. These types of risks directly effect test object.

Where as Project risk effect overall success of a project. For example, potential staffing shortage that could result in delay of completion of project can be classified as project risk.

In Risk Based testing we use the risk identified with the level of risks identified for each risk item to help our testing.

Risks can be used to guide testing in different ways like:

1.  Based on the risks identified we can identify test objects which require more effort and accordingly allocate more time to high risk items.

2. Test reporting based on Residual Risks. Like which tests have not yet run or have been skipped? Which tests we have run ? What passed and what failed.

Example:

Just like you buy insurance when you are worried about potential risks like accident or premature death. Similarly we should test areas/objects that are worrisome and ignore the ones which are not having any risks.

Benefits of Risk based testing

  • As testers we often face time pressure and seldom find enough time to run the test we want to. Mostly all testings are generally time boxed. Risk based testing provides a way to prioritize and triage test at any point of time.
  • Risk based testing provides a smart way to choose finite number of tests from an infinite set of tests.
  • Often when there are time pressure, risk based testing provides a way to drop test intelligently while also providing a way to discuss with project stake holders the risk inherent in doing so.
  • Risk based testing helps in determining acceptable level of residual risk rather than relying on inadequate metrics like bug and test counts.

 

The Economies of Software Testing

What do you mean by Economies of Software testing?
Why software defects are costly to find and fix?
How early defect detection can save much cost in fixing defects?

There is a definite economic impact of software testing. One economic impact is from the cost of defects and the another is the way we perform testing. Cost of defect is a very real and very tangible cost. The second point the way we perform testing will not consider it now.

Where defects originate?

defect_originate

One of the most commonly understood facts about defects is that most defects originate in the requirements definition phase of a project. The next runner-up is the design phase.

Some problems in getting accurate, clear, and testable requirements are:

  • Many people do not have a solid requirements gathering process
  • Few people have been trained in or understand the dynamics of requirements
  • Projects, people, and the world around us change very quickly
  • The English language is ambiguous and even what we consider clear language can be interpreted differently by different people.

 Where testing resources are used?

In the previous section we saw that most defects originate in requirements and design, but most of the testing effort occurs in a traditional “testing” phase toward the end of the project. The problem with the big bang approach to testing is that defects are not found until toward the end of the project. This is the most costly and risky time to fix defects. Some complex defects may even be impossible to fix.

 Relative cost of fixing defects

One of the known facts about software defects is that the longer they go undetected, the more expensive they are to fix. Although research differs on the exact ratios, the general rule is 1:10:100. That is, if a defect costs one unit (hour, dollar, etc.) to fix in requirements and design, it costs 10 units to fix in testing (system/acceptance) and over 100 times to fix in production. The cosThis cost of defects doesn’t even take into account the impact cost of defects. These t of fixing in production may be even higher than 100 times. Costs could be attributed to lost revenue, reimbursements, fraud, lost customers, bad public relations, and litigation.

cost_of_defect_fixing

Conclusion

  • Most defects are created in the early stages of a project
  • Most defects are found in the later stages of a project
  • It costs 10 to 100 times as much to fix a defect in the later phases of a project.

So, what does all of this mean? The main conclusion is that most people perform testing too late in the process. These people wonder why testing is so expensive and why their projects are often over budget.

If you really want to make your testing more efficient and reduce the overall cost of testing and defects, test early in the project and continue testing throughout the project.