UI Automation using Appium and WinAppDriver

Introduction to Appium

Appium is an open source cross platform test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. Platforms supported are:

  •  iOS
  • Android
  • Windows
  • FirefoxOS

Step 1: Installing Appium in Windows machine

npm install -g appium  // get appium
npm install wd   // get appium client
appium &         //Start appium

Appium install

Note: Appium will install WindowsAppDriver automatically.

You can explicitly install WindowsAppDriver.exe from 

https://github.com/Microsoft/WinAppDriver/releases

Step 2: Writing your Test code( classic Windows app)

You can use any Selenium supported language and specify the full executable path for the app under test in the app capabilities entry. In my example I will be using C#.

Following is a sample code to create test session for Windows Notepad.

 If you are using Appium as mentioned in Step 1, do not forget to add /wd/hub in ur URL.

You can get the complete sample project at my Github site.

namespace NotepadExample
{
    [TestClass]
    public class NotepadBase
    {
        // Note: append /wd/hub to the URL if you're directing the test at Appium
        protected const string WindowsApplicationDriverUrl = "http://127.0.0.1:4723/wd/hub";
        protected static WindowsDriver<WindowsElement> NotepadSession;

        [ClassInitialize]
        public static void NotepadBaseSetup(TestContext testcontext)
        {
            if (NotepadSession == null)
            {
               //launch the Notepad app
               DesiredCapabilities appCapabilities = new DesiredCapabilities();
               appCapabilities.SetCapability("deviceName", "WindowsPC");
               appCapabilities.SetCapability("app", @"C:\Windows\System32\notepad.exe");
               NotepadSession = new WindowsDriver<WindowsElement>(new Uri(WindowsApplicationDriverUrl), appCapabilities);
            }
        }
        [TestMethod]
        public void TestMethod1()
        {
          NotepadSession.FindElementByClassName("Edit").SendKeys("This is an automated text......");
          Assert.IsNotNull(NotepadSession);
        }
        [ClassCleanup]
        public static void NotepadBaseTearDown()
        {
            if (NotepadSession != null)
            {
                NotepadSession.Dispose();
                NotepadSession = null;
            }
        }
    }
}

Using Appium you can write tests with any WebDriver compatible language such as Java, C#, Nodejs, PHP, Phython, Ruby or Perl With the Selenium WebDriver API. It removes limitation of Selenium by providing support to Windows based desktop applications.

Watch the video for easy reference.

 

How to Measure DevOps Success?

How do you measure DevOps success ?

You may not know if you are achieving success with your DevOps implementation unless you measure it and also manage it continuously to keep the key performance indicators (KPI) in right balance.

Following are the few KPIs that help in measure DevOps success.

  • KPI 1 Quality
  • KPI 2 Speed
  • KPI 3  Application Performance
  • KPI 4 Customer experience
  • KPI 5 Business Success

Quality

Underline principle of DevOps is faster and better software delivery.

Shorter development cycles and close collaboration under a DevOps approach mean you catch problems before they go to production, and so spend less time fixing them.

Software quality is directly correlated to customer- impacting issues, so it pays to invest time and energy into addressing quality issues earlier in the cycles. Anyways you do not want customers to be the ones discovering your issues.

Following metrics  can help track and quantify the quality of  your software’s overall continuous improvement efforts:

  • Deployment success/failure  rate
  • Application error rates
  • Issue severity
  • Outstanding bugs

Since many quality issues do not appear until the software hits production, the ability to compare all metrics pre/post deploy is extremely helpful in understanding the impact of a specific release. This approach also allows teams to respond quickly and rollback a release or provide a quick resolution to any incidents that have occurred.

Speed

DevOps teams need to focus on the speed of development, delivery, and response to issues that occur in production. Metrics to consider for helping to track progress and success in this area include:

  • Lead time for changes
  • Frequency of code releases
  • Mean time to resolution

Application Performance

Catching performance problems before they manifest requires tracking number of performance metrics like

  • Up-time (availability)
  • App response time
  • Database response time
  • % of transaction time spent in database
  • Slow SQL queries
  • Resource usage

Customer experience

A poor application performance will result in bad customer experience. Customer experience KPI is directly related to Application performance.  Following are the few metrics that can help measure customer experience.

  • Response time of key transactions
  • Frequency of key transactions
  • Number of visits per user/per week
  • User growth rates
  • Amount of time spent in app

Business Success

Business success can be measured by measuring goals that are important for the company. For example a software can be valuable to the company when it helps grow the customer base, increase revenue, reduce customer service costs,  or some other objectives.

These objectives can be identified by talking to the business stakeholders and understand their business goals and how they will be impacted by delivering the software quickly.

Measure DevOps Success Picture

In a nutshell, all the KPIs are interrelated to each other and impact each other. If we have quality  and or Speed issues on production it will result in bad customer experience which in turn will lead to reduced business success. So, if we measure the 5 KPIs we can find out if our DevOps implementation is resulting in Business success.

KPIMetrics
Quality1. Deployment success/failure rate
2. Application error rates
3. Issue severity
Outstanding bugs
Speed1. Lead time for changes
2. Frequency of code releases
3. Mean time to resolution
Application Performance1. Uptime (availability)
2. App response time
3. Database response time
4. % of transaction time spent in database
5. Slow SQL queries
6. Resource usage
Customer Experience1. Response time of key transactions
2. Frequency of key transactions
3. Number of visits per user/per week
4. User growth rates
Amount of time spent in app
Business SuccessDepending on Business Goals.

What is DevOps

What is DevOps?

DevOps is a term used to refer a set of practices that emphasize the collaboration and communication of software developers and Information technology(IT) professional while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building. Testing and releasing software  can happen at a high velocity and more reliably.

Note that it does not comprises of any one tool neither it excludes any one activities that we do in traditional delivery model.

One of the goal of DevOps is to having an environment where releasing more reliable application can happen more frequently.

To implement DevOps we may need many tools. Many of the tools we already used in one or the other activities.

Under a DevOps model, development and operations teams are no longer “siloed.” Sometimes, these two teams are merged into a single team where the engineers work across the entire application lifecycle, from development and test to deployment to operation. Testing/QA  and security teams also become more tightly integrated with development and operations. They use different set of tool which help them operate and evolve applications quickly and reliably. These tools also help engineers independently accomplish tasks (for example, deploying code or provisioning infrastructure)

Benefits of DevOps

  • Shorter time to Market
  • Speed
  • Rapid Delivery
  • Reliability
  • Scale
  • Improved collaborations
  • Security

Transitioning to DevOps requires a change in culture and mindset. At its simplest, DevOps is about removing the barriers between two traditionally siloed teams, development and operations. In some organizations, there may not even be separate development and operations teams; engineers may do both. With DevOps, the two teams work together to optimize both the productivity of developers and the reliability of operations. They strive to communicate frequently, increase efficiency, and improve the quality of services they provide to customers.

 

What are DevOps essential practices?

  1. Frequent but small updates
  2. Continuous integration
  3. Continuous Deployment/Delivery
  4. Microservices
  5. Infrastructure as a Code
  6. Configuration management
  7. Policy as a code
  8. Monitoring and Logging
  9. Communication and collaboration

 

DevOps tool set

There are no single tools that can achieve all practices. We need different set of tools. Some of the tool categories are listed below.

  1. Code — Code development and review, version control tools, code merging [ eg: GitHub, TFS ]
  2. Build — Continuous integration tools, build status [ eg: TFS, Jenkins]
  3. Test — Test and results determine performance [ eg: TFS test tools, Selenium etc]
  4. Package — Artifact repository, application pre-deployment staging
  5. Release — Change management, release approvals, release automation
  6. Configure — Infrastructure configuration and management, Infrastructure–as–Code tools [Eg: Docker, Puppet, Vagrant ]
  7. Monitor — Applications performance monitoring, end–user experience [ eg: NewRelic, Azure Application Insights]

For more details on DevOps refer the following sites.

Testers – what value add we provide?

As a tester what value add we provide? To answer this we need to answer few questions like
What is a business value of testing?
What is cost of quality?

Running tests by itself has no value add. Testing has value when it connects with some other goals or objective of the organization.
Some of the goals are listed below:

  1. Finding must-fix defects before release. This will reduce long term defect related cost.
  2. Finding less critical defects which have workaround. These workaround can be documented and reduce tech support and helpdesk cost.
  3. Reduce risk by running tests and giving confidence to delivery manager in releasing it to customer. This will give an assurance that the software will also pass the test on customer environments and probability of failure is less.

To measure the quantitative value add and efficiency of testing we need to look into Cost of Quality.

Cost of quality can be understood as cost of poor quality. Cost of quality shows that cost of poor quality is more and that good quality saves money. Cost of quality can be classified as

  1. Cost of prevention – Cost incurred to prevent bugs from happening. Example: training to development team.
  2. Cost of detection – Expense incurred in finding bugs and would include even if we do not find bugs. Examples: Test planning, design and execution etc.
  3. Cost of internal defects/failures – Expense incurred in re-work / bug fixing and expense of re-testing.
  4. Cost of external failures – Expenses we incurred because we did not found and removed all defects before release and there are defect leakages.

Spending effort on external failures are less if we spend more effort in defect prevention, detection and internal failures. It increases confidence that probability of external failures is less.

So it’s very clear that cost of quality is high if there is no internal testing. To phrase it correctly, no proper internal testing as you may argue that developers do testing. By Proper I mean complete end to end testing and not just unit level testing.

We clearly see that there is a need of testers and some value that testers can provide.
Also as a testers we should move out of our comfort zone and think ourselves as an independent advisor to customer and provide trusted advice to customer in terms of quality and quality improvement process.
Feel free to provide your thoughts on this.

 

Multi browser testing using Visual Studio

Multi browser testing using Visual Studio can be achieved by using Selenium and NUnit plugins. Below are step by step details of how it can be achieved.

Step 1

In order to achieve multi browser testing using Visual Studio, we need to install few extensions and plugins. First is NUnit adapter ( used in test execution and NUnit Test framework) and Selenium Web Drivers.

Install Nunit adapter for Visual Studio from Tools-> Extensions and updates..

Step1 extensions and updatesStep1 extensions and updates 2

Similarly install NUnit framework extension.

Step 2

Create an empty Test project in Visual Studio as shown below.

Step2 create project

Select the newly created project and click on “NuGet package manager” and search for Selenium. It will list selenium WebDriver for different browsers. Select them and install. In my example I have selected for browser Edge, Chrome, and Firefox.

Step2 select package managerStep2 select selenium webdrivers 2Step2_installedAfter Installation of Web drivers, they will be automatically added to project references.

Alternative way to add references is by manually downloading the respective Web Drivers and adding references in the project.

Step 3

Now we will start on writing code for launching the browsers. Add a new item of type class file and add following code.

using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Edge;

namespace Sample1
{
    class Sample1_Edge
    {
        private IWebDriver driver;
        [SetUp]
        public void Setup()
        {
            driver = new EdgeDriver();
        }

        [Test]
        public void SampeBrowserLaunch1()
        {
            driver.Navigate().GoToUrl("http://digitalab.org");
        }
        [TearDown]
        public void TearDown()
        {
            driver.Quit();
            driver.Dispose();
        }
    }
}

Note the 3 important sections. Setup, Test and TearDown. Test will repeat for multiple tests.

For other browsers, you need to change the following code.

driver = new EdgeDriver();

to

driver = new FirefoxDriver();
driver = new ChromeDriver();

Step4

Build the code and run the test using Test Explorer.

Step4 Test execution

 

In the next post we will structure this and use xml file to control multiple browser settings.

In this example of Multi browser testing using Visual Studio, we have used VS 2015. But this will also work with version 2013. It should also work with VS 2012 but I have not tested it.

Getting Started with Ruby Cucumber on Windows 10

Ruby can be a great language for testing. With libraries like Capybara for driving web apps and JSON, RestClient, SOAP, and others for interacting with service apps, you’ll find testing in Ruby requires much less code than in C# or Java.

Following instructions can be used to setup Ruby Cucumber on windows 10. Following instruction have been tested on windows 10. Should work on Windows 8.1.

Step1

As a first step download and install Ruby from http://rubyinstaller.org/ . Install the latest release of version 2.2.4 64 bit or 32 bit whichever applicable From <http://rubyinstaller.org/downloads/> . I will be using 32 bit version. The 64-bit versions of Ruby are relatively new on the Windows area and not all the packages have been updated to be compatible with it.

Installing Ruby cucumber on Windows 10

Make sure you check the option to “Add Ruby executables to your PATH.”

Step 2

Also install the Ruby Development Kit from the same place, which will allow Ruby to build native extensions for libraries.

Instructions are available here: https://github.com/oneclick/rubyinstaller/wiki/Development-Kit.

Note: while installing, check if you are installing the right version of DevKit.

C:\devkit> ruby dk.rb review
C:\devkit> ruby dk.rb init
C:\devkit> ruby dk.rb install

Step3: Install Ruby Cucumber Gems

To install Cucumber, first update current gem setup.

C:\Users\pintu>gem update --system:

Next, install the gems you need for cucumber web testing. Following are recommended

C:\Users\you> gem install --no-ri --no-rdoc rspec
C:\Users\you> gem install --no-ri --no-rdoc win32console
C:\Users\you> gem install --no-ri --no-rdoc watir-webdriver
C:\Users\you> gem install --no-ri --no-rdoc cucumber
Run Cucumber
C:\Users\you\Documents> cucumber

If you try to run Cucumber at this stage you will get an error of missing features. Refer the screenshot.

Ruby Cucumber on Windows 10 -2

C:\Users\you\Documents> cucumber --init
C:\Users\you\Documents> cucumber
You are now done.
Next -> Writing  your first test using ruby & cucumber on windows 10

How to configure Visual studio load test Database manually?

To configure Visual studio load test Database manually perform following steps:
  1. Open a Visual Studio Command prompt. Type the following text: On a 32-bit version of Windows type: cd “c:\Program Files\Microsoft Visual Studio 12.0\Common7\IDE” -or- On a 64-bit version of Windows type: cd “c:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE”
  2. In that folder, type the following text:
    • SQL Express: SQLCMD /S localhost\sqlexpress /i loadtestresultsrepository.sql
    • SQL: You can install the database to any existing SQL server. For example, if you had a SQL server named Server1 you would use the following command: SQLCMD /S Server1 -i loadtestresultsrepository.sql

Caution The parameters are case sensitive. You must type uppercase S and lowercase i. SQLCMD /S pintu-home -i loadtestresultsrepository.sql

You might also need to specify a user name and password so that you can connect to Server. If so, you would use the following command: SQLCMD /S Server1 -U -P -i loadtestresultsrepository.sql Caution: The parameters are case sensitive.

configure Visual studio load test Database manually

  1. On the Load Test toolbar, choose Manage Test Controllers. The Manage Test Controllers dialog box is displayed.
  2. In the Load Test Results Connection String, click the browse button (…) to display the Connection Properties dialog box. Note:If you change the connection string for a controller, then you must select the controller.
  3. In Server Name, type localhost\sqlexpress or the name of the server that you used in step 2 such as Server1.
  4. Under Log on to the server, choose Use Windows Authentication.
  5. Under Connect to a database, choose Select or enter a database name. Select LoadTest from the drop-down list box.
  6. Choose OK.
  7. Choose Close in the Administer Test Controller dialog box

 

New features in JMeter 2.13?

JMeter 2.13 is now here for sometime now. There are many new features in JMeter 2.13 . Some of the important and major enhancements are listed below.

Following are the Major enhancements in JMeter 2.13 version.

New Element – New Async BackendListener with Graphite implementation

A new Async BackendListener has been added to allow sending result data to a backend listener. JMeter ships with a GraphiteBackendListenerClient that allows sending results to a Graphite server using Pickle of Plaintext protocols.

New JMeter 2.13

New connect time metric

Starting with this version a new metric called connectTime has been added. It represents the time to establish connection. By default it is not saved to CSV or XML, to have it saved add to user.properties:
jmeter.save.saveservice.connect_time=true

ConnectedTIme

95 and 99 percentile addition on Aggregate Graph and Report

The listeners Aggregate Graph and Aggregate Report previously showed only the 90 percentile (historical behavior), the 95 percentile and the 99 percentile have been added and are customizable. To setup the percentiles value you want, add to user.properties:
aggregate_rpt_pct1=90
aggregate_rpt_pct2=95
aggregate_rpt_pct3=99

New JMeter 2.13

Https test script modification

Now component is able to detect authentication schemes and automatically adds a pre-configured HTTP Authorization Manager with the correct Mechanism

IOT Testing and its Challenges

What is IOT?

As per Wikipedia “The Internet of Things (IoT) is the network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data.”

The Internet of Things allows objects to be sensed and controlled remotely across existing network infrastructure, creating opportunities for more direct integration between the physical world and computer-based systems, and resulting in improved efficiency, accuracy and economic benefit. Each thing is uniquely identifiable through its embedded computing system but is able to interoperate within the existing Internet infrastructure.

Internet of things or IOT covers all those device which are connected to the internet like automobile, washing machine, watch etc. All of these tools were available before. We are simply adding the internet to old things.
The smart watch is a good example of an IoT device. People are using it to track not only distance of run but uses GPS and wireless connection to upload the data to the internet. These information can be used later for analysis. Similarly you can control your home/office lights through internet. Most of you might already be using Smart TV. IOT devices have already started creeping in our day to day activities.

What is IOT testing?

IOT testing involves testing across wide variety of devices with different hardware/software combinations, across different types of network connectivity etc.

Challenges in IOT esting

1. Wide variety of devices – There are thousands or more different internet connected products now. All of them are collecting and sending and displaying data all the time. Testing on all this devices is not feasible for companies developing software’s for it.
2. Connectivity issue – Internet connectivity is what makes this network of things possible, but in many cases that network is intermittent or unreliable. It is important to know that data will be saved and stored correctly if you unexpectedly lose a connection, and is delivered correctly when your connection is restored. You can simulate this by using airplane mode or temporarily disabling the internet connection on the device. Even better than simulation is taking a walk in and out of wifi connections and seeing what really happens when you really lose a connection.

3. Different types and bandwidth of connectivity – There are so much variety of connectivity that testing all possible cases is a challenge

4. Security concerns – With the increase in connectivity the inherent problem of internet will boil down to IOT as well. According to some study, approximately 70 percent of devices in the Internet of Things are vulnerable to security problems — missing data encryption, minimal password requirements, and access to the user interface without passwords.

IOT testing Expectation –

IOT testing is more closely related to how user interact with the devices. Testing scope should also include human experience. “Human Experience” testing has the following components of human interaction with the device.

  1. It should include all physical things, including size, shape and gender of the users.
  2. All sensory reactions including sight, sound, and touch.
  3. Orientation, or the interaction with human movement

Other than that we must also plan for testing in various geographical locations, different weather conditions and contexts.  Finally we must also consider value and test thoroughly in terms of the users perceptions, mind sets, biases and emotions when interacting with the device.