What is performance testing?

“Performance testing is defined as the technical investigation done to determine or validate the speed, scalability, and/or stability characteristics of the system under test.”

Types of Performance testing

  • Performance test
  • Load test
  • Stress test
  • Spike test
  • Capacity Test
  • Soak/endurance test

Performance testing process

Performance testing starts with identifying test environment and performance acceptance criteria. Different steps that are required in performance testing are as follows:
perfromance_testing_process

Performance counters and their meanings

Some important performance counters are listed below:
  • Hits per second (HPS)
  • Transaction per second (TPS)
  • Total transaction per second (TTPS)
  • Connection Per second
  • Page download per second
  • Throughput (w.r.t. response data)
  • Throughput (w.r.t. requests)
  • Throughput (w.r.t. Business Process transaction)
  • Response time
 
Resource Utilization
CPU
  • Processor Time
  • Privileged time
  • Interrupt time
  • Processor queue length
  • Context switching/Sec
 
Memory

  • Available Bytes
  • Page reads/Sec
  • Page/sec
  • Pool non paged bytes
 
DISK I/O
  • Avg Disk queue length
  • Avg. disk Read/Write queue length
Network I/O

  • Bytes total/sec
  • Server bytes total/sec
Popular tools to measure the performance counters
  1. Loadrunner
  2. Visual Studio Ultimate
  3. JMeter
  4. Silk performer
  5. Grinder

 

Performance counters and there meaning in JMeter

Common performance counters and there meaning in JMeter.

Elapsed time JMeter measures the elapsed time from just before sending the request to just after the last response has been received.  It does not include the time needed to render the response, nor does JMeter process any client code.

LatencyJMeter measures the latency from just before sending the request to just after the first response has been received. Thus the time includes all the processing needed to assemble the request as well as assembling the first part of the response, which in general will be longer than one byte.

It is not just the network latency. It is much closer to the time which is experienced by a browser.

Median is a number which divides the samples into two equal halves. Half of the samples are smaller than the median, and half are larger (Some samples may equal the median). This is a standard statistical measure. The Median is the same as the 50th Percentile.

90% Line (90 th Percentile) is the value below which 90% of the samples fall. 90% of the request have response time that is either equal to or less than “90% line response time”. This is a standard statistical measure.

Standard Deviation – is a measure of the variability of a data set. This is a standard statistical measure. JMeter calculates the population standard deviation (e.g. STDEVP function in spreadsheets), not the sample standard deviation (e.g. STDEV).

Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time). This is same as Hits/sec as reported by some other performance testing tools.

IP Switching and Load Testing

IP Switching in Load testing

The most common use for IP Switching is when load testing against a load balancer. Load balancer typically use the IP address to route requests to a particular Web server in the farm.
So if you have 2 agents driving load to 3 Web servers, since all traffic is coming from two IPs (one on each agent), only two of the web servers would get all the traffic.
IP Switching provides a way to have traffic come from multiple IPs on the same agent, enabling the load balancer to balance load across the farm.
IP Switching will only work if you are running the test from the Controller/Agnet(remote execution).
To confirm current execution settings in VS2010:

    1. Under Test – Edit Test Settings – open the test settings that you wish to use. Check the Roles, and under Test Execution Method, make sure you have that set to Remote Execution then the Controller edit box should appear for you to type in the macine name. You can also save this as a new test settings.
    1. Under Test – Select Active Test Settings – make sure you set the test settings active for the one you edited above.
    1. In the Menu Test-Manage Test Controller all the agent machine should be listed
  1. When you submit your LoadTest, make sure that you see in the menu Test-Manage Test Controller that it shows all agent machine and the status switches to “Running Test”.

For more details refer the Visual Studio Performance testing quick reference guide.

Saving perfmon data

If you are using perfmon tool in windows, you may sometime want to save the data for future analysis. To accomplish this you need to follow the following steps.

 

  • Launch the perfmon tool.  To open Performance Monitor, click Start, click in the Start Search box, type perfmon, and then press ENTER.
  • Add the counter that is required to be saved.
  • Right-click anywhere in the Performance Monitor display pane, point to New, and click Data Collector Set. The Create New Data Collector Set Wizard starts. The Data Collector Set created will contain all of the data collectors selected in the current Performance Monitor view.

Perfmon 1

 

 

  • Enter a name for your Data Collector Set and click Next
  • The Root Directory will contain data collected by the Data Collector Set. Change this setting if you want to store your Data Collector Set data in a different location than the default. Browse to and select the directory, or type the directory name.

perfmon 2

 

  • Click Next to define a user for the Data Collector Set to run as, or click Finish to save the current settings and exit. By default it is run as system user and as a best practice this should be selected.
  • Click Finish to return to Windows Performance Monitor.
  • To start the Data Collector Set immediately (and begin saving data to the location specified), select Start this data collector set now.

To view the saved information, right click on performance pane and click on Properties. Select source and select log file option and browse folder where the file is saved.

perfmon 3

Note: The Performance Log Users group must be assigned the “Log on as a batch user” user right. By default this should be already set. If not follow the following steps.

 

  • Click Start, click in the Search box, type secpol.msc, and press ENTER. The Local Security Policy snap-in will open in Microsoft Management Console.
  • In the navigation pane, expand Local Policies and click User Rights Assignment.
  • In the console pane, right-click Log on as a batch job and click Properties.
  • In the Properties page, click Add User or Group.
  • Type Performance Log Users in the Select Users or Groups dialog box and then click OK.
  • In the properties page, click OK.

 

 

Load Testing using The Grinder

Introduction

The Grinder is an open source load testing tool. The Grinder uses Jython scripts and the scripts are executed using agents which can be distributed over the network. Property file is used to define tests and load.

The Grinder framework comprises three types of process or programs;

    1. Worker Process: Interpret Jython test scripts and perform tests using a number of worker threads

 

    1. Agent Process: Manage worker processes

 

    1. The Console:
        1. Coordinate the other processes
        1. Collate and display statistics
        1. Script editing and distribution

 

 

Configuration

We will create a couple of batch files to conveniently start up the agents and console.

    1. Extract the zip file and copy at any location. E.g. C:\grinder
    1. First batch file “setgrinderenv.bat” will have following commands.

set GRINDERPATH=”C:\grinder”

set GRINDERPROPERTIES=”C:\grinder\grinder.properties”

set CLASSPATH=%GRINDERPATH%\lib\grinder.jar;%CLASSPATH%

    1. Second batch file “startconsole.bat”

call “C:\grinder\setGrinderEnv.cmd”

java -cp %CLASSPATH% net.grinder.Console

    1. Third batch file “startagent.bat”

call “C:\grinder\setGrinderEnv.cmd”

java -cp %CLASSPATH% net.grinder.Grinder %GRINDERPROPERTIES%

How to update Property file?

The load test is defined using the property file. The configuration entry grinder.script = http.py indicates the path of the Jython script which will be run. Other interesting configurations include the number of processes created by each agent (grinder.processes) and the number of threads for each process (grinder.threads).

I would also recommend to explicitly specify the log and the temporary directories:

# The directory in which worker process logs should be created.

grinder.logDirectory = log

 # Additional arguments to worker process JVMs.

grinder.jvm.arguments = -Dpython.cachedir=”C:\\ grinder\\tmp”

The Test

The load test can be written manually or can be generated through the TCP proxy provided with The Grinder.

Starting the TCP proxy to generate the load case could be done through yet another batch file:

“Tcpproxy.bat”

call “C:\grinder\setGrinderEnv.cmd”

java -cp %CLASSPATH% net.grinder.TCPProxy -console -http > proxy-output.py

When TCP Proxy is started, display the following information:

Initializing as an HTTP/HTTPS proxy with the parameters:

Request filters:  EchoFilter

Response filters: EchoFilter

Local address:    localhost:8001

Engine initialised, listening on port 8001

This indicates that the TCPProxy is listening as an HTTP proxy on port 8001 (the default, but you can change it with -localPort).

Preparing the Browser

In the browser options dialog, set the proxy host to be the host on which the TCPProxy is running and proxy port to be 8001).    Run through your test scenario on your website. Having finished your run through, press “Stop” on the TCPProxy console and the generated script will be written togrinder.py.

The grinder.py file contains headers, requests and a logical grouping of requests into pages, of the recorded tests.

Once you’ve recorded your script you have two methods that you can use to replay your script:

    1. You can create a simple grinder.properties file and you can replay the recorded scenario with The Grinder. Your properties file should at least set grinder.script to grinder.py.
    1. Alternately you can use the console to distribute your script to an agent and set it as the script to run. Each agent will still need a simple grinder.properties file containing the console address, though you will not need to set the grinder.script property.

Will be continued….

 

Importance of Latency and Bandwidth in Web response time

The six parameters of Web response time are listed below:

  • Page size: Page size is measured in Kbytes, and on the surface, the impact of this variable is pretty obvious: the larger the page, the longer it takes to download. When estimating page size, however, many people fail to consider all the components that contribute to page size. Images, Java and other applets, banners from third sources, etc.
  • Minimum bandwidth: Minimum bandwidth is defined as the bandwidth of the smallest pipe between your content and the end user. Just as the strength of a chain is determined by its weakest link, the effective bandwidth between two end points is determined by the smallest bandwidth between them.
  • Round trip time: In the context of Web page response time, round-trip time (RTT) indicates the latency, or time lag, between the sending of a request from the user’s browser to the Web server and the receipt of the first few bytes of data from the Web server to the user’s computer. RTT is important because every request/response pair (even for a trivially small file) has to pay this minimum performance penalty. Typical Web page requires several request/response cycles.
  • Turns: A typical Web page consists of a base page [or index page] and several additional objects such as graphics or applets. These objects are not transmitted along with the base page; instead, the base page HTML contains instructions for locating and fetching them. Unfortunately for end-user performance, fetching each of these objects requires a fair number of additional communication cycles between the user’s system and the Web site server each of which is subject to the RTT delay.
  • Server processing time: The last factor in the response time formula is the processing time required by the server and the client to put together [i.e. generate and render] the required page so it can be viewed by the requester. This can vary dramatically for different types of Web pages. On the server side, pages with static content require minimal processing time and will cause negligible additional delay. Dynamically created pages (e.g., personalized home pages like my.yahoo.com) require a bit more server effort and computing time, and will introduce some delay. Finally, pages that involve complex transactions (e.g., credit card verification) may require very significant processing time and might introduce delays of several seconds.
  • Client processing time: On the client side, the processing time may be trivial (for a basic text-only page) to moderate (for a page with complex forms and tables) to extreme. If the page contains a Java applet, for example, the client’s browser will have to load and run the Java interpreter, which can take several seconds.

 

Page response time = (Page Size/Minimum Bandwidth) + (Round Trip time * Turns) + Server Processing time + Client Processing time

There are few factors that can determine the turn count

  •  Http 1.0 or http 1.1. Http 1.1 uses persistent TCP connections so that the browsers do not repeatedly closes and reopen TCP connections with a server and for large files TCP segments will be larger. This will reduce turns.
  • Recent browser versions open parallel 2 or more connections for each distinct server domain.
  • Client side caches in browsers.

So in nutshell

Page Size: larger the page size and or complex data longer the response time.

Bandwidth: Low or congested bandwidth, the longer the response time.

Turn : Higher the number of turns, longer the response time

Round trip time: Longer the distance between user and server, longer the response time

Server processing: The more stressed or busy the server, longer the response time

Client processing: More the client sever busy/low end client, longer the response time.

Crux

Page Size/bandwidth – As the Bandwidth increases, this factor becomes smaller. With a very high bandwidth and very small page size this can approach to zero.

Turns * RTT – RTT or ping time cannot be zero though it can be very less for faster connections. If the ping time is say 100 ms and there are 20 turns. It comes out to be 2 sec for network latency.

So not only the server processing time or the client processing time that contributes to longer response time. If the web page size is large or numbers of turns are too many, it can also increase the response time.

Jmeter limitations with Monitoring server on load and how to overcome?

JMeter has option to add monitor test plan to monitor application servers. But it only works with Tomcat 5 status servlet. But any servlet container that supports JMX (Java Management Extension) can port the status servlet to provide the same information.

Also if user want want to use the monitor with other servlet or EJB containers, Tomcat’s status servlet will work with other containers for the memory statistics without any modifications. To get thread information, MBeanServer will require change in lookup to retrieve the correct MBeans.

But still it is not possible of Windows server IIS.

One of the way to overcome this situation is using Nagios. This can also be used in conjuction with JMeter. That is load is generated using JMeter and Nagios is used to monitor application server performance on load.

Now, the question is what is nagios? How it works ….

Nagios is System and network monitoring tool. It watches hosts and services that is specified and alerts when things go bad crosses threshold value ( this threshold value can be configured) and when they get better.

It works on Linux and linux like system.. BUT it can also MONITOR WINDOWS SERVER. This is the most important aspect of it.

The only requirement of running Nagios is Linux machine or its variant and c compiler.

On the next article I will explain the details of how it can be used to configure and monitor windows machine.