Performance-Testing

2011-05-12T09:27:00.000-07:00

I mentioned recently I would say more about Agile performance testing, including phasing approaches.

Traditional approaches often leave testing until a fairly late stage in the application development lifecycle. The Acutest approach to testing services involves testing early in the lifecycle, and that means we often have to test component parts of a system in isolation, before everything is complete.

Sometimes this can be as simple as choosing a small number of business processes and running a test cycle geared around those. We expect to concentrate on finding application bottlenecks such as coding/design issues in the early stages. The majority of the tests would be ramp tests designed to find bottlenecks by intentionally looking for performance limits in specific areas of interest. This works best when we can combine this with our unique risk-based approach and test the most likely areas of failure - especially those most critical to users. Sometimes that can be a challenge as developers will have a tendency to release the easy bits first !

One thing I always recommend is running a combined load test as early as possible, even if not all parts of the system are stable enough to test and have to be excluded, or throughput requirements are still being agreed. You always learn something - either about the system under test, the test environment, or the test tool. Typically we would spend 20% of time in the first cycle on this, building up to 80% in later cycles. In later cycles we would however leave some time for new ramp tests to be executed, and re-runs of failed tests from earlier cycles.

Generally working co-operatively within a project or programme brings best results. It's recognised by the leading vendors of performance testing tools that a team approach is needed to successfully deliver performance tests on time. Our experience is that benefits can come from including software developers as well. Agile performance testing is possible, with short cycles, tight configuration management and good communication. Recording and customising scripts in short timescales is a challenge but can be achieved. This approach also builds performance awareness into development teams, which brings benefits down the line.

So there are some thoughts on phasing of performance tests. Let me know if these subjects are of interest.

2011-04-25T02:15:00.000-07:00

What a breath-taking roller coaster it's been !

Still it's good to be busy, as we are at Acutest. Anyway, I'm reviving performance-testing.blogspot.com and interested in covering in more detail some of the things we've found beneficial over the years such as:

-Agile performance testing, including phasing approaches
-Configuring virtual users and scripts in various performance testing tools
-Approaches to Business process testing, including financial and ERP software from Elite, a Thomson Reuters business, and SAP.

More later folks!

2008-08-04T05:49:00.000-07:00

Some of the links to articles by Alberto Savoia mentioned in my earlier article:

Web load testing (article 1) - how to keep the load steady

have moved. Here are the up to date links:

“Web load test planning”
“Trade secrets from a web testing expert”
“Web page response time 101”

Performance monitoring – Windows server monitoring

2006-03-27T07:15:00.000-08:00

I have been doing some research on what system monitoring is useful during performance testing.

I have concentrated on a minimum useful set of information needed to verify that no bottlenecks occur when using a performance testing tool. Even this minimal set should be useful though to enable a rough prediction of the maximum load that could be supported by the system under test.

There’s a wealth of information available on this subject. Two sites I found particularly useful were these:
http://www.sql-server-performance.com/tips_performance.asp http://www.computerperformance.co.uk/HealthCheck/

One subject on which there seemed to be a range of opinions was whether it is best to set up monitoring remotely or locally. If you set up remotely there will be some network traffic, but if you set up locally there is the overhead of running perfmon itself. Do you think it is significant either way – if so, in what situations? Perhaps this is only an issue if you are already close to resource utilization limits.

One thing which is probably always true is that it’s best to be selective and only collect the performance counters you need. This document is my attempt to propose a subset which will nearly always be useful. Take a look and let me know what you think. Is there anything obviously missing? Is it too comprehensive?

Performance testing - agile and open source

2006-03-23T08:18:00.000-08:00

These days many folk are asking about agile techniques and how they apply to performance testing. That's not surprising - they are keen to get the benefits of testing early in their development process and save costs as a result.

Often they enquire about open source performance testing tools as well. They're keen to see how they compare with some of the commercially available load testing tools and whether they can make further savings by using open source.

I put together an article on both these subjects with some links which may help if this is a subject that interests you. If it is, let me know and please feel free to suggest more subjects for similar articles in the future.

Performance monitoring - including SNMP and rstatd data in load test scripts

2006-03-23T04:16:00.000-08:00

Occasionally questions come up about how to monitor UNIX servers during load testing. SNMP, rstatd, vmstat - it's all so confusing! Fortunately there are resources to help.

I came across a nice little article recently on this - and there's some interesting references to SNMP techniques:
http://www.myloadtest.com/collecting-performance-stats-with-snmp

To add to the knowledge base, here is a link to an article I wrote a year or so ago on the same subject which may be of interest. Following a proof-of-concept, the conclusion was that a script could be written to extract SNMP counters and record them via user-defined counters for later analysis. The evaluation was done using Compuware's QALoad but should be portable to Mercury LoadRunner.

The link to the article:
http://www.users.globalnet.co.uk/~cater/dave/acutest/TN1.doc

Related references that may be useful:
Open Source project Net-SNMP http://www.net-snmp.org
Linux version of rstatd(1M) http://rstatd.sourceforge.net

Let me know what you think of the article and ideas for the future.

Web load testing (article 2) - be generous with pacing

2006-03-21T10:25:00.000-08:00

This is the second article explaining a practical approach to
avoiding misunderstandings concerning concurrent user load.

If you read the first article, you will be aware of my obsession with achieving even pacing for each test user. Here I will explain how to design tests to achieve even pacing.

First, a digression on thinking time and system time.

Most load testing tools allow the user thinking time to be simulated. In fact, most allow thinking time to be recorded during a user session so it can be replayed during the load test. I always include thinking time in the web session. This ensures that the total duration of each user session roughly corresponds with reality. This is important where the web system maintains sessions for users – you want the number of concurrent sessions to be realistic.

I also like to use transaction timers to measure ALL activity that is not user thinking time. In other words, I use timers to measure all system activity. Not surprisingly, I call this “system time”.

I estimate the minimum system time by running the web test script for a small number of users and adding up the times of all transactions. I multiply the system time by three to allow for slow-down under load, and add the thinking time. Then I add a comfortable margin for error, and end up with a pacing time that should be achieved– even under heavy load.

So there is my tip – allow plenty of time for each user session. Be generous with your pacing. I do everything I can to ensure there is a pause between user sessions so pacing is maintained.

The astute amongst my readers will have worked out that being generous with pacing implies a larger number of test users. Let me explain. To achieve a specified load (say 720 user sessions started per hour) with a pacing of 5 minutes requires 60 test users, whereas with a pacing of 10 minutes requires 120 test users. You should take this into account during test planning.

Web load testing (article 1) - how to keep the load steady

2006-03-21T09:10:00.000-08:00

I recently came across a series of articles written in 2001 by Alberto Savoia which impressed me very much. If you search for these titles, you can still find them:
“Web load test planning”
“Trade secrets from a web testing expert”
“Web page response time 101”

The second of Savoia's articles covers three main topics:
Misunderstanding concurrent users
Miscalculating user abandonment
Over-averaging page response times

When I read these I was interested (and somewhat relieved!) to find that much of what Savoia recommends aligns with my own approach to web performance testing.

In this article, I’ll outline a practical approach to how I deal with avoiding misunderstandings concerning concurrent user load.

Savoia rightly pointed out that when defining load for a web performance test, the starting point should be “number of user sessions started per hour”. (It matters less how long each of these individually takes from start to end, though as I will point out in a subsequent article, it cannot be completely ignored.)

Most of the well known load testing tools allow for “pacing” of a test user. You can arrange for a test user to repeat the same session with the start times for each session spaced apart at the “pacing” interval.

It is tempting to ignore this and perhaps try to disable pacing so as to start each session immediately the previous one has completed. Believe me, this is almost always not a good thing to do. The reason is essentially quite simple. As there are variations in the time each user session will actually take (particularly under load) the rate at which user sessions start will be uneven, and you will be unable to explain to anyone after the test what load you applied actually.

So keen am I to ensure that the load applied by a test user is even, that I employ a trick taught to me by a seasoned load testing professional (you know who you are Neil!) to measure the pacing achieved in the test. All decent load testing tools allow you to time transactions by inserting a “start” and “end” in a test script. All you do is start a transaction and immediately end it, at the very start of your script processing loop. This creates a transaction whose duration is always zero. At first sight this does not appear very useful. However the time interval between these transactions should align with the pacing interval. After the test, you can extract the transaction data for the test users, pop them in an Excel file and add a column which just uses a simple subtraction to calculate the intervals between the transactions. This should match the defined pacing.

Another trick to watch out for in the same area concerns the ramp-up and ramp-down graphs you will often see generated by load testing tools. I like to make each test user perform an exact number of sessions and using the pacing I can predict the time by which the last one should have ended. If just one of the sessions took longer than expected, you will see some sort of unevenness during the ramp-down. If all the sessions take longer, all test users are affected and you will see the ramp-down delayed. I have seen this in real life and it always points to either unexpected behaviour in the web system being tested or incorrect setup of the pacing.

Let me know what you think of this approach and have any ideas for future articles in the same area.