Performance monitoring – Windows server monitoring

I have been doing some research on what system monitoring is useful during performance testing.

I have concentrated on a minimum useful set of information needed to verify that no bottlenecks occur when using a performance testing tool. Even this minimal set should be useful though to enable a rough prediction of the maximum load that could be supported by the system under test.

There’s a wealth of information available on this subject. Two sites I found particularly useful were these:
http://www.sql-server-performance.com/tips_performance.asp http://www.computerperformance.co.uk/HealthCheck/

One subject on which there seemed to be a range of opinions was whether it is best to set up monitoring remotely or locally. If you set up remotely there will be some network traffic, but if you set up locally there is the overhead of running perfmon itself. Do you think it is significant either way – if so, in what situations? Perhaps this is only an issue if you are already close to resource utilization limits.

One thing which is probably always true is that it’s best to be selective and only collect the performance counters you need. This document is my attempt to propose a subset which will nearly always be useful. Take a look and let me know what you think. Is there anything obviously missing? Is it too comprehensive?

Performance testing - agile and open source

These days many folk are asking about agile techniques and how they apply to performance testing. That's not surprising - they are keen to get the benefits of testing early in their development process and save costs as a result.

Often they enquire about open source performance testing tools as well. They're keen to see how they compare with some of the commercially available load testing tools and whether they can make further savings by using open source.

I put together an article on both these subjects with some links which may help if this is a subject that interests you. If it is, let me know and please feel free to suggest more subjects for similar articles in the future.

Performance monitoring - including SNMP and rstatd data in load test scripts

Occasionally questions come up about how to monitor UNIX servers during load testing. SNMP, rstatd, vmstat - it's all so confusing! Fortunately there are resources to help.

I came across a nice little article recently on this - and there's some interesting references to SNMP techniques:
http://www.myloadtest.com/collecting-performance-stats-with-snmp

To add to the knowledge base, here is a link to an article I wrote a year or so ago on the same subject which may be of interest. Following a proof-of-concept, the conclusion was that a script could be written to extract SNMP counters and record them via user-defined counters for later analysis. The evaluation was done using Compuware's QALoad but should be portable to Mercury LoadRunner.

The link to the article:
http://www.users.globalnet.co.uk/~cater/dave/acutest/TN1.doc

Related references that may be useful:
Open Source project Net-SNMP http://www.net-snmp.org
Linux version of rstatd(1M) http://rstatd.sourceforge.net

Let me know what you think of the article and ideas for the future.

Web load testing (article 2) - be generous with pacing

This is the second article explaining a practical approach to
avoiding misunderstandings concerning concurrent user load.

If you read the first article, you will be aware of my obsession with achieving even pacing for each test user. Here I will explain how to design tests to achieve even pacing.

First, a digression on thinking time and system time.

Most load testing tools allow the user thinking time to be simulated. In fact, most allow thinking time to be recorded during a user session so it can be replayed during the load test. I always include thinking time in the web session. This ensures that the total duration of each user session roughly corresponds with reality. This is important where the web system maintains sessions for users – you want the number of concurrent sessions to be realistic.

I also like to use transaction timers to measure ALL activity that is not user thinking time. In other words, I use timers to measure all system activity. Not surprisingly, I call this “system time”.

I estimate the minimum system time by running the web test script for a small number of users and adding up the times of all transactions. I multiply the system time by three to allow for slow-down under load, and add the thinking time. Then I add a comfortable margin for error, and end up with a pacing time that should be achieved– even under heavy load.

So there is my tip – allow plenty of time for each user session. Be generous with your pacing. I do everything I can to ensure there is a pause between user sessions so pacing is maintained.

The astute amongst my readers will have worked out that being generous with pacing implies a larger number of test users. Let me explain. To achieve a specified load (say 720 user sessions started per hour) with a pacing of 5 minutes requires 60 test users, whereas with a pacing of 10 minutes requires 120 test users. You should take this into account during test planning.

Web load testing (article 1) - how to keep the load steady

I recently came across a series of articles written in 2001 by Alberto Savoia which impressed me very much. If you search for these titles, you can still find them:
“Web load test planning”
“Trade secrets from a web testing expert”
“Web page response time 101”

The second of Savoia's articles covers three main topics:
Misunderstanding concurrent users
Miscalculating user abandonment
Over-averaging page response times

When I read these I was interested (and somewhat relieved!) to find that much of what Savoia recommends aligns with my own approach to web performance testing.

In this article, I’ll outline a practical approach to how I deal with avoiding misunderstandings concerning concurrent user load.

Savoia rightly pointed out that when defining load for a web performance test, the starting point should be “number of user sessions started per hour”. (It matters less how long each of these individually takes from start to end, though as I will point out in a subsequent article, it cannot be completely ignored.)

Most of the well known load testing tools allow for “pacing” of a test user. You can arrange for a test user to repeat the same session with the start times for each session spaced apart at the “pacing” interval.

It is tempting to ignore this and perhaps try to disable pacing so as to start each session immediately the previous one has completed. Believe me, this is almost always not a good thing to do. The reason is essentially quite simple. As there are variations in the time each user session will actually take (particularly under load) the rate at which user sessions start will be uneven, and you will be unable to explain to anyone after the test what load you applied actually.

So keen am I to ensure that the load applied by a test user is even, that I employ a trick taught to me by a seasoned load testing professional (you know who you are Neil!) to measure the pacing achieved in the test. All decent load testing tools allow you to time transactions by inserting a “start” and “end” in a test script. All you do is start a transaction and immediately end it, at the very start of your script processing loop. This creates a transaction whose duration is always zero. At first sight this does not appear very useful. However the time interval between these transactions should align with the pacing interval. After the test, you can extract the transaction data for the test users, pop them in an Excel file and add a column which just uses a simple subtraction to calculate the intervals between the transactions. This should match the defined pacing.

Another trick to watch out for in the same area concerns the ramp-up and ramp-down graphs you will often see generated by load testing tools. I like to make each test user perform an exact number of sessions and using the pacing I can predict the time by which the last one should have ended. If just one of the sessions took longer than expected, you will see some sort of unevenness during the ramp-down. If all the sessions take longer, all test users are affected and you will see the ramp-down delayed. I have seen this in real life and it always points to either unexpected behaviour in the web system being tested or incorrect setup of the pacing.

Let me know what you think of this approach and have any ideas for future articles in the same area.