Don't guess, test it!
Why performance test:
• to know the performance (e.g. can I handle 100 users, how fast I can refresh the GUI)
• to know how well it scales (e.g. how if I got 10x more customers)
• for tuning the parameters, design / architecture decision, queries
Tips:
• instead of postponing the performance test until the development finish you can do the performance test early in the architecture design phase / spike solutions, to choose which framework to use e.g. JSP or GWT/Ajax, hibernate or ibatis. In the early phase of architecture definition, the cost of a wrong decision can be expensive/irreversible. Consider that if you wait until you finish with the product and then you discover that you have made a wrong choice in the early design then your developer need to spend time again to redo the works (e.g. redo the JSP to GWT/ajax).
• With agile/test driven development spirit, even you may incorporate the performance test to your continuous integration / regression test.
• environment specific: the test result in your development PC will be different with the result in the production server, so use test environment as close as possible mimicking the production
• use different hardware to measure the statistics than the hardware which is being tested
• remove data which damage the quality of statistics (e.g. the initial burn-in period)
• use random think time
Steps:
1. preliminary test: explore (e.g. range #users), set test environtment params (e.g. jvm heapsize)
2. baseline test: typical usage for base comparison
3. stress test to the limit: increase the load until breaking, to know the limit, the bottleneck, the behavior in degradation mode
4. endurance test: load test (high load but not yet overwhelm) for long hours, to detect memory leak, resources/connections which are not properly closed, to know the fail behaviour (e.g. error handling)
5. stress test to fail: deprive the resources (e.g. overwhelm the cpu with other heavy tasks, turn off the networks), to know the fail behaviour (e.g. error handling)
6. vary the parameters (e.g. java heapsize, #clusters) / architecture (e.g. load balancer) / sql query / business logic, then repeat the baseline and stress test to limit fase, compare the results if it improves.
monitoring (to know where's the bottleneck):
• application: eclipse tptp profiler
• jvm: java profiler
• database: oracle trace, query optimizer
• os: top, vmstat, perfmon
• networks: packet sniffers (e.g. tcpdump), netstat, netprotocol analyser (e.g. ethereal)
How
1. define the metrics
2. simulate usage. Run a testrun, which run testscripts (which consist of requests/queries to simulate usage profile)
3. define sampling methods (e.g. fixed #cycles or fixed time window)
Performance metric
• use a clear/specific performance criteria, e.g. not "the web should be as fast as possible" but "the user get a confirmation after click the submit button with max response time 10 sec (given 100 simultaneous users activities)"
• for GUI, the metric can be max response time given #users
• for back-end (e.g. web services or database), the metric can be #transaction per sec (TPS)
• you can define also over limit behaviour e.g. 10% degradation response time with 1000 users.
The statistics:
a. Average response time (ART): aritmetic mean for all user
caveat: if the ART is within limit it doesn't mean that all user instances are within limit.
suggestion: draw the ART vs time (test cycles)
b. Average ART (AART): normalize ART with number of requests, so it will be comparable between different testscripts.
c. define the throughput transaction per sec (TPS): e.g. #(group)queries/sec in database or #messages/sec in JMS
d.Quality: standard deviation / average, prefer bellow 0.25.
e.g. compute standard deviation of ART(as a function of cycles) and divide by ART average over #cycles.
Tools, examples:
test: soapui, httpperf,junitperf,grinder,jmeter
Please write your comments about your experiences with performance testing.
Source: Steve's blog http://soa-java.blogspot.com
References:
My blog which mentioned about the use of performance test in the countinous integration test:
http://soa-java.blogspot.com/2011/03/soa-continuous-integration-test.html
J2EE Performance Testing by Zadrozny et.al.
Grig Gheorghiu's blog, a good place for information about testing, python and linux tips.
http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html
Performance Engineering
http://performance.punebids.com/
No comments:
Post a Comment