performance engineering is a complex discipline encompassing applications, infrastructure, security, and more. To truly optimize your performance, your organization needs to address a broad range of issues. To make informed decisions as you move from performance testing to performance engineering, your teams and your organization need to ask themselves the 102 questions below.
This list is intended to be inclusive but not exclusive, and apply across all dev/test/ops approaches. No matter what your role, persona, and interest, this list should help you understand how your solution is engineered for performance, stability, and scalability. My goal is to help you, your team, and your stakeholders gain a common nomenclature for performance engineering so you can define your path and direction together.
Objectives
Objectives are measurable tasks that, once completed, will achieve goals. Common test objectives can be better met and understood by answering these questions.
Server sizing
1. How many application servers are needed to support the customer base?
2. What is the optimal ratio of users to web servers?
3. What is the optimal web server to application server ratio?
4. What is the maximum number of users per server?
5. What is the maximum number of transactions per server?
Server tuning and optimization
6. Which specific hardware configurations provide the best performance?
7. How can vendor default configurations be tuned to suit this specific infrastructure and application?
8. What system resources need tuning to give optimal performance?
Capacity planning
9. What is the current production server capability?
10. Is there room for growth?
11. What hardware or software can be added to achieve the next level of performance or capacity?
12. Is there excess capacity? Can a server be removed without compromising performance?
Third-party validation
13. What is the current ISP and network capacity?
14. Can the ISP deliver on the service level agreement that was signed?
Security exposure
15. Can system vulnerabilities be identified and minimized?
16. What is the failover for firewalls?
17. Are there new vulnerabilities when excess user load is added to the application?
18. How susceptible is the system to DoS attacks?
19. If a DoS attack occurs, how will the system respond?
20. If the system goes down due to a hacker attack, how effective are the recovery procedures?
Infrastructure
Here are some of the questions you should ask regarding the infrastructure.
Browser/user profile issues
This subsystem is known as the user community profile and consists of business process definitions.
21. What do the users do? (These are business process definitions.)
22. How fast do the users do it? What are the transaction rates of each business process?
23. When do they do it? What time of day are most users using it?
24. What major geographic locations are they doing it from?
25. Is the application browser- or interface-dependent?
26. Is modem, WAN, or LAN emulation necessary?
27. Are there asynchronous communications between the browser/client and the back-end servers?
28. Are there any non-HTTP(s) communications between the browser/client and back-end servers?
Internet issues
29. What are the peering issues associated with the client's hosting/bandwidth provider?
30. What is the hosting strategy?
Site web pipe issues
31. How much bandwidth does the site have?
32. Who is the client's bandwidth provider? (Peering issues)
33. Are there multiple web pipes?
Border router issues
34. What kind of load balancing are the multiple pipes configured for?
35. Does it use the same inbound pipe as outbound pipe?
36. Is there equal distribution for outbound regardless of inbound pipe?
37. Is there the same outbound pipe regardless of the inbound pipe?
38. Are there multiple border routers?
39. What is the failover configuration for multiple border routers?
Load balance issues
40. What type of load balancing scheme is used? (Round robin, sticky IP, least connections, subnet based?)
41. What is the timeout of LB table?
42. Does it do any connection pooling?
43. Is it doing any content filtering?
44. Is it checking for HTTP response status?
45. Are there application dependencies associated with the LB timeout settings?
46. What failover strategies are employed?
47. What is the connection persistence timeout?
48. Are there application dependencies associated with the LB timeout settings?
49. What are the timeouts for critical functions?
Peripheral systems issues
50. Is the LAN/WAN system dedicated or shared with other applications?
51. Are there any shared production resources?
52. Are there any web pipes, ERP systems, mail servers, file systems, DNS servers, etc.?
53. Does it share databases with other applications?
54. Does it share hardware with other applications?
External systems issues
55. Are there any outside vendors that provide content distribution systems (CDS)
for the architecture?
Distributed hosting issues
56. Are these multiple mirrored sites?
57. Is any site configured for failover operation?
58. How is the traffic load balanced across the sites?
59. Are there architecture components on shared WAN connections?
60. What is the failover and recovery behavior?
Firewall issues
61. What is the throughput capacity?
62. What is the connection capacity and rate?
63. What is the DMZ operation?
64. What are the throughput policies from a single IP?
65. What are the connection policies from a single IP?
IDS: Intrusion detection systems
66. Is there statistical content sampling?
67. Is there an inverse relationship between throughput and security?
68. How is content filtering achieved?
Application
Here are some of the questions you should be prepared to ask regarding the application.
Web server issues
69. How many connections can the server handle?
70. How many open file descriptors or handles is the server configured to handle?
71. How many processes or threads is the server configured to handle?
72. Does it release and renew threads and connections correctly?
73. How large is the server's listen queue?
74. What is the server's "page push" capacity?
75. What type of caching is done?
76. Is there any page construction done here?
77. Is there dynamic browsing?
78. What type of server-side scripting is done? (ASP, JSP, Perl, JavaScript, PHP, etc.)
79. Are there any SSL acceleration devices in front of the web server?
80. Are there any content caching devices in front of the web server?
81. Can server extensions and their functions be validated? (ASP, JSP, PHP, Perl, CGI, servlets, ISAPI filter/app, etc.)
82. Monitoring (Pools: threads, processes, connections, etc. Queues: ASP, sessions, etc. General: CPU, memory, I/O, context switch rate, paging, etc.)
Application server issues
83. Is there any page construction done here?
84. How is session management done and what is the capacity?
85. Are there any clustered configurations?
86. Is there any load balancing done?
87. If there is software load balancing, which one is the load balancer?
88. What is the page construction capacity?
89. Do components have a specific interface to peripheral and external systems?
Database server issues
90. Have both small and large data sets been tested?
91. What is the connection pooling configuration?
92. What are its upper limits?
Security
Here are some of the questions to ask when addressing security issues.
Firewalls and multiple DMZs
93. Does the firewall do content filtering?
94. Is it sensitive to inbound and/or outbound traffic?
95. What is its upper connection limit?
96. Are there policies associated with maximum connection or throughput per IP address?
97. Are there multiple firewalls in the architecture (multiple DMZs)?
98. If it has multiple DMZs, is it sensitive to data content?
IDS: Intrusion detection system
99. Is there any content filtering?
100. Is the system sensitive to inbound and/or outbound traffic?
101. What are the alert thresholds?
102. What are the acceptable security thresholds?
Naturally, these questions are only a starting point you also need to come up with answers and they don't cover every possible issue in performance engineering. How will you use these questions? What would you add to the list?
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
