Friday, December 13, 2013

Some thoughts about DHCP testing.

Writing test cases and even use cases against DHCP servers are not so easy. That is becasue most testers don't have proper networking knowledge or experience and likewise, most network guys don't know about test plans and use cases.

This is an old report I had created few years ago, but I think it still has some useful words.

DHCP Server
Suggested use cases for test cases

Functional test:
Suggested Tools:
·         In Linux Ubuntu: dhcpdump+ dhcping link
·         In Windows: Wireshark + ipconfig / dhcpcheck.exe from link

Description (windows steps)
Expected Results
Verify the DHCP is enabled and active
1- In Windows (client) open cmd panel.
2- Type: ipconfig /all
3- Look for the adapter connection related to the test subject
DHCP Enabled -> Yes

Assign a new IP address automatically
1- record your client current ip address
2- In windows cmd, type:
ipconfig /release
3- Type command:
Ipconfig /renew
4- Check if new ip is assigned. (Use Wireshark to capture and read packets if no response received)

Make sure no firewall or static ip is active for the client.
To have better results, reduce the ip expiration config in dhcp server.
Request for a new valid ip address.
1- Download dhcpcheck.exe from
2- Use cmd as administrator:
dhcpcheck.exe –host:dhcp host –clientIP:valid ip address
3- Check if the new ip is assigned. (Use Wireshark to capture and read packets if no response received)

Request for invalid ip address

1- Use cmd as administrator:
dhcpcheck.exe –host:dhcp host –clientIP:invalid ip address
2- Check if the expected response from server is received. . (Use Wireshark to capture and read packets if no response received)
3- Repeat the test with invalid IPs, special IPs, random string, null

Request for Duplicated ip address.
1- Use cmd as administrator:
dhcpcheck.exe –host:dhcp host –clientIP:duplicated ip address
2- Check if the expected response from server is received. (Use Wireshark to capture and read packets if no response received)

Change hardware address field to invalid values
1- Use cmd as administrator:
dhcpcheck.exe –host:dhcp host –clientIP:valid ip address –clientMAC: invalid MAC
2- Check if the expected response from server is received. (Use Wireshark to capture and read packets if no response received)

Sending request to an invalid host
1- Use cmd as administrator:
dhcpcheck.exe –host:invalid dhcp host –clientIP: valid ip address
2- Check if the expected response from server is received. . (Use Wireshark to capture and read packets if no response received)
3- Repeat the test with invalid IPs, special IPs, random string, null

Send a request with invalid timeout, min timeout, max time out.
1- Use cmd as administrator:
dhcpcheck.exe –host:valid dhcp host –timeout:1
2- Check if the expected response from server is received. . (Use Wireshark to capture and read packets if no response received)
3- Repeat the test with valid minimum timeout (based on server configuration and requirement)

Manipulate packet of requests.

This test needs more creativity. You’ll need to download Bit-Twist link
1- Capture a packet of a valid DHCP request with Wireshark.
2- Modify the packet carefully using Bit-Twist
3- Send the packet with Bit-Twist
4- Check if the expected response from server is received. (Use Wireshark to capture and read packets if no response received)

This test is checking the security and vulnerability aspects of the DHCP server and might need advance approval.

Performance Test:
Suggested Tools:
·         In Linux Ubuntu: dhcperf+ dhcpdump
·         In Windows: IXIA (if available - Recommended)
Performance Metrics
To determine the performance of a DHCP server, several performance metrics will be used in a test plan. The following terms are defined and suggested to use in the test plan and test report:
·         Connection – A single TCP connection between two end hosts, using connection establishment (3-way handshake).
·         Transaction – A single request for an object from a client to server. A transaction is made within an established Connection.
·         Connections-per-second – The rate at which new TCP connections are initiated per second.
·         Transaction Response Time – The amount of time elapsed between the time a client sends a request and the time an acceptable response to it is received.
·         Complete Transaction Response Time – The amount of time elapsed between the time a client sends the first request and the time the last acceptable response is received successfully from subsequent requests.
·         Client Capacity – Number of clients that a server can support. The maximum client capacity of a server is the threshold beyond which the server can no longer support any additional clients.
·         Success Rate – The ratio of successful transactions calculated over the total attempted transactions.

To do tests using IXIA: Follow this LINK
To do test using dhcperf: Follow this Link

Use Case 1, Max client Capacity:
To support the growing requirements of a DHCP server to handle various load conditions, it is critical to characterize its peak capabilities.
This test will help determine the maximum steady-state DHCP requests/transactions handling capability of the server. The maximum client capacity is reached when no additional clients can be realized. At this operating limit, the server will sustain a steady flow of DHCP requests, and then drop or not process any additional packets, achieving a 100% success rate.

Use Case 2, Max Transaction Rate:
This test will help determine the maximum transaction rate that the DHCP can sustain without any errors or packet losses (100% success rate). The maximum transaction rate will include all transactions for a new DHCP client to request and obtain its network configuration information from the DHCP server.

Use Case 3: Optimal Client Response Time:
This test will help determine the optimal client response time by verifying an acceptable level of performance for the DHCP server. The maximum client capacity and the maximum transaction rates will be used to help determine the upper limits of the DHCP server.
To achieve the maximum client capacity, the rate of arrival of DHCP requests at the server may be reduced to allow the server to service more requests. Therefore, to highest client capacity may require a reduced rate of arrival (transaction-rate) of DHCP requests.

Friday, November 22, 2013

Visual Studio Ultimate vs SoapUI+LoadUI


1-    Common advantages
Here are the important features that BOTH tools are supporting equally:
-          Unlimited users:
Both tools can handle unlimited users. Most other load testing tools intentionally limit their services to a certain number of concurrent virtual users.

-          Correlation (Automatic testing the actual results against expected results)
Correlation ability in load testing tools, helps to have a simple “functional testing” during the load test to make sure all pages and responses are as same as what we have recorded before. In other words, it compares actual results with expected results.  This feature saves time, energy and reduces mistakes.

-          Parameterization
This is a key feature of a dynamic test. Having this option enables us to make the test more flexible based on dynamic results. It is also the vital part of a good data-driven test.

-          Scenario execution
It is very beneficial if we can run different usage scenarios with various types of users. Both tools provide lots of option to do that.

-          Automatic transaction measurements.
Recording all necessary metrics (i.e. response times, data traffic, packet sizes, etc.)

-          Real-Time analysis graphs (Multiple views)
During long tests such as stress testing, it would be very handful if we can see the live results in a workbench. This can help us to interrupt a test if something is wrong, without having to wait the test finishes. It also helps to do better determination for system’s behavior during some “events”.

-          Additional agents (External load injectors)
Load injectors (Agents) can be used for various purposes such as:
·         When we need to put a massive load on a system that one test machine can’t produce that load alone.
·         If we need to see the system’s behavior in an environment much closer to reality, because each agent will be setup in a different computer (even in offshore) and naturally would have different IP address, ping latencies, data flow, etc.
·          If the client(s) locate in another geographical area while our server is somewhere else, we can put agents remotely and do the load and performance tests “from there”.

2-    Differences, Cons and Pros

Visual Studio Ultimate 2012 + MSDN
LoadUI Pro (+ SoapUI Pro)
Good: Tool’s License is perpetual and we won’t need to upgrade it to a new version for a long time (unless change to .Net development). Yearly MSDN renewal may be required for TFS.
$10,900 to $13,300 (13300 is the vendor’s price but there are some other stores or licensors who can offer a bit less)
$800 to $4300 ($800/yr is enough for TFS Server maintenance)
Bad: License will expire after one year and requires renewal with almost the same price. There are Two-Year and Three-Year licenses available but the cost would be multiplied as well.
Standard license:
$8900 to 9900 for one seat (user), per year.
Floating license:
$17900 to 19800 one floating seat, per year.
Licensing limitations & restrictions.
Bad: License is user-based and will be provided to one individual user. Therefore, we are “not allowed” to share the license to others. However, one license can be used to install the VS on many PCs (Work, Home, Portable …).
Worse:  License is not only user-based (like VS), it also is limited to machines. We will be allowed to “move” a license from a computer to another one, only 2 times per year that means we can install the tool on up to 3 machines.
Test function coverage

Good: Visual studio covers all Load and performance testing functions we need for Web Services and Web pages.

Bad:  LoadUI is VERY powerful to test web services, but it is very weak and limited for Web load testing. There is the web load feature in the LoadUI but it is good only for very simple pages with few elements. To do a Web load testing as good as what Visual Studio offers, we’d need to use LoadUIWeb Pro ($4900 to $7500 per year)

Bad: Weak  Ramp downs (it can be  done but needs to write heavy code for load profile plugin)
Needs scripting for randomized or distributed scenarios.
Good: Very comprehensive and flexible tools for Ramps and distributed scenarios.
Bad: We’ll need to use .Net scripting to create good test cases.

Worse: Groovy is the official scripting language, which is almost Java with some occasional differences.
Good: There is outstanding documentation for .Net programming with or even without MSDN available.
Bad: Poor documentation for using groovy scripts in the SoapUI is available.
Server monitoring
Bad: Easy server monitoring is only available for windows-based servers. For other webservers that use different operating systems (e.g. Linux), we need either to write codes to read the system counters dynamically or use another (not-integrated) resource monitoring tool during the test.
Good: Supports Server monitoring for all common operating systems, needs to install Net-SNMP on Apache.

-          Full MSDN library and forum support (one year)
-          Four times live supports with a professional.
-          There is lots of external website to help.
-          Pro user forum and offline support.
-          Live support and Ticket desk.
-          Limited resources are available in other location.
Ease of usage
Bad: Needs .Net development experience and more programming knowledge.
Good: By providing more visual and Drag &  Drop features and  ready-to-use libraries,  less coding is needed.
3-    Other aspects:
a.      LoadUI other benefits:
-          Includes a Free SoapUI Pro. That means if you buy a license of the LoadUI Pro, you will get a free SoapUI pro as well (around $350 value)
-          LoadUI provides better reports with flexible graphs and customized data.

b.      LoadUI other flaws:
-           Scripts created in Pro versions can’t be used in free versions. Therefore, we won’t be able to use our files if the license expires (Although we can temporary use trial versions). In addition this forces us to run our tests only from one system. Visual Studio Ultimate has the same problem. Please read the provided solution in “d. Visual Studio Ultimate Flaws”.
c.       Visual Studio Ultimate benefits:
-          $150/Mo Credits for Windows Azure cloud and VM processing power (that is enough for our normal test purposes). This can help to avoid purchasing unnecessary powerful test machines that be used rarely.
-          Ability to perform Unit testing, Functional testing and Code coverage testing on developed web pages with HTML, JavaScript and CSS
-          Visual Studio can be used for many other development purposes in the future. The only flaw in this benefit is not being Java compatible.
o   It comes with one year full MSDN subscription that provides Team foundation server (TFS) + one client access license. Please read section 4 for more information about TFS.
Other following provided features are not necessary. To maintain only our Team Foundation Server (TFS) in next years, we can renew MSDN subscription without following extras ($800 year):
o   Exchange server
o   SQL Server
o   Office 360
o   Windows, Windows Server
o   Team foundation service with 5+ CAL
d.      Visual Studio Ultimate flaws:
-          Same as LoadUI pro, we can’t run Load and Performance tests in other machines that have no Visual Studio Ultimate installed. Other tests can be used in Visual Studio Test Professional.
There is a solution which is having a “portable test machine”, like a good laptop (intel i7 + 12GB RAM + SSD HDD). You can install Visual Studio (or LoadUI) on it and provide it to anyone who wants to perform the test in the facility.  

4-    Important note about TFS (Team Foundation Server/Service):
TFS is a powerful and the main part of Microsoft application life cycle management (ALM). It integrates with SharePoint and Outlook and gives ability to:
-          Version control, for managing source code and other deliverables that require versioning.
-          Work item tracking, for keeping track of such things as defects, requirements, tasks, and scenarios.
-          Agile development: Compatible with agile methodologies and common agile frameworks like SCRUM, KANBAN, XP and Crystal.
-          Project management functions, which allow the shaping of a team project based on a user-specifiable software process, and which enable planning and tracking using Microsoft Excel and Microsoft Project.
-          Team build, for enabling a common process for building executable products.
-          Data collection and reporting, which aid in the assessment of a team project's state, based on information gleaned from Team Foundation Server tools.
-          The Team Project Portal, which provides a central point of communication for a team project packaged as a Microsoft Windows SharePoint Services site.

Although TFS is included with the Visual Studio, but the tricky part is that provides with 1 CAL (Client Access License) and we will need additional CALs for more CALs. (Cost of each CAL is $500, less for volume purchases.).
The bright side of the TFS and CALs:
·          Each CAL can be used by more than one user (2-3 in a group with some arrangements).

·         Users (e.g. Stake holders) can see reports and output analysis data without having a CAL.