|
A critical part of ensuring the successful launch and delivery of new multimedia services is comprehensive testing. Not only are emerging multimedia applications fairly complex compared to more traditional cellular services, they tend to be data-intensive, performance sensitive, and require more involved user interactions. As a consequence, developers need to test not only the mechanisms used to transport data, but also the perceived quality of the user experience.
Subscribers expect the flawless delivery of cellular services, which is difficult to guarantee when services can be accessed and used in a myriad of ways. For example, a photo taken on one handset may need to be transcoded in order to be viewed on a handset with a different screen resolution. The cost of failure (i.e., an image is clipped or of poor quality) is high: If subscribers encounter difficulties in using the exciting new features for which they specifically purchased a premium-priced handset, their willingness and enthusiasm to continue to pay for enhanced services will quickly fade, potentially leading to their adopting a new carrier with more consistent and reliable service delivery.
Implementing a comprehensive testing methodology, however, is not a trivial process. Testing of even simple functions requires a great many distinct test cases when one considers the array of different ways in which services can be initiated and interact with each other. For example, a user could desire to send an image that has either just been taken, is stored in memory, or is an attachment of another message. The message could be a new message, a reply, or a broadcast message. The message could be initiated from a secondary menu, quick button, or as a reply to an existing message. The handset could also be in any of number of various states, including low battery or roaming mode.
From this example, it can be seen that even the simple action of sending an image can be executed across a large number of variations. Note that the act of sending an image isn't what needs to be tested, per se. Certainly, the general functionality will have already been proven during application development. Testing each variation is important because in each case, the overall system is utilized and stressed in a different manner. A stack overflow, pointer error, or failure to meet real-time deadlines may only manifest under certain operating conditions such as when a specific subset of functions are operating simultaneously or memory resources are stretched.
While testing every possible case would be cost-prohibitive, developers still need to test a representative sample of cases as well as all identifiable extremes. However, this still result in a test bench consisting of thousands of individual test cases. In order to provide sufficient testing to give carriers the confidence that handsets will act consistently and reliably as expected, developers need to abstract test case definition and creation, automate test case execution, and implement a framework which enables the simplified management of test assets and results.
Simplification through abstraction
While the number of test cases required for comprehensive testing can be seem daunting, each test case shares many elements in common with other test cases. For example, when testing text messaging, a test bench will confirm receipt of short and long messages. What is different between these two classes of tests is the message used in the test case. By abstracting the data used in the test case, both of these tests can be described using the same logic, with a single data parameter for the particular message to send. The actual test case need only be created once since it can then be adapted into its many variants.
However, this is but one dimension of possible abstraction (see Figure 1). If the type of message is abstracted as well, the same logic can be used to substitute images for text messages. Note that this abstraction also carries over to how the results of the test case are analyzed. For example, a received text message is compared against a reference copy to see if it arrived intact. The number of errors can be specifically counted and compared against an acceptable threshold. A received image, however, can withstand a limited amount of degradation and still be considered high quality. Therefore, it is not compared directly against a reference image but rather analyzed using software that objectively determines the common-perceived quality of the received image based on the reference image.
Figure 1. An Adaptive Test Case methodology reduces complexity across the entire testing process by adapting test assets at three different levels.
This type of abstraction can be referred to as a Test Verb. Test Verbs define core functions. Not only do they define how a particular function is initiated and used, they also captures differences between platforms, such as how many screens it takes to select a function. For example, the Test Verb "Send Text Message" can be represented by the menu choices required to select the function from the main menu and complete the command. By substituting the "Send Image" Test Verb for the "Send Text Message" Test Verb, the same test logic can be used to define a whole new class of tests leveraging many or all of the tests already created for testing text messaging.
|
