 |
 |
 |
The number and type of wireless applications envisioned for enterprise and mass market users grows daily. With each of these types of applications, there are a number of factors that must be considered when testing for conformance to functional, user and performance requirements. Any good tester knows to pay attention to the basics, which can include:
• verifying the baseline functionality and features
• checking the design and proof-of-concept solutions against user requirements early in the development cycle
• testing under tightly controlled conditions to validate executable code against design during later stages of the development lifecycle
• compatibility testing all known, planned variations in the software and hardware configurations where the application will run
• exposing the entire system or application to unexpected events, faults in dependent databases, networks or applications, or unpredictable user behavior
• subjecting the software to volume, load and stress conditions to gauge performance at the boundaries of its designed capacity and measure actual limitations of that performance
• determining if the application or system not only meets formal design requirements, but also whether it will be usable and meet the (perhaps undocumented) needs of its users.
Not all of these factors are within the scope of all testing engagements, but a good test plan will at least address them so as to make all assumptions about test scope visible.
When it comes to wireless enabled applications, however, there are factors that require the testing strategy to be modified significantly in order to ensure that hardware and software elements of the solution under test are suitable for its target environment and users.
A strategic approach to testing mobile solutions takes into account a number of characteristics unique to the mobile paradigm:
• the increased complexity of emerging handheld devices
• the greater sensitivity to security and load related problems in wireless infrastructure
• increased complexities of scale
• verifying the baseline functionality and features
• checking the design and proof-of-concept solutions against user requirements early in the development cycle
• testing under tightly controlled conditions to validate executable code against design during later stages of the development lifecycle
• compatibility testing all known, planned variations in the software and hardware configurations where the application will run
• exposing the entire system or application to unexpected events, faults in dependent databases, networks or applications, or unpredictable user behavior
• subjecting the software to volume, load and stress conditions to gauge performance at the boundaries of its designed capacity and measure actual limitations of that performance
• determining if the application or system not only meets formal design requirements, but also whether it will be usable and meet the (perhaps undocumented) needs of its users.
Not all of these factors are within the scope of all testing engagements, but a good test plan will at least address them so as to make all assumptions about test scope visible.
When it comes to wireless enabled applications, however, there are factors that require the testing strategy to be modified significantly in order to ensure that hardware and software elements of the solution under test are suitable for its target environment and users.
A strategic approach to testing mobile solutions takes into account a number of characteristics unique to the mobile paradigm:
• the increased complexity of emerging handheld devices
• the greater sensitivity to security and load related problems in wireless infrastructure
• increased complexities of scale
Increased Complexity in Handheld Devices
Perhaps no other area within the realm of mobile computing has changed quite as fast as the presentation device. The first handheld computing device to achieve any real market presence was the Apple Newton, introduced in 1993 and much beloved - but only by 200,000 people. Since then, this space has evolved to the point where there are literally thousands of different combinations of hardware devices, software capabilities and wireless networking features. In fact, the handheld computing platform has now become a far more complex element of the overall mobile services technology chain on its own, than the entire networked-applications architecture of a generation ago. Specific characteristics of the handheld device that make wireless computing a challenge include:
Perhaps no other area within the realm of mobile computing has changed quite as fast as the presentation device. The first handheld computing device to achieve any real market presence was the Apple Newton, introduced in 1993 and much beloved - but only by 200,000 people. Since then, this space has evolved to the point where there are literally thousands of different combinations of hardware devices, software capabilities and wireless networking features. In fact, the handheld computing platform has now become a far more complex element of the overall mobile services technology chain on its own, than the entire networked-applications architecture of a generation ago. Specific characteristics of the handheld device that make wireless computing a challenge include:
• Device and firmware diversity: today's devices have not just firmware, but an operating system, browser and even an applications runtime layer (in the case of Java or BREW enabled phones). Recently, some very expensive and customer impacting quality problems for wireless users have been traced to firmware or operating system bugs in cell phones. Testing for compatibility between these layers, as well as with the wireless network interface, is paramount to deploying a trouble-free mobile service.
• Support for multiple interface and rendering standards: device manufacturers are moving away from network specific handsets toward dual- or tri-mode phones capable of supporting multiple networks. And, despite the recently announced M-Services meta-standard, applications rendered in HDML, WML, WAP 1.2, WAP2.0 and cHTML and xHTML will continue to require focused testing at the presentation device.
• Support for the embedded base: imagine desktop applications which were forced to be compatible with Windows 3.1, Win95, Win98, Win2K, Windows Me and Windows XP - plus the Macintosh. A similar challenge confronts developers and testers of applications targeted for handsets and PDAs with multiple, older versions of software.
• New paradigms in the client applications platform: the client software on a network-connected PDA or cell phone was once relatively simple; most processing was performed in the network on WAP or custom mobile server platforms then rendered on the client over low speed, text- based interfaces to the device. With the advent of highly functional PDAs and handsets that have rich client execution environments and persistent data store, testing becomes much more complex.
• New security concerns: with the ability to download applications and execute code on the device comes the corresponding risk associated with user authentication, enterprise data and transactional security as well as viruses and other malicious code. The newfound power of the mobile desktop increases the need for testing and verification of security mechanisms, both in the devices themselves and in the network infrastructure.
Increased Complexity in Mobile Networks
Paging was once defined as a one-way service for short text messages, and cellular phones provided analog voice services only. Today, paging can be two-way, utilize many different network types (digital cellular, PCS, packet radio, etc) and offer the types of messaging services once reserved for enterprise private networks. Likewise, cellular isn't cellular anymore, with changes to the signaling, data transport and switching techniques that bring added complexity.
Over the next five years, every major region of the world will undergo a network evolution that moves it through a series of technology upgrades on the way from second generation (2G) networks to third generation (3G). Along the way, we will see networks which now offer low speed (9.6 Kbps), circuit-switched services using dedicated cellular channels and wireless modems transformed into high speed (up to 2Mbps), packet-switched systems with direct digital interfaces. We will witness modifications of radio base station equipment, implementation of overlay networks and entirely new architectures and communications protocols (including IPv6, mandated by 3G standards), and deployment of signaling, protocol and applications gateways to move data between operator networks and public/private internet-based infrastructures.
Changes in handheld devices will be required to accommodate this unsynchronized global shift, and verification of end to end services will be a challenge even for those with the proper access to in-country test facilities. Mobile applications will also need to survive cell or network handover problems which are bound to arise and the inevitable mid-session drop back to slower, second generation networks lacking the signaling features of 3G.
Lastly, the development of piconet technologies such as Bluetooth and Wi-Fi (802.11b) will soon lead to deployment of networks and access points for these unlicensed radio bands and inclusion of their protocol stacks into devices. For wireless applications, verification of interoperability alone will be a challenge. It is easy to see that both functional and performance testing of mobile applications prior to deployment will be a necessity for firms whose customers have come to expect a seamless user experience, even while mobile.
• Support for multiple interface and rendering standards: device manufacturers are moving away from network specific handsets toward dual- or tri-mode phones capable of supporting multiple networks. And, despite the recently announced M-Services meta-standard, applications rendered in HDML, WML, WAP 1.2, WAP2.0 and cHTML and xHTML will continue to require focused testing at the presentation device.
• Support for the embedded base: imagine desktop applications which were forced to be compatible with Windows 3.1, Win95, Win98, Win2K, Windows Me and Windows XP - plus the Macintosh. A similar challenge confronts developers and testers of applications targeted for handsets and PDAs with multiple, older versions of software.
• New paradigms in the client applications platform: the client software on a network-connected PDA or cell phone was once relatively simple; most processing was performed in the network on WAP or custom mobile server platforms then rendered on the client over low speed, text- based interfaces to the device. With the advent of highly functional PDAs and handsets that have rich client execution environments and persistent data store, testing becomes much more complex.
• New security concerns: with the ability to download applications and execute code on the device comes the corresponding risk associated with user authentication, enterprise data and transactional security as well as viruses and other malicious code. The newfound power of the mobile desktop increases the need for testing and verification of security mechanisms, both in the devices themselves and in the network infrastructure.
Increased Complexity in Mobile Networks
Paging was once defined as a one-way service for short text messages, and cellular phones provided analog voice services only. Today, paging can be two-way, utilize many different network types (digital cellular, PCS, packet radio, etc) and offer the types of messaging services once reserved for enterprise private networks. Likewise, cellular isn't cellular anymore, with changes to the signaling, data transport and switching techniques that bring added complexity.
Over the next five years, every major region of the world will undergo a network evolution that moves it through a series of technology upgrades on the way from second generation (2G) networks to third generation (3G). Along the way, we will see networks which now offer low speed (9.6 Kbps), circuit-switched services using dedicated cellular channels and wireless modems transformed into high speed (up to 2Mbps), packet-switched systems with direct digital interfaces. We will witness modifications of radio base station equipment, implementation of overlay networks and entirely new architectures and communications protocols (including IPv6, mandated by 3G standards), and deployment of signaling, protocol and applications gateways to move data between operator networks and public/private internet-based infrastructures.
Changes in handheld devices will be required to accommodate this unsynchronized global shift, and verification of end to end services will be a challenge even for those with the proper access to in-country test facilities. Mobile applications will also need to survive cell or network handover problems which are bound to arise and the inevitable mid-session drop back to slower, second generation networks lacking the signaling features of 3G.
Lastly, the development of piconet technologies such as Bluetooth and Wi-Fi (802.11b) will soon lead to deployment of networks and access points for these unlicensed radio bands and inclusion of their protocol stacks into devices. For wireless applications, verification of interoperability alone will be a challenge. It is easy to see that both functional and performance testing of mobile applications prior to deployment will be a necessity for firms whose customers have come to expect a seamless user experience, even while mobile.
Security, Performance and Scale
Although enterprise networks and applications have invested both time and money in recent years to beef up their infrastructure so they can handle security and performance challenges, many of the same issues have yet to be overcome in the mobile services world. Until now, the lower processing and datastore functionality of handhelds along with the network operators' unwillingness to open up more control of the wireless infrastructure have kept security concerns in check, for better or worse.
However, concern over the robustness of WAP security mechanisms (Wireless Transport Layer Security, or WTLS), for example, has impacted enterprise user adoption. Small devices without the processing power to handle strong security methods such as RSA 1024 and SSL and the risk of loss and theft mean the challenge of deploying and testing for security in mobile devices is inherently different from that of other client devices. The move toward more robust operating systems, like Java, will bring both greater support for enhanced security such as SSL and IPSec
Although enterprise networks and applications have invested both time and money in recent years to beef up their infrastructure so they can handle security and performance challenges, many of the same issues have yet to be overcome in the mobile services world. Until now, the lower processing and datastore functionality of handhelds along with the network operators' unwillingness to open up more control of the wireless infrastructure have kept security concerns in check, for better or worse.
However, concern over the robustness of WAP security mechanisms (Wireless Transport Layer Security, or WTLS), for example, has impacted enterprise user adoption. Small devices without the processing power to handle strong security methods such as RSA 1024 and SSL and the risk of loss and theft mean the challenge of deploying and testing for security in mobile devices is inherently different from that of other client devices. The move toward more robust operating systems, like Java, will bring both greater support for enhanced security such as SSL and IPSec
and increased security risk of device-based processing and data store. Security concerns will only increase with the availability of enterprise data extended to the device.
Similar concerns relate to the expected increase in scale of mobile applications deployment, which will be much larger in the future than anything seen to date in the cellular data arena. Internet-ready handhelds are becoming pervasive worldwide as content providers or enterprises can be reached from many parts of the globe. Therefore, it will be ever more critical for enterprises and mobile service providers alike to thoroughly test to ensure that performance levels both in the mobile network and the applications infrastructure remain acceptable.
Similar concerns relate to the expected increase in scale of mobile applications deployment, which will be much larger in the future than anything seen to date in the cellular data arena. Internet-ready handhelds are becoming pervasive worldwide as content providers or enterprises can be reached from many parts of the globe. Therefore, it will be ever more critical for enterprises and mobile service providers alike to thoroughly test to ensure that performance levels both in the mobile network and the applications infrastructure remain acceptable.
Why Mobile Testing is different?