 |
 |
 |
The application testing and review process that is used to approve BREW applications for distribution to customers is well known for its rigorous requirements.
Operators know that an application that successfully makes its way through the certification process will perform well and will download over-the-air to the device as it should. It will not interfere with a handset's voice services, disrupt the handset's hardware or firmware, or cause problems to the network. And the application will be digitally signed by an approved developer, which ensures that it is an authentic BREW product.
The process, called "True BREW Certification," has been in place since the inception of BREW in 2001, when it was used to evaluate the applications that ran on the first BREW-enabled handset. Since those initial implementations, thousands of applications have been introduced by more than 80 developers, whose products are offered on devices from 45 manufacturers for use on 60 operator networks. Today, True BREW testing must meet those volumes and it must consider, as well, how applications use newer technologies and features now popular on mobile phones, such as Bluetooth, streaming media and GPS-enabled location-based services, and how the software interacts with the network or a content provider's server.
Despite the market growth and some increasingly sophisticated tests, the True BREW testing process itself and the basic testing criteria haven't changed much over the years, says Kathy Braegger, head of developer relations at Qualcomm Internet Services.
"But obviously, the more complex and rich the applications are becoming, the more we have to test for," she said. "Always the goal is making sure that the end user has a good, quality experience with that application. That's huge for BREW."
All applications are tested on actual handsets in San Diego by a Intertek NSTL, which has been the sole vendor providing True BREW testing services since the 2001 BREW launch. Several testing levels are offered, depending on the intended use of an application. The levels include a "full test" of all True BREW test procedures that is used for every application; approaches to extend a full test of a single application to multiple handset platforms to eliminate redundant tests (which can eliminate up to one-third of the tests needed after the first full test); and self-testing, in which qualified developers evaluate their own applications according to True BREW test criteria, prior to submission, to streamline the certification process. Self-tested applications are subject to spot-testing process to ensure they comply with True BREW performance and quality standards.
Operators know that an application that successfully makes its way through the certification process will perform well and will download over-the-air to the device as it should. It will not interfere with a handset's voice services, disrupt the handset's hardware or firmware, or cause problems to the network. And the application will be digitally signed by an approved developer, which ensures that it is an authentic BREW product.
The process, called "True BREW Certification," has been in place since the inception of BREW in 2001, when it was used to evaluate the applications that ran on the first BREW-enabled handset. Since those initial implementations, thousands of applications have been introduced by more than 80 developers, whose products are offered on devices from 45 manufacturers for use on 60 operator networks. Today, True BREW testing must meet those volumes and it must consider, as well, how applications use newer technologies and features now popular on mobile phones, such as Bluetooth, streaming media and GPS-enabled location-based services, and how the software interacts with the network or a content provider's server.
Despite the market growth and some increasingly sophisticated tests, the True BREW testing process itself and the basic testing criteria haven't changed much over the years, says Kathy Braegger, head of developer relations at Qualcomm Internet Services.
"But obviously, the more complex and rich the applications are becoming, the more we have to test for," she said. "Always the goal is making sure that the end user has a good, quality experience with that application. That's huge for BREW."
All applications are tested on actual handsets in San Diego by a Intertek NSTL, which has been the sole vendor providing True BREW testing services since the 2001 BREW launch. Several testing levels are offered, depending on the intended use of an application. The levels include a "full test" of all True BREW test procedures that is used for every application; approaches to extend a full test of a single application to multiple handset platforms to eliminate redundant tests (which can eliminate up to one-third of the tests needed after the first full test); and self-testing, in which qualified developers evaluate their own applications according to True BREW test criteria, prior to submission, to streamline the certification process. Self-tested applications are subject to spot-testing process to ensure they comply with True BREW performance and quality standards.
"Self-certification is much more prevalent today than it has been ever before, because as the test plan gets known and developers get their own processes down for quality assurance testing, they're able to satisfy the requirements," said Brent Melson, vice president of technology for Intertek NSTL (National Software Testing Labs).
Wherever possible, QIS has also built efficiencies into the certification process to help meet increasing certification demands, Braegger said. For example, it has created a way to identification device families, such as the Motorola RAZR, which have similar screen sizes, keypads, or other features that can benefit from a multiple platform testing strategy. It has also eliminated the need for re-testing of specific software components, such as the proprietary BREW module information file (MIF) that every BREW application must have, when a particular application is tested for multiple devices.
Wherever possible, QIS has also built efficiencies into the certification process to help meet increasing certification demands, Braegger said. For example, it has created a way to identification device families, such as the Motorola RAZR, which have similar screen sizes, keypads, or other features that can benefit from a multiple platform testing strategy. It has also eliminated the need for re-testing of specific software components, such as the proprietary BREW module information file (MIF) that every BREW application must have, when a particular application is tested for multiple devices.
Aaron Rosenzweig, an Intertek NSTL project manager who oversees True BREW testing at the company's San Diego labs, said that the vast majority of the tests, around 80-85%, are conducted to evaluate the interactions between the application, the handset and the software that is installed on the device by the manufacturer.
According to Rosenzweig, some of the most severe problems are those that affect the handset power cycle, for example when a key press shuts off the phone off or causes it to freeze up. Other failures might stem from faulty functionality of the application, as when a ring tone, once purchased, won't run on the handset, or when the use of an application prevents the use of voice calls or text messages.
The certification is not intended to make sure a BREW application is likeable or even useful. Nor is it a guarantee that operators will adopt an application. Ideally, a developer would have a distribution agreement and pricing plan negotiated with an operator before submitting an application for True BREW testing, which must be paid for by the developer.
The testing process is also more extensive and lengthier than the testing regimens used by the Symbian Signed, Windows Mobile Mobile2Market, and Java Verified certification programs, according to Intertek NSTL, which provides testing services for all of these. Braegger, of QIS, said that an application that passes through True BREW testing without any failures can be certified in a day if it is prioritized for an operator. Otherwise, a developer should expect it to take about 4-6 days.
Ultimately, an application's potential to attract the interest and commitment from an operator will depend on the product's basic market potential. GlobalLogic, a software engineering company that creates applications and conducts QA testing for BREW developers, also offers usability testing to make sure its customers' applications will be liked and used by consumers.
According to Rosenzweig, some of the most severe problems are those that affect the handset power cycle, for example when a key press shuts off the phone off or causes it to freeze up. Other failures might stem from faulty functionality of the application, as when a ring tone, once purchased, won't run on the handset, or when the use of an application prevents the use of voice calls or text messages.
The certification is not intended to make sure a BREW application is likeable or even useful. Nor is it a guarantee that operators will adopt an application. Ideally, a developer would have a distribution agreement and pricing plan negotiated with an operator before submitting an application for True BREW testing, which must be paid for by the developer.
The testing process is also more extensive and lengthier than the testing regimens used by the Symbian Signed, Windows Mobile Mobile2Market, and Java Verified certification programs, according to Intertek NSTL, which provides testing services for all of these. Braegger, of QIS, said that an application that passes through True BREW testing without any failures can be certified in a day if it is prioritized for an operator. Otherwise, a developer should expect it to take about 4-6 days.
Ultimately, an application's potential to attract the interest and commitment from an operator will depend on the product's basic market potential. GlobalLogic, a software engineering company that creates applications and conducts QA testing for BREW developers, also offers usability testing to make sure its customers' applications will be liked and used by consumers.
"It's one thing for an application to pass certain tests that are driven by True BREW certification [and other] testing requirements," said Chet Kolley, an area vice president at GlobalLogic. "It's another thing for the application to work well and look good for the consumer," he said.
The most important test case to keep in mind, in other words, is how a customer will use the application on their phone.
The most important test case to keep in mind, in other words, is how a customer will use the application on their phone.
TBT - True BREW Testing