Real tests use real devices

It's not that testing with real devices is the gold standard, though that's one way to look at it. The fact is, there is simply no way to emulate the behavior of a mobile app in the field unless the device is literally in the field. System overhead, memory usage, CPU speed — a host of variables impact app functionality in ways that just can't be reproduced in a lab. To get real test results, you have to deploy real devices.

But creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.

This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.

Source: http://keynote.com/benchmark/mobile_wireless/article_mobile_app_performance.shtml

Performance Testing for Retail

In the broadest sense, testing falls into two categories: ongoing evaluation and tweaking to optimize daily performance, and peak load testing to determine overall site capacity and potential breakpoints.

In either case, the only way to quantify user experience is to measure what users are actually experiencing. Unless you’re the local bike shop serving only your immediate area, the testing needs to be done across a wide geography and multiple backbones. And it needs to use an actual Web browser, and go through the same type of page view sequences and transactions as a typical user would. There’s simply no other way to get a true perspective on what users are really experiencing.

GETTING A PERSPECTIVE ON EVERYDAY PERFORMANCE

Ongoing performance testing, done on a routine basis or following upgrades, patches, or redesigns, can be as simple or complex as the site itself. Assuming the back of the house is in order — that servers are performing up to speed and all of the infrastructure is optimized — the focus of the testing is on the front end. Individual pages are tested and analyzed, and those performing below par are deconstructed to identify the structural problems or elements that are causing the slowdowns.

A single element could be dragging down the page load, or it could be less-than-ideal placement of JavaScript on the page. The point is that testing and identifying the problem pages or transactions is only half the problem. The other half is being able to pinpoint the causes and correct them. A robust performance evaluation regimen accomplishes this.

RETAILERS SHOPPING FOR STRATEGIES

A few retailers are feeling upbeat about the coming holiday season. Best Buy is planning to do more holiday hiring than last year, partly because of strengthening demand for flat-screen TVs, smartphones, and netbook computers, and partly because of market share they captured following last year’s demise of Circuit City. Target plans to have 26 new stores open in time for the holidays, including five “Super Target” stores with full grocery departments; this, and more aggressive pricing, are moving Target closer to a head-to-head brawl with Wal-Mart.

Most retailers, however, are simply looking for ways to maximize their share of holiday revenue. Deep and earlier discounting is expected to be widespread. And in a throwback to an earlier era, and at least partly in response to a growing paucity of consumer credit, the layaway plan is making a comeback. K-Mart and its parent company, Sears, are both offering not only in-store layaway plans, but also online versions as well. Consumers can reserve their merchandise and make payments online, and when they are paid in full, pick up their merchandise in the store. Whatever it takes, even down to giving away free gift cards, retailers are doing to get consumers to buy, either in the store or online.

Source: http://keynote.com/benchmark/online_retail/christmas_article.shtml

Internal Testing Program Using Real Devices

Creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.

This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.

But when a company has particular security concerns and is reluctant to expose its data on a shared infrastructure, or if it has ongoing testing needs, a "private cloud"solution is in order. In this case, the test provider procures devices and contracts to create a private test network exclusively for the use of a particular client company.

In either case, the client team has ready online access to the entire process, from scripting to running the tests to reading results.

Cloud-based app testing is an automated process. There's no human being working the phones out in the field; rather, the devices are remotely operated by machine, performing the interactions specified in the test scripts and providing feedback on the devices' responses.

Read More at http://keynote.com/benchmark/mobile_wireless/article_mobile_app_performance.shtml

Internet Retailers and Website Performance

How many seconds does it take to lose a shopper to a competitor’s site? How long will a business user wait for Javascript to execute so she can see the data she’s searching for? How many times will a user tolerate delays in downloading a bank transaction, or registering a bid, or completing a form, before they abandon the site?

The cost of poor site performance is not just lost visitors, it’s lost money. In a recent survey, nearly three-quarters of Internet retailers correlate poor site performance with lost revenue, and more than half with lost traffic.

Just a few short years ago, evaluating website performance was a fairly simple affair. “How fast did the page load?” was often the first and last question that needed to be asked. User expectations were far lower, and patience much higher, when the experience of accessing information or making a purchase online was new and different and amazingly convenient.

Today, however, user expectations are stratospherically higher. With the Internet now tightly woven into the fabric of everyday life, and a multitude of Web sites available to satisfy any given need or desire, users expect not only virtually instant page-loads, but fast and flawless execution of transactions and enhanced functionality that delivers a “rich” site experience.  In the intense competition to attract and keep site visitors, web performance is now a critical business driver for site success.

Read More at http://www.keynote.com/benchmark/index.shtml

Enhance Web Performance with Best practices: Methodology & Modeling

To really understand how your website performance will hold up—or not—under holiday stress, and to understand what the experience will be like for users, use an arrival rate methodology and factor in behavior models for the many, many types of users and tasks your site will serve.

Behavior modeling results in numerous permutations (often thousands) combining these variables:

• Familiarity:  experienced users vs. newcomers
• Connection speed:  super-fast FIOS vs. super-slow mobile device, and everything in between
• Latency tolerance:  patience of users with slow site response
• Interaction speed:  complexity of the page to navigate, and attention level of the user
• Tenacity:  willingness of users to stick with a task through completion

Test in the real world—all of it.

To know how your site will perform for users dispersed across the country or the world, load testing must be done over the Internet, from the same geographic locations as your users, not from behind the firewall.  There’s simply no way to simulate the vagaries of Internet backbones, third-party content feeds, CDN performance, and signal transmission through the critical last mile—unless you are at the end of that mile, with a browser.

With testing agents dispersed where your users are, you get an accurate picture of variations in performance, and overcome the danger of looking at averages.  An average page-load time of three or four seconds may seem OK, but that kind of average could mean your page is loading in one second for someone in New York, but taking six or more seconds for someone in Chicago.  And that is not likely to be acceptable.  The solution is to test from multiple, geographically dispersed locations, look at the data, and address any local or regional bottlenecks.

The holiday shopping season is the culmination of many hard hours of work for the IT/Web department.  And no matter how well things are planned, no matter how rigorously everything is tested, there’s always the chance that the unexpected will happen and something will go wrong.  So it makes good sense to have technical personnel on hand and on call during all the critical shopping periods to handle any emergencies, and to have extra computing capacity standing by just in case it’s needed.

Read More at http://keynote.com/benchmark/new_media/article_streaming_for_primetime.shtml

A need for speed on mobile websites

While the majority of smartphone users want to shop with their phones, only a small minority of retailers offer either mobile apps or websites. One study puts the number of mobile-friendly retailers (app or site) at 32 percent; another counts less than 5 percent of retailers with a mobile-specific site.

Retailers need to catch up in a hurry. The smartphone tipping point was officially reached in spring of this year, according to Nielsen, with 55 percent of new cell phone purchases now being smartphones. Smartphone users express a strong preference to use their phones to shop, but a majority of them find the mobile shopping experience unsatisfactory — 54 percent say mobile apps and sites are “ineffective and difficult to use.”

A Harris Interactive study found that, of adults who conducted a mobile transaction in the past year, 4 out of 5 experienced a problem. And yet 85 percent of the same survey group expects the mobile experience to be equal to or better than using a computer.

It doesn’t have to be such an unsatisfying experience for users. It’s not rocket science to create a mobile site that loads with acceptable speed and delivers the features consumers want. What it requires is a change in mindset from desktop Web thinking to mobile Web thinking. Techniques that make for a rich, and yet still high-performing, experience on the desktop can render the mobile experience painfully slow or even unusable.

The good news is that on the mobile Web, user expectations of content are likely more modest. They are typically more focused on a task — finding an address, getting a price, seeing a product, buying a single item — rather than the more leisurely browsing or immersive experience they might be looking for on a desktop or laptop. So from the start, mobile content can be pared down without compromising likely use cases, so you can already be ahead of the game simply by selectively choosing content for mobile.

Source: http://keynote.com/benchmark/mobile_wireless/article_speed_up_your_mobile_site.shtml

Retail and an Intro to Mobile Strategy

Mobile Performance management starts with how the pages are built, which often presents a dilemma for retailers. With more and more products approaching commodity status and available at multiple online outlets, site experience becomes a key differentiator.  Retailers want to create a rich experience for visitors with interactivity, dramatic product presentation, perhaps Flash, personalization or other features to set themselves off from the competition.  But a heavy load of features and functionality can drag site performance down, often because of third-party content, and, instead of making visitors sticky, can drive them to leaner, faster competitive sites.

The successful retailers this year will have built mobile into their strategy right from the start—not just as an afterthought to the “main” site, but side-by-side with it.  Shoppers carrying smart phones are using them to check prices, locate products, find deals, look at reviews and, more and more, to make purchases. 

Many retailers were surprised at the amount of mobile traffic they got during the 2009 holiday season.  And there will be millions more smart phones in the hands of shoppers this year.

With regards to mobile website availability, the inherent slowness of cellular networks and devices, mobile sites need to be even leaner and meaner than wired Web sites.  It takes some hard decision-making and analysis of what is essential for users when they are browsing on the go and what it takes to satisfy them, including their need for speed.  Search results can be confined to return four or five results, for example, instead of the 40 or 50 that might be delivered on the wired Web.  And perhaps tracking pixels are needed only on the landing page and cart page, instead of every page on the site.

Read More