"Performance" to cloud providers typically means only availability

One reason cloud performance monitoring is so critical is that cloud provider service is so nebulous. "Performance" to cloud providers typically means only availability; and even availability is only loosely guaranteed. For Amazon Web Services, as an example, unavailability means no connectivity at all during a five-minute period; if your user has a lousy, erratic, miserably slow connection, as far as Amazon is concerned, they've delivered. And availability means availability when it leaves Amazon's door; whether or not anything actually reaches your user is not Amazon's problem (regardless of their choices for ISP and connectivity). Oh, and the burden of proof is on you, the customer. For all intents and purposes, Amazon is not even checking to see if you even have service.

This is not to dump solely on Amazon. The same guarantees, or lack thereof, are typical of many cloud providers. In addition to the caveats above, scheduled and emergency downtime is excluded from availability guarantees; penalties for unavailability are minimal, and certainly not commensurate with potential business damages; and any other kind of performance is not included in the service level agreement.

An ideal cloud-client working relationship includes substantial SLAs, external monitoring of SLA parameters that is visible to both provider and client, and meaningful recourse if the service falls short. In lieu of this ideal, however, the onus is on the enterprise to put cloud monitoring and measurement in place and to hold their provider accountable – so they can either get the service level they need, or switch to a better provider.

Read More 

The Elephant In The Cloud: Performance

For all of its tremendous potential, there’s one thing that will make or break the SaaS model both for vendors and users: Web Performance. No longer is technology contained within the walls of the enterprise, running on its own proven network, controlled closely by its own IT department. Now, the nerve center of the enterprise, the productivity of workers, the integrity of information assets is controlled by an outside entity, flows through the various pipes from a remote data center over the Internet into the enterprise’s network and ultimately, into a browser. And that presents challenges both for performance and user experience.


The problem with the browser being the front-end for SaaS applications is that users have very clear expectations of a browser experience, based on their use of the Web. Users go to a site and expect it to load fast — in two seconds or less. They don’t like to wait, and won’t. Google says that for every additional 500ms of delay, the site loses 20 percent of its traffic. With fast broadband and wireless connections everywhere, users expect blazing speed when they fire up their browser.


Read More http://keynote.com/benchmark/SaaS/article_industry_focus_cloudy_applications.shtml

Tablet Problem? Or Tablet Opportunity?

Yes, it’s another form factor, and more operating system permutations, still more network connection considerations, and additional development expense. But tablets could actually be a bright spot for content owners looking to attract audience and advertising dollars.

This past holiday season, iPad users showed themselves to be valuable customers, buying more and spending more than other mobile shoppers. And a number of studies indicate that tablet users are avid content consumers as well.

More than three-quarters of tablet owners use their device every day, on average for 90 minutes. More than half use it daily to get their news, and many now spend more time on news than they did pre-tablet. In fact, the only thing they’re more likely to use their tablets for than news is to browse the Web.

These numbers are likely skewed, as most current tablet owners have pricey iPads; they tend to be higher-income, better educated, often middle-aged consumers. As the market expands down to lower-priced Kindle Fires and others, tablet demographics will likewise gravitate toward a more mass consumer profile.

Read More: http://keynote.com/benchmark/SaaS/article_three_screen_optimization.shtml

Mobile Web: Load Testing and Ongoing Performance Monitoring

Many retailers are making efforts to enable genuine mobile commerce, as opposed to trying to drive traffic from mobile to other channels to complete the transaction. And many are bringing their mobile development efforts in-house, after testing the waters with a third-party solution. Both of these scenarios involve significant risk, making it all the more imperative to perform robust load testing and ongoing performance monitoring.

The big challenge that mobile sites face that’s different than the desktop Web — in addition to the inherent slowness of over-the-air (OTA) signals — is the plethora of devices, operating systems, and carriers. A mobilesite or app has to be vetted in all the major configurations that ultimately control what is displayed in the user’s hands.

Any outage or excessive slowness during peak holiday periods can inflict serious pain on a retailer’s bottom line, whether the retailer is big or small, online-only or hybrid. The only way to make sure a website can handle an extraordinary rush of traffic is to thoroughly test it well in advance and every time changes are made. The most solid test regimens subject a site to traffic that’s multiples of the highest demand projected by the sales and marketing department.

Read More

Real tests use real devices

It's not that testing with real devices is the gold standard, though that's one way to look at it. The fact is, there is simply no way to emulate the behavior of a mobile app in the field unless the device is literally in the field. System overhead, memory usage, CPU speed — a host of variables impact app functionality in ways that just can't be reproduced in a lab. To get real test results, you have to deploy real devices.

But creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.

This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.

Source: http://keynote.com/benchmark/mobile_wireless/article_mobile_app_performance.shtml

Performance Testing for Retail

In the broadest sense, testing falls into two categories: ongoing evaluation and tweaking to optimize daily performance, and peak load testing to determine overall site capacity and potential breakpoints.

In either case, the only way to quantify user experience is to measure what users are actually experiencing. Unless you’re the local bike shop serving only your immediate area, the testing needs to be done across a wide geography and multiple backbones. And it needs to use an actual Web browser, and go through the same type of page view sequences and transactions as a typical user would. There’s simply no other way to get a true perspective on what users are really experiencing.

GETTING A PERSPECTIVE ON EVERYDAY PERFORMANCE

Ongoing performance testing, done on a routine basis or following upgrades, patches, or redesigns, can be as simple or complex as the site itself. Assuming the back of the house is in order — that servers are performing up to speed and all of the infrastructure is optimized — the focus of the testing is on the front end. Individual pages are tested and analyzed, and those performing below par are deconstructed to identify the structural problems or elements that are causing the slowdowns.

A single element could be dragging down the page load, or it could be less-than-ideal placement of JavaScript on the page. The point is that testing and identifying the problem pages or transactions is only half the problem. The other half is being able to pinpoint the causes and correct them. A robust performance evaluation regimen accomplishes this.

RETAILERS SHOPPING FOR STRATEGIES

A few retailers are feeling upbeat about the coming holiday season. Best Buy is planning to do more holiday hiring than last year, partly because of strengthening demand for flat-screen TVs, smartphones, and netbook computers, and partly because of market share they captured following last year’s demise of Circuit City. Target plans to have 26 new stores open in time for the holidays, including five “Super Target” stores with full grocery departments; this, and more aggressive pricing, are moving Target closer to a head-to-head brawl with Wal-Mart.

Most retailers, however, are simply looking for ways to maximize their share of holiday revenue. Deep and earlier discounting is expected to be widespread. And in a throwback to an earlier era, and at least partly in response to a growing paucity of consumer credit, the layaway plan is making a comeback. K-Mart and its parent company, Sears, are both offering not only in-store layaway plans, but also online versions as well. Consumers can reserve their merchandise and make payments online, and when they are paid in full, pick up their merchandise in the store. Whatever it takes, even down to giving away free gift cards, retailers are doing to get consumers to buy, either in the store or online.

Source: http://keynote.com/benchmark/online_retail/christmas_article.shtml

Internal Testing Program Using Real Devices

Creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.

This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.

But when a company has particular security concerns and is reluctant to expose its data on a shared infrastructure, or if it has ongoing testing needs, a "private cloud"solution is in order. In this case, the test provider procures devices and contracts to create a private test network exclusively for the use of a particular client company.

In either case, the client team has ready online access to the entire process, from scripting to running the tests to reading results.

Cloud-based app testing is an automated process. There's no human being working the phones out in the field; rather, the devices are remotely operated by machine, performing the interactions specified in the test scripts and providing feedback on the devices' responses.

Read More at http://keynote.com/benchmark/mobile_wireless/article_mobile_app_performance.shtml