Performance Testing for Retail

In the broadest sense, testing falls into two categories: ongoing evaluation and tweaking to optimize daily performance, and peak load testing to determine overall site capacity and potential breakpoints.

In either case, the only way to quantify user experience is to measure what users are actually experiencing. Unless you’re the local bike shop serving only your immediate area, the testing needs to be done across a wide geography and multiple backbones. And it needs to use an actual Web browser, and go through the same type of page view sequences and transactions as a typical user would. There’s simply no other way to get a true perspective on what users are really experiencing.

GETTING A PERSPECTIVE ON EVERYDAY PERFORMANCE

Ongoing performance testing, done on a routine basis or following upgrades, patches, or redesigns, can be as simple or complex as the site itself. Assuming the back of the house is in order — that servers are performing up to speed and all of the infrastructure is optimized — the focus of the testing is on the front end. Individual pages are tested and analyzed, and those performing below par are deconstructed to identify the structural problems or elements that are causing the slowdowns.

A single element could be dragging down the page load, or it could be less-than-ideal placement of JavaScript on the page. The point is that testing and identifying the problem pages or transactions is only half the problem. The other half is being able to pinpoint the causes and correct them. A robust performance evaluation regimen accomplishes this.

RETAILERS SHOPPING FOR STRATEGIES

A few retailers are feeling upbeat about the coming holiday season. Best Buy is planning to do more holiday hiring than last year, partly because of strengthening demand for flat-screen TVs, smartphones, and netbook computers, and partly because of market share they captured following last year’s demise of Circuit City. Target plans to have 26 new stores open in time for the holidays, including five “Super Target” stores with full grocery departments; this, and more aggressive pricing, are moving Target closer to a head-to-head brawl with Wal-Mart.

Most retailers, however, are simply looking for ways to maximize their share of holiday revenue. Deep and earlier discounting is expected to be widespread. And in a throwback to an earlier era, and at least partly in response to a growing paucity of consumer credit, the layaway plan is making a comeback. K-Mart and its parent company, Sears, are both offering not only in-store layaway plans, but also online versions as well. Consumers can reserve their merchandise and make payments online, and when they are paid in full, pick up their merchandise in the store. Whatever it takes, even down to giving away free gift cards, retailers are doing to get consumers to buy, either in the store or online.

Source: http://keynote.com/benchmark/online_retail/christmas_article.shtml

Internal Testing Program Using Real Devices

Creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.

This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.

But when a company has particular security concerns and is reluctant to expose its data on a shared infrastructure, or if it has ongoing testing needs, a "private cloud"solution is in order. In this case, the test provider procures devices and contracts to create a private test network exclusively for the use of a particular client company.

In either case, the client team has ready online access to the entire process, from scripting to running the tests to reading results.

Cloud-based app testing is an automated process. There's no human being working the phones out in the field; rather, the devices are remotely operated by machine, performing the interactions specified in the test scripts and providing feedback on the devices' responses.

Read More at http://keynote.com/benchmark/mobile_wireless/article_mobile_app_performance.shtml

Internet Retailers and Website Performance

How many seconds does it take to lose a shopper to a competitor’s site? How long will a business user wait for Javascript to execute so she can see the data she’s searching for? How many times will a user tolerate delays in downloading a bank transaction, or registering a bid, or completing a form, before they abandon the site?

The cost of poor site performance is not just lost visitors, it’s lost money. In a recent survey, nearly three-quarters of Internet retailers correlate poor site performance with lost revenue, and more than half with lost traffic.

Just a few short years ago, evaluating website performance was a fairly simple affair. “How fast did the page load?” was often the first and last question that needed to be asked. User expectations were far lower, and patience much higher, when the experience of accessing information or making a purchase online was new and different and amazingly convenient.

Today, however, user expectations are stratospherically higher. With the Internet now tightly woven into the fabric of everyday life, and a multitude of Web sites available to satisfy any given need or desire, users expect not only virtually instant page-loads, but fast and flawless execution of transactions and enhanced functionality that delivers a “rich” site experience.  In the intense competition to attract and keep site visitors, web performance is now a critical business driver for site success.

Read More at http://www.keynote.com/benchmark/index.shtml

Enhance Web Performance with Best practices: Methodology & Modeling

To really understand how your website performance will hold up—or not—under holiday stress, and to understand what the experience will be like for users, use an arrival rate methodology and factor in behavior models for the many, many types of users and tasks your site will serve.

Behavior modeling results in numerous permutations (often thousands) combining these variables:

• Familiarity:  experienced users vs. newcomers
• Connection speed:  super-fast FIOS vs. super-slow mobile device, and everything in between
• Latency tolerance:  patience of users with slow site response
• Interaction speed:  complexity of the page to navigate, and attention level of the user
• Tenacity:  willingness of users to stick with a task through completion

Test in the real world—all of it.

To know how your site will perform for users dispersed across the country or the world, load testing must be done over the Internet, from the same geographic locations as your users, not from behind the firewall.  There’s simply no way to simulate the vagaries of Internet backbones, third-party content feeds, CDN performance, and signal transmission through the critical last mile—unless you are at the end of that mile, with a browser.

With testing agents dispersed where your users are, you get an accurate picture of variations in performance, and overcome the danger of looking at averages.  An average page-load time of three or four seconds may seem OK, but that kind of average could mean your page is loading in one second for someone in New York, but taking six or more seconds for someone in Chicago.  And that is not likely to be acceptable.  The solution is to test from multiple, geographically dispersed locations, look at the data, and address any local or regional bottlenecks.

The holiday shopping season is the culmination of many hard hours of work for the IT/Web department.  And no matter how well things are planned, no matter how rigorously everything is tested, there’s always the chance that the unexpected will happen and something will go wrong.  So it makes good sense to have technical personnel on hand and on call during all the critical shopping periods to handle any emergencies, and to have extra computing capacity standing by just in case it’s needed.

Read More at http://keynote.com/benchmark/new_media/article_streaming_for_primetime.shtml

A need for speed on mobile websites

While the majority of smartphone users want to shop with their phones, only a small minority of retailers offer either mobile apps or websites. One study puts the number of mobile-friendly retailers (app or site) at 32 percent; another counts less than 5 percent of retailers with a mobile-specific site.

Retailers need to catch up in a hurry. The smartphone tipping point was officially reached in spring of this year, according to Nielsen, with 55 percent of new cell phone purchases now being smartphones. Smartphone users express a strong preference to use their phones to shop, but a majority of them find the mobile shopping experience unsatisfactory — 54 percent say mobile apps and sites are “ineffective and difficult to use.”

A Harris Interactive study found that, of adults who conducted a mobile transaction in the past year, 4 out of 5 experienced a problem. And yet 85 percent of the same survey group expects the mobile experience to be equal to or better than using a computer.

It doesn’t have to be such an unsatisfying experience for users. It’s not rocket science to create a mobile site that loads with acceptable speed and delivers the features consumers want. What it requires is a change in mindset from desktop Web thinking to mobile Web thinking. Techniques that make for a rich, and yet still high-performing, experience on the desktop can render the mobile experience painfully slow or even unusable.

The good news is that on the mobile Web, user expectations of content are likely more modest. They are typically more focused on a task — finding an address, getting a price, seeing a product, buying a single item — rather than the more leisurely browsing or immersive experience they might be looking for on a desktop or laptop. So from the start, mobile content can be pared down without compromising likely use cases, so you can already be ahead of the game simply by selectively choosing content for mobile.

Source: http://keynote.com/benchmark/mobile_wireless/article_speed_up_your_mobile_site.shtml

Retail and an Intro to Mobile Strategy

Mobile Performance management starts with how the pages are built, which often presents a dilemma for retailers. With more and more products approaching commodity status and available at multiple online outlets, site experience becomes a key differentiator.  Retailers want to create a rich experience for visitors with interactivity, dramatic product presentation, perhaps Flash, personalization or other features to set themselves off from the competition.  But a heavy load of features and functionality can drag site performance down, often because of third-party content, and, instead of making visitors sticky, can drive them to leaner, faster competitive sites.

The successful retailers this year will have built mobile into their strategy right from the start—not just as an afterthought to the “main” site, but side-by-side with it.  Shoppers carrying smart phones are using them to check prices, locate products, find deals, look at reviews and, more and more, to make purchases. 

Many retailers were surprised at the amount of mobile traffic they got during the 2009 holiday season.  And there will be millions more smart phones in the hands of shoppers this year.

With regards to mobile website availability, the inherent slowness of cellular networks and devices, mobile sites need to be even leaner and meaner than wired Web sites.  It takes some hard decision-making and analysis of what is essential for users when they are browsing on the go and what it takes to satisfy them, including their need for speed.  Search results can be confined to return four or five results, for example, instead of the 40 or 50 that might be delivered on the wired Web.  And perhaps tracking pixels are needed only on the landing page and cart page, instead of every page on the site.

Read More

The impact of web load testing on performance

What drives the financial impact of a site outage or performance issues is abandonment. It’s important to understand that visitors experiencing a Website with issues don’t result in lost revenue, per se. Only when those visitors don’t return, and/or go somewhere else is revenue lost. A shopper’s tolerance for errors is called “tenacity” in Web load testing parlance. Low tenacity shoppers bail from slow searches and hanging shopping carts in a dash.

If a load test had been run that adequately modeled the impact of the planned launch, the damage would have been done outside business hours and Target management would have been able to decide whether to make changes to their systems and retest, postpone or restructure the launch or just "risk it" and see what happens.  We don't really think they ever had the chance to make those decisions.  Surely they didn't conduct a load test that predicted such an epic fail.  But should they have?

Realistic Web load tests model site usage and shopper behavior. Systems are deployed to simulate high levels of demand from multiple geographically disperse areas. Once the load is generated, the infrastructure and application’s response are watched carefully to identify bottlenecks and breakage points as the entire mesh of the Website’s interconnecting parts are stressed. Only this level of testing can accurately inform e-commerce teams of their preparation adequacy.

Read More

Website Performance: More Than Just Speed

In addition to evaluating customer experience on a subjective level, the Keynote research assessed seven factors related to the site’s service levels:

1. High-Speed Response
2. Dial-up Response
3. Response Time Consistency
4. Geographic Uniformity
5. Load Handling
6. Availability
7. Outage Hours

Car rental sites and the travel sector in general lag in technical quality, page performance, and in the delivery of an error-free user experience. Geographic consistency and load handling as key advantages, users know that wherever they are in the country or what time of day they visit, they can count on consistent website performance.

Three bottlenecks that block traffic that car rental sites, and Web sites in general, need to focus on in order to improve technical performance:

• Too many technical elements on a page: From small non-visual images to java scripts to unnecessary encryption, too many individual elements on a page can stifle site performance

• Java overload: The ubiquitous coding language is an important tool for developers, but every Java script can act like a tiny speed bump for browsers. “Sites may have five, six, seven or 10 JavaScript files in a page, and every time the browser hits that, it slows by almost a factor of two.

• Proliferation of third-party tags: The rising number of third-party tags – DoubleClick ads and calls to third-party analytics services – can also hinder web page performance, minimizing them whenever possible is a good idea. Another option would be to move the tags to the bottom of the page where they are less likely to impact user performance.

Monitoring Web Pages & 3rd Party Content

If you are the owner of the Web site, there are at least two ways to improve overall site performance. One is to optimize the Web site itself for each widget so that those widgets running on the page will be more efficient, a method that is not cost-efficient. The other, more cost-effective option is to use continuous and focused performance monitoring of your Web site. Breaking down performance by time and by component category allows you to pinpoint the components that adversely impact Web site performance.

The benefits of web page monitoing and third-party components are significant indeed. First, operations can target these issues quickly and efficiently, which can reduce potential downtime and loss of revenue. This metric, known as Mean Time to Identification, can be tracked. Second, business unit managers can track the performance of all content, both internal and external, which can establish SLA accountability with the third-party vendors, saving money on lost downtime or the cost of rebates. Another benefit is the accountability that can also be established internally on components and content that has been developed on your site. Third, development and QA teams can save money by tracking these issues in real time. Modifications to code on the Web site or to the widget have been known to adversely affect a previously well-performing Web site, and monitoring can nip these issues in the bud, saving time and therefore money. Finally, the user experience can suffer due to bad website performance. This can cause a loss of viewers, both because of direct experience and by word of mouth. This impact can be potentially devastating to the bottom line.

Read More

Web Performance Varies by Geography and Network

Unlike Web content downloads to desktops, mobile downloads can vary dramatically based on time of day, network operator used, and geographic location. For example, a mobile Website can take twice as long to download in San Francisco as compared to another operator in New York, London, or Tokyo. If a content-monitoring strategy does not include monitoring web content from various geographic locations, it is impossible to know what the end users are experiencing.

In the Web world, both the user interface and the delivery mechanism have been standardized for years, keeping mobile browser compatibility in mind. However, mobile content must be routed to the user through an operator network, and additional operator specific content may be added during the download. Also, different devices render content differently, so when the Web server detects a specific device type it may choose to send a variant of the generic content to the requesting device.

Read More

Testing Mobile Devices Cost Effectively

Although manual testing is a need with real devices and real networks, avoiding it whenever possible is always a good idea. Manual testing is expensive and slow, it also lacks the necessary instrumentation to isolate application problems of your product to ensure a quick delivery. Instead, how about considering a solution that combines some manual testing, some remote-manual testing, and a lot of testing using emulated devices.

Mobile testing with an emulator is cost effective as it can be done very quickly and efficiently. Diagnostics on the tool is a must have, this lets you isolate problems and ensures flexibility in network stacks you will need to test different network options. Ensure that your emulated device solution contains a high – level scripting solution to allow you to repay your test cases over and over. Also look for an emulated device that lets you change device profiles quickly.

Emulation Technologies and Mobile Device Monitoring

Many Websites and services require only a browser to render the page and do not depend on the operating system of the device (for example, SMS). To accurately monitor such services on a broad range of possible combinations in the mobile ecosystem, the mobile monitoring solution needs to be able to emulate the vast array of devices available. A diagnostic capability helps to quickly pinpoint the root causes of failures when something goes wrong. Emulation technologies must contain, along with a wide range of device profiles, an awareness of the multiple factors comprising the end-user experience.

Emulated mobile device monitoring can be performed by content owners in two separate modes:

● Over the air mobile monitoring for true end-user experience

This requires a measurement solution that truly emulates the entire technology behind mobile downloads—including the actions of the operator network that sits between the Web server and the end user’s device.

● Direct over the Internet monitoring to measure only the availability of content without the impact of operator networks

By using this method, you can quickly identify the root cause of failures experienced by end users.

Read More on Mobile Monitoring

Mobile Web Optimization

Testing mobile web applications lets you in on several challenges in the mobile web. Decision making is crucial when it comes to your testing options. It is necessary to explore all the challenges first, to understand them. This helps is explore our technology options and to manage each challenge in an effective way. We will need to measure positives and negatives of each challenge before we choose the combination of testing options that suits best. Mobile testing challenges include devices, network and scripting.

One of the biggest and the most obvious mobile testing challenges is when it comes to mobile devices as its used by customers. There are thousands of client devices that could be used on your mobile site, therefore all are to be considered while testing your mobile applications. If we try and reduce this number, we will be taking a chance that out application might not work on a few devices and that will lock out a number of potential customers.  To handle this challenge, we will need to test using real devices or use emulated devices.

The next challenge is a regional challenge, the Network. Considering about 400 mobile operators in the world some being GSM, some being CDMA and the remaining use local or less common networking standards. Since each network has a unique combination of network, it wouldn’t be possible to discuss a challenge without discussion location. Also travelling to each network would be expensive. There are several ways of dealing with network challenges. (More about Network challenges in separate post)

Call scripting being the last challenge is actually used to execute the test script. Script execution can be manual or automated where real devices require manual scripting while emulated devices can use automated scripting.  

We need to understand a lot more about these challenges on mobile web testing. To lay down a path, once we understand the above challenges, we would need to focus on how to utilize the information and what would be the testing strategy for mobile application testing.

Root Cause Analysis with Mobile Monitoring Solutions

The quality of the subscribers’ mobile experience can be ensured by monitoring, benchmarking performance and availability of services. Now, this needs to get down to every step of a transaction.

Mobile monitoring solutions provide real-time information for transactions involving root cause analysis, competitor benchmarking and alerts in case of service downtimes.

Let’s get into a few details regarding ‘Root cause analysis’.  In case you there is a service downtime or a degrading performance issue how do you determine the cause for the problem?   It could be the handset, the carrier network or your content portal.

Mobile monitoring solutions can quickly take you through an intuitive portal if the problem is resulting from a specific transaction; or if your service in every location is down; if the problem is local to one city. You can also drill down further into the components that make up that transaction to find out what exactly caused the issue.