Thalasar Ventures

I now have roughly 5 months of data dealing with the beta test of my comparison shopping engine. In the short time it’s been up, with ZERO advertising dollars, it has achieved a Page Rank of 6 and an Alexa ranking of 150,000. That said I think there is room for serious improvement.

The point of a beta is to gather data and refine your product. This is the step that many net businesses simply drop the ball on. Given the amount of effort spent on developing an interface or a specific product, their is a great deal of emotional commitment invested in the current product. People are emotionally invested in both the layout (which they undoubtedly had a hand in developing) and view change as dangerous.
This is a common enough problem and quite frankly to achieve success, you need to spend a huge amount of time using analytics and user testing in order to understand how people are actually using your site. I am conducting some informal user testing right now. It consists of asking people at a coffee shop to use my site. It’s cheaper than typical user testing (which will run you $15,000 – $20,000) and will work if you keep your hands away from the keyboard, resisting the urge to say (here’s how you do that) you will learn a ton about what’s wrong with your web site.
Jakob Nielson has pointed out that even small user tests of five users tend to find 80% of usability problems.
Here’s a few things I have learned. (Please note for competitive reasons, I am not going to divulge everything I have learned).

• Pricing triggers are too hard to set up.
  Users are typically just click on the lowest price. While setting up a pricing trigger seems like the better way to save money, it’s not being used. It’s just buried too deep.

• Generalized search
  Users like a search that operates across all categories. Sometimes they simply cannot know the category they are searching ahead of time.

• Pay attention to what users do
  Users may say one thing but it’s how they interact with the site that important. Many times what the user says contradicts what the user is doing. The behavior is the important factor.
  

So if you are going to set up your own informal user test you should do the following. I have set up my informal testing at the local coffeeshop. Even this basic testing is better than nothing. I don’t have the budget for earlymiser.com for even a small usability test in a formalized setting. So here are some rules for setting up a coffee shop usability test. You should periodically do this for your site and blog since it will undoubtedly improve the web site experience for your users.

• Bring a notebook computer that all users will use
• Give users time to get familiar with your computer.
• Bring a mouse, since users might not be familiar with a touchpad.
• Offer an incentive for testing your site – free coffee and pastry. You would be amazed at how well this one works.
• Make sure you get individual users to test first. That’s where you are going to find out where your product needs work. Groups tend to better problem solvers.
• Take notes as people are using the site. Your presence is probably less intrusive than a video camera and most coffee shops are not gonna allow you to tape.
• Have a task list you would like the user to accomplish. These tasks should should be what the site is about and how you think a user should use the site. This list should be very short so that the user can do the next point.
• Let the user’s play with the site. This contradicts the above point but if a user comes to your site, they should be able to figure out what the site is about right? If they can’t, that’s a problem right there.
• Resist the urge to help. When users are stuck on part of your task list, ask them how they think they should proceed. Note the sticking point in your notes. If necessary, help them through the part and then move on.
• Users often want to get things “right”. So they often look at a test as something that they can be embarrassed by if they don’t get it. You need to reassure them that you are testing to make improvements and changes, don’t worry about getting it right, simply use the site.
  

Please note you should be conducting regular informal coffee shop testing for your blog as well. Too many people neglect even simple user testing and quite frankly it can be done cheaply while giving you most of the benefits of a more formal user test. Bloggers can do this and even perhaps get a new reader or two at the same time. You should conduct multiple tests obviously. After your first usability test, test again after your changes. This is to make sure you haven’t introduced new problems to the site. Small iterative testing is much more productive than a large elaborate test.

Update
I just realized of course that unless you have conducted a usability test before, this advice is less than helpful. I reccommend that you start with usability testing without a budget for the beginner. Doing a write up and taking clear notes on while people are using the site are the keys for making this a success.