The Hidden Dangers of Tree Testing

Tree testing and its related research activity, the card sort, are often the first research lines of defense for content strategists. We love seeing what people who might use a site or app think about terminology, hierarchy, and categorization. But sadly, I’ve been in many a research phase where we looked at the results and said “wait – was that a problem because we used the wrong words?” or “Did they put those together because of how we phrased them or what they associate with that term?” or even “You thought that card meantĀ that? I always use that term forĀ this… what were our participants using it to mean?”

This is the hidden danger in card sorting: confusing the search for terminology with the search for organization.

What are card sorts and tree tests?

If you’re already familiar with the basics of these activities, go ahead and skip this section. But for the uninitiated, a brief introduction:

Card Sorts

A card sort is a type of user research involving 20-30 digital or physical cards, each with a word written on it. Participants (either as individuals or in groups) sort the terms into categories, to show the researchers which items they think are related or would expect to find together.

In a closed card sort, the moderator also provides the categories. In an open card sort, participants create their own category titles. In this way, card sorts are useful in creating an information architecture or sitemap.

Tree Tests

In a tree test, the moderator (or program, for unmoderated tests) provides a top-level site map to the participant, and a task for the participant to complete. The participant then clicks down through the sitemap until they find the page or section where they believe they would be able to complete the test. With tracking software like Optimal Workshop, researchers can see what percentage of people went down wrong paths, and how many people chose the same wrong path. In this way, the team can understand which labels are being misunderstood and/or what is being placed in the “wrong” category.

You can see why suddenly realizing post-facto that participants and moderators had different term connotations would cause problems.

The Hidden Dangers

Card sorts and tree tests provide the answers to two different questions:

  1. How does my audience interpret these terms?
  2. How does my audience group this information?

However it is paramount to remember that a single card sort or tree test can only answer one of these questions at a time. If you don’t yet know how an audience will interpret terms, you can’t truly test whether the information has been grouped correctly. Equally, if you aren’t sure whether items are grouped correctly, you can’t truly select appropriate labels.

Sounds like a catch-22, no? Here’s how to get around it.

  • Option 1: Use a card sort for organization, and a tree test for labels
    In the card sort, don’t worry about calling things by their potential final labels. Err on the side of verbose and call an article “Article about how to mow the lawn” rather than choosing a catchy title. Call a blog “Marketing blog,” rather than “Insights” or “Perspectives.” Focus on nothing but how people categorize the different information chunks, and remove any tricky language. Then, once the IA is on its way to final, test out labels in a tree test, and see whether people connect the top level nav labels to the items housed within.
  • Option 2: Learn the labels first, and tree test for navigation
    In this variation, a card sort may not be needed at all, but rather a user interview that includes asking the participant to define terms and name certain types of content. From this, the content strategy team will learn what vocabulary participants are comfortable with. Then, with the confidence that comes from knowing participants and moderators have a shared vocabulary, set up a tree test for sample IA.

Don’t waste your time with a tree test based on uncertain labels and navigation! Divide your tasks, and conquer tree testing.

Marli Mesibov

Marli is a content strategist with a passion for the user experience. Her work spans websites, web applications, and mobile. Marli is the VP of Content Strategy at the UX design agency Mad*Pow, where she helps healthcare, finance, and educational organizations communicate with their audiences. Marli is a frequent conference speaker, and has spoken at conferences including Content Strategy Forum and LavaCon. She can also be found on Twitter, where she shares thoughts on content strategy, literature, and Muppets.

Leave a Reply

Your email address will not be published. Required fields are marked *