Step-by-step guide to conduct card sorting analysis

Reading Time: 7 minutes

Imagine you’re designing a new online store. You have many ideas for categorizing products, but how do you know which makes the most sense to your customers?  This is where card sorting analysis comes in.

It’s like a detective game for understanding how people organize things in their minds. 

By watching participants sort information into groups, you get a glimpse into their thought process, how they naturally categorize the world around them. 

This secret knowledge is the key to crafting an information architecture (IA) that feels intuitive and effortless for your users to navigate.

Let’s start by understanding card sorting methods.

Open vs. closed card sort

The analysis approach hinges on the type of card sorting you conducted:

Open card sorting: Participants create categories for the cards, revealing their mental models for information organization. 

Closed card sorting: You provide pre-defined categories, and participants sort the cards into those categories. This helps assess how well your existing IA aligns with user expectations.

Open and closed card sorting influence the focus of your investigation rather than the exact procedure. Emerging open card sorting analysis trends will guide you through creating an information architecture from scratch. Closed card sorting validates or disproves the existing information architecture.

With closed card sorting, you should pay attention to how users organize concepts and if this aligns (or diverges) from your existing information architecture or structure. If particular cards are constantly placed in several categories, it shows that they are confusing to users and might be separated into multiple topics.

You may like to read more about- What is card sorting?

Steps to conduct card sorting analysis

Card sorting analysis is a worthwhile technique for understanding how users categorize information, which can inform the structure of a website or app.

Here are four steps to conduct card sorting analysis effectively:

Step 1: Take a big-picture look

Unsurprisingly, examining your card sort data is the first step in finding meaningful insights. Start by reviewing your results and looking for intriguing trends or standout groupings. Rather than becoming too technical, look for more significant implications for user experience.

Look for the most often-created categories in your customized participant database if you carried out an open card sort (this is especially beneficial if you have undertaken repeated card sorting sessions). After that, you can build your information architecture on top of these categories.

Trends for closed card sorts might need to be more prominent. Since you are working with pre-existing categories, you must compare your outcomes. In your user research, look for anything that stands out—for example, a lot of cards that are regularly categorized in a category that differs from the ones on your website or categories that are not utilized at all.

Other recurring themes and patterns could be:

Normative clusters: 

If participants consistently arrange the cards into the same categories, they may have a shared mental picture of how these topics should be placed on your website.

Categories that are overused: 

If participants place too many cards in one category, you should create a second, more focused category for these ideas, indicating that your current category needs to be more specific.

Combining categories: 

Card sort analysis in several categories indicates that different points in your information architecture should be able to access these concepts.

Step 2: Clean and standardize your data

You require precise card sorting insights rather than general ones to advance your information architecture. To obtain that, you must compile all of your data into one location and ensure it is flawless before submitting it for more examination.

This is particularly crucial if you intend to analyze your data later and look for numerical trends. Without the proper tool to arrange your data regularly, it is easy to get lost in a sea of numbers.

To consolidate your data, you must convert the results of your card sort if it used physical cards to a digital format. Most designers and researchers use spreadsheets for this purpose, but if you want quicker results, register for a dedicated card-sorting application. 

Consider that you have conducted a card sorting analysis event in person. An infinite number of rows and cells in a spreadsheet creates the framework for standardizing and arranging your data. 

When entering the results into your spreadsheet, remove any incorrect entries or cases where a participant left the sort unfinished, as this can distort the data.

Be aware that a high volume of incomplete data may suggest that your test has to be reassessed due to potential confusion. Put these blank entries away for evaluation at a later time.

Users’ assigned categories must also be standardized if you have used open or hybrid card sorting. 

As an illustration, a category may have been labeled “about us” by some users and “about the firm” by others.

Find categories with similar meanings and choose a standardized version to use. As an illustration:

a. Reach out to reach out

b. Customer service/assistance desk

c. Plans/Pricing/Products/Our offerings

You can now manually analyze your card sort after your data has been cleaned and standardized.

Step 3: Examine your card sorting by hand

Examining your card sort analysis by hand can be a meticulous but insightful process.

Here’s how to approach this step effectively:

Quantitative analysis:

Finding essential patterns and commonalities is the primary objective of the quantitative analysis for your card sort analysis. Determine metrics, examine the data, and determine which groupings are more prevalent. 

Three beneficial techniques and diagrams belong within the statistical analysis category when it comes to quantitative analysis for card sorting analysis:

Clustering 

Clustering entails grouping analysis results together. The similarity between the objects in a cluster increases with distance. Using this strategy, cards will show up as clusters when you analyze your card sorting findings based on how often participants grouped them.

Dendrograms

Dendrograms are diagrams that resemble trees and show the hierarchical connections among several data points. They help us see how people naturally arrange popular sets of cards into several categories.

Spreadsheets with matrices

A matrix spreadsheet can determine which categories cards were sorted into the most often. Although percentages can be used instead of numbers, they facilitate understanding of a group’s affinity.

Several online card sorting analysis and statistical analysis applications include integrated card sorting results to assist in creating clusters, dendrograms, and matrix spreadsheets.

Qualitative analysis:

Qualitative card sorting analysis can provide more in-depth insights than quantitative research, but processing data and making sense of the results can be time-consuming. However, because qualitative analysis does not require statistical generalizations, you can run card sorts with fewer individuals.

Step 4: Analyze agreement scores

Analyzing agreement scores helps quantify how consistently participants grouped the same items.

Here are some things to follow for efficient card sort analysis:

Calculate agreement scores:

a. Calculate agreement scores using a card sorting analysis tool. These scores measure how often participants grouped the same cards.

b. Higher agreement scores indicate participants had a similar mental model and consistently grouped specific cards. In comparison, lower scores suggest variability in how participants perceived the organization of the content.

Identify high and low agreement areas:

a. First, focus on areas with high agreement scores. These clusters can serve as a strong foundation for your information architecture.

b. Analysis areas with low agreement scores to understand why participants disagreed. This may indicate that the cards are ambiguous or participants had different interpretations of the categories.

Consider the context:

a. High agreement scores in specific categories can highlight the most intuitive groupings for users, providing a clear direction for your site’s structure.

b. Low agreement scores can signal the need for further research or revisions to the card labels or categories to improve clarity.

Use heatmaps:

a. Visualize agreement scores using heat maps to quickly identify strong and weak areas of consensus.

b. Heatmaps can help stakeholders quickly grasp where users agree and where more attention is needed.

Step 5: Share your findings with the stakeholders

To ensure that your findings become solutions, share them with stakeholders. Creating a UX research report is the most efficient and straightforward approach.

Your reports should contain all of the necessary information from your card sorting, so include the following:

a. Your chosen card sort method, why you chose it, and how it fits within your research plan and objectives

b. The demographics of the testers and the method you used to acquire research participants

c. Detailed results, discoveries, and insights from your card sorting

Include a section for solutions based on your insights and how they will affect your UX roadmap and research plan. 

Conclusion

You can transform raw card sorting data into actionable insights by employing these analytical steps and considering the “why” behind user behavior; card sort analysis empowers you to create an information architecture that aligns with user mental models, resulting in a website or application that is intuitive, navigable, and ultimately, a delight to use.

Frequently asked questions

1. What is card sorting analysis, and why is it important?

Card sorting analysis involves interpreting the data collected during a card sorting session. By analyzing how participants grouped information, you gain valuable insights into user mental models, how they categorize and organize information in their minds. Card sort analysis is crucial for designing an information architecture (IA) that feels natural and user-friendly.

2. What are the different types of card sort analysis?

There are several methods for card sorting analysis, depending on the type of card sort conducted (open vs. closed) and the desired level of detail:

Fundamental analysis: Examining participant groupings, identifying common categories, and calculating agreement scores (e.g., Inter-Rater Reliability).

Data visualization: Creating visual expressions of the data using techniques like affinity diagrams, tree diagrams, and reheat maps.

Advanced Techniques: Utilizing clustering algorithms to automatically group cards and dendrograms to visualize cluster relationships (beneficial for large datasets).

Qualitative analysis: Review notes and observations from the card sorting sessions to understand the rationale behind user choices and uncover underlying needs.

3. What tools can I use to analyze card sorting data?

Various tools are available, from simple spreadsheets to dedicated card sorting software. Spreadsheets offer an essential way to organize and analyze data, while software can automate tasks like calculating agreement scores and creating data visualizations.

Get the list of the best free and paid card sorting tools to utilize for card sorting analysis.

4. What are some key things to look for when analyzing card sorting results?

Common categories and card placements: This reveals how users naturally group information.

Agreement scores: Measures consistency among participants and identifies strong consensus or disagreement areas.

Miscategorized cards: Highlights potential issues with your IA or card labels.

Qualitative insights: Understanding the “why” behind user sorting behaviors provides valuable context for interpreting quantitative data.

Read next on:

Confirmation bias explained: Importance and examples

Top 6 participatory design methods

What is field research? Meaning, examples and methods