Etsy’s mission is to Keep Commerce Human and every day we work to build tools to help creative entrepreneurs do just that. Everything we do centers on supporting you and your business because when you succeed, we succeed. Experiments are a tool we use to ensure that when we make updates on Etsy, they’ll have a positive impact on sales and traffic or make Etsy’s tools easier to use.
What do we mean we say experiments? Experiments are one way we gather data about how Etsy buyers and sellers interact with a new feature and measure how changes to that page or tool affect an end result. By testing new features with a small percentage of Etsy users, we gather valuable data that helps us ensure that the changes we make will have a positive impact for the majority of Etsy sellers. In most cases, we use experiments to confirm a hypothesis that we formed with the help of research and data, not to test a completely new idea.
When we make updates, our first priority is building a better Etsy for everyone. We use experiments to test almost every change we make to the buyer experience on Etsy to ensure that they increase sales, boost traffic, or make a shopper’s experience on Etsy better. When we do experiments on the tools you use to run your business, it’s because we want to be sure that the changes we’re making will make it easier for you to manage your shop and find the tools you need.
We know that you rely on Etsy’s tools to run your business, so having to change your workflow because you’re part of an experiment can feel like an unnecessary addition to your already packed to-do list. But every time you encounter an experiment on Etsy, you’re helping to build better tools for the entire community. If you’re asked to share feedback on an experiment via a survey—add your thoughts! All that feedback is shared directly with the team working on the feature or experiment.
An experiment case study
Let’s look at an example of how we used an experiment to test our hypothesis about improvements to the listing process.
Our hypothesis: During research studies, we discovered that many sellers like to upload photos to their listing before adding a description or other elements, so they could look to them for reference while completing the rest of their listing. Knowing this, it didn’t feel right to us that uploading photos came later in the listing creation process. Our hypothesis: making photos the first step of creating a listing would save sellers time.
How we confirmed our hypothesis: Before making a change to the listing process that would affect all sellers, we wanted to confirm that the change would be positive. So, we designed an experiment during which 25 percent of sellers would see the photo step first at the top of the listing creation page, 25 percent would see the photo step at the bottom of the page, and 50 percent would see the photo step in the middle of the page, where it had always been. This option, called the control, would help us prove that moving the photo step to the top of the page is what actually caused the positive result.
The results: After testing the three versions of the listing process for a few days, we found that our hypothesis was correct. Sellers who saw the photo option first completed new listings more quickly and created more listings than the sellers in the other groups.
By running a short experiment, we determined that the change would have a positive impact on a majority of sellers.
How we talk about experiments
Logging into your Etsy shop and finding something out of place can be disconcerting, but experiments are an integral part of building great tools. They allow us to quickly confirm our hypotheses, so we can make impactful changes on a regular basis. We know it can be frustrating to be surprised by a change in your shop when you’re part of a test, but announcing experiments before they happen can bias the results. If you knew we were testing a new tool, you might use it differently, impacting the results of the experiment.
We regularly test changes to the buyer experience on Etsy to help make it easier for buyers to complete their purchases, so you can make more sales. Something as seemingly small as the color or location of a button or the language used in a menu can have a measurable impact on the number of completed purchases.
Other ways we gather feedback
When we test tools that might mean big changes to your workflow, we use prototypes to gather feedback and test our hypothesis so that we don’t risk asking you to change your processes for a tool that might still need a little work. Prototypes are one way you can choose to test a new Etsy tool to help make it even better before it’s launched to all sellers.
Unlike a prototype, you can’t opt-out of an experiment. Whether you have one sale or thousands, you could be part of an experiment. By including all kinds of shops, we can feel confident that the results of the test will have a positive impact on everyone.
We also use other kinds of research methods like interviewing sellers about their processes or watching a small group of sellers interact with a new tool to gather feedback to help us make informed decisions. And, of course, our Community team keeps on eye on trending conversations in the Etsy Forums, and shares key themes with teams throughout the company.
Words by Taylor Combs
