From Alpha to Beta: Continuous Deployment at Threadflip
by Samantha Lynch & Elizabeth Trongone
Having prior product management experience in companies like Birchbox, Warby Parker and Group Commerce, we sought to continue to hone our product management skills through the lens of Launching Technology Ventures. Our specific goal for our LTV project was to refine our skills in usability testing and customer interviews, while being the voice of the customer throughout the initial stages of product development. As both of us are searching for product manager roles for after graduation, we wanted to be able to point to our own learnings as well as experience with working with a very early stage company. Neither of us had worked with a start-up before they reached product market fit, and we wanted that early experience. Therefore, in searching for our “ideal” LTV project, we looked for start-ups that had not yet launched in beta and landed on an opportunity with Threadflip.
Threadflip is an online destination for buying, selling and sharing fashion with friends and like-minded fashionistas. Threadflip enables users to easily weed through their closets, recoup some of their investment and make room for new finds. We began working with Threadflip right as they launched in alpha and partnered closely with the Co-Founder/CTO and VP of Product from the beginning. Because Threadflip was founded by an engineer/product guy (and because Eric Ries is one of their advisors), we did not have a difficult time convincing the Threadflip team to work with us weekly and bring us into their product development process as extended team members. We set aside dedicated days for user testing and interviews that tied into Threadflip’s software development process. (e.g. Threadflip sprints start on Monday and end on Friday. We tested interest in new features before the sprint started through interviews and then after the sprint ended using the real product). Throughout our time working with Threadflip, we collected learnings on working with engineers, recruiting users and administering tests and interviews.
We also quickly observed a vast difference between sessions with internet savvy/frequent online shoppers and those that were less so. Existing users of the alpha stage ecommerce site tended to be familiar with start-ups and online models such as ETSY, eBay and Pinterest. Among the new users, some immediately referenced these sites and their other online shopping experiences while others clearly were not accustomed to online shopping and found the model, picture feed and features (virtual closet, “saving” items, and sharing) to be more confusing than the internet savvy group that caught on quite quickly. This demonstrated for us the challenge in creating a product that offers a great user experience for a range of users and satisfies the power adopters’ desire for new and cool features but crosses the chasm to the mainstream users’ desire for simplicity and ease of use. We saw that concentrating testing more heavily on one group versus the other could bias feedback and recommendations; it is important for product managers to keep this in mind when recruiting and testing users and to understand a site’s target demographic at different stages.
We also found that existing users required more specific questions in order to uncover motivators and barriers to certain behaviors while more general questions allowed new users to share initial insights and responses to their first experiences on the site.
Lastly, we found that repeated patterns emerged from even a small number of users. Certain critical pain points and areas of confusion came up continuously. For example, the tedious process of packing, describing and photographing items and uploaded images was a significant barrier to entry to selling on the site for many users. This repeated finding confirmed Threadflip’s interest in launching a white glove service for picking up and orchestrating the sale of items that they are currently rolling out with the site’s official launch.
Having prior product management experience in companies like Birchbox, Warby Parker and Group Commerce, we sought to continue to hone our product management skills through the lens of Launching Technology Ventures. Our specific goal for our LTV project was to refine our skills in usability testing and customer interviews, while being the voice of the customer throughout the initial stages of product development. As both of us are searching for product manager roles for after graduation, we wanted to be able to point to our own learnings as well as experience with working with a very early stage company. Neither of us had worked with a start-up before they reached product market fit, and we wanted that early experience. Therefore, in searching for our “ideal” LTV project, we looked for start-ups that had not yet launched in beta and landed on an opportunity with Threadflip.
Threadflip is an online destination for buying, selling and sharing fashion with friends and like-minded fashionistas. Threadflip enables users to easily weed through their closets, recoup some of their investment and make room for new finds. We began working with Threadflip right as they launched in alpha and partnered closely with the Co-Founder/CTO and VP of Product from the beginning. Because Threadflip was founded by an engineer/product guy (and because Eric Ries is one of their advisors), we did not have a difficult time convincing the Threadflip team to work with us weekly and bring us into their product development process as extended team members. We set aside dedicated days for user testing and interviews that tied into Threadflip’s software development process. (e.g. Threadflip sprints start on Monday and end on Friday. We tested interest in new features before the sprint started through interviews and then after the sprint ended using the real product). Throughout our time working with Threadflip, we collected learnings on working with engineers, recruiting users and administering tests and interviews.
Working with Engineers
Threadflip’s continuous tech deployment and iteration allowed us to see our recommendations implemented through the course of the project and to gather feedback in real time. Threadflip’s engineering team works in short one week sprints; we conducted our user sessions earlier in the week, shared our feedback mid-week in line with these meetings and recruited a new set of users at the end of the week to test the latest site iteration. We shared our takeaways and recommendations from our user sessions and customer interviews with the CEO and his engineering team through a constantly updated Google doc. The CEO would respond to us in real time, indicating which changes they had iterated on or were in the process of implementing. We then were able to probe on these changes and features in our next set of testing with new users. Working with an engineering team that firmly believed in lean principles and fast iterations in response to continuous feedback allowed us to truly impact the site development process from alpha to beta. Such changes included introducing new filtering categories, improving checkout process, as well as clarifying the “About Us” page and the “saving” feature.Recruiting Users
Recruiting existing users to participate in interviews turned out to be a far more difficult challenge than we had expected. The response rate was extremely low even for active users on the site. Threadflip shared their email list with us sorted by active/inactive users and buyers/sellers. We began by emailing 100 from this list offering a $25 credit. Our response rate was so low (less than 3 users), that we adjusted the credit to $50 and emailed an additional 100 users, enabling us to reach a total of 7 sessions. The need to truly incentivize even existing users to participate was an eye-opening learning for us as students interested in feedback-driven lean models. In our previous work with later stage companies such as Bonobos and Warby Parker, whose brand names already benefit from significant press and hype, we had found it far easier to illicit responses. This difference in ability to garner resources for a later versus early stage start-up was very illuminating for us.We also quickly observed a vast difference between sessions with internet savvy/frequent online shoppers and those that were less so. Existing users of the alpha stage ecommerce site tended to be familiar with start-ups and online models such as ETSY, eBay and Pinterest. Among the new users, some immediately referenced these sites and their other online shopping experiences while others clearly were not accustomed to online shopping and found the model, picture feed and features (virtual closet, “saving” items, and sharing) to be more confusing than the internet savvy group that caught on quite quickly. This demonstrated for us the challenge in creating a product that offers a great user experience for a range of users and satisfies the power adopters’ desire for new and cool features but crosses the chasm to the mainstream users’ desire for simplicity and ease of use. We saw that concentrating testing more heavily on one group versus the other could bias feedback and recommendations; it is important for product managers to keep this in mind when recruiting and testing users and to understand a site’s target demographic at different stages.
Administering Tests & Interviews
We found that our execution of interviews and tests greatly improved over the course of the project. We quickly realized we often had begun with leading questions that led existing users to certain explanations for their behavior and that led new users in certain directions. Over time, we began to give less initial guidance but became better at probing more deeply and asking follow-up questions on the spot to pinpoint areas of confusion and causes of behavior.We also found that existing users required more specific questions in order to uncover motivators and barriers to certain behaviors while more general questions allowed new users to share initial insights and responses to their first experiences on the site.
Lastly, we found that repeated patterns emerged from even a small number of users. Certain critical pain points and areas of confusion came up continuously. For example, the tedious process of packing, describing and photographing items and uploaded images was a significant barrier to entry to selling on the site for many users. This repeated finding confirmed Threadflip’s interest in launching a white glove service for picking up and orchestrating the sale of items that they are currently rolling out with the site’s official launch.
Comments
Post a Comment