Experience | Robert Riedl Product

Agile / Scrum

As a certified scrum master, I have trained technical, product, and design teams in agile concepts and scrum processes, and have professionally used scrum for over 10 years with 18 products. As a product manager, I have collaborated with my teams to create scrum artifacts including product goals, user stories, prioritized product backlogs, acceptance criteria, definition of done, and burndown charts.

While there are great principles in the agile manifesto, for me the most important one is that the best architectures, requirements, and designs emerge from self-organizing teams. Ultimately the team is making the commitment for delivering the product and should be empowered to define how they accomplish this monumental task.

Automation

From my six years working with operations teams at Amazon, I've learned that it's best to create an automation plan from Day 1. Even if the plan cannot be achieved within the short-term, it still helps teams be proactive rather than reactive about improving efficiencies and not be caught 'flat footed' if they need to scale quickly to meet a new demand.

Below are two case studies where I've developed both tactical and strategic initiatives for automation: Scaling Publisher Outreach and Integrating the Human in an Automation Loop.

Scaling Publisher Outreach

At Amazon books, I worked in the Specialty Reading group that had initiated a program called Great on Kindle to highlight nonfiction books that had high quality characteristics such as high resolution photographs or supplemental information through the "X-ray" feature on Kindle.

The process of letting publishers know about issues that needed to be fixed to include a book in the program was manual: our account manager would collect the information from our operations teams, create spreadsheets of the issues, send them to publishers, and track their responses. This manual process took time and limited the number of publishers and books we could include the program.

Amazon already provided a tool called the Quality Issue Dashboard (QID) that showed publishers their book defects. I reached out to the tool owners and collaborated with them to create a plan to include issues from our program in the tool. I also worked with our operations, tech, and design teams to onboard our issues onto QID. Amazon Book's account and vendor managers were concerned about publisher and author confusion between book defects and issues for the Great on Kindle program, so I ran usability testing as well as three beta testing cohorts to ensure that users were not confused as well as to ensure that our team was ready both technically and operationally.

By automating the reporting of book issues for our Great on Kindle program to a self-service tool that publishers and authors were already familiar with, we had a 5X higher participation rate in the program, expanded the program to 3X more publishers and 5K independent authors, and the time our account manager spent on outreach for book issues was reduced by 90%.

Integrating the Human in the Automation Loop

Across the billions of transactions that pass through Amazon, some are fraudulent or not compliant with company policy. They can range from a fraudster using a stolen credit card to a criminal group laundering money. An Amazon group called TRMS provided internal businesses such as Kindle or Zappos with tools to automatically shut down known fraud patterns as well as tools for human operators to manually investigate new or ambiguous patterns.

I was hired by TRMS to create a new platform called Nautilus to provide core services for Amazon's businesses to perform manual investigations and allowed businesses to customize their investigations with their specific business logic and UX. In addition to launching this platform and two customized investigation tools, I defined the platform's strategy to eventually automate all manual investigations.

After speaking with stakeholders including process engineers, operation managers, and data scientists I concluded that complete automation would not be possible since fraud methods are constantly evolving. Instead, I developed the vision of a human-in-the-loop framework to integrate the speed and efficiency of machine intelligence for known fraud patterns with the adaptability and broad thinking of human intelligence for new fraud patterns. I created a long-term plan to achieve this vision as well as initial projects for my product team to launch to automate processes but also provide data scientists with more data about the manual investigation process. These initial projects included the integration of real-time translations into the manual investigation process to reduce dependencies on investigators with language specializations but also provide data scientists with localized data to help them build ML models for automation; and guided workflows, where the procedures investigators used were broken down into their atomic parts and integrated into the investigation process. This not only would help reduce operator errors but also provide data scientists with insight into the data points the investigators used as well as allow them to test different models with the investigators directly.

Leadership

I have experience in leadership roles for products, teams, and people. I've found that a singular, well articulated, and understood vision that resonates with organizational and individual values is a key component for successful leadership. Below are three use cases from my experience in leadership.

Nautilus: Leading a new product

Situation: The platform that my group at Amazon, TRMS, had provided for the manual investigation of fraud was problematic:

my tech team had trouble supporting the platform because it had been pieced together over several years and by different teams using outdated technology. As a result even simple changes were complex and prone to error.
users of the platform (investigators) found it to be unreliable, difficult to use, and slow.
developers from other businesses trying to build features on the platform found it difficult to use with only 40% agreeing that they knew how to integrate their content into the platform.
operations managers were frustrated by the time it took for our team to build new platform features and the time it took for their dev teams to build on our platform. As a result improvements for efficiency and automation were slow to implement.

To address these issues TRMS funded the development of a new platform, Nautilus, and hired me to be its product manager. I found that my tech team had low morale, a high churn rate (65%), and was consistently one of the lowest teams in job satisfaction based on an annual internal survey. I also found that investigators and operations leaders did not have confidence in the tech team to deliver a useable product.

Solutions: In addition to building a best-in-class platform, not only did I need to earn the trust of my stakeholders but also create buzz and confidence around the tool. I accomplished this by:

Meeting one-on-one with end users of the tools: developers building tools for the platform, and investigators tools hosted on the platform to understand pain points and their hopes for the new platform
Building a product team of former investigators who had used the prior tools and were known by operations teams
Collaborating with my product team to run a SIPOC with cross-functional team members to identify current usage of the platform and tools hosted on it as well as feature gaps and opportunities for improvements and automation.
Using the information from the SIPOC and one-on-one meetings to identify core values for the platform and articulate them in a vision document for the platform
Creating a weekly meeting to engage with developers of tools for the platform to share progress on platform
Working with a UX Designer to build high-res concepts of the new platform to share with team leaders
Engaging with change management to build and maintain anticipation for the new tool through a cadence of communications
Meeting with ops managers and internal business stakeholders on-site to share learnings from end-users, the vision document, and progress
Collaborating with my software development manager and program managers to improve our agile/scrum process to give the tech team ownership on commitments and estimation

Results:

My tech team missed the original ship date by three months but due to improved forecasting through agile, we met our revised launch date.
90% of the businesses that used our old platform had confidence in our new platform, Nautilus, to fund migration to it.
85% of developers migrating to Nautilus reported they had a clear understanding of how to work with the platform compared to 40% with the old tool.
Ops managers had the confidence in the new platform to provide investigators to test it
My tech team's churn rate was reduced from 65% to 25% and they were a top 3 team in job satisfaction.

Nancy Drew Mysteries: Leading a team

Situation: When I started at Her Interactive, we were a small scrappy product development team of three artists, one program manager, and one developer. We needed to grow the team to meet the cadence of releasing two Nancy Drew mystery games per year so I took on the role of Director of Product Development to build the team as well as establish processes for production and design. We did not have a dedicated game designer since we had always designed the games as a group and we kept this process even as we grew. But as our games became larger and more complex, team members were taxed more to contribute to the design and after making 11 games, team members were burnt out with the process and saw it more as a chore than an enjoyable activity.

Solutions: I revamped our design process where there was a single-threaded owner for all design aspects. I took on the role of Executive Producer and was responsible for all elements of design: logic, story, puzzle specifications. Not wanting to take all creative input from the team, I polled them at the start of the ideation of each game to find out what interests they had either from a content or technical perspective and then created three different options for the next game which we selected as a group.

The week before production began, I presented a pitch deck for the game describing its vision: mood, setting, characters, objective, new features, and user feedback I had received to justify my decisions.

Results: I was able to take away the drudgery of game design from the team but still incorporate their thoughts and ideas so they felt engaged with the product. The presentation of the pitch deck was a popular event that became ingrained in our culture and continued even after I left the company. We often complemented the presentation with food and decorations like King Cake for a game set in New Orleans. Sales and Marketing also leveraged parts of the deck for their own pitch decks.

Amazon: Leading people

Situation: At Amazon in the TRMS group, I hired a team of product managers with no prior product experience but who had experience using the internal products we were replacing. Because of issues with prior attempts at building these types of products, there was a high bar for them to succeed. I needed to set them up for success.

Solutions: In addition to weekly individual one-on-ones, I scheduled daily stand ups with the product team to share their wins, losses, priorities and questions so that I could quickly course correct any problems. I also modeled this behavior by sharing my wins and losses. In addition to these daily meetings, I set up weekly 'lunch time learnings' where I would share best practices and methods in product management. Lastly, I worked with my reports to identify career goals and a path for promotion.

Results: My team was able to deliver their products on time and within 2 years of reporting to me, 60% of the product team was promoted. The lunch time learnings were so popular that other members of TRMS joined the sessions regularly and three of them eventually became product managers.

Scaling

Scaling in technology often refers to mechanisms to manage increases in processing or data storage, sometimes called 'load scalability.' But there are other types of scaling such as maintaining a system, process, or product while extending to another geographic area. My experience in scaling has been about reducing costs or managing growth so that costs do not increase at the same rate as growth. This can be accomplished in several ways such as delegating processes to lower cost resources, standardization, and automation.

Below are two cases where I implemented mechanisms to scale for growth. See the section in automation for additional use cases.

Her Interactive: Reducing COGS

Situation: I was one of the first hires for a start-up called Her Interactive that had acquired the license to use the teenage detective book property, Nancy Drew, to make video games where the player would take on the role of Nancy Drew and investigate a crime by grilling suspects, finding clues, and solving puzzles. The first game was delivered on two CD-ROMs, but leadership wanted to reduce the game size to one CD-ROM to reduce our COGS and tasked me with finding a solution.

Solution: I did not want to reduce the game content as that would affect the quality of the product, so I looked at what assets accounted for the greatest size and found that almost half of the footprint was taken by movies of animated characters responding to Nancy's questions. As I looked closely at each animation, I noticed that the animations re-used the same core movements. So if a character waved once for one conversation and waved twice in another conversation, we would have two separate files for each response. I came up with the idea of creating a library of each individual movement that we could call up as many times as needed. So if a character waved one to 100 times in different conversations, we would only need a single file of a wave that we could render over and over again rather than render 100 separate files. I worked with my developer to create a proof of concept and found that there was no performance issues and we developed a workflow for animators to create 'direction sheets' on how to string the individual movements to create a full animation.

Results: We were able to get the game to a single CD-ROM which reduced our COGS by 100%. We were also able to quickly add more animations later in production with no additional cost to footprint except for the accompanying sound file.

Amazon: Managing Growth

Situation: When Amazon would expand into new marketplaces, my organization (TRMS) would need to hire operators with specialized language skills to perform risk investigations in that locality. These operators had a higher cost for their language skills, were often difficult to find at the volumes needed, hiring them often required the establishment of a new local office, and it usually took longer to ramp up the operators than forecasted resulting in a delay in launching a new marketplace or higher fraud costs. I was tasked with finding a way to reduce the dependency on specialized language operators so that TRMS could scale for geographic expansion.

Solution: I spoke with process owners and operators to understand what types of investigations required locality knowledge vs language knowledge and learned that less than 20% of investigations required locality knowledge. I worked with my product manager to design experiments across several languages to compare error rates between cases where non-English content was translated into English versus the same cases having no translation and found that for all but one language, the error rates were similar. Lastly, I worked with my tech team to determine the cost of adding in a just-in-time translation for cases to be localized and retooling our workflow system to integrate a 'language neutral' queue to route tasks to operators without language specialization.

Results: While we were not able to remove all dependency on language specialization operators either for locality knowledge or escalations, we were able to reduce the need for language specialization operators by 50%. This resulted in our ability to quickly ramp up to support geographic expansion.

Strategy

Sometimes product managers are fixated on the immediate problem that their products are trying to solve and miss larger opportunities or in other words, miss the forest for the trees. This can happen in organizations as well. Creating a broader vision or strategy for tactical solutions helps identify opportunities to broaden impact and make informed decisions about current and future investments.

Below are two examples where I helped develop define tactical approaches within a broader strategic vision.

League of Women Voters: Saving Trees

The League of Women Voters has identified the conservation of mature forests as a primary tool to mitigate climate crises. One of the local chapters in my area was in the planning phase to create a campaign to stop the immediate auction of a state owned 100 acre parcel for timber harvesting.

When I started volunteering on their leadership team and looked through documentation about mature forests, I discovered that the state owned over 6,000 acres of mature forests that were unprotected. Moreover, an additional 14,000 acres of unprotected mature forests were owned by the US Government and private entities. I wrote a strategic document advising that a 3-pronged strategy for state owned, US owned, and privately owned mature forests be considered to protect all of the forests in the county rather than a small parcel.

The League pivoted their approach to focus immediately on protecting state owned lands and lobbying efforts were redesigned for this outcome.

Amazon: Integrating Ownership Signals into the Store

While working on a project to provide Amazon book readers a view of all of their acquisitions across channels (store, library, subscription) and formats (eBook, print, Audible, audio), I realized that while the digital bookstore did an excellent job facilitating a book purchase, it did a poor job communicating with customers post-purchase. For example, user research revealed that readers often would return to a book's "detail page" to find other books similar to it but all of the primary content in the detail page focused on selling the book again. It would be as if a customer bought a book at a local book store and came back and said, "Hey I just bought this book and found it interesting" but the bookseller responded by trying to sell the book again.

I talked to owners of different Amazon book products such as Audible and the subscription service, Kindle Unlimited, and affirmed that they also had issues with this approach. As a result, I collaborated with them and my designer to create a strategic vision document (PR/FAQ) for the systemic integration of ownership signals into the store to support an improved post-purchase experience. I also included a tactical roadmap outlining 15 experiments to both validate the approach and establish an ownership foundation in the store such as providing easy ingress to read eBooks or a library of all book acquisitions from which readers could select certain ones for book recommendations.

The proposal was backed by over 15 internal organizations, was funded at a VP-level goal, and was a forcing function to address a significant technical debt with book acquisition data.

User Research

I have designed, executed, and analyzed over 100 quantitative and qualitative user studies employing methods such as Kano, mental models, surveys, A/B testing, usability testing, beta testing, and focus groups. I've also collaborated with in-house and third party researchers for large scale studies.

Below are use cases where user research helped shape my products (and avoid some errors).

Kano: Saving Nancy Drew's Notebook

Situation: When I was the Executive Producer at Her Interactive overseeing the development of the Nancy Drew mystery video games, I was always looking for ways to streamline our production process including improving or removing game features. One feature we had introduced early on in the game series was "Nancy's Notebook" that had a running commentary of information the player, as Nancy, had discovered during gameplay. I introduced this feature as a response to players who said that they sometimes lost track of the narrative after leaving the game for a while. We later complemented the notebook with a checklist of tasks to perform.

I had introduced an analytics system into the game to track usage of features and knew that the checklist was used more often than the notebook and that most players only read about 40% of the notebook's entries. I also knew from beta testing that the notebook feature was not a 'delighter' since most people found it 'OK.'

As a result of this data, I planned on removing the notebook feature to free up design and production time for other features; however, I wanted to perform one last analysis to ensure this would not be disruptive to the players.

Solution: In our beta testing survey, I included Kano questions about the notebook to ascertain how players ascribed importance and value to the feature. I also asked testers how they used the notebook.

Results: I found that the notebook was a 'must have' and was an expected feature of the games. While players admitted that they did not use the notebook all of the time, they liked having it during the game to view Nancy's thoughts which in turn would sometimes spark their own investigations and thoughts about the case. As a result of the testing, we kept the feature.

Mental Maps: Justifying Page Numbers

Situation: When I worked at Kindle, I found that a majority of books did not have page numbers but instead used Kindle Locations which were a measurement of the amount of content in the book. This caused reader confusion because an eBook whose equivalent print book would have 200 pages would have 20,000 Kindle Locations. I was drafting a proposal to create a synthetic pagination method that would be applied to any book without page numbers and wanted to pre-empt a challenge from my leadership questioning the need for any page numbers since Kindle offered other ways for readers to determine their place in a book: time left reading for the chapter or book, % complete, and a progress bar.

Task: I wanted to understand how readers used page numbers so I interviewed Amazon workers, colleagues, friends and families about their reading process using the mental models method. While not a representative sample, this would provide at least a signal and insight into reading behaviors and preferences which I could turn into a formal study if leadership needed greater confidence in the findings.

Results: Applying the mental model technique, I asked readers to describe their last reading event and prompted a question about the method they used to determine where they were in the book. I found that half did not use page numbers and half did. When I asked the readers who used page numbers why they did not use the other methods to determine reading progress, they responded that either they did not trust the computation for reading time or that the progress features in Kindle were overinflated because it included back matter. I shared this information with leadership who felt the anecdotal information was sufficient to maintain page numbers.

Usability: Getting Customers What They Need

Situation: At Kindle, I was managing a product to show independent authors quality issues with their books. My UX designer created mock ups in Sketch but I could not book her time for usability testing because the project was too small.

Task: I reached out to the group that managed independent authors and discovered they had a group of authors who were available to review upcoming products. I ran usability tests with a sample of these authors where I sent them an interactive PDF of the mocks and asked them to perform tasks.

Results: I found that the messaging about image quality issues did not provide sufficient information for the independent authors to locate the image. In the mocks and with publishers, we gave the page number of the image to help them find the issue; however, this did not work with independent authors because their source file (usually word docs) had different page numbers than the epub file that either they generated from the source doc or that they had a third party to create. They asked for the first sentence preceding the photo so they could search for it to locate the photo with the quality issue.