Case study

Voice communications in gaming today lack the mechanisms and support to provide safe interactions between players.

Screens of Safe Voice (OTO) before the aquisition.

In essence:

• Unity is primarily targeting Moderators that are employed by game studios, but Moderators can also be volunteers from a game’s community.
• Moderators need to review countless numbers of incoming reports and are therefore constantly exposed to toxicity.
• With the current solutions on the market, game studios struggle to find the most efficient way to collect evidence to help Moderators make the right judgment.

Overview of
the Design Process

Infographic of the double-diamond design process.

Our work-in-progess Moderator User Profile.

The Opportunity

In summary, Unity has a unique chance to empower moderators by enhancing its Machine Learning capabilities. Moreover, Unity can distinguish itself in the market by providing audio visualization, a feature that sets it apart from competitors.

Screenshots from competitors.

Defining the Solution

After interpreting all of our insights from the discovery phase, we aligned user needs and problem into action points to focus our energy on what will bring the most value.

A/B Test

While generating ideas to provide Moderators with tools to refine the Machine Learning, we decided to test 2 solutions to align the team and validate assumptions. To save valuable time, we decided to conduct an internal unmoderated A/B test to better understand which feedback flow to implement with 14 participants, including 3 community managers at Unity.

Key findings:

• Over 70% of users preferred design B. The design was more intuitive and used less clicks than design A.
• The current design layout didn’t meet users expectations. Users expected to view audio tracks as the first thing they would see when landing on the page.
• Users were experiencing issues with the Figma prototype, such as the clicking area and some time delay.

Screenshots of both flows used for A/B testing.

Alpha Launch

After the A/B test results, we launched the product with our first customer (Hi-Rez) incorporating design B. We also decided to wait and gather more insights from users before making any major changes to the current design layout.

The Results

• Analyzed 18K sessions and detected over 10K sessions with toxic moments in the first 45 days
• Enabled moderators to review 15-minute-long games with multiple players in under a minute
• Identified 2.8K sessions with audio disruptions, highlighting a previously unknown player experience issue

“Safe Voice gives me visibility into what’s happening in my game, not only from a trust and safety perspective, but also from a general gameplay perspective. I’m a big fan of the extra data points that we get in the Safe Voice dashboard. The tags help us understand what to look for when reviewing an incident.”

—Tony Jones, Lead Producer, Hi-Rez

Learnings and Reflections

Throughout this experience, I discovered that it's not because someone previously worked with designers necessarily comprehends the design process. When I joined the team, they already had existing designs, but the team was not aware that each change request required restarting the design process to ensure we addressed the right issues effectively.

Following the product launch, I organized a workshop session with my team to present the Design Process and how we collaborate with cross-functional teams. By the end of the session, my peers left with a clear understanding of the significance of the design process and why it is essential for successful outcomes.

On another note, striving to create responsible algorithms while challenging our biases was an invaluable aspect of this journey. I actively encouraged the team to consider gender as a multifaceted and diverse concept, avoiding reinforcing binary perspectives in the machine learning process. There is room for improvement, and I believe we can do better at inclusion by protecting all gender identities from toxicity.