Skip to content

Video Games TUG

Video Games Tableau User Group 20 Mar 2026

Level 3 : March Madness

1h 25m 15,743 words transcribed

Summary

The March 2026 Video Games TUG, a bimonthly virtual user group for people working in the games industry or hoping to get into it. Louis Yu of 2K Games, a newly named 2026 Tableau Visionary, and new co-lead Dawn Harrington of the Tech Tips Girl blog open the meeting. Louis recaps the GDC keynote by former Blizzard chief creative officer Rob Pardo on building a lasting game: make a game the team loves, seed ideas from the whole team, and solve the core gameplay loop first with a minimum lovable product.

Jeff Waldron, who leads online platform engineering at Gearbox Software, reprises a GDC talk on how Gearbox load tested its online platform for a major launch. The platform, mostly Go services on MySQL and Redis, had never seen launch-level traffic, so the team built one load test to rule them all: replaying real player behaviour checked with Wireshark packet captures rather than synthetic load, and leaning hard on observability so any problem could be diagnosed in about two minutes. Jeff covers the vertical slice and the dynamic events that ate the most time, why the k6 scripts became a brittle application, Redis as the biggest scaling bottleneck and the move to serverless Redis, keeping cost in view, and the fifteen-month campaign that ended in the smoothest launch Gearbox has had. The lessons: treat the load test as an application, make sure the team owns its tools, and make cost observable.

Sahana Subraya, Director of Product for Dragon City at Socialpoint, presents "Data Talks to You", a guide for product managers on deciding with data: you don't need to be a data scientist to find where users drop out of a funnel, size the impact of a fix before building it, read retention and market size together (a Game A vs Game B case shows relative metrics only mean something next to absolute scale), measure success with pre/post comparisons and A/B tests while avoiding tests that are too short or too small, learn enough SQL to pull your own numbers, and never trust an aggregate that hides the segments underneath. Q&A covers comparing success across very different kinds of games.

Louis then highlights Tableau Desktop Free Edition and video-game entries from a recent Makeover Monday, Dawn previews Tableau Conference 2026 in San Diego, and Louis closes with the Games Night Viz community project.

Listen to this TUG

Prefer to listen? This meeting is also an episode of the Tableau TUG recordings podcast.

Key points

  • Build a game the team loves, involve the whole team in ideas, and nail the core loop first: a minimum lovable product, not just a viable one.
  • Load test with real player behaviour, verified with packet captures, rather than generic synthetic load.
  • Invest in observability so a problem during a load test (or a launch) can be diagnosed in minutes.
  • A load-test harness that grows complex is an application, and needs to be engineered like one.
  • At scale, small inefficiencies add up: Redis was the biggest bottleneck at millions of simulated users.
  • Make load-test cost visible, so the team optimizes where the money actually goes.
  • Find where users drop out of a funnel before deciding what to build.
  • Relative metrics like retention only mean something next to absolute scale.
  • Define success metrics before you ship, and avoid A/B tests that are too short or too small.
  • Learn enough SQL to pull your own numbers, and look beneath the aggregates.

Jump to a chapter (33)

  1. 0:00 Welcome to the Video Games TUG
  2. 0:22 Meet Louis Yu and Dawn Harrington
  3. 3:16 What the Video Games TUG is
  4. 4:33 GDC 2026: Rob Pardo on building a lasting game
  5. 8:46 Jeff Waldron: launch load testing at Gearbox
  6. 10:09 Gearbox's online platform
  7. 13:39 Goals and constraints
  8. 15:02 One load test to rule them all
  9. 17:15 Packet captures and observability
  10. 19:45 The load test in five chapters
  11. 25:04 Working with k6
  12. 26:18 Scaling, and the Redis bottleneck
  13. 30:29 Cost
  14. 31:47 Fifteen months to a smooth launch
  15. 34:32 Troubleshooting in two minutes
  16. 38:11 Lessons learned
  17. 39:26 Q&A
  18. 41:31 Sahana Subraya: data talks to you
  19. 43:24 Using data to understand users
  20. 46:07 Finding the drop in a funnel
  21. 49:37 Sizing the potential impact
  22. 53:38 Retention and market size: Game A vs Game B
  23. 56:46 Measuring success: pre/post and A/B tests
  24. 1:00:42 Common A/B testing mistakes
  25. 1:03:01 Why product managers should learn SQL
  26. 1:05:50 Aggregates hide the story
  27. 1:07:53 Key takeaways
  28. 1:08:18 Q&A: comparing different kinds of games
  29. 1:10:20 Tableau Desktop Free Edition
  30. 1:12:18 Community, and Makeover Monday video game vizzes
  31. 1:17:33 Dawn Harrington: Tableau Conference 2026
  32. 1:21:50 The Games Night Viz community project
  33. 1:24:30 Wrap-up

Transcript

0:09 Dawn Harrington: So welcome everybody to Video Games Tableau User Group.

0:13 Very excited. This is my very first one of the… I'm attending.

0:17 Um, so I have high expectations, Louis.

0:20 High expectations.

0:22 Louis Yu: Well, now, now you're giving me pressure a bit.

0:28 All righty. Let's keep… Okay, so let's do a little bit of intro. So, uh, I'm Louis. I am a manager at 2K Games where I'm part of a team that does, like, a lot of the dashboards. Uh, so we… The, the team itself is called the Visual Analytics team.

0:45 Uh, I'm also a Tableau Visionary. So Dawn and I, we have just been crowned the 2026 Visionaries for this year, so really excited and happy to kind of lead the committee and grow, uh, and nurture them.

1:01 So, uh, expect more great stuff from us, uh, from, from now on.

1:07 Um, I'm also a co-lead for a Games Night Viz community project. So if you are keen on practicing your data viz, uh, skills using video games data set, uh, we have a couple of slides at the end of the, uh, call, and I will walk you through that.

1:23 Um, apart from that, I am a Asia mentor for the Tableau Sandbox, Sandbox Buddy Program, which is ran by Prasann and Toray.

1:33 Um, and I'm also a podcaster for Secrets of the viz, where I kind of talk to authors on their viz and kind of get behind their creative processes and how they actually created wonderful viz that you've seen on Tableau Public and, you know, other platforms as well.

1:48 Um, my partner in crime, uh, Maha, who started this whole talk with me, is unable to join the call today, but he is, um, from the parent company, Take-Two, uh, so he's kind of like my counterpart over there. Um, and then we have our latest and precious, uh, co-lead, Dawn, who has joined us this year, but I'll leave her to give her intro by herself.

2:15 Dawn Harrington: So yeah, uh, Tech Tips Girl blog, um, is like the, my main site where I give tips and tricks on Tableau, uh, Alteryx, and some other Microsoft Office stuff. Um, I've been doing this for a long time, so it's older versions of Microsoft Office, I would say.

2:35 Um, like tips like, you know, that you can block like, um, Outlook invites so people can't forward them or they can't reply all to an email. Like that's another quick tip that I think everyone should look up how to do.

2:48 Um, but that's in my blog site. Um, I've been a presenter at TC '24, TC '25, and now TC '26. Oh my goodness. Um, and now I'm also the Games Night Viz co-lead, so I've been bugging Louis for a while, being like, "Hey, hey, hey, can I be a part of this?" So…

3:08 Louis Yu: Yeah. Yeah. Just-

3:09 Dawn Harrington: Please

3:09 Louis Yu: … going in with a full package, Games Night Viz and Video Games Talk, so there you have it.

3:17 All righty. So who are we? We are a Tableau user group that is focused on the video games industry. So this is actually curated for anyone working in the industry or looking to get, looking to get into it. So we wanna kind of facilitate and nurture a community that can kind of learn from one another. So anything to do with, you know, things that are happening in our industry and, you know, how people are implementing, uh, analytics or AI stuff, uh, in, in our industry.

3:45 So it will be a virtual first meetup, uh, that is gonna be bimonthly.

3:50 Uh, we do have a LinkedIn group, so you can actually go to that link, uh, tiny URL, and it'll point you to the LinkedIn group.

4:00 Uh, and you know, do engage with us over there. Uh, we'd love to chat with you on that.

4:05 Uh, in terms of the cadence of the meetups, uh, we are in March, so the next one will be in probably in June, followed by September and December. So the June one would be kind of a post-Tableau Conference coverage because I know not everyone is going to Tableau Conference, but hopefully you're going. Uh, we have, we do have some slides at the end to kind of talk about Tableau Conference, so do stay to the end and Dawn will take you through that.

4:33 Uh, in terms of our industry cadence, right? Uh, we have just passed the GDC, the Games Developer Conference every year. Uh, it's kind of one of the biggest one, and this year they actually rebranded quite a bit. Uh, Jeff was there, uh, last week, uh, to give a talk, and he's here today to redo that, uh, amazing talk that he had over there. Uh, in April we have the BAFTA Games Award and then followed by some other really huge e-games, uh, events like Evo, uh, Gamescom and so on, so forth, right?

5:05 Uh, and we'll continue to kind of give coverage on those.

5:08 So if you have been on, uh, at GDC or you haven't been, uh, there was a keynote that kind of talked about the how to build a lasting game, which I thought was really interesting. Uh, and this was actually done by Rob Pardo, the ex-Chief Creative Officer of Blizzard.

5:28 So he has a lot of experience in building games like World of Warcraft, StarCraft, uh, Hearthstone, and what other games he built? Uh, Overwatch as well.

5:38 Uh, and he talked about basically three pillars, right? Uh, that, uh, a lot of studios are kind of not doing it right. Uh, and I think this is, this is very, it, this is a, a uprising kind of trend in our industry, uh, that I think should be considered, right? So he talked about, you know, starting with the kind-Team itself, right? You build a game that the team loves and not what you think the player wants.

6:07 And he basically, uh, you know, talked about this in a way where, uh, the play test has to get your developers and designers and people in your team excited about. If they play the game, if, if they play a half-polished game, uh, and they are more excited about that than, you know, something that's out there in the market, then you have kind of like a winner.

6:30 Uh, the other thing he also talked about is, uh, involving the team because, uh, everyone has a lot of different ideas, right? Creative, creativity comes from everywhere, and what he did was that he created this virtuous cycle of ideas and development, uh, through what they call by seeding. So everyone comes up with different ideas, and they kind of put Post-It notes to the different ideas to vote, you know, what should go into the game and stuff like that.

6:55 Um, and then the third thing is, uh, he talked about, you know, tackling the hardest challenge of a game, which is the core gameplay loop, right? Uh, so you wanna have like a minimum Lovable product. Uh, I, I know that, you know, in, in product development, they always talk about minimum valuable, uh, minimum viable product, which is like the bare minimum that you should go to the market.

7:18 But the most, the minimum viable doesn't mean that it's the optimum one, right? So which is why he talked about minimum Lovable product, meaning that if your team loves, you know, something that is unpolished and stuff like that, you know, you can continue to iterate it. There's the passion to drive that. And we've seen this happen very recently in a game that some of you might be very familiar with, uh, called "Clap Skill." And I managed to find a data set that talks about how many games, Game of the Year awards that was given to, uh, "Clap Skill." And officially, uh, as of March, 2026, it has overtaken "Elden Ring," which previously held that, uh, title.

8:02 And some of you might feel, or some of us might feel that, you know, uh, Game of, Game of the Year awards sometimes might be biased and stuff like that. But if we take a look at player choice, they are also at the top, right? So this, this actually shows us that, you know, if the studios and developers actually put passion and love into the game itself, they, they were building the games that they want to play, not what they feel that the player wants to play.

8:28 And this has helped them, uh, and this has resonated very well with the community. And for those that have not played "Clap Skill," I strongly, strongly encourage you. So Nick, if you are looking to buy a PS5 and you don't know what games to play, "Clap Skill" is the number one game that I would recommend. Right. So without further ado, uh, I'll leave it over to Jeff, uh, who is the technical director at Gearbox Software, who is gonna talk to us about how, you know, uh, the team actually did load, uh, launch load testing to help optimize networking and, uh, matchmaking and stuff like that.

9:04 So over to you, Jeff.

9:08 Jeff Waldron: I also apologize if you hear some chirping in the background. I got some baby chicks on Wednesday, and just like a normal baby, they are very loud.

9:19 Louis Yu: That is pretty random.

9:22 Jeff Waldron: Yeah, sorry. They're, uh every… Hopefully noise canceling captures it and you don't quite see it, but always that risk.

9:32 All right.

9:33 Louis Yu: Perfect.

9:34 Jeff Waldron: All right. Uh, thanks for that introduction. As mentioned before, my name is Jeff, and despite struggling to present this through technology, I am a technical director at Gearbox Software.

9:44 I'm in charge of engineering on our online platform. I've been at Gearbox seven and a half years. I started on "Borderlands 3" and have been working on shipping games on our platform ever since.

9:55 I'm really excited about what I get to share today, but first and foremost, I wanna note, I'm, I'm not the main character. I get to work with an amazing group of people, and it, it is their accomplishments that I get to share with y'all today.

10:09 What I'm hoping you walk away with is how Gearbox approach load testing. We've had some successes and some stumbles, and overall, what kind of concerns and constraints we had as we went through this process.

10:21 There is gonna be some technical discussion here, but the, the technical stuff is pretty light. This is mostly a process, uh, related thing to give you an idea on load testing, what it is, why we're doing it, less about the technical issues that we dove into here. So hopefully, if you've never heard of load testing, you'll be able to get an idea what we did, why we did it.

10:41 Now, GDC asked me to include a note about silencing your cell phones. I left it in here just as another reminder. This is a condensed talk from what I gave at GDC. If you thought this was cool and interesting and you wanna see a talk that goes into way more detail, uh, the recording of that talk should be up on GDC Vault.

10:59 In case you've never heard of an online platform before, I wanna talk quickly about what Gearbox's online platform is, which is publicly branded as Shift. So if you've ever seen that logo or branding before, uh, that's what it's talking about.

11:11 But it starts with awesome and amazing Gearbox games, uh, like "Borderlands 4." And actually, the Shift platform was first introduced in "Borderlands 2" and has been in every Gearbox game since.

11:22 And we have our amazing players that without them, none of this would work.

11:26 And this online platform is in between, as we help to connect players to the game and to each other. We have all sorts of features, whether that's managing accounts and all of the legal requirements and challenges globally, to a cross-platform friends list that players can work on.

11:39 So what my team does is we focus on integrating these features into our games so that things are smooth, seamless, and work well, and then we help engage with players, make sure they have their friends, their matchmaking works well for them, uh, we have cross-play multiplayer, and everything works well.

11:56 Let's talk a little bit technically about the platform.

11:59 We start with our players, and when they play our game, they make HTTP calls to various APIs. We have about 20 publicly facing APIs that our game clients will talk to, but in total, we have 56, as there's a lot behind the scenes that goes on These are running in Docker containers, mostly in Golang. We have a couple of older applications on Ruby on Rails.

12:20 And we are 100% cloud, 100% AWS, and Fargate is our compute solution that we use.

12:27 Generally speaking, we have a lot of MySQL databases for our persistent relational data, and Redis for our more ephemeral caches. We have a couple of other things floating around, a DynamoDB or two for a non-SQL database, but for the most part we're MySQL and Redis.

12:44 I'm gonna start by talking about some background information, why we're load testing, and the challenges we saw.

12:49 It's been a while since we've done a large scale load test. The last one we did was for Borderlands 3, and we've made a lot of improvements on our platform since then. Switching to more serverless options, and generally just dealing with all the upgrades that Amazon has, uh, delivered since then.

13:04 Borderlands 4 also has a lot more online features than previous titles in the Borderlands franchise before it, meaning that those features had not actually seen launch level traffic in production before.

13:17 And we know these games are popular with our players. We'll release a DLC or an update, players will come back, and we wanted to have a platform that could scale up to those players, but also scale back down, uh, in between content drops.

13:30 All of this led us to desiring a more rigorous load testing approach. We wanted to really give this a lot of focus and make sure that it would work well.

13:38 So we made a plan.

13:40 And, and like a lot of good plans, we had some solutions criteria to explore.

13:44 The first is scaling the load test. This was a pain point for us in Borderlands 3. We spent probably about half of our time investigating issues with the load test itself and not the platform, which means half our time was spent on something that wasn't directly benefiting players.

13:59 Now, this is never gonna be zero, but we wanted to reduce that from 50% quite a bit.

14:05 We also wanted a load test that was accurate, which feels like a no-brainer. Like, why would you do a load test that's not accurate? But because of some assumptions and mistakes that we have made in the past, we focused too hard on load testing trees, and we missed load testing the forest together.

14:21 And lastly, proactive observability. It's really great if you have an online platform that can respond to issues, but we wanted to focus on what issues were coming up so we could avoid them happening altogether.

14:33 And we had some constraints as well to work through.

14:36 Like pretty much everything, time and money are this forever constraint that we have to work around.

14:41 But we also decided to do our load test later on in the game's development. We did this so we can have a more accurate load test.

14:48 But by doing so, it meant we were doing this load test while the game was getting shipped, and all of the crazy chaos and challenge that comes with shipping a game.

14:57 With this plan, we decided to build something new and had a different approach.

15:03 And that approach has two founding principles that shape this load testing effort. The first is one load test to rule them all. We wanted a single load test that did everything a Borderlands 4 player does. The idea was to give us the accuracy that we wanted by simulating the players.

15:21 Next was observability and equal importance to scalability with a two-minute benchmark to identify issues. I'm gonna talk about observability a ton, but first I wanna talk about that first pillar, one load test to rule them all, and I can do that best with a picture.

15:36 First, I'm gonna talk about what we did in Borderlands 3.

15:39 We used a framework called Locust, and that load testing script would talk to one API, and then that API would talk to a MySQL or Redis. It would talk to other APIs, uh, whatever it needed to in order to perform its functions.

15:52 And we recreated this over and over and over until all of our APIs were represented and load tested.

15:58 For Borderlands 4, we had a single K6 script, and I'll talk about K6 a little bit on the next slide, and this one script talked to absolutely everything a player talks to, all the different APIs, all the different workflows, and then those talk to all the other APIs and MySQLs all behind the scene, representing one test to rule them all.

16:19 Fundamentally, we wanted to simulate lo- we didn't wanna simulate load. We wanted to simulate players. Think about players, how they work, how they interact with things, and this represented how we framed this load test by s- thinking about player actions. And that's actually how we represented this in JIRA. We had a matchmaking epic and a boot up the game epic, uh, to represent these player actions as opposed to an accounts API or a matchmaking API.

16:44 We found 24 different unique player actions that we represented in this script. To give you an idea on the scope of one of those actions, I'll talk about adding a friend.

16:53 That starts with making a search for a friend, so you type in a display name, you get some results back.

17:00 You grab one of those results, you send them a friend request, and then that person either accepts, declines, or ghosts that friend request. That whole workflow encompasses one of these 24 actions, and that's one of the simpler workflows.

17:15 You can think you know what your game is doing, or you can know what it's doing, and we used packet captures via Wireshark to make sure that our load tests were accurately capturing what the game client did.

17:27 K6 was the framework we used. It's written in JavaScript, and it's very common. There are a lot of big customers that use K6, and we wanted something that was tried and true and battle tested.

17:38 To give you an idea, when we created one load test to rule them all, we were able to simulate 12,500 users on a single container. If you're not sure what 16 CPU means in this context, that's because we use AWS Fargate, and that's the metrics that it uses as well.

17:53 So if you use Amazon, that'll give you an idea on, uh, how many users we were able to simulate.

17:59 I'm gonna talk about observability quite a bit. In case you've never heard about observability, it's sometimes, uh, written as o11y, which is because observability, the word, starts with the letter O, has 11 characters, and then ends with a Y.

18:14 It's also sometimes called O11Y if you've heard that phrase.

18:17 And there's a lot of examples of observability here.

18:21 I won't dive into them really in too much detail, but generally if you have something that gives you insight onto how your platform is running and operating, it's considered observability. And we wanted to do better here. We wanted to level up and make sure that we could have clear insight into how the platform was working.

18:39 But some of that is with tools. And if you have this awesome, great tool, but nobody is using it, it's not really providing value. So we wanted the team to be leveled up using these tools, which led us to a focus on observability of equal importance to scalability.

18:53 We came up with a two-minute benchmark on how long it would take to identify an issue with the idea that you would have two different one-minute tools or one-minute screens, as these are often tools that are digitally presented.

19:06 If you have an alert that is fired, someone has reported an issue, the idea is any engineer could hear that information, go to a specific tool, spend one minute on that tool, figure out the info needed, and then jump to a second tool to actually find the issue also within a minute, leading to two minutes to identify an issue.

19:27 It's worth noting though, we really wanted to focus on proactive observability, find the issues before they showed up. It's just a little harder to create a benchmark for something like that without saying absolutely nothing breaks. And saying absolutely nothing breaks is a pretty tough metric to work and test around.

19:46 So I'm gonna talk about this load test in five different chapters based off of, uh, Elements load test. The first was our vertical slice that we performed.

19:54 We were doing something new and different, and we wanted to make sure that this different approach was still valid, or, and we wanted to not throw a bunch of eggs in one basket here.

20:05 So we borrowed from game development. We created s- a vertical slice for load test. If you're not familiar, a vertical slice is generally where you represent a part of the game completely and fully. So we wanted to do that with our load test.

20:19 We grabbed boot and sit, the first thing players did, and we wanted to fully represent that feature, scale it to a very high CCU, which for us was 3 million CCU with a 15-minute ramp-up rate.

20:30 And we wanted to see, is this gonna work? Our one load test to rule them all, our big observability focus, everything we were trying to do, we wanted to see how does it work.

20:39 And this is how it works.

20:41 Now, I know this graph doesn't mean too much, but I'm gonna walk you through, uh, what these different sections mean.

20:46 Firstly, the different l- uh, colors each represent a unique HTTP call.

20:53 Our 15-minute ramp-up time is represented here, and there's a lot that happens when a player starts up the game. They authenticate, they get their news, they get their playlist, they get their entitlements, they get their information, their friends, all of that. There is a ton going on when a player starts up the game.

21:07 And when a player sits at the main menu, there's pretty much nothing going on except for these two.

21:12 We have telemetry that we collect on our players and what they're doing in our game. Now, for this load test, we're collecting telemetry that says they sat at the main menu and did nothing.

21:21 Not very exciting, but it's still data that we would submit to our data pipeline.

21:26 Next, we have an API that is an HTTP long poll that is gathering messages for the player, matchmaking information and things like that.

21:34 Well, at this point, there is no messages or no information being given. The client wouldn't know that, and it would still make this request for that info, and we wanted to simulate that.

21:44 And we ran this for an hour after we had the ramp-up time, and then finally ending with a 15-minute ramp down.

21:52 So as we came out of this vertical slice, we wanted to see how are things going, and we were pretty happy here. The strategy was working. We were able to hit a pretty high CCU in a pretty short amount of time, and we represented a ton of calls. If you recall from that previous graph, there's a lot of different colors, and we had represented so much here.

22:11 So we said, "Yeah, this is great. This is working." Moving forward, we wanted to continue doing this. We were onto something. This was the right path, and we were wrong.

22:20 We had some incorrect assumptions that we made early on that hurt us later on.

22:26 At this scope and scale, we were comfortable with k6. Even though we represented a ton of information, we were pretty happy.

22:33 But the challenge came with the call count complexity versus the workflow complexity.

22:39 Now, we know call count and complexity weren't gonna match exactly, but we assumed if most of the HTTP calls are represented, perhaps most of the complexity is represented as well, which was absolutely incorrect, and it really hurt us when we were scaling the load test client.

22:57 So we started with one action, and we wanted to add the remaining 23 actions in k6, representing everything there. Some of these were simple, like adding a friend. But matchmaking and all the different things that could happen, a player dropping out, uh, somebody disconnecting, and all of those workflows being handled was very tricky.

23:18 That and working with k6 provided challenges we didn't anticipate.

23:22 So I asked the team what were some of the challenges they had working with k6, and I got a long list of issues. I won't dive into them all in particular, but the big takeaway is this was a death by a thousand cuts for us.

23:36 We were making incorrect assumptions. We thought things would be easier than they actually were, and this ate up a lot of valuable time.

23:43 Now, it's one thing to eat up time because of an incorrect assumption, but we're doing this getting close to shipping a, shipping a game, and that time is way more precious.

23:52 That being said, we did get this to work, and here's an example of one of the final, uh, load test runs.

23:59 In the middle here, uh, you see there's a lot of players doing more things. Now they're matchmaking, they're adding their friends, they're communicating with other players, and that's what happened for the bulk of the load test.

24:10 We also, towards the end of the load test, changed it so users would ramp down and, or sorry, restart the test instead of ramp down. We did this so we could run a soak test over a longer period of time to catch more issues. Now, in this screenshot, this is actually not from one of the soak tests, but in theory, this is what players would be doing.

24:29 So as we worked on scaling the client, our vertical slice messed up here. It didn't capture the full load test complexity. And because it didn't do that, we invested a lot of energy into a tool that wasn't right for the job. Trying to get the dynamic events is really what took us a lot of time, because we had to have a Redis that had all the load test clients talk to each other, and that was tricky to implement and get right.

24:51 And trying to diagnose that during a load test was also very challenging.

24:57 There was a lot of trial and error, and our estimates were often wrong, making it very difficult to plan around.

25:05 Overall, thinking about k6, it all founds on that one principle that we had for one load test to rule them all.

25:12 User simulation versus load generation. And remember, we wanted user simulation here. That was our goal with the load test.

25:20 But k6 is actually not good at that. It's very good at load generation, and that's how it's been used in the past.

25:25 And if you think about what happened with Boot and Sit, where players are just starting their game client, really load generation and user simulation are more or less the same thing. But as we added dynamic events, that's where it started to drift, and that drift adds a lot of challenges.

25:40 I also have called this a load testing script, but it's doing everything a game client does. That's not a script, that's an application. We should have thought of this like an application using framework and tools that are just like the applications we develop.

25:54 But at the end of the day, it worked.

25:58 We did have one load test to rule them all. It did accurately represent player traffic.

26:03 It was just brittle.

26:05 Engineers are no stranger to taking a square peg and getting it through a round hole. That's what we did here. Not ideal, but it can function.

26:14 So we took that test, and we were using it to help scale the platform.

26:19 Now, before I talk about some CCU numbers, I do wanna talk a little bit about being over-prepared. And a CCU, if you haven't heard the term before, means concurrent users, and that is what we were targeting. We started with three million concurrent users for our load test, and then we dropped that down to two million, but we kept the fifteen-minute ramp up.

26:37 At this point in our load test, we were closer to release. We could have a better idea how many users we would have, but we also wanted to be overly aggressive here.

26:47 You don't wanna be under-prepared and have a bad launch for players, but if you're over-prepared, you're just spending way more resources and energy fixing problems that will never show up. So you wanna be just a little bit over-prepared here.

27:02 And as we started scaling, easily the biggest challenge for us was Redis.

27:06 Hands down, we spent more time on Redis scaling, and that's because a horizontal or a vertical scale would take thirty to forty-five minutes. So if you needed to do both of them, that's an hour to an hour and a half for scaling.

27:19 And there's compute optimized Redis, memory optimized, network optimized, all sorts of things that maybe don't matter at a small scale, but when you have millions of users you're simulating, those optimizations really add up.

27:32 And some of these Redises didn't have percentile-based usage. So like memory, we could see, okay, you're using seventy percent memory. We can kind of figure out if we add more nodes or add more users, what that looked like. Network bandwidth was not the case. It just said throttled, not throttled, making it really difficult to try and guess and predict.

27:50 All of this led us to switching to serverless Redis, and this is the ECPU from one of our serverless Redises during our launch Borderlands four.

28:00 In the beginning, there's not a lot happening as, uh, there's users that are still there playing Borderlands three and Wonderlands and Homeworld three.

28:07 And then boom, we launch the game, and this automatically scales and handles our traffic perfectly well over the launch weekend and into the weekday when people don't play as much.

28:17 We're no strangers to serverless. Actually, our MySQL offering that I mentioned before, we've been using serverless MySQL for quite a while.

28:25 But it's worth noting that that is a more mature product from Amazon.

28:29 Serverless has always been a good case for us. Uh, actually for our data serverless as well, we've been saving money with serverless. It has scaled easier. For our workflows, everything has been simpler and faster and great. And, um, after we switched to Redis, it was a night and day difference. It solved our ch- s- s… It solved our scaling challenges practically overnight.

28:50 We ran into a couple of little issues, though. The graph I'm showing is actually a combination of two different things, your read ECPU and your write ECPU. And there are different throttling restrictions on both of those. And we'd accidentally ran into caps on one, but not the other.

29:05 So little things like that.

29:08 But there was a pretty big issue that impacted us, and that is with the connections that I also will show with the diagram here.

29:15 So imagine you have a Redis, and it's talking to a couple of different Docker containers.

29:20 Serverless Redis was missing a setting called TCP keepalive. And really what that is is just a heartbeat service between the Redis and the Docker container.

29:28 So if a Docker container disappears after that keepalive setting, it will drop that connection because it's not talking to the container anymore.

29:37 And when you don't have that setting, it means these open connections persist forever, like this, where you have all these connections that are just open, but nothing's talking to them until you eventually run out.

29:49 Now, this wasn't a problem before because on-demand Redis had the setting, and this was a solved problem we didn't have to think about until we made the switch.

29:58 We also didn't catch this until pretty late in development because we have a script that would nuke our load test environment every night and then resurrect it in the morning.

30:06 Because of that, all of these connections got cleaned up every day.

30:10 When we ran longer soak tests, that's when we started to run into this issue and noticed it.

30:16 I wish we'd been more careful here.

30:18 Uh, we could have looked at this technology, understand the differences that we had lost a little better, and gone in with more knowledge and better prepared.

30:28 Uh- Cost.

30:30 Load testing is tricky, particularly when it comes to cost. You not only have to pay to scale your platform, you have to pay to scale the load test itself, which means per minute you're spending more money on the load test.

30:42 And we had to be careful here. Sure, we could just keep throwing money at Amazon, and they would've loved that, I bet. But it doesn't mean that we're focusing our resources on the right things here, and we wanted to be smart.

30:53 So that two million tests that I mentioned, we actually only ran that every two weeks. We were running two-week sprints at the time, and our sprint demo was just a two million load test to make sure it worked.

31:04 When we weren't doing the two million demo, we did 100K CCU tests.

31:09 Makes it easy. It's one twentieth of two million. So if you had something that was over 5% usage, you knew that it wouldn't scale all the way up.

31:18 And that was a nice way to help test our logic, test observability, make sure that we were doing something that would work well for us.

31:24 And when we did that two mil demo, sometimes it would fail, and if it failed, we said, "Okay, cool. Wait two months, do it again. Sorry. Wait two weeks and do it again." But then we got closer to launch, and wait two weeks was not an acceptable answer. So we ran these two mil tests more frequently.

31:41 I wanna share some cost numbers and data. I can't share exact dollar values, but I can share relative amounts.

31:48 So this load testing campaign was over fifteen months, but I'll note we didn't load test every week or every month in that timeframe. But if you did take all fifteen months, combine them together, it equaled about two and a half times the money we spent when we launched in September 2025.

32:04 But August of 2025 had over a quarter of our costs in there. When we ran into an issue, we ran that two mil test more frequently, and we were running into a lot of issues in August that we had to fix pretty quickly.

32:18 Top three applications by spend. Redis is number one. Not a big surprise. It was on demand for most of this load test. That percentage would be smaller if we only had serverless Redis represented here.

32:31 Our top three application spends. Our account service, which is a big monolith, was a quarter of our costs.

32:36 The other two services were, uh, what was ha- what had the most HTTP calls in the graphs that I represented. So it makes sense that those would also be pretty high.

32:46 All of this led to the moment of truth when we launched. Now, we did all this load testing to launch well, and I'm very proud to say this was our smoothest launch yet. We had no scaling issues. Uh, everything went well across all regions, all launch waves. Steam Tuesday, perfectly.

33:03 We had observability that gave us incredibly high confidence that if an issue would pop up, we would be alerted of it.

33:11 I also ca- uh, can't share exact CCU, but we were adequately prepared, and the thing that I am most proud of is we had no launch overtime. To me, that is the metric of success that I care about the most, is that everybody on the team felt great, and we didn't have issues that caused us distress.

33:29 Observability we wanted of equal importance to the scalability, though.

33:33 And I'm happy to say we had no technical blind spots during launch, but we also didn't have an issue to really look into. So it's hard for me to say our observability helped us catch these problems and fix things for users because they just didn't exist.

33:47 That being said, we did look into some non-player impacting issues, stuff that was very minor, uh, millisecond increases in response time for a couple of minutes, largely because we didn't have anything else to dive into.

34:01 We did have a couple of issues where we assumed serverless was the issue. A-and that was never the case. We've always ended up proving what was the real issue, never serverless.

34:12 But because of that, uh, we could've been more evidence-based in investigation. We had something that we couldn't see, and we just went to instantly blame that was the issue.

34:25 All right. Wrapping all of this up here, I had a hard time finding where to put this observability in practice as it covered a lot of different areas.

34:33 But one of the challenges that I gave the team was that we launch in one environment, so we should be load testing in one environment.

34:42 Now, that might sound, "Oh, yeah. Yeah, it makes sense at first." But the way we've traditionally done load tests is we would run a load test, then make one change, then rerun the load test and see exactly what happened in that change. It's a very scientific approach.

34:57 And because it took so long, we had multiple load testing environments, so everyone could be working in parallel.

35:03 The problem is that that scientific approach works when you have a controlled environment. Now, when you ship a game in production and have people playing that game, that is anything but a controlled environment.

35:14 So by saying you have to use one environment, you had to make sure that excellent observability was in place. When you made a change, you had to have the observability on that change to make sure it's working excellently.

35:24 But we didn't always have excellent observability, particularly in the beginning. So this ate up time. We had to, we had to try and debug issues and didn't have the observability. People didn't know which of the half dozen changes that were rolled out broke that two mil load test demo.

35:39 While it did highlight a weakness and it resulted in us having a great end product, this took more time, and time was a pretty precious commodity during this. So I'm not entirely sure if this was the right call or I would suggest it.

35:53 But I will say load testing is amazing training. I was so happy with how much the team's knowledge and troubleshooting skill scaled as with the platform, and that included using observability tools.

36:04 By making sure that the team was using these tools during our load tests, it helped give us confidence to use them during launch. And when those tools were not sufficient, it would highlight them.

36:14 Whenever we did a load test and then we had a problem, and if it took us more than two minutes to diagnose that, now we have two problems, and during our demos we'd run into an issue, we would write two JIRAs to fix the next time. Fix the underlying cause and improve the observability so we can find this problem easier in the future.

36:33 Some final reflections co- connecting all of the different chapters here.

36:37 Be aware of some missing challenges or complexities when you are evaluating a new approach. We stuck with what players did first, and what we should have done is stick with what was the most complicated thing to implement.

36:51 Scaling load test client, really think about your dec- decision here. Do you want to have load generation or user replication? And then build something around that.

37:00 But also don't stress it if you're wrong here. What we did was brittle, but it did technically work.

37:06 I'm very happy with a lot of the bleeding edge technology we used to scale the platform using newer serverless Redis, but be prepared to be cut a little bit on sharp edges when you're using something more bleeding edge.

37:18 This one, uh, really stands out as an obvious thing to me, and I feel bad we missed it.

37:23 Cost observability should have been first and fore- foremost. We should have had cost observability before we ran a single load test in our load test environment. Having an alert on daily issues opposed to Slack or to Teams or email saying how much you spent and in what different areas, that would've helped us every day understand where we could be optimizing things.

37:43 Also, this was a successful launch for us, the most successful one that we've ever had. I give load testing and our approach here a lot of reason for that. I shouldn't be too critical on all the things that we could do better, but also I wish we had been better on using our observability tools because we did some science and evidence-based investigation that we missed just because we blamed the thing we couldn't see.

38:07 I wanna wrap this up with what I have as the three most important messages and takeaways to me. This is what I would tell the past version of myself if I knew how to time travel, and these are the things that I'm carrying with me to the future to make sure that the next load testing effort at Gearbox is even better.

38:23 One load test to rule them all was awesome. It worked well for us. It was valuable. We used that load test to use practice launches to test our alerts. It was just an incredibly valuable tool that had wide range in use.

38:37 But it's an application, not a script. Should have treated it like an application and used the application languages that we use.

38:45 Next, that observability training is so key and vital.

38:48 If you have the most amazing observability tool that tells you everything you need to know and nobody uses it in a high pressure situation like a launch, it's providing zero value. Really make sure that the team has ownership of these tools and is excited about them.

39:03 And last is cost observability. It would've been so easy to just have a post or a message saying, "Here's how much we spent," and that would've helped us out, saved so much money in optimizing for load test.

39:15 All right. That's it. That's my presentation. Uh, if anyone is interested on, like, connecting with me on LinkedIn, I've got a QR code, uh, or you can just search my name and it should work.

39:26 Louis Yu: Thanks, Jeff. That was really, really insightful. Like, I mean, for those that are in the video games industry, we know that, you know, months leading up to a launch is very, very chaotic. Uh, and I'm happy to see the behind the scenes. I never knew that, you know, load testing has so many steps to it, and, like, the way that you use observability and, you know, the tools that, uh, help you to analyze and, you know, simulate load and user replication, it's really, really, really interesting.

39:56 Um, and I think, uh, the, some of the messages that you… I mean, some of the points that you, you pointed out at the end, uh, leads back to, you know, the keynote that GDC has, uh, that Rob Pardo say, you know, lean into what your team does the best, um, and kind of leverage those skills that you have. And, you know, have everyone be on board with what you're trying to do, and that kind of builds a better product.

40:21 You know, i- in your case, builds a better process in, you know, making sure that our game doesn't crash on launch.

40:29 And for those that haven't played Borderlands 4, uh, go and play it. Uh, I, I've played Borderlands 3, but I've not played 2, but I've heard that 2 is kind of the GOAT of the franchise, and 4 is going back to the GOAT.

40:47 So I've heard a lot of good things about it. I've played through a couple of chapters. I need to finish up the game, um, hopefully in the next couple of months. Uh, there, there are a couple of DLCs that came out which is really, really interesting, so hopefully everyone has, uh, time to go and try that out as well. Uh, oh yeah, there was a question.

41:04 Oh, you, you have already… You're fast.

41:09 There we go.

41:10 Uh, is there any other questions from the folks before we pass it over to Sahana?

41:18 Sounds good. All right, cool. Thank you again, Jeff, for sharing your immense knowledge. Uh, I'll catch up with you next week in Austin.

41:31 All right. Sahana, uh, take it away.

41:34 Sahana Subraya: Uh, thank you for my- having me here. I will try to be as quick as possible, uh, but you can reach out to me later if you have more questions, okay?

41:42 Um, hi, I'm Sahana. I'm working as a director of product at Dragon City, a mobile game in Social Point.

41:49 Uh, I'm building the roadmap and strategy, uh, doing the forecasting, managing P&L, et cetera, uh, based on this roadmap, and I'm also leading a team of product managers here, um, and the designers to execute on this.

42:02 Um, I have been in this role for the past one year, but prior to this I have worked as m- um, product manager in multiple mobile games, uh, for almost, uh, eight years now. Okay?

42:13 Uh, today's session, so I'm going to talk about how data talks to you and mainly if you're willing to listen. So this is more like a guide to, uh, the PMs on using data, uh, to make better decisions on their day-to-day.

42:27 Um, I've been in situations in the past where, like, head of product or, like, the company owner who would come to us and say, "Hey, I think we should build this And the whole room would ideally just nod, you know? Uh, it's not because, um…

42:41 I mean, I'm not saying that it's happening everywhere, but it's not because people are lazy or people, uh, uh, don't want to look at data, but they rely on gut versus data sometimes because, uh, they feel that data is slower in general. But today, I want to change that and, uh, I think that data isn't slower than intuition, it's just a skill that we need to use properly in general.

43:04 Okay?

43:05 Um, cool. So, uh, why does, uh, data matter for PMs? Um, basically, I, I think it's for Mo- mainly for four different things. First one is mainly to make decisions. So when you are, um, data-driven, you start, um, having evidence and stop having opinions in general. Uh, and the second thing is for understanding the users. Um, surveys generally tell you what people say, but data tells you what they actually do, and these two are different things and have different, uh, impact altogether.

43:38 And the third thing is for prioritization, because every PM has more ideas than the time that they have. So data is how you stop, stop arguing about what to build next, and then start deciding. And the fourth one mainly is finding problems. A best PM finds the problem before user complains about it, and data is your early warning system for it. So you don't have to be a data scientist, you just need to have, uh, fluency enough to ask the right questions and read through the answers.

44:09 Um, I've organized today's talk, uh, around, uh, phases that mirrors how a PM actually works. Basically, the pre-analysis where, uh, the data work that you do, you do before building anything, then you, uh, take it live, which is basically you design and run experiments or you launch the feature where you have real-time monitoring, et cetera. And then, uh, post-analysis, where you're measuring what you have shipped actually worked or not.

44:36 I'll try to give some examples or case study to help you understand better, but yeah, I will be covering, um, these topics.

44:45 Um, so moving on. So basically, this– imagine you join a new team and you inherit a product.

44:52 Um, one of the hardest questions is, "Where do I focus?" You know? Like, um, nobody has an answer to it, and if somebody tells they do, then they're ideally guessing. So, uh, data gives you a structured way to find out, and two main tools do the most of the work for it. One is figure– funnel, which kind of, um, jots down the step user takes to achieve the goal, and then you look at where they fall off, and the biggest drop is generally telling you where the biggest problem is.

45:22 And the second one is mainly defining the metrics in general, the key metrics that you want to, um, track. Um, it could be like usage rate, open rate, session, et cetera. And, um, if the metric is de-declining, then it generally shows something is wrong. So data shows to you, um, where, um, befo– um, like where the problem is before users start complaining about it or be-before you, uh, get to know it from, uh, other form.

45:50 Um, the principle here is simple. You should let the data point, um, let the data point tell you to focus on the right place, no? Like, where do you need to prioritize and focus and, um, make sure that, okay, this is where you put your energy in, in the, in the, in the first place.

46:07 Cool?

46:08 So let me make it concrete with the case here. Like, so for example, if you, um, if you're talking about a online, um, plat booking, uh, flight booking platform, um, you don't know where to focus and, I mean, how do you arrive at the area where you want to focus on? So for example, here we have four key, um, I would say steps when– for a, for a, uh, user when they come to the platform.

46:30 First is they come and search for the flight, they get the result page, then they add the passenger details, and then they have a payment, no? So the question now is, okay, where does the, wh-where does the problem lie? Or like, basically you don't know where, um, the issue is occurring or on which device or which device type or where– what is the, what is the main, uh, problem that you, you need to focus on.

46:53 Um, so for example, imagine that you have this kind of a data. The first thing that jumps off here is that, okay, you see that mobile generates almost like seventy percent of the traffic, so whatever problem you find, it matters most on mobile, no?

47:07 So, uh, now looking at the results, um, there is a drop that you can see which happens between results and passenger details, um, especially on mobile users. But you need to also see, okay, whether this drop is also happening on desktop, is it similar or is it something that is only particular to mobile or not?

47:26 So I'm just gonna, like, I mean, um, show you the comparison here, and you can see that, um, uh, the dro– there is a drop between passenger details to the payment page and, um, the gap is so high between mobile and desktop, no? So you can see that there is clearly a problem here. Mobile users are, uh, getting to the, um, passenger details page, but they are, they are dropping off after that.

47:53 And now you know that, okay, there is a problem here, um, and you can, mm, go ahead and figure out what is that problem, no? Why, why is this happening? It could be multiple reasons. You could then do, um, test. You can, you can do a QA or you can go and ask users what has happened. It could be that, okay, in mobile, something is not showing up or some, some kind of a, a issue is happening only on mobile which is kind of leading to this.

48:18 But the data has pointed to you that, okay, this is where you need to focus on and this is where there is a problem. And, um, why you need to fix this, fix this is also very clear because you see that there is three times more traffic on mobile versus desktop. So if you fix this, uh, issue Uh, ideally you're gonna see an enormous revenue impact, and that is where you should be focusing on it, no?

48:43 So yeah, I mean, just to give you an example on how we could use data to identify problem, I think this is, this is one of the, one of the ways to, like, I mean, look at it. The key message here is that you should always try to break, break it down between different kinds of, um, I would say user base, regions and things like this, so that sometimes you will miss out on something if you look at an– at a overall level and don't deep dive enough on each of these stages in the funnel, uh, itself.

49:12 Okay?

49:13 Um, then moving on.

49:16 The second, uh, thing which we talked about was also in terms of, uh, potential impact analysis. You know, this is one of the, one of the scenarios which PMs face constantly. Basically, you'll always have to make a decision on which feature to build first. M-mm, basically, you always have limited resources, but you will have many things to build. So the main, main, um, uh, mm, criteria is that, okay, you need to understand the potential impact of what you are building.

49:46 So the– there are generally four steps that you will have to follow. One is first is you define what are the options that you want to choose, uh, or build. Um, then you ca- um, try to get data that will tell you, mm, relevancy of each of these options, um, tells you how many users are impacted or how, um, I mean, uh, how does it, um, make a difference for, mm, this kind of users?

50:09 What is the volume of it? What is the, uh, what kind of a metric it is going to move, et cetera. And then you need to estimate those impact as well. So if you are going to say that, okay, I'm gonna build option A and this is gonna move my metric by the, by X percentage, but I'm go– if I go ahead and build option B, then it is going to increment by Y percent.

50:28 Then you are having key, uh, ways to compare between the impact, and then you'll be able to make a decision which is ultimately making a go or a no-go option, no? Like, I mean, whether you go with option A or option B based on the projection in itself. So in this case, what you can avoid is that you can make, you can make decisions which is a lot more, uh, where you're a lot more confident about it.

50:50 Um, and generally, um, it is not like, okay, you're doing it based on opinion, uh, in general. Okay?

50:57 Um, this might not work everywhere, but yeah, in most of the cases, yes, it does. So I would say use it when you, uh, you find it, um, useful for you. No?

51:07 Um, for example, I'm just going to give one, uh, one case study here. Imagine that there is a, like, a cab aggregator who, uh, which has a driver app and driver needs to log in using either phone number or email. Uh, and to make changes to that, the driver has to call the call center to, like, make these changes. Now, if a PM wants to build a self-service update for, mm, as a feature but has bandwidth to build only one, now you have to make a decision on, uh, which one do I build first, no?

51:35 So if you look at the data here, first instinct would be that, hey, I mean, a lot of people are doing on email, so the volume of, uh, email is higher, so let me go and build for email, no?

51:45 But if you look at the login usage, you can see that seventy percent of the drivers use phone instead of, like, email, no? Only twenty-five percent use email. And, mm, number of unique, uh, drivers who are calling is almost identical, forty-eight thousand and forty-seven thousand. So the call volumes are similar, but phone number affects nearly three times as many drivers in their daily usage.

52:07 So you will build the phone number first and email later, no? So here the potential impact is higher even though the raw call count is slightly lower.

52:16 Um, but with the detail on all these, um, metric, you are able to make the right, uh, decision and you get the right kind of insights, no? Um, if you just look at one metric saying that, "Hey, I will just look at the calls made and then I will make a decision," I, I think you might make a wrong decision, uh, in, in that cases.

52:35 So yeah, I think this is one of the things that helps the PMs to make decisions in a much, much, um, I would say, confident way, uh, rather than just based on gut feeling, you know?

52:47 Um, cool. And the last one, which is mainly, uh, for the pre-analysis, uh, part, where it's about market sizing. You know, like trying to understand, uh, which o- which of the, mm, I would say market do I sh- uh, I mean, feature or something or even, um, product that do I, do I fast follow, no? Like which one do I think is something that I need to build, uh, in general?

53:12 Um, or you want to pitch, uh, pitch to your leadership about a new product or a new, um, mm, thing that competitor has launched, et cetera. Uh, I think this will help you, um, make that pitch in general. How, how do you do it? Basically, you need to try and answer these questions. What is the absolute market like? Uh, it could be in terms of downloads, active users, et cetera.

53:35 And then you should try to look at three key potential metrics there. One is basically retention. Um, basically to understand whether users stick around. Um, revenue potential, trying to see, okay, do you make money out of it? And the last one is how sticky is the product? Basically, um, where are players or a- are m- users coming more often to the product, no?

53:59 So these are the things that you will have to evaluate if you want to, uh, uh, get a picture of, uh, whether this, um, product or this, mm, whatever feature or this kind of a, um, this is something that, this is whether this market is worth the investment or not. So I think mostly, mm, I mean, I, I, I find this, um, happening mostly in scenarios where you have to make these kind of– when you become the person who's making the decisions on this.

54:27 Um, but I'm just gonna give you some example here. Like for example, if, um, if you are building a studio and you want to fast follow one of the games, no? Like, I mean, um, I think which, which a lot of, uh, lot of the people in the industry are doing it like, okay, you see something successful, you would want to, you would want to fast follow it.

54:47 So imagine if you're looking at these two games, um, I mean, if you look at it, you can see that the retention numbers for game A looks very impressive, no? Like these are very, very strong numbers. But if you look at, um, everything else, game B has a higher number. Like, so for game B, it ha– it is nine times higher download, twelve times higher daily active user, thirty-three times more l– monthly revenue.

55:11 Um, and also you can see that the stickiness in itself is also much higher than, uh, game A. So more engaged daily despite having lower retention. Basically, they might be coming much more often, um, but maybe they're not retaining, uh, long term. Um, but the main lesson here is that relative metrics are only meaningful in context of ab-absolute scale also.

55:33 So for example, even if the retention numbers are, uh, really good on game A, but the absolute scale of game, uh, B is much, much higher, no? Like it is at a factor of five. Um, um, mm, so even though, even though if D30 retention might be lower, m-still they have a much, much higher, um, higher, uh, engagement, much, much higher monetization and simply larger user base in itself.

55:59 So ideally, I mean, as a decision, you should be following game B, uh, itself. I have just u-used some, like an example of, okay, how you can try and do comparison of different kind of metrics, um, and try to understand what do you need to, like, take a look at when you are trying to do, like, um, assessing of the market in itself.

56:19 Okay?

56:20 Cool. So I think that was mainly on the pre-analysis part of it, like generally what you do before you are actually investing either resources from tech or you're in, like, building a team or anything, which is, like you do as a pre-analysis before you get into, um, get into, uh, action, no? Uh, the other part of it, which is the post-analysis. The post-analysis is when you are actually, mm, have done everything, but now you need to know whether you did something successful or not.

56:52 So how do you do it? Basically, you try to, like, I mean, try to measure the impact properly, no? Um, so for this to be done in a really, mm, good way, one thing you have to take into consideration is you should always try to define your success metric before the launch. I mean, uh, I mean, I, I know that I'm mentioning it now, but ideally, when you are doing a product launch or a feature launch, what you need to do is that you need to define the success metrics before and not after.

57:22 Because if you decide what success looks like after you see the results, you are actually not measuring. You are actually telling a story that fits the outcome, no? So ideally, you should try and, uh, define the success metrics earlier. The second one would be either doing a comparison of a pre-post, um, analysis. I mean, in, in most of the cases, we do A/B test.

57:43 I mean, I will talk about A/B test next. But, um, in some cases, when there is no possibility of doing it, you also do a pre-post analysis. Uh, but you should be careful about seasonality and ca-careful about what period you are comparing, et cetera. But I, I, I think this is also one way where you could try and measure between, um, your, between the se– between two different periods to measure the success of the product.

58:08 And the step three is basically, um, you should also check for side effects, no? Like basically you define a suc– a key metric that you are defining as a success metric, uh, but you should also look at other metrics, whether a feature or a product is like, um, actually hurting this other metric, no? So you should always have guardrail metrics you are watching, mm, along with your primary success metric.

58:33 So it do-doesn't mean that, okay, you are improving the revenue, but you are killing fully the retention, no? So this, this shouldn't be the case, and you should always, uh, keep this, um, side, um, like side effects into consideration when you are launching something. And the last one is basically making a decision. Um, so based on these ma-metrics, you either iterate on the product, you, uh, or you kill the product, no?

58:57 Like, I mean, you will have to make a decision once you have, uh, outcome of the, of the, um, feature that you have launched.

59:04 Um, so the w-worst outcome of an impact analysis is that, um, it's not a negative result, it is no decision. So basically, if you are doing something and not taking a decision, m-mm, that means that that's the worst thing that you can do as a PM. So it's always important for you to define the suc– metrics, measure the impact, and then take a decision based on it.

59:29 Okay?

59:30 Um, then now I was talking about A/B testing. So this was basically a way, like one main key tool for a PM, um, where they need to make sure that, okay, if you are having two different options, um, you need to try and see, okay, which is the better option or how do I, how do I validate, okay, which option is better? You kind of run a A/B test, you know?

59:53 So A/B test removes this aspect of, okay, I mean, uh, I– mm, your opinion on, okay, this, I think this would work better than the other one, no? So this kind of thing gets, um, removed when you do an A/B test.

1:00:05 Um, so when you're doing the A/B test also, you need to follow the same, mm, kind of, uh, rigor, which is you have to pick your metric first. You need to make sure what is– what you're gonna improve with this change. Um, you need to make sure that, okay, you're dividing the group randomly, ensuring that, okay, you run the experiment long enough to actually get statistically significant results.

1:00:27 You also make– need to make sure that you have, uh, guardrail metrics also tracked for both these groups and ultimately then make a decision based on that. So I think the methodology doesn't change much with an A/B test, but it is just important that you kind of make sure that you are not, um, running it, running the test too short of a time, not keeping too small of a, um, audience, et cetera.

1:00:48 To avoid making wrong decision based on the, based on the dec– based on the data. So yeah, I think this is one of the key things which, mm, PMs try to, uh, do throughout their career. Trying to prove that, okay, this pro– this, um, solution is better than that solution, and this is one of the way, uh, that they succeed to do it. I can talk about it in one of, like, m-with one of this example.

1:01:13 Uh, so for example, now, right now, mm, i-imagine that you are selling, um, some hard currency like gems for, for one dollar, but you want to know what is the right price point, no? So imagine that you want to test it for two dollars, three dollars or whatever. Mm, then what you can do is that you run it as a test with multiple groups which has different values, and then you try to look at each of these metrics to be able to make a decision.

1:01:40 Sometimes when you look at only one of the metric, you can make a wrong decision, as I said earlier as well. It's always important to know, uh, what do you have to, um, ultimately choose based on what kind of metric. So if you look at, look at this example, you can see revenue per user, hey, it's very amazing when you do a three dollar, um, test. But, um, you can look at the payer conversion.

1:02:02 New payer conversion has dropped to half, no? Like, I mean, it has become six percent. So you are… If you make a decision that, okay, three dollar is the right purchase, uh, point, mm, price point, so let me go ahead with it, you will lose the long-term impact because, uh, over the period of time, you are gonna lose the people who are converting.

1:02:21 So ultimately, you're not going to, um, make revenue. So that's, that's the key thing that you'll have to identify when you are, mm, running the test.

1:02:31 Uh, you always look at metrics, mm, not just for short term, across time horizons and, um, and also try and look at balancing both short-term impact and long-term impact, and then make a decision ultimately. So yeah, I mean, this is one of the example that I could share where, I mean, you can see that there is– there are key metrics that you, m-might see doing really well, but, uh, there are other metrics that you will have to, um, take a look at if you have to make the right decisions.

1:03:01 Okay?

1:03:02 So that's, uh, that's mainly about the, mm, two, uh, sides of a life of a PM where you do analysis on both the sides when you're doing something, before you do something and after you do something. Why I have put SQL here is basically, I mean, I, I think some of you might have– might be thinking if, if I'm a PM why do I need to write code, no? Like, I mean…

1:03:25 But here's my answer, because SQL is a language for asking questions, you know. Like, I mean, every time you write an SQL query yourself, you are actually getting that answers much faster than waiting for an analyst or to get or, like, waiting for somebody to give you those answers. So I, I personally consider PMs who are the, who are, who are the ones who are able to write queries and get these answers, pull their own numbers, um, are the ones who are, uh, really, really, mm, dangerous, you know.

1:03:57 Like, I mean, they, they, they are the ones who can make it make a really big difference in making decisions. So you don't need to be expert, but you just need to know the basics. So I mean, I, I, I think this is pretty, uh, simple, um, kind of logic that you need to apply. So you just need to know what you need to, like, I mean, what you, what simple queries that you can write to get this information.

1:04:21 And especially, I think one of the, one of the most used thing that, uh, I see that I am using or my PMs are using are basically joins of tables because you have, uh, mostly all the information available. But you need to know a way to, like, try and join these tables to get the information that you want, no? So I think, mm, for a PM, um, if, if, um, if they, if, if they have to, mm, focus on something in SQL, they need to make sure that, okay, they know the basics and they know how to, like, make these, uh, small simple joins to be able to get the information that they need.

1:04:58 And I think that would be, um, skill set which will be valued a lot.

1:05:02 Um, so yeah. I mean, I think that this was just a brief of what a, what data means to the PM. So I would want you to close… I mean, take, uh, take, take out these four things from, from this, uh, discussion. One is first thing that I mentioned, which is you need to define your metric before the experiment and not after. Uh, this sounds very obvious, but it's, like, mm, violated constantly, you know.

1:05:28 Like, sometimes I have seen a lot of examples where people don't do it.

1:05:31 Uh, so you have to decide what success looks like before you see the results. Um, and th- I think this is a must. Mm, and then, um, I also think you should segment everything and try to see the data, uh, by, like, segmenting in different kind of, um, um, n- through different kind of, um, metrics. Uh, you, you cannot just look at an aggregate and decide, "Okay, mm, I'm gonna make this decision." So sometimes aggregate numbers hides the truth, so it is very important for you to segment it, mm, have a deep dive on different metrics to be able to make decisions.

1:06:06 Um, then the third impos- important thing that I want you to take away is that pre-analysis is a must.

1:06:13 Um, this saves a lot of engineering time.

1:06:17 Um, mainly, uh, I have seen that engineering is one of the costs that you wouldn't want to, um, utilize it without having a, a good conviction that, okay, this is going to work, no? So it is one of the costliest resource that you can, um, utilize, no? So you make, build something, and then you see that it's not working. I think, uh, I think it's a stupid decision to make, uh, ultimately.

1:06:40 So pre-analysis is really, really important for a PM.

1:06:44 Um, and then post-analysis ideally earns trust, no? Like, so if you do a post-analysis and say that, uh, hey, this was the results Um, I saw that these things worked and these, th- these things didn't work. I think you're building credibility. So you are building credibility that, okay, you are making decisions based on this, uh, information that you are getting.

1:07:03 So I think this is, uh, the third thing that I would want to say. And the last one is SQL is your direct line to data, so I would say own it. You don't need to be an, an expert in it. You don't need to like, I mean, um, be a data scientist for it. You just need to be competent enough to pull your own numbers, uh, maybe on a Sunday when no one is…

1:07:24 else is around, but make sure that okay, you are able to do it. Um, and, um, you get your own data because data doesn't replace good judgment. It just makes it sharper. Like, I, I don't think, um, data will tell you everything, but the best PMs are the ones who use data to ask better questions, uh, not to avoid, um, making decisions, but to make them con- uh, make their decisions more confident, no?

1:07:51 So yeah. I think these are the four key takeaways I would want to, um, put forward for you guys. Um, and I hope that you, uh, you enjoyed the talk.

1:08:03 If you have more questions, you can reach out to me or like, I mean, you can type in if you have other questions, but yeah.

1:08:10 Thank you.

1:08:11 I think I went really fast, but yeah, hopefully, hopefully it covered everything.

1:08:18 Louis Yu: Thanks, Sahana. I think there's a question from Nick, uh, on recommendation for folks who need to compare metrics against like two different types of game. Like it could be a sequel to a popular game five years ago versus, uh, a DLC. Like how would you compare the suc- or, you know, the success criteria for, you know, two different types of games?

1:08:40 Sahana Subraya: Yeah. I mean, if when you're comparing, if you're talking about two different ki- types of games, you could, you will for sure can, you for sure cannot compare, uh, um, I would say, uh, hyper-casual game with, um, probably, um, RPG strategy game, no? So the metrics will look very, very different. So in the… But the point is that you will al- you will try to make sure that you do this comparison to understand where the difference comes from, no?

1:09:10 Like, I mean, so for example, in some of the games, um, maybe if there are shorter session lengths, still they ha- they would have, um, higher stickiness. Imagine they come, um, a lot more frequently. Like basically they come every day, but they have shorter session lengths. Whereas in an RPG game, they might have longer session lengths, but still come every day, you know, like on a everyday basis.

1:09:33 So you need to define what is the thing that you want to chase after, no? Like, I mean, what is, what is the key for you to make this game a success? So it's not like, mm, you, you look at an RPG game saying, and say that, "Hey, they have much higher session length, so I also need to make my hyper-casual game have a lot longer session length." That, that's not how it would work.

1:09:54 So you, you would ha- you, you need to use your, um, um, understanding of what needs to be compared and then make decisions based on it. Um, and it's good to know what are those things where it should be different, but at the same time you should know where you need to like, I mean, do a comparison and see, oh, okay, this is where it should be similar.

1:10:14 So yeah. I, I don't know if I answered the question, but, uh, this is what I think.

1:10:20 Louis Yu: Awesome.

1:10:22 Yeah, I think this is really interesting because, uh, data is very important for us to understand what happened, how to measure it, and how to improve.

1:10:31 Um, so I think like you've touched on all of the key aspects of being, you know, having that analytical mind.

1:10:38 Uh, the one thing I would add is if people aren't aware, Tableau Desktop is now free, so you can connect… Like the, the, the best, the best SQL statement you can write is select star from table. Uh, and you can put that into Tableau Desktop and you can explore to your heart's desire. You know, drag and drop different columns into, you know, the rows and measures and dimensions and, you know, just having that curiosity mind, uh, to kind of explore your data to find something that, you know, would tell you a story about what's performing and what's not.

1:11:16 So if you have yes, Nick. Uh, you… Yeah, in order not to, you know, burst your Snowflake credits or Databricks credits, you, you'll need a date filter as well.

1:11:27 But, um, yeah. So for those who don't know, you know, uh, go download Tableau Desktop, uh, today. Uh, the only restrictions is, uh, you have, you do have the full authoring experience of, uh, a, a creator license. It's just you can't actually publish to a server or cloud.

1:11:44 Uh, but once you kind of upgrade it to a, a license, uh, you can actually do that. So, but other than that, you know, you can local save, you can export the image as a, um, uh… You can export the chart as a image and, you know, embed it anywhere you want. So, uh, try it out.

1:12:04 Awesome.

1:12:05 Uh, thank you both our speakers, Sahana and Jeff, uh, for this amazing talk. I think it is a lot of value that is, uh, within, you know, what you guys have shared. Uh, I'll just add a little bit, uh, before we wrap it up.

1:12:19 So for those who are not already in the community, uh, you can actually join the DataFam on Slack. Uh, so you can go search for Tableau Community on Slack. It will, it'll get you the link to it, um, and you can ask questions there. So it's basically a community platform where, you know, uh, everyone gathers to share, um, you know, knowledge and all.

1:12:45 Uh, the other thing is we are not the only Tableau user group out there. Uh, there are some that are location-based. So if you're based in US, there are a few different state, uh, level Tableau user group as well as industry. So ours is considered industry Tableau user group. Uh, but if you're interested in like healthcare, finance, uh, and whatnot, uh, there are other user groups as well.

1:13:09 Um- Now there are, if you haven't started, if you haven't start a Tableau Public profile, I would strongly recommend that. Uh, there are a couple of different community projects that have existing data sets that you can practice and all. Uh, you know, Tableau Desktop is free now, so you can actually do that. Um, now I have a few, uh, kind of video games focus visualization that I wanna kind of do a spotlight on.

1:13:38 So last week, uh, for Makeover Monday, which is one of the community projects, they had a Super Mario data set, right? So it's, it's fun thing. So it, it looks at the unit sales for Mario games from like 1980 something to like, uh, 2026, I don't know.

1:13:56 And this viz by Nobu, uh, actually is very fun because he embedded some, uh, web objects, uh, as a, a GIF, animated GIF. So you see like the little mushroom, the star, uh, and little Mario over at the, at the left-hand side. So it's a very cute way. Uh, the size of the coin actually represents the size of the unit sales, so I think this is, uh, a pretty thematic design, but a very unique way of showing the, uh, you know, the, the sales within, within Tableau itself, right?

1:14:28 Uh, the next one I wanna show is by Prashant. Uh, and this is, I mean, if you don't pay close attention, uh, you probably wouldn't notice, but the bars actually have kind of like a glowing effect around each of them. Uh, and this is actually done using map layers. So you have like one map layers, uh, and the one on the outer edge of the bars are of a lower opacity, so it creates the illusion of that glowing effect.

1:14:54 So I thought that was a really cool way of kind of doing bar charts. Um, obviously you can't do this for everything, uh, but it's a very fun way of showing it. Um, the next one is by Yoshihito, uh, and this is, uh, a very straightforward dashboard, but I really like the way that he has this dynamic switch over the top where you can look at sales, uh, by units.

1:15:15 You can look at the different, uh, dimensions itself, and it kind of sorts itself, uh, by the different decades and the games itself. Uh, and it's a very fun way of looking at it.

1:15:25 Uh, the next one is by Tsutomu, uh, and this one actually has a kind of like a timeline series on it. So you can actually click on the years on the lower portion of it, and it just, it, it kind of shifts that little dot on the left-hand side, uh, on how high the sales, uh, are. So that was a very cool animation trick that he used. Um, the next one is not exactly Mario, but Rudy has actually done a Donkey Kong Bonanza viz.

1:15:57 Uh, if you haven't played this game, this game is fairly new. I think it came out like a couple of months ago. Uh, I've heard a lot of good things about this game, like it's really fun and all. Uh, so if you, you are keen, you can actually check out the game as well.

1:16:10 Um, now for Tableau Conference, uh, Dawn will talk a little bit about that. Uh, but we have finished, uh, in February actually, they announced the top three finalists for Iron Viz, and that is Kevin, Anne, and Brian. So this year is gonna be really exciting because all three of them have been in the community for quite a while, and they have been doing a lot of awesome work.

1:16:36 Uh, if you've not checked out their Tableau Public profile, you should because they are doing a lot of, uh, kind of Tableau magic, like things that you would not think that are doable in Tableau, they have done it. So I'm really excited to see them, uh, and what they have to offer to us on, on stage this year.

1:16:53 Uh, so over, uh… Oh, and if you wanna explore the rest of the entries, uh, this is the link. I'll, I'll share the, the recording after this as well, but you can actually explore. I think there's about, uh, 122 entries for this year. Uh, so you can go ahead. There's, there's a lot of really cool stories that people have created. Or if you would like, you can actually, um, check out the podcast that I do.

1:17:18 Uh, I've interviewed some of these amazing vizs, so you can actually take a look, uh, or listen to, you know, how they have come up with their ideas, their inspiration, and the development process, and challenges, and stuff like that. So it's a pretty interesting way of, of, uh, diving into it.

1:17:33 So over to you, Dawn, to give us a little bit of, uh, intro to what Tableau Conference is and what to expect.

1:17:41 Dawn Harrington: I didn't have any cool GIFs for my slide. I'm keeping it very old school with a print screen, so you know, you guys can shame me later. It's fine.

1:17:51 So if you haven't gone to Tableau Conference, it's amazing. Uh, it's, it'll be like my third time going.

1:17:58 Um, it's very big. How, how big do you think it is, Louis? Like 20,000 or…?

1:18:05 Louis Yu: Probably, I would say. Yeah.

1:18:06 Dawn Harrington: Yeah.

1:18:07 Um, so there's a couple of us that are presenting, some famous names. Um, it's three days, so it's Cinco de Mayo, the 5th of May to the 7th.

1:18:20 Um, but it really has everything Tableau. Um, and one of the reasons why I wanted to get into the community was just because like how welcoming and how friendly everybody is. You're meeting like other, you know, people that are interested in data like you are.

1:18:34 Um, because Tableau isn't just about y- it's, it's about so m- so much more than just, um, a visualization tool. It's a community, like DataFam community, and you're learning neat things. Yes, there's the Iron Viz, um, where you see live people have to build like a dashboard in 20 minutes. Um, I couldn't do it 'cause I'd probably forget my name, and then there, there we go.

1:18:57 Like I just, I wouldn't be able f- be, I'd be starstruck on stage and just wouldn't be able to continue, but I don't know how they do it, but it's amazing. You should sign up. Um, can we go to the next slide?

1:19:08 Louis Yu: Uh, yep. There you go.

1:19:09 Dawn Harrington: Yeah, no worries. Um, so the registration link is there. There's a coupon, I think, still, um, and like $300 off. So if you can register, you're not too late.

1:19:20 Other sessions already, um, such as a session with me and David Kelley. Uh, I may or may not be wearing my light up, uh, shoelaces that I got one year from InterWorks. Um, just, you know, TBD there.

1:19:32 Um, and the FleurLash twins, they always have great sessions. There's some other famous names, but you know, I wanted to share the more important ones with you all, you know, right here. So,

1:19:45 I'm just kidding.

1:19:46 Um, so our session's about future-proofing your Tableau data model. That sounds really, you know, techy, but it's, it's all about like how can you design a dashboard that won't fail when new data is added? 'Cause a lot of the things that were talked about today were, are, are excellent, uh, points to make. Like how do you, um, how do you ensure a successful dashboard?

1:20:11 Well, a lot of the things that Sahana talking about are still applicable in the Tableau dashboard world too, 'cause it, it's all methods for, um, you know, impact analysis. Like is your dashboard successful? So those are great tips. Even Tableau world, even though we're not doing cool things like design video games, Jeffrey, you know, we're not doing anything like that.

1:20:32 So, um, if you wanna go to the next slide. Do I, did I add another slide? Yes. Okay. So the other cool thing that we did, um, very important here, is we designed some pins for Tableau Conference for me to, to give out. Um, it was the cats versus the sheep llama thing here. It was supposed to be a llama, but Microsoft Creator made it Mo- look more like a sheep.

1:20:56 Um, so which one was more popular? We had a poll.

1:21:00 It was the cat. Not biased there at all. I have two cats. Um, I didn't hit vote like 10 times. No. Um, a lot of users said it was the cat. So, but anyway, I deci- I made a, an executive decision and I'm gonna have both pins. So if you see me at Tableau Conference, talk to me about your cats, your dogs, show me pictures. That's cool. Talk to me about your Tableau dashboards, and ask me if you want a pin, so you can have one of these lovely pins.

1:21:28 I will give them out. But um, I only ordered 50 of each, so try to find me early and, and talk. Show me your cat pictures and, and so on. So, um, the cat won, but we just decided to go with both. All right. And I… Is there any more slides that I have?

1:21:46 Louis Yu: No.

1:21:47 Dawn Harrington: Nope.

1:21:47 Louis Yu: No.

1:21:47 Dawn Harrington: This is over to you. Over to you, Louis.

1:21:50 Louis Yu: Thank you. Yeah, so that's, that's a very strong, um, reason to head over to Tableau Conference this year.

1:21:57 Um, so I just wanted to add a little bit about Games Night Viz. So as I talked about this earlier in the call, uh, this is a community project. Uh, and we have kind of like, it used to be monthly, but we kind of stretch it a little bit, so it's kind of like quarterly themed challenges. Uh, we have a, a few ones like Spider-Man, Disney, Pac-Man, uh, Mario even.

1:22:21 Uh, and this year, this month, uh, we are actually restarting the Pokémon challenge that we did, um, couple months back. Oh, wait, sorry, last year. Um, because, you know, Pokébia, this is the 30th anniversary of Pokémon. Uh, Pokébia is out. It's taken the world by storm. I'm playing Pokémon, uh, Fire Red with my daughters. Uh, I thought, you know, this would be a fun way to kind of reintroduce that.

1:22:47 Uh, we did have some badges. Uh, I won't be at Tableau Conference this year, but, um, let's see. I don't have the badges here on my, on my desk, but, uh, what we're gonna do is, uh, you know, create a viz, uh, about Pokémon. It can be, you know, about your first Pokémon game, about how the franchise evolved, your favorite party, your favorite Pokémon, and we will choose our favorite, you know, three vizs, and we'll send you the badges.

1:23:17 Uh, so it's a very nice kind of badge that you can put on your bag, and we might print some notebook. We'll see. Yeah. But these three kind of images is something that we created last year. Uh, so gotta viz them all. Uh, Chat Desert users Data Blast, and viz viz a chew. Yeah. viz a chew. It's always a hard thing to pronounce. viz a chew is super effective.

1:23:36 So this is kind of our play of words of, you know, the, the themes of Pokémon. So, uh, it was very popular last year when I was at TC. Everyone was grabbing the badges off my bag and, uh, we thought that we might just redo this. So, uh, whe- when you kind of submit your vizs to Tableau Public, just remember to tag me, Dawn, and Will Sutton, uh, on, on LinkedIn or Twitter, whichever works for you, or even Slack if you want to.

1:24:05 Uh, and we'll get to… We'll, we'll constantly look out for vizs, but if you need help, uh, there are curated data sets already. We'll provide a official announcement probably like next week or something. Uh, but you'll have like a list of existing data sets that you can play around with or bring your own data sets. Uh, whichever is fine.

1:24:27 Uh, I think that's probably, yep, the last slide I have. Uh, thank you so much from me, Mahath, and Dawn, and to Sahana and Jeffrey for giving us your time, uh, to run through amazing speeches that you had, guys have. Uh, if there is no other questions, we can end the call, and we'll see everyone in June.

1:24:52 Thank you.

Shorts for this TUG

Leave a comment

Your email address will not be published. Required fields are marked *