Sea of Thieves: Testing a Core Game Mechanic
Prior to PlayStation London Studio’s closure, I was shortlisted as a potential candidate, for the final stage of their Player One Internship programme in November 2023. I was invited to the studio to attend an exclusive open day, which offered an insight into games development and allowed me the opportunity to meet their Development Support (QA) team to learn about their role.
As part of this, I was asked how I would test a game mechanic and was tasked with creating a test plan. This was a fun project, which really got me thinking about the testing process. It taught me that the testing cycle doesn’t have to be a linear process - and if it is not achieving the results we desire, we need to ensure we adjust our path.
The Task:
We would like you to think about how you would test a core mechanic from a game you know well/your favourite game. This can be done in any form: Bullet points, test plan, mind maps or any alterative you can think of. There are no wrong answers, and the aim is to see how you think through things and any considerations that may apply. (e.g., internet connectivity, multiplayer, control schemes any others that you think apply)
What Game…
There were a few games that came to mind which I thought would be quite interesting to explore further, in terms of their core gameplay mechanics. At first, I thought of some of the recent games I’ve been playing, such as Jusant, Assassin’s Creed Mirage, Payday 3, and even THE FINALS’ recent open beta - choosing just one is quite tricky for me, as the more I think about them the more I want to investigate each of them!
However, I keep coming back to Sea of Thieves due to its complex interconnected gameplay mechanics.
Sea of Thieves, developed by Rare, holds a very special place in my heart and has been an all-time favourite of mine since their first release as a closed beta in 2018. It’s a shared-world action packed pirate sandbox game that has an open world and a heavy focus on teamplay – this is the exact reason why I love the game so much! It has a powerful ability to bring complete strangers together with a common goal. The game’s fun and social approach to teamplay, combined with highly interactive navigation and ship management systems (like lifting the anchor, adjusting the sails, navigating by compass, or repairing the hull) all come together and make for a truly unique experience.
This is a game I know well. It has many complex mechanics at its core – plus it is always changing with its live service approach. Let’s take the core mechanics of Sailing and Navigating, something which is fundamental to the game. There’s a lot involved with Sailing and Navigating, with a variety of smaller mechanics coming together in unison.
That’s me!
To start, before we think about testing, I believe it’s important to first establish what the core elements of Sailing and Navigation are within the game – as well as the different gameplay loops and the expected outcomes of each mechanic, plus how they interact with each other. For this I’m going to use a mind map on Miro to explore the quantity of options and see how they interconnect with each other.
Click above to view my Mind Map on Miro.
After just a short time drawing this out on Miro, I can see that Sailing and Navigating has so many interconnected parts, and I might be biting off a bit too much to start off with such a broad area. So I think drawing it in a little and focusing on one area might be best!
Let’s take one part and start from there: such as the Ship’s Wheel / Steering.
Planning and Strategy:
So how would we start testing the Ship’s Wheel? That’s where I’d say effective planning and strategy come in! As QA is quite new to me, I’ve found the QA Quest website particularly helpful with deciding on the best method – by adopting their step by step approach.
1. Plan
We should start by developing a comprehensive test plan which covers user interaction, different scenarios and environment variables.
o This should outline the approach that we are going to take.
o This should also make it clear what is not going to be tested at this stage.
o We should liaise with those involved with production to ensure that this aligns with the current road map that the team is working towards.
o We should use Risk Assessment / Risk Management to determine the priority of items in the test plan.
o Ensure that we first establish what the level of quality should be within the game, and then use this to check if the current quality meets this standard.
2. Design:
The next step would be to design test cases covering various situations to ensure the Ship’s Wheel and Steering functions consistently and is reliable in various scenarios.
3. Implement & Execute:
We should then implement and execute the test cases we have designed (repeatedly!)
o Consider making use of automation!
o Include Localisation Testing.
o Write up bugs.
o Check through the logging system.
o Updating documentation regularly.
o Writing feedback for other teams.
4. Report:
Record the quality of the game:
o Establish what the current level of quality is.
o Take into consideration the timescale requirements – will we reach the correct level of quality for release or release of the next update?
o Consider how many bugs we are finding.
o How quickly and accurately are the fixes being made?
o Consider if we are testing enough.
o Consider Defect Management and the process of handling the bugs we find.
5. Closure or Exit Criteria:
This is not necessarily the end of the testing process, rather it would be when the cycle of testing has been completed and our goals have been met:
o For a full release, we would ensure that the game does everything defined in the game design document to an acceptable quality level.
o For the end of a milestone, we should revisit the planning stage for the next stage to establish the next test plan and where we go from there.
I’d also like to note that this isn’t set in stone and it’s just as important to check the work we are doing to ensure we are achieving the desired results - we should change the process if this is not working! This might mean starting the process again.
Relating this to Sea of Thieves - Scope and Objective:
So now that we have thought a bit about each stage of the test process, let’s relate the planning directly to Sea of Thieves – specifically the Ship’s Wheel and Steering mechanics.
First, we should define our scope and key objectives of the test plan:
Scope: Testing the Ship’s Wheel and Steering
· We are only going to test this initially on the ‘Sloop’ ship type.
o Larger Ships such as the Brigantine or Galleon are to be excluded at this stage, as they handle differently.
Objective of our test:
· To ensure that the Ship’s Wheel functionality is responsive, intuitive, and contributes to the immersive sailing experience.
· To ensure that the Ship’s Wheel allows players to accurately steer and control their ship.
· To ensure that the experience is frustration free for players.
What to Explore:
Now, let’s think about what we might want to explore in our tests:
1. Basic Functionality:
o Verify that players can interact with the Ship’s Wheel and Use it:
i. Start Interact.
ii. Cancel Interact.
iii. Player interaction should be cancelled if damage is sustained by the player.
iv. Ensure that players can interact even if they are holding an in game item, inventory tool, resource or physical loot item.
o Ensure that the Wheel can turn left and right and that limits are in place for max turn.
o Verify the responsiveness and accuracy of the Ship’s Wheel when steering the ship both stationary and at speeds.
o Test the Turning Radius, speed adjustments, and the ability to stop the ship from turning.
o Wind and Sail effects on Sailing and Steering.
o Ensure interact button’s function when bindings are changed.
2. Environmental interaction:
o Test how the Ship’s Wheel responds in different weather conditions (stormy, calm seas).
o Assess any impact of environment variables on steering, such as:
i. Hitting an obstacle
ii. Natural Waves
iii. Wind direction
3. Player Interaction and Control Schemes:
o Test the Ship’s Wheel across various input devices (keyboard and mouse, controller, Xbox adaptive controller etc.)
o Check for ease of use and ability for new players to understand how it works.
o Verify that only one player can use the wheel at one time.
o Ensure that all members of the crew can interact with the wheel.
o Ensure that enemy players can interact with the wheel on other player ships.
4. Team Coordination:
o Assess how well the Ship’s Wheel integrates with crew coordination (communicating orders, working in sync, radial chat options etc).
o Check how the Ship’s Wheel reacts when other players interact with a different part of the ship at the same time.
5. Cosmetic Changes:
o As Sea of Thieves key method of rewarding players is through different in-game cosmetics, we need to ensure that the Ship’s Wheel remains unaffected by changes to the Cosmetics: consider changes to player clothing, ship livery, and items etc.
Other Things to Consider:
There are also several other area’s we should consider:
· Internet Connection:
o We need to ensure that the server stability doesn’t affect the responsiveness of the Ship’s Wheel.
o To ensure that if a player loses connection whilst operating the wheel that the wheel is still usable by other members of the crew.
o To ensure that the wheel movements are consistent across server side and client side – so all players see the same thing!
o We should consider the performance of the wheel when the server is under a heavy load, with max number of players carrying out simultaneous intensive actions (non-controlled environment).
· Player Experience:
o We also want to gather feedback on the satisfaction and immersion that the Ship’s Wheel offers in terms of functionality – does the player enjoy operating it?
o Realism vs. Gameplay - we should consider the balance between realism and enjoyable gameplay with the steering mechanics.
· Accessibility and Inclusivity:
o Ensure that the Wheel mechanics are accessible to all players, such as those with a disability which can affect their mobility, vision, coordination etc.
o Assess the Wheel mechanics when using assistive technology devices.
o Ensure that key bindings can be re-mapped, and work as intended when re-mapped.
· Live Service Updates:
o Ensure that any changes made to other parts of the game do not impact on the current performance of the ship’s wheel.
o Consider implementing a way for new players to provide feedback on core mechanics such as steering and using the Wheel (possibly in the form of forums or Discord etc)
· User Training:
o Evaluate how new players learn to use the ship’s wheel and consider how this can be used to implement in-game tutorials or guidance prompts.
· Consistency Across Platforms:
o As Sea of Thieves is a multi-platform game, across Xbox One, Xbox Series X/S, and PC (Steam & Microsoft Store) - we need to ensure that the Ship’s Wheel works consistently across different platforms.
o We need to consider different controls for controller and keyboard/mouse.
Test Cases
With the scope and key objectives of our testing in mind, let’s get started with what a potential test case could look like.
Test Case ID: 001
Test Title: Using Ship’s Wheel - Anchor Up and Sail Raised
Environment: Xbox Series X with standard Xbox Controller
Pre-conditions: Load into the game on a ‘sloop’ ship type with only 1 player on an empty server. Place ship in calm water close to a starting Outpost away from obstructions. Ensure player is on ship’s deck facing the wheel, with no tools or items currently equipped. Anchor raised with sails up.
STEP 1.
INSTRUCTIONS
EXPECTED BEHAVIOUR
The Ship’s Wheel will remain still, and the ship will remain in place.
Observe Ship’s Wheel with no gamepad inputs.
STEP 2.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will move towards the Ship’s Wheel and the prompt ‘Use Wheel’ will appear with the Primary Interact controller button icon X.
Move Left Thumbstick forward (short movement).
STEP 3.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will interact with the Wheel, moving into position, and begin using it.
Press X button once on gamepad (short press).
Summary of steps 1-3.
STEP 4.
INSTRUCTIONS
EXPECTED BEHAVIOUR
Ships wheel will begin rotating left, causing the ship to begin rotating left in the water. The speed of turn will increase until maximum turn has reached 360°. Player will hear audio and receive vibration when max turn is reached, and wheel will stop turning.
Ship will continue to rotate left whilst remaining stationary.
Move Left Thumbstick to the left and rotate wheel 360° until max left turn is reached (long movement).
STEP 5.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Wheel will begin rotating to the right, causing the ship to begin rotating right in the water. The turn will begin to stop once the wheel reaches its original centre point. Player will hear audio and receive vibration when centre point is passed.
Move Left Thumbstick to the right and rotate wheel 360° until centre point is passed (long movement).
STEP 6.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Wheel will rotate a small amount to the left, past the centre point. Ship will not rotate. Player will hear audio and receive vibration when centre point is passed.
Move Left Thumbstick to the left if centre point is passed (short movement).
STEP 7.
INSTRUCTIONS
EXPECTED BEHAVIOUR
Wheel will rotate right to the centre point. Ship will not rotate. Player will hear audio and receive vibration when centre point is passed. Audio and vibrations will stop once movement is stopped. Central point of wheel will align with ship’s mast.
Move Left Thumbstick to the right if centre point is passed again (short movement). Release Left Thumbstick once central point on wheel is aligned with mast.
STEP 8.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Ships wheel will begin rotating right, causing the ship to begin rotating right in the water. The speed of turn will increase until maximum turn is reached 360°. Player will hear audio and receive vibration when max turn is reached, and wheel will stop turning. Ship will continue to rotate right whilst remaining stationary.
Move Left Thumbstick to the right and rotate wheel 360° until max right turn is reached (long movement).
STEP 9.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Ships wheel will begin rotating left, causing the ship to begin rotating left in the water. The speed of turn will increase with movement. Ship will continue to rotate left whilst remaining stationary. The ships turn will begin to stop once the wheel reaches its original centre point. Player will hear audio and receive vibration when centre point is passed.
Move Left Thumbstick to the left and rotate wheel 360° until centre point is passed (long movement).
STEP 10.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Ship and wheel will both remain stationary.
Observe Ship’s Wheel with no gamepad inputs.
STEP 11.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will move away from the Ship’s Wheel and cancel interaction. The prompt ‘Use Wheel’ will appear above the wheel with the Primary Interact controller button icon.
Press B button once on gamepad (short press).
Expected Results of Test Case: All controller inputs and button presses should result in the expected wheel and ship movements, without any unwanted behaviour.
Now that our first test case is complete, we can begin to change things up for another test case!
This time, we could consider changing the controller bindings for the interact button and perhaps drop the sails so that the ship is moving forwards when turning the wheel.
Test Case ID: 002
Test Title: Ship’s Wheel With Alternative Interact Key Binding - Anchor Up and Sail Raised
Environment: Xbox Series X with standard Xbox Controller
Pre-conditions: Load into the game on a ‘sloop’ ship type with only 1 player on an empty server. Place ship in calm water close to a starting Outpost away from obstructions with sails down and anchor raised. Ensure player is on ship’s deck facing the wheel, with no tools or items currently equipped.
Ensure that Primary Interact Controller binding is set to RB.
STEP 1.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The Ship’s Wheel will remain still, whilst the ship moves forwards with the sail down.
Observe Ship’s Wheel with no gamepad inputs.
STEP 2.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will move towards the Ship’s Wheel and the prompt ‘Use Wheel’ will appear with the new updated Primary Interact controller button icon RB.
Move Left Thumbstick forward (short movement).
STEP 3.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will interact with the Wheel, moving into position, and begin using it.
Press RB button once on gamepad (short press).
STEP 4.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Ships wheel will begin rotating left, causing the ship to begin turning left in the water whilst moving forward. The speed of turn will increase until maximum turn has reached 360°. Player will hear audio and receive vibration when max turn is reached, and wheel will stop turning. Ship will continue to turn left whilst moving forward.
Move Left Thumbstick to the left and rotate wheel 360° until max left turn is reached (long movement).
STEP 5.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will move away from the Ship’s Wheel and cancel interaction. The ship will continue turning left and moving forward at speed. The prompt ‘Use Wheel’ will appear above the wheel with the Primary Interact controller button icon.
Press B button once on gamepad (short press).
STEP 6.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Wheel will remain stationary, whilst the ship will continue turning left and moving forward at speed. The prompt ‘Use Wheel’ will remain above the wheel with the Primary Interact controller button icon.
Observe Ship’s Wheel with no gamepad inputs.
STEP 7.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will interact with the Wheel, moving into position, and begin using it.
Press RB button once on gamepad (short press).
STEP 8.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Wheel will begin rotating to the right, causing the ship to begin straightening out. The turn will stop once the wheel reaches its original centre point, and ship will continue moving straight forward. Player will hear audio and receive vibration when centre point is reached.
Move Left Thumbstick to the right (long movement) and rotate wheel 360° until centre point is reached and aligned with mast.
STEP 9.
EXPECTED BEHAVIOUR
INSTRUCTIONS
The player will move away from the Ship’s Wheel and cancel interaction. The prompt ‘Use Wheel’ will appear above the wheel with the Primary Interact controller button icon.
Press B button once on gamepad (short press).
STEP 10.
EXPECTED BEHAVIOUR
INSTRUCTIONS
Wheel will remain stationary, whilst ship continues moving forward at speed.
Observe Ship’s Wheel with no gamepad inputs.
Expected Results of Test Case: All controller inputs and button presses should result in the expected wheel and ship movements, without any unwanted behaviour. Ship will continue moving at all times.
That’s a wrap!
So now we have a basic idea of what a test case could look like for testing the core mechanics of the Ship’s Wheel and Steering. I do want to note that these test cases are by no means exhaustive (they never can be!) and there would likely be several variations of the same test and other scenarios which these would need to be run for. It would also be important to test how these mechanics interact with the network / server as we would need to ensure what is happening for one player is the same as for another player.
We also need to consider what stage of the development cycle Sea of Thieves is at, especially as it is a live service game with regular updates that was originally released in 2018. Rare will have set up an automated cycle for core mechanics such as steering and using the ship’s wheel (in a controlled environment, possibly with no obstructions etc) as these mechanics form a core part of the game and are vital to ensure the game functions as intended.
As it is an online live service game that is constantly changing, with core mechanics being altered and new content being added, it would be necessary to perform frequent ‘smoke’ preliminary tests. This would be in addition to the automated tests and would ensure that a broader area of the game can be tested and would allow us to reveal simple errors/issues by touching upon the main functions as new content and updates are added.
It is also vital that we can perform testing in an uncontrolled environment, due to Sea of Thieves being an online game. This is where community test environments come in - such as the Sea of Thieves Pioneer Programme. These types of environments are important as they allow the development team to test new features in a separate isolated build from the main game, allowing the game to be tested before and changes are made to the current live build of the main game. It also lets the team test the severs in a similar set up that the full game would.
Finally I want to stress that the testing cycle doesn’t necessarily have to be a linear process, as it is just as important that we are continually checking that the work we are doing is achieving the results we desire - and if it is not, we need to ensure that we revisit previous steps and adjust the testing plan as needed. Picture it as a path – we would change our route if we notice that we are going off course!
