Server issues have always been a hot topic within the Apex Legends community, with players highlighting lag issues, code: net errors, and – more recently – DDoS attacks on ranked servers. To address all of these concerns, Respawn has now released an in-depth developer analysis aimed at “demystifying” some of the issues and explaining what the team is doing to fix them. You might want to put the kettle on, as this one is long to read.
Written by Chief Engineer Samy “Ricklesauceur” Duke, the blog post notes that Respawn has made some changes to the way it tracks server issues. The introduction of a performance display in Season 6 allowed players to report issues more accurately, and Respawn has also started tracking key performance metrics for players and servers. Over the past few seasons, Respawn has also used a data science team to systematically analyze performance data. But despite all of this, Duc said this approach means Respawn can still be slow to resolve issues.
“From the moment you start noticing issues, it can take us up to two weeks to find a fix, and even longer to deploy the solution if it requires a server fix,” said Duc. “We can do better. We will do better. So let’s talk about solutions.”
In addition to a weekly report, Respawn now switches to real-time alerts to allow the team to immediately resolve hardware issues, investigate issues, and work on fixes. A new unique server ID is also added to the game’s performance bar, allowing Respawn to locate faulty servers a bit faster. These changes should come in the course of the new season.
Duc anticipates that issues with idling servers should be resolved faster with this approach, as real-time detection will allow Respawn to take misbehaving datacenter machines offline faster. A new tool capable of creating RPROF files, meanwhile, should help Respawn optimize servers to reduce latency issues, as those files give developers a glimpse of what the server is doing on each frame.
Some solutions for more complex problems are still in the works: Respawn is hoping that automation will help detect congestion points for packet loss issues, for example, but it looks like there are still more solutions to it. in the “early stages”. Duc explained that the shots that don’t register, known as no-reg, may be due to a bug in the game’s physical simulation. These bugs may be hard to spot, so Respawn is working on more. automated tools to find them – and asks gamers to submit issue detection clips to help track down those bugs.
However, some problems like offset compensation will never have a perfect solution. (This is often the cause of getting shot behind a door or wall, by the way.) Apex is designed so that players with low ping do not have an advantage over players with high ping – a Respawn deliberately designed compromise in the system. Duc noted that this was “really important for rural players, or for players in areas where connectivity is unstable.”
The infamous “code: net” connection error is apparently a generic message that could signify a number of issues, and Duc has stated that they “often have more to do with Respawn’s services supporting the game.” Steps have apparently been taken to reduce the likelihood of code: net bugs occurring, but Duc recommends reporting these messages to the EA help site.
As for the tickrate debate (how often a server updates what the player sees), there is an extremely detailed paragraph explaining why simply increasing the server tickrate for Apex Legends, like others line shooters did so, would not make a big impact on performance.
“Put all of these calculations together and you realize that 20Hz servers cause about five frames of delay and 60Hz servers cause three frames of delay,” said Duc. “So to triple the bandwidth and processor costs, you can save two frames of latency at best. The benefit is there, but it’s not huge, and it wouldn’t do anything for simple lag issues (like getting shot from behind cover), ISP-level issues, or bugs (like with hit reg and slow-mo servers). ”
The post adds that Respawn is investing in its online infrastructure and working with “server-level to ISP-level” partners to improve the overall experience. A few issues that weren’t covered in the post, such as DDoS attacks, were covered in a Q&A on the Apex Legends subreddit. Communications Director Ryan K. Rigney explained that Respawn is improving the tools to identify DDoS attacks, closing a number of loopholes, working with experienced partners to develop solutions, and investigating more serious consequences for perpetrators. attacks, including legal action.
Phew, that’s a lot of information – and this is only the condensed version. If you are looking for full mathematical and technical extravagance, you can find the blog post here.