Part III: Launching and Managing a Game
Chapter 12. The Live Development Team
KEY TOPICS
Live Development Team Responsibilities The Publishing Process
The Publishing Plan
Patch Creation and Publishing Schedules The Live Test Server
How Often Should You Publish?
Critical Bugs and Exploits Bug-Fixing Versus Nerfing
Planning and Implementing Major Expansions Implementing an Expansion
Sometime from one to six months after launch, the live development team will take over responsibility for the game from the development team proper. By this time, the game should be stable, the initial crush of subscribers absorbed, and the intake of new subscribers settled out to a low, but consistent, level. The live development team should have a relatively calm environment in which to begin its work.
And there is a lot of work to be done. The overall responsibilities of the live development team are three-fold:
Maintain the game's current content and tools
Design and add new content and tools on a regular basis
Design and develop major game expansions (either retail or download)
[ Team LiB ]
[ Team LiB ]
Live Development Team Responsibilities
Let's explore each of the responsibilities in a little more detail:
Maintaining the game— Maintenance is a fairly simple process: Identify and fix bugs as they are reported, monitor the game servers for stability and anomalies in the code, and make sure the administrative tools used by player relations and others continue to work as intended.
New content and tools— The live development team designs and develops new content and features for inclusion in the game on a regular basis (every two to six weeks) and confabs with the player relations and community relations teams to identify new tools or capabilities needed by those worthy of them. This is the meat of the matter for both the players and the team; regular refreshment of the content is vital to the good health and growth of the game.
Expansion pack development— For games with a retail component, it is necessary to keep a steady flow of expansion packs on the shelf. This not only keeps a presence at the retail level to attract a steady stream of new potential subscribers, but allows the retail software to be recompiled and updated, avoiding exceptionally large downloads for new players.
How often to ship a new box to retail is a matter of resources and how quickly the world-building tools allow for new content to be built. The available evidence seems to suggest shipping an expansion every nine months to a year as optimum. As examples, Sony Online's EverQuest (EQ), the largest US game, has been out for a little over three years and already has three expansions with a fourth in development as of January 2002; Ultima Online (UO), out for almost five years, has shipped four of them; and Asheron's Call (AC), the smallest of the three examples, has been out just over two years and only recently shipped its first expansion.
With all this activity going on, it is necessary to be organized and coordinated to avoid a constant process of "patch, freak out, fix all the bugs that should have been fixed on the test server, and calm down the players who want to hang us all again." The heart of all of this is the publishing process.
[ Team LiB ]
The Publishing Process
Patch day: These words strike terror into the hearts of players all over the world and with good reason; the industry doesn't have a great record of stable patches, nor one of informing the players just what changes will be made in each one. More often than not, patches aren't well-tested, they are filled with bugs, and they contain wagonloads of unannounced changes, some of which look like bugs to the uninformed players.
In previous chapters, we discussed cowboy programming and other bad development practices that foster this kind of self-defeating chain of events. If the producer of the live team can manage to control the team's cowboy programming and its equally noxious, shoot-from-the-hip cousins, it will be by strict enforcement of a publishing process. The publishing process structures fixes of the current/old code, design changes, and additions that represent new code, plans for QA testing, and lays out when and when not to do an emergency patch.
[ Team LiB ]
[ Team LiB ]
The Publishing Plan
In some ways, the process for creating and publishing a game patch should resemble the initial development process discussed in Chapter 2, "Planning." Such a thorough process has not been the standard to date, resulting in unstable patches that crash games and nasty bugs that sometimes corrupt entire character databases. This often necessitates a rollback to a backup version of the game, which results in the loss of many hours of player in-game time and accomplishment. Now you know why players dread patch day.
A patch should be meticulously planned, developed using best-of-breed version control and CCPs, and thoroughly tested and debugged on a live test server before being inflicted on the subscribers. Patches published online to the player base won't be as large as the initial development, but they still need to be charted in a task schedule program by a project manager. And the live development team, like the initial development team, tends to bite off more than it can reasonably chew for a patch.
[ Team LiB ]
[ Team LiB ]
Patch Creation and Publishing Schedules
The steps that should go into the publishing process can be outlined as follows:
Creation of a live production plan. The plan should include the following:
Bug list: Bugs detected and to be fixed
Balance issues: Fixing out-of-balance regions, items, and classes New features: Interface capabilities, bells, and whistles
New content: New mechanics, dungeons, skills, crafts, and so on New tools: To assist members of the live team
Tentative schedule of patches by date Regular schedules for fixes Regular schedules for new content 1.
Development and documentation of changes. This phase should include the following:
Version control of changes and additions to the code Change control process (CCP) documentation of changes Commenting of new code, recommending of changed code 2.
QA plan. This includes the following:
CCP vetting of new content and features Internal test server implementation Live test server implementation Fix problems found in testing and retest 3.
Publishing the patch: Patch day! Substeps include the following:
QA sign-off Producer sign-off
Authorization to initiate a publish Emergency patch procedures 4.
Step 1 is the planning and design stage: when the fixes and new content are planned, prioritized, and scheduled for publishing. It is important to note that this document will change frequently as problems are fixed, new concepts and content are planned, and previously planned content and features are created, tested, patched to the game, and removed from the schedule. There is also some tentative scheduling of when the game patches will be published and what will be in each patch.
NOTE
It is recommended that you schedule "fix" patches separate from new content and feature addition patches. Mixing them can be convenient, but imagine what would happen if a "fix" isn't really fixed or causes unanticipated problems and has to be rolled back to an earlier version? If the new content is critical to the ongoing storyline or is highly anticipated by the subscribers (as it almost always is), being forced to revert to former versions will cause a major hoo-rah, guaranteed.
Step 2 is the development phase, when the changes and additions are developed and documented.
Step 3 is the testing process, controlled by QA. QA should perform initial tests in-house on a closed server, perhaps assisted by a small crew (25–50) of trusted outside testers. Once problems detected in this testing have been fixed and retested, the changes can be placed on a live test server, with open access to the subscribers. The process should be fairly rote from that point: find the bug, fix the bug, retest to make sure the bug is really fixed, repeat until finished.
Step 4 is the actual publishing of the patch to the live player servers. Once QA and the producer have signed off on a patch, authorization is given to initiate the "publish" according to the schedule.
Occasionally, nasty problems with the code will develop out of nowhere and require an emergency fix to stabilize the game or eliminate a particularly nasty bug. Following the dictates of The Law of the General Perversity of the Universe, an emergency will usually happen at 3 a.m. on the Saturday morning of a four-day holiday weekend. If an emergency patch is needed, there should be an authorization process documented, even if it is just a panicked junior engineer calling the producer for permission to make a quick fix and reboot the
servers.
[ Team LiB ]
The Live Test Server
Some teams like to hide upcoming changes from the player base. As discussed earlier in the section about managing the expectations of the players, this generally causes more problems than not. Remember that the players are stakeholders, too, just as much as any executive or member of the team. Their input is important, if only so they can say they had some.
Aside from the consideration of not taking the player base by surprise with changes, live testing with as many people as you can get is a necessary component of the live phase development procedure. Remember the bugs that were found when thousands tested in the late stages of open Beta, on code that you were positive was stable? Why would anyone believe, then, that making changes to code interlocked with many other moving parts can be done successfully with just a few dozen testers?
So, a live test server is needed. There are a few guidelines to follow:
Players who regularly inhabit a test server can become proprietary about it, in the same manner they do with their characters and possessions on the production server. It is recommended that you wipe the character database clean often, just to get players in the habit of it.
It can be difficult to attract and retain players on the test server; they only have so much time to play, and the testing doesn't develop into a permanent character. A system of rewards can go a long way to attract people onto the server. For example, you can offer a tester-only item of clothing for use in the live game after a certain number of hours of testing is completed;
players enjoy this kind of recognition among their peers, as it implies close contact with the god-like developers.
Something that many teams forget is that the player base exists at all levels of experience and character growth, and with all manner of item inventories. After wiping a database clean, all characters are back to zero in growth and inventory, so the main testing is done by low-growth characters with "newbie" items. Be sure the live test server has player-activated options for raising and lowering test characters to various stages of growth and easily adding items to the inventory, so balance and bug issues can be tested in all ranges. This will also help attract players to the test server, as players love to test out character classes and items they haven't touched before.
Listen to what the testers say. One of the most common mistakes teams make is to ignore what the testers are telling them.
The communication has to flow both ways for the test server to be of any real use to the team.
[ Team LiB ]
[ Team LiB ]
How Often Should You Publish?
After the initial launch phase, when the game has settled into a stable state, you should institute a regular publishing schedule of patches. With a regular schedule necessary to manage the expectations of the players about patch publishing, the question becomes how often you should publish a patch.
We've seen every schedule from once per week to every two or three months. In our experience, once per week is too often and once a month is not often enough. A happy medium, depending on the resources available, is every two to four weeks, not counting emergency patches.
The important thing is that publishing patches happens on a regular schedule and that schedule is kept, if at all possible. There will always be exceptions; some scheduled publishes won't make it through QA. Constant communication about the state of an upcoming patch with the players will help you manage these occasional situations.
[ Team LiB ]
Critical Bugs and Exploits
What is a critical bug or exploit? When it comes to bugs, anything that crashes either the servers or game client has to be fixed as soon as possible; that's just plain common sense. We don't know any developer who wouldn't work as long as it takes to find and fix a crash bug. Some exploits, especially attempts at duplication of items, require players to intentionally crash servers through the introduction of lag or what looks like a denial of service attack (streaming huge amounts of data at a server until it slows to a crawl or crashes from the load) and should be treated the same as crash bugs.
The same criticality goes for duplication bugs that don't require crashing the servers to implement; anything that allows a player to duplicate game items outside the context of the gameplay should be found immediately. Whether you close it right away is a more interesting issue; live teams have been known to put in logging code just to see who is using the dupe bug consistently, so those people can be removed from the game at a later time.
Beyond that, we get into the realm of theory and speculation, at least from the developer's point of view. What may be a critical exploit or harmful bug to the community relations or player relations team—one that must be fixed right now—might not be a priority at all for the live development team. Is being able to stack flour sacks around a boss monster to prevent him from moving and making him easier to kill a critical exploit that must be fixed immediately, or can it wait until the next scheduled patch? How about players using a teleportation spell to strand non-magic users on an island? You certainly want to stop that kind of activity, but should it be fixed right now, or can it wait?
These are tough areas for decision makers, and the best guideline we can give is this: If a bug or exploit thoroughly screws up the balance of the game and/or is used to prevent other players from playing normally and requires gamemaster (GM) intervention to set right, it probably ought to be fixed as soon as possible. In other words, if it generates help calls to the GMs, it ought to go to the top of the
"fix" queue.
[ Team LiB ]
[ Team LiB ]
Bug-Fixing Versus Nerfing
"Nerfing" is a generic term used by the community to describe the act of changing an ability or feature in such a manner that it takes away power from players (for example, changing crossbows into Nerf crossbows). While the term "nerfing" existed before 1999, it came into general use by subscribers to describe the actions of Verant's live team early in EQ's lifecycle, when it was nerfing a character class, skill, ability, or weapon seemingly on a weekly basis. Nerfing does not include vital bug fixes, although the players may perceive them as such if not properly forewarned.
Developers most often use the term "game balancing" to describe this activity. They are pretty much the same thing with the same effect: They neutralize dozens and even hundreds of play hours for individuals. There's nothing like working for weeks to achieve a skill level and having the developers change the rules overnight because they feel it unbalances the game. It may even be true that a skill or ability is horribly unbalanced and a change is necessary; doing it without warning and discussion with the players, in a for-pay game, is guaranteed to cause outrage and the occasional death threat.
It is also a mistake that almost everyone in this industry makes. For some reason, developers seem to delight in springing takeaway surprises on the players, as if knowing about them in advance will somehow neutralize the change or give the players an advantage.
However, nothing angers players more than being presented with a surprise nerf. Predictably, developers seem shocked at the ticked-off reaction from the players every time because they believe they are doing good things for the game.
Some changes are necessary—even nerfs—but there is no excuse for not utilizing the community management team to prepare the players for them. If the players know a week ahead of time that a certain change is coming, they can avoid wasting their time playing to advance skills, weapons, or whatever is about to become less useful. And, in the end, it is all about not wasting the player's time.
[ Team LiB ]
Planning and Implementing Major Expansions
Throughout the whole process of regular patches and adding new content, the team needs to be planning the major expansion packs.
Most often, these expansions are developed as new retail units to retain vital shelf space and keep the "buzz" about the game alive.
Most expansions are also far too large to conveniently download, which is another recommendation for planning them as retail offerings.
The computer/video game press will almost never rereview an online game simply because new content has been added on a regular basis. However, the press is in the habit of reviewing new retail units. Issuing an expansion as a retail unit is almost certain to get it reviewed by the major game press.
These expansions should cover two main areas:
All the fixes and content/features additions to date, to bring the shelf unit as close to the current code base as possible New content and features found only in the expansion
What kind of new content and features should be added? Following are the considerations you should ponder:
Do you want to primarily appeal to new players or service your current players?
Remembering that most new players will churn out before the expiration of the free trial offer, you get your best return on investment (ROI) from your current players. Keeping current customers around should be a primary concern, so heavily weighting new content to retain current customers is not a bad idea. The idea is to keep them in the involvement phase of their lifecycle and not let them slip into the boredom phase. Be careful not to add just top-level content; those in the mid-range, who are most likely the players who want to stay but don't have 20 hours per week to indulge, can use some refreshment, too.
If you're keeping a unit on the shelf, you'll have a steady influx of new customers anyway, so don't sacrifice the satisfaction of current players on the altar of higher numbers. That doesn't mean you can't add content for newer players, especially if you need new tutorials or should redesign the new player experience. This is important to look at in the first major expansion but less vital in subsequent ones.
What are the customers asking for and what needs to be added?
If the community relations team has been doing its job, you should have an ongoing list of the major features and additions requested by the players and know which items in the envisioned or upcoming section of the four-step notification plan are most discussed. By now, you should have also set up your own player satisfaction matrix (PSM), as discussed in Chapter 8,
"Getting into the Design."
Both should figure heavily in planning the features and content for an expansion, as long as the bias of the vocal minority on the forums is being accounted for. If something is called for by both the players and the PSM, you should probably include it.
For example, if both the PSM and the players are calling for player-owned housing, it should be in the expansion.
[ Team LiB ]