1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Greetings Guest!!

    In order to combat SPAM on the forums, all users are required to have a minimum of 2 posts before they can submit links in any post or thread.

    Dismiss Notice
  3. Greetings Guest, Having Login Issues? Check this thread!
    Dismiss Notice
  4. Hail Guest!,
    Please take a moment to read this post reminding you all of the importance of Account Security.
    Dismiss Notice
  5. Author Wes Locher has teamed up with Stratics for a giveaway of his new book Braving Britannia. This book explores the history and impact of Ultima Online and includes interviews from current and past dev team members as well as many UO and Stratics community members. Click here for more details!
    Dismiss Notice

[Developer Blog] Observing the "Burn Jita" Player Event

Discussion in 'EVE News' started by EVE News, May 2, 2012.

  1. EVE News

    EVE News RSS Feed
    RSS Feed

    Apr 12, 2011
    Likes Received:
    Things have been afoot in the Jita solar system recently...

    For quite some time, the Jita solar system has been the most populated solar system in EVE Online on a sustained basis, having organically risen as the main player trade hub in game due to its location and the basic human need to gather together and trade stuff.
    It's had a varied reputation as [wretched hive of scum, villainy and scammers], [representative product of EVE's emergent sandbox game design], [indefatigable cog of economic prosperity], and [rallying point for impassioned protestors].
    And, starting last Friday, it was on fire.
    Actually, Jita itself wasn't on fire, but thousands of players who normally find it a "safe" haven of trade and economic gameplay found their ships on fire - everything from a lonely hauler on up to the simply massive, highly-armored freighters moving thousands of USD equivalents of ships and goods.
    As you can see from the below "kill" heatmap of the EVE universe (where each dot is one of the 5,431 solar systems in EVE's known universe), it was a pretty big deal.
    Click image to enlarge
    And it all happened because a few players wanted to make it happen and then, after convincing thousands of others to join them from around the world, they made it happen.

    For several months now, a coalition of thousands of players have been planning the "Burn Jita" event, aimed at disrupting the biggest trade hub in game in order to wreak some terrific, universe-wide changes by shaking the very pillars of EVE's economy at its metaphorical heart. Surely there was some sort of larger, meta-economic objective that likely would cause them great profit. This planning happened both in game and outside of game, via voice over IP chat, secret forums, Twitter and more. A sci-fi plot of the new information age if there ever was one.
    Using game mechanics and some very impressive logistics, the coalition, lead by a group known as Goonswarm, amassed a tremendous amount of manpower and ships in a concentrated effort that would probably make many real life military strategists jaw-drop. Bravado-saturated propaganda was also a pretty big part of their plot.
    Fast forward to Friday: Player fleets comprised of high-damage-dealing ships began to destroy high-value industrial ships during a sustained campaign with single, coordinated volleys, sacrificing themselves in waves because the in-game "police" warps in almost immediately with harsh, reprimanding justice.
    Amidst this orchestrated chaos, opportunistic salvagers profited from the wrecks left behind and EVE's deep, talented player community began blogging about, filming, screenshotting, livestreaming and celebrating/decrying the event. Some opposing fleets rose up to fight the invading hordes as well - to mixed success.
    All of it occured in the same framework of unrestricted player movement and limitless player choice in a single, shared game universe. As our senior producer put it "f*cking brilliant". As a participating player put it: "There’s just something special about building 15,000 spaceships and loading their guns with 1 round of ammo to shoot. And doing it right in front of the police."
    As developers we watched in awe at another amazing thing our players brought to the universe we created. Yet there was a lot more going on behind the scenes on our end besides popcorn and a comfy chair...

    Since the "Burn Jita" event was announced well in advance and CCP wants to support player-driven events (as long they are within the rules of the game), we reinforced the Jita solar system on our beefiest hardware, further reinforced all neighbouring systems and set out to monitor the event and provide the best experience we could to willing (and unwilling) participants. Then things started to unfold a day earlier than announced on Friday morning. We gathered data and fine-tuned the systems and as CCP Veritas put it: "It's okay, didn't want that Friday night anyway."
    As we talk about the players' experience, I want to give you a bit of an insight into two projects in EVE in relation to the "Burn Jita" event: A different perspective on Time Dilation and a brief introduction to EVE Metrics.
    We introduced Time Dilation to be able to handle large-scale fleet fights in a more graceful manner. Basically it slows down overall "game time" to make sure each player action is properly handled and executed in order by the servers - keeping it fair and orderly. CCP Veritas of the EVE Dev group called Team Gridlock was the driving force behind Time Dilation (TiDi) and has written a number of devblogs on TiDi:
    Time Dilation was born from the need to better support large-scale fleet fights in 0.0, the lawless area of space where player alliances fully rule and frequently battle in numbers that dwarf entire server populations in other online games. While we have had Time Dilation kick in in low-sec (slightly less lawless area of space) then this is the first time that Time Dilation kicks in for any sustained periods in high-sec (where the police show up) and it's been very beneficial for us to observe TiDi in this environment - a digital gold mine of sorts for our dev team.
    The server CPU profile and general usage profile of Jita has always been very different from the rest of the universe. There are 4 large groups of server-load profiles:
    1. 0.0 fights - where players typically bring big numbers to the dynamically shifting battlefields of lawless, alliance controlled space
    2. mission hubs - where lots of players are running missions for popular NPC agents for fun and profit
    3. market hubs - where the financiers, traders and manufacturers do a lot of collective business (Jita, Amarr, Rens and more...)
    4. general solar systems - just your "average" solar system
    We configured Time Dilation to deal with 0.0 fights although those same parameters mostly work for trade hub server loads like Jita - but could be improved. As a specific example: TiDi has to kick in very aggressively and very quickly during large fleet jump-ins, where a fleet simultaneously transfers between two solar systems. This "use case" doesn't typically apply to Jita. Instead we have to deal with Crime Watch monitoring (i.e., who is "allowed" to kill whom, which also applies to low-sec but not 0.0) and CONCORD response (i.e., the police, which doesn't apply to low-sec or 0.0) for short-burst smaller engagements (but potentially multiple such activities). Different code and different usage profile.
    So we reconfigured TiDi specifically for Jita, only on the node hosting Jita. CCP Veritas provides the details:
    Here's how we handled the server load of a few hundred murderous savages valued customers making ships explode in the most populated system in EVE Online.
    Having a lot of people in Jita is nothing new. Being EVE's main market hub these days, many folks come to sell their produced goods or fit out their spaceships for adventures elsewhere. Sometimes, the strain from all that commerce has been enough to overload the server it runs on, and so we've made a single exception in our "no population caps" rule so that things can remain nice and orderly. This is generally considered an okay thing, since preventing someone from jumping in to make some deals isn't nearly as bad as preventing people from jumping into a huge fleet fight. I'm just giving some historical perspective, of course, because ever since the Inventory Setification optimization that accidently halved Jita's normal load profile, we haven't really been at the cap very much.
    In any case, going into this event we had two dials to adjust - the application of Time Dilation and where the population cap is set. Let the graphs begin!
    Click image to enlarge
    The event kicked off in the very early morning on Friday, 27 April. We weren't entirely sure when exactly it was going to happen, so we had Jita configured just like any other normal day, with a 2,300 player cap and generic TiDi parameters. I probably would have lowered the population cap once things got down into the 20% range, but I was getting my ANSI Standard amount of sleep(), so that didn't happen.
    However, if you showed me this graph without the title and said it was from a fleet fight out in 0.0, I would believe you. The only thing that looks out of place is that the load was 60% before the fighting opened. TiDi is set up to keep the fleet fight load profile in the 80-100% CPU utilization range, which as you can see here it did quite well. From what I hear, the early hours of the event were heavy with counter-play, so I suppose it makes sense that the performance looks like a fleet fight.
    Fast forward a few hours 'til after downtime (and coffee), and we're back up with a much more aggressive population cap - 1,850:
    Click image to enlarge
    We wanted to keep things fairly low at this point as we had found a few bugs in how Crime Watch worked under Time Dilation. It's not something it's asked to do very often so these issues hadn't come up since TiDi's introduction months ago. However, even with such a low cap (1,850 is low these days, how far we've come...), TiDi is kicking in. Unfortunately, it's not keeping load in the nice 80-100% band like I like to see, it's holding around 60-90%, with just a couple spikes here and there keeping it down.
    Later that afternoon, CCP Masterplan hot-patched Jita with a fix for the nastier issue we had found with Crime Watch and I got to work tweaking TiDi and upping the population cap. Essentially, we're looking at a load profile with high ambient load and small spikes. The trick was to find the sweet spot that would keep the ambient load at around 80% CPU and have TiDi be less sensitive to spikes as they were almost guaranteed to be small. Normally we have to be much more aggressive when a spike starts because it could be a huge jump-in or a really effective bomber run, but in this particular use case we're limited to ganks or smartbombings involving tens of ships, not thousands.
    Basically speaking, I made TiDi's tendency to slow down time more relaxed and less frequent, and made its mechanism to speed back up slightly more aggressive. Here's the results from Saturday with a 2,200 player cap:
    Click image to enlarge
    Sweet spot achieved!
    So how was the experience in Jita? Lazarus Telraven, one of Goonswarm's main Fleet Commanders, says "This weekend has been a milestone for EVE, the first time anyone has declared war on a solar system. The game worked wonderfully. Among the Jita Burners complaints are few and praise is plentiful, but we all agree that without TiDi none of this would have been possible. In fact, after asking all of our allies what they thought I only received one complaint! The only real problems that I was asked to mention was blackscreening on undock and not being able to make a war target only overview. When I asked our fleets to offer comments on the performance the response was overwhelmingly positive. GSF member Bagdon said that "opinion on performance for ccp: ****ing brilliant. all brackets on, all effects on, I might even start playing more after this!" The number of people in Jita spiked between 1500-2300 pilots the entire weekend with fighting, fleet warping, fleets undocking, and just loads of everything in the system but it all worked beautifully as TiDi was up and down in prime times. With everything being said we can't wait for the rest of the changes coming with Inferno."
    The comments on the client performance ("****ing brilliant. all brackets on, all effects on") were heart-warming to us since we have recently put a lot of effort into client performance, per CCP Veritas' devblog Your Client, Made Gooder, starting with the release of EVE Online: Crucible 1.2 in February. In part, that effort focuses on increasing the client performance but we are also spending good time on identifying and removing frame rate spikes and drops; making the client run more smooth, i.e., be more responsive and more consistent.

    For those that are interested in some numbers here are some statistics about the weekend in Jita gathered using a new system, called EVE Metrics, that is currently in development. CCP Atlas provides the details:
    First off, the top 20 weapons/ammunition used over these three days:
    Weapon Charge # Shots Damage
    1400mm Howitzer Artillery II Republic Fleet Phased Plasma L 706 2,276,304
    1400mm Gallium Cannon Republic Fleet Phased Plasma L 449 1,710,348
    1400mm Howitzer Artillery II Republic Fleet EMP L 1,007 1,464,334
    1400mm Gallium Cannon Arch Angel Phased Plasma L 365 1,164,404
    Heavy Missile Launcher II Scourge Fury Heavy Missile 2,353 985,594
    Large EMP Smartbomb I 6,383 979,762
    Heavy Missile Launcher II Caldari Navy Scourge Heavy Missile 2,581 966,753
    280mm Howitzer Artillery I Republic Fleet Fusion S 2,699 940,617
    1400mm Howitzer Artillery II Tremor L 1,411 829,474
    1400mm Carbine Howitzer I Republic Fleet Phased Plasma L 205 828,913
    Large Graviton Smartbomb I 5,879 775,961
    1400mm Gallium Cannon Republic Fleet EMP L 225 683,016
    1400mm Carbine Howitzer I Republic Fleet EMP L 247 565,172
    280mm Howitzer Artillery I Republic Fleet EMP S 1,473 549,903
    Large YF-12a Smartbomb 2,909 542,685
    Neutron Blaster Cannon II Void L 653 527,318
    Mega Pulse Laser II Scorch L 1,135 519,119
    425mm Railgun II Spike L 1,132 509,942
    1400mm Prototype Siege Cannon Republic Fleet Phased Plasma L 196 495,839
    280mm Howitzer Artillery II Republic Fleet EMP S 1,722 472,955
    Next, let's compare weapon module activations this past weekend with the previous weekend:
    Click image to enlarge
    Click image to enlarge
    Click image to enlarge
    Click image to enlarge
    Note that these graphs do not show how often these modules fired but rather how often people clicked the button to start them cycling. The total damage inflicted over these three days amounted to 45,117,952 hit points delivered in 249,021 distinct hits.
    Here are the damage and hits timelines this past weekend:
    Click image to enlarge
    Click image to enlarge
    Note that in all the graphs above we're looking at one hour moving averages.
    Finally, here we see the number of market orders placed in Jita per day since the beginning of March until this past weekend:
    Click image to enlarge
    The red dotted line signifies what we would have expected to see during a normal weekend.
    So while many ships were destroyed this weekend, it sure feels good to be the hosts of such massive amounts of spaceship carnage. We'll always be prepping for the next big event our players throw at us.
    Make your stories happen!
    - CCP Explorer // Software Director // Follow on: Twitter / Google+
    (Devblog co-authored by CCP Atlas, CCP Manifest and CCP Veritas)

    New to EVE? Start your 14-day free trial today.
    Returning pilot? Visit Account Management for the latest offers and promotions.

    Continue reading...