Thursday Interviews: ISD – ECAID

We bring you a series of interviews with some of the guys and gals that make the squeeky weels in EVE Online less squeeky by generously donating their spare time to make your spare time more enjoyable.

The Interstellar Services Divisions (or simply ISD) are the voluntary players that fill out a variety of roles in EVE. Read on to find out more about the ECAID (Bug Hunters) and find out more about their day-to-day responsibilities.

 

Vince: What ISD name do you usually go by?

Garik: I go by just Garik. The official (RP) team name is Equipment Certification and Anomaly Investigations Division but we go by Bug Hunters. Easier to pronounce.

Vince: The Interstellar Service Department has been in service for quite a while now. For how long have you been on the Bughunter team?

Garik:  I’ve been with ISD since 2005 and with the Bug Hunter team since 2007. I had to look that up, I couldn’t remember it myself – it’s been so long!

Vince: Have you been part of other ISD programs before you joined the ECAID?

Garik: Yes, I was a part of the forum moderation team for a couple of years before its sunset.

Vince: We’re aware that the voluntary programs require you to each donate some of your own free time, how much would you say that this amounts to on a weekly/monthly basis – for you?

Garik: The amount of time depends on the situation. It’s always higher around a release, or in the middle of recruitment I spend a significant amount of time overseeing various matters with the QA team and the rest of the BH leads. I don’t think I have specific “number” because for a lead we never ended up keeping track.

Vince: Are you aware of how much time the other volunteers use?

Garik: That depends from one vol to another. Some of our most active volunteers spend more time on the test servers and on the bug list than the live servers! But on average it isn’t more 3-4 hours a week. Its not all spent on core bug hunting though. There’s enough of idle chit-chat and gossip in our irc channels.

Vince: What are “the special perks”? 😉

Garik: I get a shiny indestructible ship and a rank I would never have in real life. Seriously, the ship is really shiny… and pointy.

Vince: So outside of being a regular player – what does a regular day of bug hunting consists of during the busy days?

Garik: Reviewing player filed reports and digging through the submitted reports and updates to find the cause of the bug. A lot of time is spent on the test server trying to reproduce other bugs. Often we just play around with new features finding bugs without any user input – this is a key component of our testing mandate. It helps the QA catch a lot of issues before they go live.

Vince: There seem to be some degree of networking involved as well. Do you get information from moderators, GM’s, CSM’s or just the developers?

Garik: We sometimes liaise with GMs on certain special occasions. We get reports from the GMs when they start to encounter reports of what could be bugs, or users encountering strange game quirks (hello backwards flying ships!) Ofcourse there is our close involvement with the QA team on an almost daily basis — we require their help to our daily work.

Vince: Can you tell us the process of bugs life, from its hatched till its fully assimilated into the drone regions (ie. fixed)?

Garik: The lifecycle of a bug can be broken down into 4 distinct steps

1. Reporting of the bug – This when the issue is spotted and reported by a user or by a member of the BH team. If it is from the end user we spend a little more time getting detailed information like DxDiag reports (computer specifications) and any log server information. Depending on the bug we may ask for more info like Agent name, system etc.
2. Filter & Test Bug Report – Once we have the required information, a bug hunter will assess the report. This can involve recruiting additional help from another BH or member of the QA team. The bug hunter will try to reproduce the bug on the test servers in various different scenarios. If the bug cannot be reproduced or is an intended feature – it is closed and email notice sent to the user informing them of this fact.
3. Defect – Once the bug is reproduced a detailed report is created with additional insider information helpful to the developers. This report, called a “Defect” is submitted to the CCP team. We contribute screenshots, hardware specifications, detailed logs to help the developers narrow down the source of the bug.
4. Life as Defect – A member of the CCP Dev Team (usually the Dev Manager) will assign a developer to analyze the defect and  work on a fix. Once a fix has been found for this bug, it is deployed on the test server and passed on to the QA team and the BHs who verify that this fix has solved the problem. If it isn’t solved it goes back to the developer and this goes back and forth till a proper solution has been found.
   Post – Bug Cycle.
5. One final stage is the release of this fix. Often fixes stay on the test server for a while till they are pushed on to TQ. Bug fixes go out in small patches called “Hot fixes” or they go out in a minor release – which is when user gets to see them. The release of the bug fix depends on the severity of the bug – high priority bug fixes often get out really fast.

Vince: What was the most interesting bug you’ve encountered so far, and what made it stand out compared to other bugs?

Garik: One of the more interesting ones I recall was related to lag. One of our more dedicated Bug Hunters spotted a possible memory leak which could induce lag and client freezes during large fights. The bug was hidden in a sneaky location near the ship’s exhaust trails. This bug was found due to a very detailed observation of the performance monitor and how the ships were used. It was great catch and even got a mention on one of the dev blogs.

This bug stood out because it wasn’t directly reported and it dealt with an ever present problem in Eve – Lag. The bug helped reduce the memory footprint during big fights and it took a sharp eye to spot something this far off the reservation.

Vince: How fast are bugs usually squashed? Is the process quite long or do you sometimes get those bugs that are fixed in 4 minutes as well?

Garik: That depends on the definition of “fixed”. High priority bugs are fixed are quickly and get pushed quickly in server-side patches or hotfixes in client patches. Minor issues get fixed on the test servers soon enough, but they may take a while to show up on TQ based on CCP’s release cycle. There are some bugs that are so complicated to reproduce (requiring some specific hardware/os combo) that it will take quite a while to fix. Other bugs are just too low on the totem pole for getting developer time.

Vince: As a player, what would you say has been the most rewarding experience in EVE Online for you?

Garik: The most reward experience has been the community and the nature of the game. I’ve met a lot of new people and gotten to know them in-game and out of game from all around the world. The Eve game in itself, I’ve enjoyed the sandbox nature and its let me do a whole lot of whacky stuff that I wouldn’t have before.

Vince: Do you have any other interesting stories that you would like to share?

Garik: There are plenty, I could tell you… but then I’d have to kill you 🙂

Vince: Some topics of interest on the forums seems to be NeX, FiS vs WiS and the coming null-sec development plans. We realise that they are contentious topics but we’re going to ask you anyway: “What are your thoughts on these topics, personally?”

Garik: I’ve not been big on WiS and such because I liked “old school” Eve. WiS a good improvement towards Eve and look forward the full feature, but you shouldn’t forget what makes Eve strong – FiS. Can’t say more without revealing who I am 🙂

Vince: What are you looking forward to in the next year of playing EVE Online?

Garik: I wrote a lengthy comment here, but after Hellmar’s recent blog and Zulu’s follow up blog post I’ve removed it. I think it’s good to see changes coming in to good ol’ regular PvP. I’m going to look forward to these changes a lot.

Vince: Why do you play EVE Online?

Garik: Ah that’s interesting. I started Eve just as way to kill time and try out a new style of gaming, it was my first MMRPOG. Soon I met a lot of people and the sense of comaraderie and group dynamics in Eve was too good to pass up. That’s why I play Eve and will probably continue doing so. The game has evolved beyond regular “pew-pew” and has become a little metaphor for old “Wild Wild West” within our lives.

 

We’d like to thank Garik for being willing to share additional spare time with us by participating in this interview and for shedding some more light on the work involved for a Bug Hunter! Feel free to post comments, likes and questions for the next batch of interviews with the Vols!

If you want to touch base with Garik yourself, he usually hangs out in the IRC channel.