Forums › English Language Forums › General › General Discussion

Search

Shooting oneself in the foot. Alt title: how testing servers for this game is fundamentally flawed

34 replies [Last post]
Tue, 05/13/2014 - 14:36
Lunamon's picture
Lunamon

Now, I usually don't post here, (Though that might have to do with the general low quality of the forum community.) but I kinda want to bring this topic up anyway, since it seems no one else is doing it.

Locking the test servers behind a pay wall is kind of dumb, and in a way, prevents testing to actually be done.
Now, if anyone have any preconceptions that having people pay you to play test your game is a valid way of doing things: I suggest you actually consider the implications of that.

Now, the problem with locking play testing behind a pay wall is that you limit the amount of players, and what kind of players, that can test your content. There is a very small user base that is allowed on the test server, and it's limited to those who have spent any money on the game in the latest few months. That is running on a lot of assumptions, for example, that they are the most competent ones at providing test feedback, that they are a wide variety of players that have a wide variety of opinions and points of view, etc.

Don't get me wrong, I understand the reasoning with the choice:
A. You want players who are dedicated to the game to the be the ones to test.
B. You also, ideally, want a rather small selection of players to test your game in order to keep track of all the data.

However, I don't agree with that reasoning, nor the solution. First of all, the people who pay money in a game like this comes in two types: Those who value their time more than their money, and those who wants to support the developer. I personally fall into the latter, but the former is by far the most common. Although, to the majority it's a mix of both, to some degree. As such, I don't think listing people who have payed in the recent times is any proof of dedication. In fact, I'd go as far as to say that if someone has put any real money into the game is irrelevant, and is no proof that they are dedicated to it.

I would also like to point out that listening to feedback from only dedicated players in the first place is fundamentally flawed. The game is clearly advertised and promoted in a way that is to bring in new players who may be inexperienced with the genre, or games in general. As such, feedback from players who are yet not dedicated to the game is of great value! Or to sum up: Inexperience =! irrelevance. (Though, getting test feedback from this demographic is by far the hardest. But almost entirely locking them out is not exactly helping.

The problem of not wanting to get flooded with data... well, when you run a continuously updated game, people are going to have very strong opinions on it! But it's also important to take into account that people don't always know what they want, and they don't always know what data YOU need. The forum approach that you're running today works, but it's primitive. Anyone who takes up game design today knows that making sure you get the right feedback is very important. Surveys are a great way to do that, (though, not the only way, as should be obvious!) as long as you still leave room for players to provide their own thoughts and opinions.

That is all! ~Lunamon

(And now we wait for people to make completely irrelevant replies with argumentation and reasoning that is not based on any logic in this realm of existence!)

Tue, 05/13/2014 - 14:52
#1
Flowchart's picture
Flowchart

Now, I usually don't post here, (Though that might have to do with the general low quality of the forum community.) but I kinda want to bring this topic up anyway, since it seems no one else is doing it.

This actually comes up all the time. Heck, I knew what your complaint would be when I saw the subject.
But if you don't regularly post here, that can be excused.

Here's the last topic about this, if you want to see the replies there
http://forums.spiralknights.com/en/node/100140

(And now we wait for people to make completely irrelevant replies with argumentation and reasoning that is not based on any logic in this realm of existence!)

So you come here, preinsult everyone here, and complain about the low quality of the forum community.

Tue, 05/13/2014 - 14:55
#2
Poomph's picture
Poomph

"(Though that might have to do with the general low quality of the forum community.)"

I'm sorry we have displeased you your majesty! Please forgive us! We're unworthy, we're unworthy!

"(And now we wait for people to make completely irrelevant replies with argumentation and reasoning that is not based on any logic in this realm of existence!)"

Oh no! We are not forgiven! Save yourselves, none shall escape the wrath!

Tue, 05/13/2014 - 15:10
#3
Pandafishie's picture
Pandafishie
hai guise...

i lost the game.

Tue, 05/13/2014 - 15:17
#4
Avenger-Of-Troy's picture
Avenger-Of-Troy
Aww... but you're so wrong!

We're lucky that they even GIVE us test servers.

It works like this:
OOO: Hey guys, thanks to everyone who payed in the last month or so, wanna have a free browny for helping us out?

So basicly, it's a reward for paying.

Tue, 05/13/2014 - 16:17
#5
Autofire's picture
Autofire
You are experiencing a PICNIC ERROR!

@ random posters
/sigh
You guys are proving the OP's point. Can't you prove the OP wrong? Why do you even have to act poorly in the first place? :P

@ OP
Anyway, I do agree that putting testing servers behind a pay wall is a problem. However, it should depend on what they're testing. If they're testing content aimed at new players, it should be available to all players so that new ones can try it. Others, though, like this gunner update might be better off tested by the ones in the know. You could then access the testing server either through paying real cash (like now), energy, or a random drawing. I really think it should depend on what's being tested, rather than everything being the same.

EDIT: At the same time, saying that the same forum community which you are addressing has low quality, you're pretty much baiting for insults and off-topicness, just to spite you. I suggest to retract what you've said if you want to keep this a bit more on topic.

Tue, 05/13/2014 - 17:19
#6
Lunamon's picture
Lunamon
Avenger Of Troy:Actually,

Avenger Of Troy:

Actually, no, we are not lucky at all that they are giving us a test server. And in fact, having such attitude is problematic. You see, play testing is something you have to do in order to be sure you deliver a good product. Without play testing your product will be of less quality, and as such it will sell less. (This works slightly differently for constantly updating games, but the concept still remains.)

And considering that they are also using the test server for beta testing... well, that means they are having paying customer doing payed work for free. So basically, by saying it's a reward for paying, you're devaluing not only other peoples work, but also the value that the feedback actually has.

Autofire:

Randomly drafting a play testing team should be fairly easy. Even if OOO don't keep track of their own metrics (but I assume they do.) Steam provides some. Ideally, they would pick players who have X hours put into the game and have been active X hours in the last few months. Regardless, I think any system would be better than the currently established one. I believe that the majority of the people who play this game care about it, and as such would look positively if approached with the offer of testing. And even those who don't, or those who can't test for X reason, will still be a minority, and new testers can be selected and approached with the offer of testing. Obviously, it would have to start quite a while before the test server goes live, so planning is key. (Which, admittingly, does not seem to be OOO's forte.) And, this is purely speculation, but I think such system would bring a sense of wonder/excitement within the community.

@Edit:
I was going to get those kind of replies regardless, so I figured I might as well address them in the OP so I never have to do it again. And I have no qualms with letting others know how detrimental I think such behaviour is to the community.

Tue, 05/13/2014 - 17:49
#7
Autofire's picture
Autofire
You are experiencing a PICNIC ERROR!

Maybe they could use random drafting combined with access-with-cash...or maybe not. I'm not really sure. Maybe they could have a testing server...to test ways to access it? xD

Now that I think about it, there was a time where people were getting gifted 30 days of free play time. (They gave out elevator passes to random people, probably as a way to see how things would look before removing mist energy.) Because of that, they definitely have means to pass out testing server rights.

Also, about 'preinsult'ing the community, as flowchart put it: I'm just saying that you might end up with more comments that you mentioned, which may reduce the people actually considering what you've posted. But I digress; it's your call, so I won't be a nag.

As a matter of fact, I remember a while ago when I actually did a similar thing. My goal was to bait out a specific kind of player, who would then respond by being the opposite of how I said they would be. It had mixed results...but I'll drop it. This is off topic.

Tue, 05/13/2014 - 21:45
#8
Fehzor's picture
Fehzor

If we are to view this as acceptance testing, then it is supposed to be for the client and those who will use the product. In this case, we hit some portion of both. The testing server has to be closed off somehow as well. If it wasn't, then EVERYONE could come onto it, and we'd have to support it, it would ruin the update with spoilers, you know.

But more to the point..................

When the developers go in, they're not looking for feedback on what they should do, they're looking for feedback on what they've done. In other words-- does it work correctly? They don't care about us. If they want to go in and destroy the shard bombs with garbage then they've already made the decision to do so, and they think that it'll be best for the game. But if we find a bug with the bombs? That's on them. Maybe they'll read what we write about it and change stuff based on it. But that'd be up to them, really.

Wed, 05/14/2014 - 02:16
#9
Lunamon's picture
Lunamon
Fehzor: The test server, as

Fehzor:

The test server, as it is today, already spoils everything anyway. So, I can't really understand that point of argument. However, if one wants discretion, you could just have everyone who signs up on the test server sign a none disclosure agreement. Though, that seems, to me, to be more effort than it's worth. Let's also not forget that people dig out the content of this game by reverse engineering it long before stuff is announced.

The crystal bomb issue is an interesting one. I actually agree with the choice of changing it from how it was, since it was not only a bug, the behaviour was weird. Whether or not one likes the execution, well, that's a personal preference. (Personally, I like how they work, it's just that I feel they deal way too little damage to be useful.) I think, that when OOO goes into testing, there are things that they won't falter on or change. This has to do with their perspective of the game, and ideally a better sense of game balance than the players. The feedback they got from the shard bombs range from, "it's stupid you should bring back the old ones" to "these are underpowered". It let's them know some general things, but honestly, I doubt that this was the feedback they were looking for, hence, why a survey would work better.

The issue here is not their decision though. Rather, I feel it's the lack of openness to the community. While we might not agree with it, I feel it would be for the better if they provided a reasoning behind their choices, at least the more controversial ones. It would show that they value feedback and shows that they respect others that disagree. As it is now, OOO is mostly a giant silent wall. That may work when making some games, but for Spiral knights, I can't agree. But hey, we are talking about the company that makes changes to their game but don't bother letting people know in the patch notes. *sigh*

We are getting a bit off topic here, but that's fine a suppose.

Wed, 05/14/2014 - 02:56
#10
Avenger-Of-Troy's picture
Avenger-Of-Troy
Here's the answer:

Life's unfair. Deal with it.

Wed, 05/14/2014 - 03:13
#11
Mtax-Forum's picture
Mtax-Forum

There are two basic explanations;
First one is that OOO do fix only serious and visible bugs. That was totally visible after Battle Sprites test - theres no serious game affecting bugs, but tons of graphical ones for example, so they limit accepted bug reports to the most common ones of small ammout of players.
Second one is server issue. All of us know that servers are hosted on potato and if you would play at least one evening CET you would see how laggy it is even when that little ammout of testers plays. At last Gunner test I was dancing around with "Revive now" note. Preview server also causes lag to main one. Lifting limit of paid players would be exactly shooting oneself in the foot.

5$ is price of one small pizza, everyone can affort this without big lose.

Wed, 05/14/2014 - 04:50
#12
Hellscreams-Forum's picture
Hellscreams-Forum
@OP

Pay to get into test server is a great idea...I mean if it's free, What would super special update be...

Wed, 05/14/2014 - 04:53
#13
Dermog's picture
Dermog
@panda

....Now I have to tell everyone else in my family that I lost the game....

Jokes aside, why even come to the forums to complain about something that everyone knows about already and expect the forum community to be a bunch of [snowballs]? I mean yeah, some people here don't have the greatest English, but that doesn't mean they're stupid...

Such condescend, much meanie, wow.

Wed, 05/14/2014 - 06:10
#14
Arcknightdelta's picture
Arcknightdelta
I started a very similar

I started a very similar thread to this. Unfortunately someone in our "wonderful" community decided to talk about what was in the Gunner update. No names but I'm sure the comments were around Midnight if that helps refresh some memories.

The 'unfortunately' part is, because of that, my thread was banished to the shadow realms. I can see it, I just can't post in it.

Wed, 05/14/2014 - 07:00
#15
Glittertind

"Locking the test servers behind a pay wall is kind of dumb, and in a way, prevents testing to actually be done." -Lunamon

I see where you're coming from with this, but I really want to nitpick on how Three Ringos don't call it "testing server", but preview. "Preview server".
Sure, they have a testing forum and they're asking for feedback and whatnot, but it is still called preview server... If you ask me, it sounds like it's only for previewing upcoming changes for a price, and that's honestly how I've thought about it. Yes, I'm on thin ice with this, but still. Couple this with Three Rings' grave-silent attitude and notoriety of pushing out buggy, unwanted updates...

Wed, 05/14/2014 - 10:55
#16
Autofire's picture
Autofire
You are experiencing a PICNIC ERROR!

@Glittertind

Maybe they should change the name of the server, then. :P

(Yes, I agree with you. I think right now it's not true to its name.)

Wed, 05/14/2014 - 11:11
#17
Whimsicality's picture
Whimsicality

Access to the preview server is a luxury (heck, having decent internet access is luxury enough). Luxuries, by their very nature come at a cost. Other companies restrict their testing groups to whom they select; Three Rings accepts everyone dedicated enough to their game to have dropped a mere $5 in the past three months (seriously? that's a lot?). Whether or not that's lenient enough is personal opinion.

Wed, 05/14/2014 - 11:19
#18
Poothis's picture
Poothis
My thoughts

OOO needs an incentive for people to pay to gain funds. Also, if everyone could test, then they are already trying out the update already. They might as well just release it already if that was the case and take all the smack from how unbalanced everything would be.

Wed, 05/14/2014 - 14:12
#19
Feyi-Feyi's picture
Feyi-Feyi

You are not entitled to anything except the game. It is free, have you noticed?
If you have no trust in the people testing things, you're always allowed to actually pay for the game and join us in the test server.

It is not because a community does not agree with you, that we are of low quality.
Opening with a statement like that is bait, and says more about you than the community.

I won't waste any more words on this.

Wed, 05/14/2014 - 15:13
#20
Nosgoth-One's picture
Nosgoth-One

*applaud Feyi-Feyi opinion*

I honestly believe (not saying everyone do) that the reason who people want to play on "preview server" its because they just cant wait for the new guns update and honestly i dont think they would give feedback at all about the weapons (again not like every tester do that) people just cant wait for the update only because OOO made the mistake about the release date, and now people believe "They are entitled" to test the content because they were "tricked" by OOO and they "deserved" to play the content without pay.

Do people want so badly play the "content that its UNDER TESTING?" get 5 bucks and get some energy and there you go. you can play as much you want even if you dont give any feedback.

Dont want to pay because you believe a game should be free? okay, get TF2, people these days have energy, i mean the play vana like mindless zombies right? get some of your CE traded from another player who can give you TF2 Premium, get some metals, make it refined metal, then trade it for keys, then sell it at the steam market, get 5 bucks and there you go. only need time.

Wed, 05/14/2014 - 19:45
#21
Welux's picture
Welux
*Random Noises*

Well, allowing people that pay to get into the test server will make people spend lots of that juicy cash on CE. Making OOO moonies,

But, if you let it open to the public, you will get people that are generally uneducated about the game not only getting a freebie to see whats in the update, but they're also going to give biased and generally uneducated feedback. I could have gotten into the test server, but i didnt because im not very knowledgeable about any of the guns, being mainly a sword user, so my feedback would have been biased, uneducated, and worthless garbage.

Now imagine 100+ posts of that. people saying that something broken or blatantly overpowered is ok, and not to fix it. All of the educated, game-smart people that were gunners or super endgame (have like every weapon/armor) would be drowned out, ignored, and they'd be the minority, and not many listen to the minority.

Now, i'm probably gonna get "So you're saying if you don't pay you're not knowledgeble?"
No, but im just making an observation, as most people that pay and post feedback are long time players, are part of the forum community, and this hasn't been their first rodeo.

Wed, 05/14/2014 - 20:10
#22
Klipik-Forum's picture
Klipik-Forum
*tries to dodge firestorm of hate*

I would prefer the test server be a closed-beta thing with NDA's and all that, both to get people who definitely care about the game and to get rid of all the ridiculous leakage problem OOO has with new content. For heaven's sake, anyone who wants to can browse around their computer library and find almost all of the planned, removed, unreleased, canceled, testing, and conceptual content OOO even has. That combined with OOO's bad habit of announcing things way, way before they're ready and then delaying them several times creates a bitter community. If OOO were to restrict access to downloading the preview server files to only the people allowed into closed beta testing, and NDA'ed them to not tell or show anyone about it, and actually announced things and stuck to a schedule that made any amount of sense, none of these things would be a problem. The current community negativity stems from two main things: botched announcements and bad community management (I'm looking at you, UI update. Still no explanation...). Fix one, and the other practically fixes itself.

I feel like that got a bit off-topic... in short, the current test server format is both bad for testing and bad for security. Fix both.

Thu, 05/15/2014 - 18:11
#23
Magnum-Frost's picture
Magnum-Frost
-Generic Subject Line-

>playing on the test servers
>actually testing

I didn't pay good money to go to the testing server and do things like math, I spent hard earned ca$h on that. Screwing around and hyperventilating over spoilers FTW.

Thu, 05/15/2014 - 18:36
#24
Klipik-Forum's picture
Klipik-Forum

Then there's no reason for OOO to let you on the test server.

Thu, 05/15/2014 - 18:58
#25
Welux's picture
Welux
^

Well, Magnum paid, which means that Magnum is allowed to go into the server if he/she want's.
If Magnum doesn't want to test, it's not exactly needed, since others are doing the testing.

I only didn't do it because i WANTED to be able to give feedback and help the dev team. But, i was inexperienced. So i didn't.
Also, i don't play LD. So, yeah.

Thu, 05/15/2014 - 22:57
#26
Fehzor's picture
Fehzor

There are still a surprisingly large number of players that don't even know what the new guns will be. OOO doesn't want us to spoil the update for them, and I guess that's their decision.

But I would really have to agree with you on the whole feedback thing- they should be honest and open with their community, especially on shaky things like the shard bombs and the future of the game as a whole.

Fri, 05/16/2014 - 01:30
#27
Mookie-Cookie's picture
Mookie-Cookie
Beep.

More or less everything's been covered by everyone else that's posted prior to me, but I figured I'd throw my 2 cents in for the hell of it.

The pay wall is not really fundamentally flawed, it's just a methodology of establishing your demographic. They're throwing some new stuff together, and they want the people that actually financially support them to be the first ones to see it, and get to have a play with it. That's their call, and it's not actually a bad idea. It's just demographically awkward.

Think of it this way. Assume you have 5000 players. 200 of them have paid something towards the game over the period that they've played. From a very simple point-of-view, that implies those 200 deemed the value they get for their money was worth it; or they just felt like it would be good to support the developer (albeit, it's a good idea to base numbers on the former reason, not the latter).

Now, assume you have a new aspect of your game that you're about to roll out. You could:
1) Only ever debug internally (ie. alpha test) and fix issues as they arrive after it goes live.
2) Release externally to smaller group of players, and fix issues as they're reported. Then roll out the debugged version.
3) Release externally to the entire group, but have the tests run independently of the main game (so essentially the same as 1, but the roll-out is for testing, not to "go live").

Now, 1 is the worst possible idea a development team could use. It'll just lead to customer dissatisfaction and serious damage to the team's reputation if issues arise. So there's 2 and 3, right? Well, obviously OOO is going with 2. 3 would just be a mess of bug reports and issues, which complicate a team's ability to determine the actual issue.

So, 2! Seems like a pretty sweet way to go, right? Well, how do you determine who should belong in that group? You could:
1) Pick at random, from a list of players online within a set period of time.
2) Apply a constraint allowing players who seem to support the game to test.

So, you're in favour of method 1, whereas OOO has opt'd for method 2. In terms of demographics, you're probably more likely to find players that actually do have an interest in the game using method 2, than you would in 1. If you never spend a single dollar on the game, from a perspective of someone who has no idea of your actual connection to the game, you look like you're not particularly bothered about the game. Those who have paid, it's the exact opposite.

So! After all of that rubbish, I understand why you feel the pay wall is a bad thing; but I'm here to tell that it's fundamentally a pretty solid idea. Chances the players actually dedicating money to the game will be of more use than players that don't spend anything, are just better.

You have to remember OOO's cannot go around asking every recent player if they fancy the idea of testing new aspects of SK before OOO rolls them out. It's simply unrealistic for them.

Fri, 05/16/2014 - 06:06
#28
Fehzor's picture
Fehzor

While I'm on here, this is relevant:

http://en.wikipedia.org/wiki/Software_testing

We don't know what they're hoping to test. If I had to guess, I'd say what we do is a mix of acceptance and system testing. They want to know if its ready and good for the customer and if there are any outstanding defects that we can find. Anyone can find defects, so why not have the customers do all the testing?

Fri, 05/16/2014 - 11:55
#29
Uberer-Alt's picture
Uberer-Alt

Maybe choose who gets to play on the preview server based on play time or something else that shows how experienced you are.

Fri, 05/16/2014 - 12:17
#30
Polychromous's picture
Polychromous

@Uberer-Alt Yes, how many times we have grinded Vanduke solo without dying. It seems so silly that it might end up being an accurate way to choose players, hah.

Sat, 05/17/2014 - 07:17
#31
Pepperonius's picture
Pepperonius
While I am in favor of the pay option for the preview server

Another option would be for those who completed the SK survey to have access. Money isn't the only way to show you care, as not all of our players have access to cash. Our 13 year olds have parents, sure, but some of them won't allow in game spending. At the same time, money!=passion for the game, but taking the time to do the survey means at least on some level, they care.

That would be my suggestion.

Also, i second literally everything Mookie-Cookie said. Solidarity, my ginger.

Sun, 05/18/2014 - 01:08
#32
Alphappy's picture
Alphappy

I think the paywall is not intended to include only those they want to be involved in the test, but to exclude those they don't want involved in the test. Generally speaking, if someone can't pay $5, they either don't care enough about the game to actually test, or they're too young. It is not hard to open a checking account and link a card to it, and $5 every three months is nothing for someone who is paying for internet in the first place.

There's not really any meaningful alternative to a paywall, anyway. Using hours played would encourage idling, grinding or just generally doing nothing useful, which would imply(or soon cause) disinterest in the game. The survey might work if they announced it in-game(I don't blame people who don't want to burn brain cells on this forum), but they would have to really change its content. The last survey was "what do you want to see," "did you like the prize boxes," and "are you active on social websites"(????). It doesn't really ask about your knowledge of the game or even your opinion on existing gear. Additionally, a paywall defines a single, static bar for admission, whereas the survey would be subjective selection and feelings would probably be hurt.

Sun, 05/18/2014 - 02:01
#33
Ewbte's picture
Ewbte
/o

/e dont see any reason to talk with people who don't have 5$ once in a 3 months for game they love to play. Oh excuse me 4.95$, guess that's a big difference.

Sun, 05/18/2014 - 08:33
#34
Klipik-Forum's picture
Klipik-Forum

The problem isn't the actual price point, it's the idea of the pay wall and even more so about the people it lets in rather than the people it keeps out.

Powered by Drupal, an open source content management system