NOTE: Due to the fact that people seem to be completely mis-reading this question, I'll quote the last paragraph up front:

Just to be clear, I'm not against the concept of story identification questions, and some (very few) of them are quite great.

TL;DR version: "My personal evaluation of intrinsic worth of questions seems wildly out of sync with the aggregate community based on the # of upvotes they receive. Why?


Now, for the full original question:

I am genuinely puzzled by the fact that questions are so well rewarded. Why? Is it good?


First, a couple of stats. As of now, the site has close to 300 of questions tagged - out of 2000 total. Some of them seem to be mis-tagged (and closed), so let's say 250.

Of those:

  • The 50 highest voted ones have 11+ upvotes each

  • The second 50 highest voted are 8-11 upvotes each.

  • The third 50 highest voted are 6-8 upvotes each.

  • 2 highest voted ones are 30 and 28 upvotes (admittedly, 28 upvotes one is a great one that even I up-voted).

  • Of the latest 50 tags that I analyzed, the average was 6.7 up-votes, with ~3 standard deviation. Here's the histogram (X-axys is the question's # of up-votes, Y-axis is # of questions with so many up-votes):

    enter image description here


My questions are:

  • Somewhat rhethorically: why are those questions so much up-voted? Are they really THAT useful to so many people or so brilliantly written? I don't know the stats for the entire site but i'd not be surprised if site-wide upvote average is lower than 6.7.

    Personally, I feel that those questions, while perfectly OK to be on-topic, are not a big win for the site and should not be stongly encouraged by lots of (in my humble opinion) undeserved up-votes. If I posted an identification question - which I may have but don't recall - I would honestly expect at most 1 upvote if it was about some obscure work with no great interest as far as overall SciFi field.

  • Does the community feel that these questions are in some way, as a class, helpful to the site? Do the drive many visitors? Do they bring in some special nuggets of knowledge that nobody knew about yet many people would find useful?

Please note that I specifically refer to upvotes on the questions themselves - I don't feel that the ones on the answers are in any way un-deserved.


Just to be clear, I'm not against the concept of story identification questions, and some (very few) of them are quite great, either due to the question's innate quality, or a really interesting take on a well known work, or just genuinely interesting (e.g. this). But a vast majority that I see are nowhere near special.

share
5  
I think part of what you are seeing is that people have a tendency to upvote questions and answers about works they like or find interesting. I've probably upvoted an identification question because it lead me to a work I didnt know yet or had forgotten. –  Mark Rogers Jan 5 '12 at 22:03
1  
@Mark - the part I'm curious about is distinction between "didn't know yet" and "was worth finding about". There are gaziooions of SF&F works I don't know about, and for 90% of them that are, according to a well known law, crap, the value of learning about it is zero. –  DVK Jan 5 '12 at 23:33
    
BTW, I'm going to run a minor experiment. I will post what I personally consider an interesting story identification question, see what happens. scifi.stackexchange.com/questions/8619/… –  DVK Jan 6 '12 at 17:17
    
I hate to be a NARC here and I respect your great contribution to the site as a writer, but the site isn't really for people's individual voting experiments. Also it seems mostly driven by angst at identification questions. Even if you score a high value on your fake question, how does that change matters really? I don't think your going to get identification outlawed. –  Mark Rogers Jan 6 '12 at 18:45
1  
@MarkRogers - Can you please explain in detail why it is a fake question? Would you have hated the same question (try to think objectively) if it came from random user, or would you have up-voted it? Also, read my comment carefully. The experiment was me posting what I would consider a good question (i.e worth of an upvote from me). –  DVK Jan 6 '12 at 18:50
    
I'm sorry man I'm not going to get into this drama. I'm not going to close the question at the moment. But you truly understand what I mean, your merely trying to spear identification supporters on your angst. –  Mark Rogers Jan 6 '12 at 18:52
1  
@Mark -you need to stop guessing at people's intentions. You're not very successful at that based on that last comment. If you carefully read my post(last paragraph), you will see that I explicitly stated that some ID questions can be good ones. I came up with an existing example of one type, and couldn't come with an existing example of a second one, so tried to create one. If you don't like it, downvote it. Don't treat it as some grand conspiracy. If I wanted to use it to drive a point, I'd have posted it before posting this thing on Meta, and created a throwaway account for the test –  DVK Jan 6 '12 at 18:57
    
@MarkRogers - also, as is the usual courtesy, if you were the one who down-voted, explain why; and if it's not the quality of the question, explain that "posting questions of OK quality for reasons other than wanting to find the answer is a no-no" to Joel Spolsky and his infamous "How do I move the turtle" question on SO. –  DVK Jan 6 '12 at 19:04
    
I'm sorry these style of questions have made you angry, I don't seek any conflict with you. I'm sure those posts are great and all, but the community decides before the owners in many cases (because they seek the support of the community) and the community has decided that story-identification is on-topic. All this legal maneuvering your attempting is theatrics that will really lead no where. –  Mark Rogers Jan 6 '12 at 20:28
1  
@Mark - please read carefully. I have VERY EXPLICITLY stated that I don't think those questions are off-topic, merely that they seem to be over-valued upvote wise relative to how I personally percieve their worthiness compared to other questions. Again, it'd be nice if you didn't attribute some nepharious intentions that are 100% obviously not there if you bother to thing or grant me >100 IQ. I wouldn't have posted my "troll" question AFTER this meta question was posted if I wanted to make a point, precisely because of a risk of overboard reaction from someone like you. –  DVK Jan 6 '12 at 21:34
1  
@Mark - I edited my Question. Please see the very beginning. They don't make me angry. They make me puzzled. –  DVK Jan 7 '12 at 0:55
    
Based on the data posted by thedaian and Gilles, it looks like this question is based on a false premise: story ID questions are not well-rewarded compared to other types of questions. –  Martha Mar 1 '12 at 23:14
    
And, given Jeff Atwood's recently revealed thoughts on the blog i'm not such a weirdo as everyone took me to be when I first posted this :) –  DVK Mar 2 '12 at 0:22
add comment

5 Answers

up vote 7 down vote accepted

Given that you said in chat

I know I'm repeating myself here, but my question wasn't "are they good" but "I don't get what all the upvoters see in seemingly - IMHO - useless questions". I explicitly said that I don't mind them being asked at all, and some are pretty good.

First, I've seen a lot more upvoting on scifi.SE than some of the other stackexchange sites I've gone to, this is mostly speculation, but I've had a lot more upvotes per answer here than anywhere else. If this fact is true, it accounts for the next part of my response.

I agree with dmckee. The questions are highly accessible, almost everyone can look at them and given that a lot of people seem to upvote questions and answers, a lot of people are going to upvote the question because hey, they know how it feels to half remember a story and they want to help out the person asking the question but they can't answer it so it gets an upvote.

The thing is, this probably happens with "normal" questions, too, but people usually only look at questions they feel they might answer (I am likely to look at a question concerning Doctor Who, but I am not likely to look at a question concerning comic books). Since people often look at questions based on their personal fandom, and questions are something that everyone could possibly answer, and look at, it causes the story ID questions to have a much higher rate of views and votes.

Personally, a lot of story ID questions seem to come from new users, and I tend to upvote new users more, because they're new, and upvoting new users is a good way to get them to come back to the site. However, if the question doesn't really provide any details beyond "it was a book about time travel" or something, then I'm likely to comment on the question, asking for more detail (or upvote a comment that's already saying the same thing)

Actual Statistical Data Edit I decided to do some calculations (or write a program to do the calculations for me...), using the StackExchange /search/ API. Interestingly, these do not back up my theory. You can look at the stats from the API in this webpage (view source if you wish, these stats are generated from data from the API). I decided to use the top 4 tags (, , , ), and ended up with 100 questions from each (side effect of the search API).

Conclusion: Average score for questions were slightly lower than questions, and far, far lower than and questions. Average views were also far lower (by about 200, at minimum). The highest score received by a question is 30, the other tags outrank that except for , which has a high score question of 25. Highest view count for all other tags are significantly higher, and in terms of lowest views, only beats out (side note: with a limit of 30 questions, questions won with a 67 lowest view count, beating out the second lowest question from Harry Potter, with 60)

The end result: these stats do not actually support the idea that questions are somehow higher rated, or even higher viewed than other questions. The only thing I do know is that there are more of them (273 questions at this point, with the next highest tag of having 238 questions).

share
    
It's an interesting theory. Can it be supported by data (E.g. average # of views on contemporaneous ID vs non-ID questions)? If so, it's worth an accept. –  DVK Jan 7 '12 at 0:37
    
@DVK: If we're in the data dumps or the Data Explorer, then yes you can flog the data into some kind of statistics. Indeed votes/view should be pretty easy. Just do it with the story-identification tag required and excluded and you're home free... –  dmckee Jan 7 '12 at 2:50
    
@dmckee - I source-scraped actual SE HTML pages to get my statistics. I could probably do the same with thediaian's ones, but I figure if the reward is an accept, I shouldn't be the one doing the hard work :) –  DVK Jan 7 '12 at 6:02
1  
@DVK I've added some statistics from 100 questions from all 4 popular tags. And put all the stats on a webpage for everyone to see. –  thedaian Jan 8 '12 at 7:49
add comment

Speaking as a moderator

Right this minute, 5% of all questions were rejected (closed and perhaps deleted), compared with our 16.3% rejection rate overall (12.3% if you discount the list and recommendation questions from the site's early days). I conclude that story identification questions are better quality than average.

I've seen the interpretation that story identification are not policed and so poor questions (with too little detail) fall through the cracks. This does not agree with my experience, and in fact is contradicted by your data: those upvotes indicate that most story identification questions do get attention, and those people who upvoted didn't vote to close as “not a real question”.

9.5% of open are unanswered (as in: have at least one upvoted answer); the overall rate is 4.4%. So, on a subject matter that's inherently difficult, we still manage a decent response rate (90% have an answer that's at least vaguely plausible).

We get the occasional “me too” non-answer on story identification questions. This shows that story identification questions are useful to people other than the asker. I don't have statistics for these, but I think they're more common for than on average. Here's a sample such non-answer:

Thankyou so very very much for answering the question.
I stayed up nights for years trying to remember the name of that book so i could track down the second book and read it.
The questions was highly relevant and direct as I had exactly the same question now finally answered. Thank you... Now I can get on with my life. :)

However, these pile-on users don't seem to stick on the site (all the examples I found were one-time posters).

Speaking as a user

I subscribe to the tag, because most questions interest me one way or another: either I know the book and I want a chance at answering, or I don't know the book and I may be interested in reading it (if it's left such a good memory to someone else that they want to find it again, that's a recommendation). Then I apply my usual rules for voting on questions: if it's a well-written, on-topic question that's likely to elicit interesting answers, then I upvote. In other tags, I can only upvote if I know enough about the background (most questions about a specific work don't make any sense if you haven't read/seen/… it). Thus are inherently more prone to being upvoted by me on average.

Speaking as a community builder

When the site started, I hoped that it would become the go-to site for story identifications. Story identification is a perfect fit for the question and answers format. My model was the Usenet group rec.arts.sf.written, which is well-known as a go-to place for story identification (YASID — Yet Another Story IDentification; if you try to search for these on google you'll miss a lot because Google strips the [YASID] prefix that's traditionally used in the subject of such questions). YASIDs work ok on a Usenet group, but not so great: sometimes the thread diverges and the answer is buried amid hundreds of off-topic possts.

I wouldn't say that we've reached the point where we can advertise on our ability to answer story identification questions. Mind you, our accept rate is a little over 50%, which is almost as high as rasfw's; but rasfw gets more of the difficult requests, because it's the last resort to many. It's a pity, because story identification is great for promotion: this question form attracts both novices (who want their question answered) and experts (who want to show off their culture). I hope that we can organically recruit more experts, so that we can claim to be a place where story identification questions get answered.

share
4  
I'm not sure I agree with your interpretation of quality. Popular does not necessarily mean good. Story identification questions are rewarded because they are popular, like the "What kind of red stapler should I buy as a programmer" questions we used to get on Stack Overflow before we hit them with the Ban Hammer. –  Robert Harvey Jan 5 '12 at 22:57
2  
@RobertHarvey The usual reproach against identification questions with respect to quality is that they have too little detail, hence NaRQ. Upvotes don't usually go along with NaRQ. Upvotes are often a sign that a question is non-constructive, but NC doesn't come up for story identification. –  Gilles Jan 5 '12 at 23:01
3  
I define quality mostly to mean that the questions are of use to future visitors. Given the highly localized nature of these questions, I have my doubts about that. They are unquestionably popular, but that popularity still doesn't necessarily equate with quality. –  Robert Harvey Jan 5 '12 at 23:04
2  
@RobertHarvey I've addressed this in my answer as well. I've heard such unsubstantiated doubts before; I'm still waiting for some semblant of evidence that the concern is a real one. –  Gilles Jan 5 '12 at 23:07
2  
@Robert: I can't speak for others, but I make a really effort to improve the titles of these questions so that they might be useful to others--because the initial text does often seem very focused on the OP's experience. –  dmckee Jan 5 '12 at 23:12
1  
@Gilles - yeah, my thinking re popular!=good was in line with Robert Harvey's. Just because "how do I add 1 and 2 in Java" gets 100 upvotes on StackOverflow, doesn't make it a good question. –  DVK Jan 5 '12 at 23:35
add comment

I was curious whether they really score higher than other questions, or if everything just scores highly.

This gathered the set of scores for all questions and all questions:

import json
import gzip
import urllib2
import StringIO
import collections

story_id = collections.defaultdict(int)
everything = collections.defaultdict(int)
page = 1
while True:
    content = json.load(gzip.GzipFile(fileobj=StringIO.StringIO(urllib2.urlopen("http://api.scifi.stackexchange.com/1.1/questions?page=%s" % page).read())))
    if not content[u"questions"]:
        break
    for question in content[u"questions"]:
        if u"story-identification" in question["tags"]:
            story_id[question[u"score"]] += 1
        everything[question[u"score"]] += 1
    page += 1

The average score for all questions is currently 10.2 and for 7.0 - however there are extreme values in general questions, and aren't for story identification.

If you look at the score distribution (I've trimmed off the questions scoring more than 20):

score dist

It looks to me like these questions are more likely to score in the 2-8 range, but less likely to score highly. I don't think it's accurate to say that these questions generally score higher than other questions. It seems more accurate:

  • to say that questions in general tend to score more highly than a long-established site (e.g. Stack Overflow) - 81% of questions get at least a 5. I think this is likely because while the site is small, it's fairly easy to look at all questions.
  • to wonder if questions are scoring too highly even though they score (a little) lower than the average question.

For the latter, IMO there are reasons these are easy to gain votes:

  • It's difficult for me to judge whether a {random topic} question is well-asked or not if I have no familiarity with the work (I might not even look at the question). It's not hard to judge whether a question is good (does it include sufficient information that it's likely that it can be answered).
  • If you're using "interestingness" as a reason to up vote as well as "well asked", some people (including me) find these questions interesting in general (there's plenty of material in meta about why this is so I won't repeat it here). I would guess that one reason that the questions don't score as highly as other questions in general (even though "well asked" is fairly simple to achieve) is that only a few people feel this way.
share
1  
Can I give it 3 +1s? One for effort, one for ability to learn how to use the API from an expert (never seen the API before), one for well reasoned arguments. BTW, the bullet points in the end was what I was theorizing, I just deemed those to be insignificant factors yet many people on this Q seem to voice them as their reasons so my estimate was way off. –  DVK Jan 8 '12 at 0:41
    
Nice! I shall not be outdone, however, so I've collected some stats of my own –  thedaian Jan 8 '12 at 7:49
add comment

It's a common experience to have read or heard or seen a story that we can't remember. And, often, we're not the only ones who remember certain scenes or characters vividly.

I know that about 10 years ago, I was trying to find an old time radio show, A Pail of Air, but kept searching for "bucket" instead of "pail." It took me a long time to find the story because I kept using the wrong word on my searches. Then, later, when I found out it was on a broadcast of "X Minus 1," I found a website that mentioned this is "one of those stories." It's one that everyone remembers parts of, but they forge the title and author.

With that in mind, and considering that many of these questions refer to seeing the show as a child or teen, I think these questions help a lot of people find works they are looking for (perhaps so they can re-read or re-watch) and if it helps the person asking the question, it likely helps others Googling for that info at some later date.

I just wish, when I was looking for the info on A Pail of Air, that SE was around and someone had asked about it. That would have saved me hours of searching and I'm sure these story-id questions will do that in the long run for a number of people.

share
1  
What I don't understand is, is it remotely possible that THAT many people are actively searching for "A Pail of Air", AND every other ID question? E.g. you found that one question useful (or would have 10 years ago), but what I am not ready to believe is that so many people would find so many questions useful for finding those exact pieces of work. –  DVK Jan 7 '12 at 0:39
2  
Not every story out there, but there are some with a reputation for being remembered but the title and author are forgotten. For example, Asimov got so many letters and calls and was approached by so many people that say, "I read a story I think you wrote and I can't remember the title." And they always described the same story (The Last Question). It happened so much that when someone said, "I can't remember the title of a story..." and he'd say, "It's The Last Question," and they'd always say, "You're right!" So not every question will help, but some will. It's hit or miss. –  Tango Jan 7 '12 at 1:05
    
but that wouldn't account for the distribution I observed. Again, I don't have an issue with an odd uber-upvoted ID question - those make sense (your comment has a good example). I can't figure out why there are SO MANY. Remember, even for RECENT ones, average was 6.7 with STD of 3. –  DVK Jan 7 '12 at 1:08
2  
I can't explain it, other than, perhaps, that people are upvoting the ones that provide good info about the story because there have been some very poorly written ones. (Heck, I still can't see why any Star Wars question is upvoted like crazy, and I ask one about material that has influenced generations of writers and it gets 1 or 2 upvotes!) –  Tango Jan 7 '12 at 1:15
1  
that one's easy. Same reason Java questions get 5-20 times the upvotes of Perl ones on SO, Bigger interested audience –  DVK Jan 7 '12 at 1:23
add comment

There is a certain amount of bikeshed effect here. These questions are highly accessible. That's a feature of crowdsourcing and you just get to live with it.

For my part, I like them because they are about something other than Star Wars, Star Trek, Battlestar, and Harry Potter. To be sure, people are entitled to like what they like, but the recent questions on those universes have been of interest only to fan boys.

The Science Fiction and Fantasy universe is so much bigger than the shows you've seen on TV and at the cinema in that last decade.

share
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .