Survey Writing Guide: Fandom Edition
Sooo! People are collecting survey data on fandom! Makes sense — fandom is an interesting population to study, and getting data on attitudes/behaviors/etc is a great way to try to find out the truth about popular points of contention about media. To get unbiased, meaningful data, though, you’ll want to make sure your survey is in good shape. This is a guide to try to help you do that.
I am assuming, if you are reading this, that you want your survey to be unbiased and interpretable, sampling from as many viewpoints as possible and getting data on opinions you might agree and disagree with. In other words, I am assuming that you want to do your fandom research in good faith. If that is not why you are making a fandom survey, then you can ignore this whole post. I would ask you to reconsider running a bad-faith fandom survey, but I can’t stop you. But just know that my assumption is that you’re out here in the honest pursuit of truth.
I’m not accusing a survey of being done in bad faith if it doesn’t follow my suggestions. I also realize most people are conducting these surveys in their spare time because they’re passionate & curious, not for the purpose of serious publication or whatever. I’m not telling you that you have to follow all or even any of the advice here. Do what you feel good about doing. Fandom should be fun!
This guide is only about surveys themselves, and will not help you get a good sample of your fandom. In other words, none of this advice will make sure your survey reaches obscure corners of your fandom, insular communities you aren’t in, or casual fans. A good survey is important, but it’s only one part of getting accurate data, and participation recruitment is a whole other thing.
This is also only intended to help with survey setup, not analysis or results reporting. You can write a great, unbiased, easy to understand survey, and still present your results in a way that is biased/doesn’t include good information/is hard to understand. Furthermore, once your results are public, they’re out of your hands: you might be well-meaning, but please keep in mind that some people might use your data to make unfair or incorrect assumptions about others, support biased arguments, or generally behave unpleasantly. That’s just part of survey work; don’t let it stop you, but don’t be surprised if/when it happens.
Now, onto the survey tips ~
Think about how to structure your questions
Use multiple choice answers/avoid free response answers as much as possible. This is going to be really helpful for you when it’s time to analyze your data: it’s so much easier to analyze responses that are just clicks of a box than responses that require you to read someone’s long explanation of an opinion and try to figure out what they mean. It also makes answering easier on your participants: clicking a box is a lot easier for them than writing out what they think. Giving an option for them to expand on their answer in a free response text box is great, but try to capture as many possible responses as you can in multiple choice questions.
Consider: should participants be able to give more than one response per question? A good rule of thumb for this is if different responses to a question are mutually exclusive or not: if they are mutually exclusive, only give people the option of clicking one box; if they’re not, let them choose as many as they want. For example, if you want to know somebody’s ONE favorite season of a show, only let them choose one season. If you want to know ALL the seasons they like, give them the option to choose multiple.
Only include one question per question
A LOT of surveys ask questions that are actually several questions in one, and that makes answering them difficult! For example, you might THINK you’re only asking one thing if you say “Are you introverted or shy?” as a yes/no question, but that is actually two questions: some people are introverted but not shy; some people are shy but not introverted; some people are both introverted and shy, and some people are neither. People who respond “yes” might mean they are one or the other or both, and people who respond “no” might mean they are one or the other or neither — and you’ll have no way of knowing, because your question was too ambiguous.
If you’re asking a question about a nuanced topic, make sure your question and possible responses are nuanced too
People in fandom have a lot of complex views, and giving them only a black-and-white way to tell you about those views means you will miss out on nuance and get inaccurate results. If you are wondering about opinions on a controversial pairing, asking “Do you think it’s okay to ship Reylo?” (for example) and only letting people respond yes or no is not going to capture the range of opinions that exist on Reylo. If you can include multiple answer boxes for people to click, do that!
Analyzing free response data (where people can write their own answer) is hard on you, as the survey writer, and people might not feel like explaining their opinions. Definitely include a free response box if you want, but also provide a lot of options for responses.
The more controversial the topic you’re asking about, the more nuance you should let your participants give in their responses. People in fandom often have very strong and very complex opinions about certain issues; you’ll want to make sure not to flatten those opinions too much, while still keeping them simple enough to analyze. Asking “do you enjoy reading fluff?” is (probably) not going to be too complicated, and you’ll likely be fine giving people 2-3 response options (yes/always, sometimes, no/never). But you should give a lot more response options if you’re asking “is it ever acceptable to ship two siblings together?”: some people will be fine with “yes, always” or “no, never,” but a LOT of people are going to want to specify conditions like “only if they are not blood relatives” or “only if it is framed as being bad” or “it’s acceptable sometimes but I personally avoid it,” etc. Make sure people without clear-cut opinions have the ability to give you their opinions too.
Use simple, clear language
Use language that is as simple and precise as possible. Keep questions short when you can. Avoid using a lot of slang and regional idioms: people come from all over the place in fandom, and might not know what you mean. People also have a lot of different reading levels: you want to make sure most of your participants can understand what you’re asking them.
There are some terms you can assume everyone in your sample will know, and some that you can’t: it’s up to you to figure out which terms are which. For example, most people in tumblr fandom space will understand what a ship is. They might not know what “IC” means though, so you might need to write it out as “in character.” Further, pretty much everyone in the MCU fandom might know what “Stucky” is, but not as many will know what “Coulwhip” is. You can make these kinds of technical terms easier for people to understand by just writing out all abbreviations and writing all pairings in the Name/Name format.
Use neutral language and keep your personal opinions out of it
Avoid inserting your own opinions into your survey, even if you expect that almost everyone participating will agree with you. Even “obvious” jokes won’t read as jokes to everyone; you might upset/offend/put off some group of participants disproportionately such that they won’t complete the survey, and you’ll miss a chunk of data you want. This is a problem, because say, for example, you really want to know if people’s favorite Supernatural ship has any correlation to fans’ personality traits — but if you poked fun at Dean somewhere else in your survey in a way that made Dean fans click off the page, then you’re going to lose a lot of Dean fan data that it would have been really useful for you to have, since Dean is a very popular character to ship with a lot of other characters.
A good-faith survey is also not the place to moralize. Fandom can be a very polarized place; you are not going to be able to predict which things you ask about are going to be morally objectionable to your participants, and if they’re going to agree with your moral judgments. Your results might surprise you! A survey where you hope to get accurate data is not the place to defend a problematic character you like or to call some group of shippers gross. Your best bet for not causing issues is to include stuff that’s controversial and to be as neutral about it as possible — for example, if you are asking about ships people like, include every reasonably popular ship (potentially excluding rarepairs/extreme rarepairs just because it’s hard to include all of those), even the ones you hate/think are harmful/whatever, and don’t comment on the goodness or badness of any of them.
Avoid “leading” questions to avoid bias
If you say, “Do you love this awesome TV show?” that’s going to be awkward to respond to for everyone whose answer is anything other than “yes.” Something like “What is your opinion of this TV show?” (with responses like: I love it, I like it, I have mixed feelings about it, I am neutral toward it, I dislike it, and I hate it) will give you a better sense of people’s genuine thoughts.
Select your multiple choice options thoughtfully
Be as egalitarian as possible in the options you provide to your fandom-specific questions that aren’t free response. If you leave out a major character on a list of favorite characters, for example, your participants are going to wonder why you did that — and you might not get data about that character, especially if you don’t include a free response option.
Avoid bias in your multiple choice answer options
If you ask people for their favorite and least favorite [something], make sure the list of [something] options for both of those questions is identical. For example, if you’re asking about people’s most liked and most disliked ships, make sure the list of ships you include for both questions is the same. Including Sylvie/Loki only on the “ship you most dislike” but not the “ship you most like” list might seem like a reasonable move to you if you don’t know anybody who (openly) likes Sylvie/Loki, but doing that will bias responses and make your data inaccurate. Some people will stop filling out the survey, some people who like Sylkie just won’t want to bother filling in the blank, some people might forget that they like Sylkie unless it’s right in front of their eyes as one of the response options, some people will feel judged and so they won’t answer your survey honestly, etc.
Follow common best practices advice for demographic information
Collecting information on respondents’ ages, genders, race/ethnicity, employment status, education history, etc is, luckily, very common, so there are a lot of standard questions about those things. I’ll just go ahead and list what I sometimes collect in my own surveys:
Age: for anonymous fandom surveys, I suggest using age groups (under 13, 13-18, 19-25, 26-35, etc) instead of asking people for their precise ages. Just make sure to include all possible age brackets. For example, don’t assume everyone in fandom is under 50.
Gender: unless you want a lot of nuance in your responses, it’s typically enough to include a list like: man/boy or male, woman/girl or female, nonbinary, genderfluid or bigender, and other. If you are interested in trans people specifically, I suggest asking that in a separate question (like: “Do you identify as transgender?”), rather than including binary trans identities in addition to binary cis identities in the same question. The exception to this is if you are including a very long list of gender options.
Race/ethnicity: (disclaimer that most of the data I collect comes from the US, so my best practices are most suited to that population. I welcome any feedback about the best way to ask about this for an international and/or non-US population.) I typically ask something like: “Which category best describes you?” and then give the options: American Indian or Native Alaskan, Asian, Black, Hispanic or Latinx, Middle Eastern or North African, Pacific Islander or Native Hawaiian, White, and other. Ideally, this list should allow multiple responses, so that multiracial/multiethnic people can give accurate reports.
Employment status: “What is your employment status?” with responses: employed full time, employed part-time, unemployed and looking for work, unemployed and not looking for work, self-employed, homemaker, student, and retired. If possible, let this be multiple choice, so that, for example, people who are students AND work can select both applicable options.
Education level: “What is your highest level of education?” with responses: some high school, high school diploma or equivalent, some college, college degree, some graduate education, Master’s degree, Doctorate degree, professional degree (e.g. M.D., J.D., M.Div, etc)
Sexual orientation: like with gender, you probably don’t need to provide a long, comprehensive list unless your survey is about sexual orientation specifically. Something like the following is probably enough to cover most of your respondents: “What is your sexual orientation?” with responses: straight/heterosexual, gay/lesbian/homosexual, bisexual, pansexual, asexual, and other.
Other questions: just try to make sure you are providing response options that will cover the majority of your participants, and include an “other” option if you’re not sure you’ve covered everything. There are a fair number of basic best practices documents for common demographics questions that you can find through Google, so if you’re not sure how to ask something, give it a quick search. You can also ask people you know who might know more about some specific demographic quality than you do to weigh in!
That’s it for now! I hope you find this helpful, either in writing your own surveys, or in evaluating surveys you are participating in!
21 notes · View notes