Last month we ran an experiment to test using a template to help new question askers ask better questions. This experiment is the first in a series addressing the top voted theme for TeamDAG to work on: incoming question quality. This experiment also is the first one that uses our new question quality measure for evaluating questions.
Given that we haven't executed an experiment around question quality before, we intentionally kept things simple so that we could focus on the mechanics of running this new type of experiment. We used a template loosely based on enderland's answer to Jon's question on how question templates could help people ask better questions. There are a lot of other interesting ideas for how we can expand on this approach. We don't in any way think that this specific template is the canonical template to use for this effort. In fact, it may be that there isn't a canonical template and as such may test multiple templates in a multivariate test and/or test a wizard approach that supports multiple templates. But that is fodder for a future meta post. This one is about the results of this experiment.
So, how'd it go?
Well, the good news is that with some tweaking of our A/B testing system the experiment was a success. That means the question quality measure worked as expected and there was a modest impact on the quality of questions asked by users who encountered the template.
The A/B test ran for a little over two weeks. Anyone with a rep below 111 was a candidate for seeing the template. 50% of the total pool was exposed to the template if they asked a question.
Editor with template
Roughly 95,000 "ask a question" click events were registered for each pool of users. Now a lot of people go to "ask a question" and don't actually do so. Generally, only about 46% of users who visit "ask a question" go on to post. For our new user pool the average was closer to 32%, which isn't too surprising. We all know that asking a question on Stack Overflow comes with a bit of fear and loathing (another topic for another day).
More interesting, there was a 3% reduction in questions asked by users who were exposed to the template. I think this is actually a really good thing. There's no way to know exactly why people who saw the template chose not to ask their question. However, my hypothesis is that people who saw the template had a better understanding of the level of investment that we expect from question askers and some of them were a bit intimidated by it. Several scenarios come to mind:
- Potential askers decided that they needed to do their homework first and come back later to ask.
- They decided it wasn't worth the effort and didn't ask their question at all.
- Maybe during the process of writing a better question, they solved their problem without needing to post it.
The other strong finding was that users exposed to the template asked ~3% fewer "bad" questions. Those questions were basically evenly distributed between "neutral" and "good". From the point of view of answerers "neutral" basically equals "bad" because no one was interested in answering it. However, from the point of view of the asker it really is neutral. They asked a question. They didn't get hammered with down votes, but they didn't get an answer either. Our focus is on improving overall question quality, so the upside here is small ~1.5% improvement in good questions.
Overall, we're happy with the results of the experiment. Our testing system is now working for question quality experiments. The experiment points to some hope that some sort of template can provide a measurable improvement on question quality. We don't think this template (alone) is the right solution, but we're interested in running additional experiments to figure out what will work. We are working on a more involved wizardy approach right now. We'll share more on that as it develops.
