Actually, it IS about confidence intervals. None of the minor issues that you had pointed out would cause a confidence interval to be so wide that a population proportion that is over 50% would generate a sample proportion of less than 10% with sample size of 40.
No, it's not. It seems to be about a hypothesis test that you have in mind but haven't been clear about. As far as I can tell, you have in mind a null hypothesis that P >= 0.5 where P is the population proportion of people who believe their ENT was more effective than the forum. Thus the alternative is P < 0.5. Beyond that, your thinking is convoluted. What you appear to be talking about is related to the probability of a Type 1 error [Pr(reject null|null is true)]. Now, your example [Pr(P_hat<.1|P>.5) where P_hat is the sample proportion] is more specific, but not in a way that is really useful. So what you really need is a p-value, not a confidence interval. This is very basic.
The other issues are not minor, they are fundamental conceptual issues that matter once one gets to survey design and survey sampling. They are, however, beyond what one would learn in a basic statistics class.
The population that this poll deals with might not be all T sufferers. As I pointed out multiple times, the population the poll deals with are all of the people who made an account here, which is all of us. If we could choose which population we want to study, we would want to study the subpopulation that we are part of.
No, the true population of interest is people visit the forum, not members. After all, individuals may be helped (or harmed) by what they read here without becoming members and they have in many/most cases seen an ENT. Thus they are also in a position to evaluate whether the information they found on the site was more or less useful than the ENT they saw. So, you still have the wrong population.
However, let's take the population as registered members. This poll (and the same is almost certainly true for any poll on this web site) will not provide unbiased information about even that population.
To see this, let's think about when a sample of members would provide unbiased information about a population parameter for the population of members: 1) if it were a random sample, or 2) if it were non-random but non-non-random in some intentional way - e.g., a stratified sample - or in a way that is completely unrelated to the outcome of interest.
Neither of these is satisfied in this case. You certainly did not randomly sample the population or draw a stratified sample. You are therefore relying on people to choose to respond, i.e., you have a choice based sample. Given that one of the most socially reinforced (and least controversial) statements one can make on this forum is that "my ENT knows nothing", the sample of people who answer the poll is almost certainly biased and is thus related to the outcome of interest.
It's actually worse than that because on any given day the people participating in the forum (and thus likely respondents) are a selected sample of people who have ever registered. I think the best analogy is to a hospital. If you walk around the various wards of a hospital and take a sample of patients (a stock sample) you will find patients who are on average worse off (less healthy and in the midst of longer stays in the hospital) than what you would find if you sampled patients upon admission (a flow sample). The latter patients would on average be healthier and would have shorter completed stays in the hospital. The reason is that most patients, even those who are admitted, will have short stays and will be relatively healthy. In contrast, when you sample people in the hospital without regard to admission date, you will find a much less healthy population because the unhealthy people 'accumulate' in the hospital.
With some exceptions of people who are "better" who stick around to help others, the sample of people on line at any given time, and thus the people"eligible" for your poll, is like the stock sample in the hospital analogy and is negatively selected relative to the population of members or visitors. (It's actually a bit more interesting because the analogy breaks down in one way - new members here are on average in great distress and would likely appear "worse" than they will ultimately turn out to be, but this only exacerbates the problem because the 'eligible' sample is then composed of people in crisis and a selected sample of longer term members.)
Thus, the sample of registered daily visitors is a non-random sample of all members and even more so of visitors, and the sample of people who feel drawn to answer a particular poll on ENTs is biased.
Consequently, the results of your poll, or any poll on this site, do not provide unbiased estimates of population parameters even when the population is members of this site.
I've thankfully never had to develop my own survey, but I work with survey data regularly and think a lot about how the data are generated - both in terms of how the surveys are developed and administered and how questions are answered.
Member
