The highest standard in many types of research on humans is the double-blind randomized experiment or 'trial'. In it, the people in all the treatment groups do not know whether they are receiving the hypothesized great, new medicine or procedure or are in the control group getting something else. Neither do interviewers or other persons in contact with those in the experiment. Only the data analysts get all the information. Typically, those analysts apply some calculations to the data to see if any difference between the groups' results has a high probability of occurring just by chance. The words "double-blind" refer to keeping the nature of the treatments secret to try to avoid psychological or other boosts just from knowing you are getting the possible new miracle drug or whatever.
The 'randomized' part of the name refers to assigning people to treatment groups randomly, using a table of random numbers or some other reliable impartial mechanical method of deciding which people get into which of the groups. Most basic statistical theory is based on the idea of sampling from a population but in real life, the nature of the entire set of people matters very much. It is not possible, actually, feasibly and within a reasonable time and budget, to get a random sample of all people there are. So the population always consists of a limited and in some ways, select group, such as Americans or college students willing and able to participate in the experiment.
I read a while back that some physicians like to prescribe the latest drug that seems to have had good results because all drugs and all inventions tend toward less power and effect over time. An article in a recent New Yorker by Jonah Lehrer called "The Truth Wears Off" explores research on humans that first seems quite striking but over repeated tries of the same setup shows a decline in power and effect. The latest issue has several letters to the editor from researchers and professors of research methods emphasizing that such a trend is to be expected. As more replications of the research take place, more variation in the type of person in the experiment occurs.
This reminds me of the comment by C.S. Lewis about the result of his visiting groups of people all over Britain to explain and defend Christianity. He says that when he discussed ideas with other Cambridge dons, much like himself, he had one sort of experience. Then, out in public, he found a much wider array of opinion, reception, and discussion mode than he was used to. There are places where humanists offer the opinion that between individual and cultural variations and psychological effects and powers there really can be no such thing as totally scientific research involving people. I don't go that far but we are tricky critters living in a complex surprising world.
The 'randomized' part of the name refers to assigning people to treatment groups randomly, using a table of random numbers or some other reliable impartial mechanical method of deciding which people get into which of the groups. Most basic statistical theory is based on the idea of sampling from a population but in real life, the nature of the entire set of people matters very much. It is not possible, actually, feasibly and within a reasonable time and budget, to get a random sample of all people there are. So the population always consists of a limited and in some ways, select group, such as Americans or college students willing and able to participate in the experiment.
I read a while back that some physicians like to prescribe the latest drug that seems to have had good results because all drugs and all inventions tend toward less power and effect over time. An article in a recent New Yorker by Jonah Lehrer called "The Truth Wears Off" explores research on humans that first seems quite striking but over repeated tries of the same setup shows a decline in power and effect. The latest issue has several letters to the editor from researchers and professors of research methods emphasizing that such a trend is to be expected. As more replications of the research take place, more variation in the type of person in the experiment occurs.
This reminds me of the comment by C.S. Lewis about the result of his visiting groups of people all over Britain to explain and defend Christianity. He says that when he discussed ideas with other Cambridge dons, much like himself, he had one sort of experience. Then, out in public, he found a much wider array of opinion, reception, and discussion mode than he was used to. There are places where humanists offer the opinion that between individual and cultural variations and psychological effects and powers there really can be no such thing as totally scientific research involving people. I don't go that far but we are tricky critters living in a complex surprising world.
