FAQs

Why don't I see XX study here?

There are a couple of possibilities. The first is that I can only use studies that provide enough information to calculate the things needed to graph them. For vaccine effectiveness, for example, I need a study with one group who got the vaccine and one group who did not and I need to know how many people in each of those groups got sick following vaccination. Similarly, for studies of MMR and autism, I need a group who got MMR and a group who didn't and the rates of autism in each group. A case study of several people who got both the MMR and autism would not be sufficient because it doesn't tell me whether this rate is more or less than in the unvaccinated.

I have also tried to avoid double-counting data, so if a preliminary study was published with initial results and then later updated, I included the latest published version of the results only (unless I messed up).

A final possibility is that since this is a side project for me and my literature search was not exhaustive, I am missing studies.

I don't think XX study is very good.

This is quite possible. The goal of this site was to give a sense of the sheer volume of studies that have been done to answer particular questions and whether there is a lot of disagreement or whether anyone is just cherry-picking particular studies that support their position while ignoring a vast wealth of other data. I did not want to over-complicate the display of information by presenting huge numbers of details about individual papers (and, to be honest, I do not have the time to read them all through thoroughly enough to do that, especially when other sites like the Cochrane Reviews have done a much better job with larger numbers of people with a range of expertises and transparent, pre-defined criteria of quality). There are many blogs out there about vaccination that spend a great deal of time on particular studies and whether they are well-conducted or not and diving into the details of various papers. I think these sorts of sites often confuse and frustrate laypeople, although conversations about good study design and whether too many rubbish papers are getting published are obviously important conversations for scientists to have amongst themselves. Therefore, I have not attempted to exclude studies or provide any sort of weighting based on quality.

However, there are a few filters and aspects of the graphs that may be useful in this respect:

1. I have noted whether studies are RCTs, cohort studies, case controls or epidemiological studies. You can see from the hierarchy of evidence that RCTs are usually given the greatest weight because they should have the least bias, but it is still possible for such a study to be set up poorly. The default behaviour of the site is to make different types of studies different colours, but you can also filter to remove certain types of studies from the graphs and averages.
2. The more people who are investigated, the higher the certainty in the study result if all other factors are equal. Of course, not all other factors are equal between studies (bigger studies tend to be epidemiological studies or retrospective studies that are more prone to bias), but the default behaviour of the site is to present larger studies as larger bubbles. If you don't like this there is an option to turn it off.
3. If you mouse over a particular bubble, there may be notes that give a brief indication about certain concerns with the study and if any conflicts of interest were declared. Whether there are notes or not, you can click on the bubble representing any study to be taken to the abstract or (if publicly available) the full study, where you can read it through yourself and try to make your own judgments of quality.

One more note: I do not currently have a requirement for the data used to be from a peer-reviewed study. Peer review is an important part of the scientific process, but in certain cases I have used epidemiological data that I found online that was not linked to any particular study. These data may come from the CDC or WHO, which I consider relatively trustworthy sources. You may feel differently. Again, you can see if this is the case by clicking through on any data point to the original source or using the filters to exclude data that have not been peer-reviewed. I have not currently had to make a judgment call on whether data from another source is "trustworthy" or not, but if I do exclude something for that reason I will note it here. I will not include data from retracted studies, obviously.

You talk about risks of catching a disease, but could you tell me what the risks are if someone actually gets the disease? Do any of these diseases actually kill people?

This is a great question, but really beyond the scope of this site. If you have been through the site, you know that your risks of catching a disease depend on a lot of things, including how many people you come into contact with, how the disease is transmitted and, crucially, how many people around you are vaccinated and whether you are vaccinated yourself. This is why I have talked about the probability of catching a disease in different scenarios - because the probability varies depending on circumstances.

This variation is even larger when trying to predict how a disease will affect someone. Diseases can have different effects in people of different ages, in those with better nutrition, in people who are already sick or have problems with their immune system. For an overview of the symptoms and complications of various diseases, I found this site to be fairly detailed without being overwhelming (this is a link to the chicken pox page; other disease can be found along the side). To try to break down these effects and complications for each age group and scenario for each disease would make the site gigantic.

Another resource I found helpful in thinking about these risks was graphs like this of "deadliness" (case fatality rate = probability of dying for certain assumptions about the person) versus contagiousness. Note that I do not vouch for the accuracy of this particular graph (for example, the case fatality rate used for mumps appears to the fatality rate for mumps encephalitis), but I think it gets across a basic idea that is often used to mislead people: some diseases, like ebola, are of concern because if you get them, you are very likely to die, but right now they do not spread that easily between people; other diseases, like chicken pox, are not particularly deadly, but they spread a lot faster and more easily so more people are at risk to start with.

Finally, a lot of anti-vaccination websites go on and on about the flu, both because the vaccine effectiveness varies so it is easy to dig up papers from a bad year, and also because they like to say flu is not that dangerous so probably a lot of these things we vaccinate for are not that dangerous. But if you look at this graph, flu is way over in the bottom left corner, meaning that normal seasonal flu is less contagious and/or less deadly than almost every other disease. So it is not a very good comparison for what other diseases are like. In fact, looking at the graph, it seems like a bad idea to try to guess the dangers of one disease by looking at another. They are all quite different and have different risks.