This was excellent. You have very clearly shown the base rate fallacy as well as Simpson's paradox. I would suggest, when looking at observational data, the 3rd item to be added to the Mt. Rushmore of biases to examine is the healthy user bias. I believe you comment on my stack so you are not surprised to know that I think this is a critical component to consider in any look at observational data.
This is why we need RCTs. Until yesterday, I was unaware of any RCTs showing a mortality benefit of the vaccines (I have a full-time, unrelated job... keeping up with all of the literature is not possible for me). Someone on Twitter pointed me to a Cochrane paper which, in turn, seemed to rely on a single RC for VE against COVID deaths for Pfizer and Moderna. Hooray! However, when I looked at the papers involved from that RCT, while it claimed VE against severe disease & death was 97%, it also claimed VE against symptomatic infections was >90%. How can I believe the former when the latter is obviously nonsense?
Commenter Wendy Leonard just pointed out the 4th great unknown of the observational datasets - Natural Immunity. Up to know I have assumed it is evenly distributed in the population although suspect it could be higher in the unvaccinated. How do you address this?
This is a good point. I'm not sure, however, if it will be that simple to include however. Seems to me there might be 3 categories of folks who have had COVID. Those who had COVID who have never been vaccinated. Those who got vaxxed, then got COVID, and those that got COVID and then got vaxxed. I suspect the amount of immunity provided is not consistent across the three groups, with the middle group probably doing the worst.
I fully agree about the Healthy User Bias being a very underappreciated aspect of observational data and I've seen you and also Matthew Crawford often discuss it since 2021.
I know there is a lot of disagreement about this, but my own personal suspicion is that the absolute sickest, weakest, frailest, most immune-compromised are, in fact, disproportionately represented in the unvaccinated. Which obviously plays havoc as a confounder when trying to compare mortality rates.
I think you bring your own confirmation bias to the numbers. You ignore the known death rate of unvaccinated in age groups, and Natural immunity gained from previous infections, so if the vax and boosters actually worked then the deaths should be falling not growing in age groups. Especially as the weakest already succumbed to the more "dangerous" variants. Also as the numbers are fudged a little to say people who die within 14 days of a shot arent vaccinated it puts it own skew on the results. Many more professionals are coming to the same conclusion that these jabs do not help, in fact cause infection rate to go higher.
Hi Wendy, thanks for reading! What exactly to you think my bias is which I thought the numbers confirmed?
I agree, if the vaccines are reputedly so safe and effective (which I do not personally believe) , then why is mortality rising. Note, I pointed out in the sub-section "Graphic (Novel Treatments)" that the base rate fallacy cannot "explain away any trends of increases in totals". It would be great to see an accurate chart of mortality rates per 100K by age group and vaccine status per month to show the trends over time. Unfortunately, that was not the case in the charts widely circulated this week.
Good point about natural immunity - it along with the Healthy User Bias mentioned by T Coddington is another major issue underappreciated when considering observational datasets. Note, it could confound the data either way depending on its prevalence or distribution in the different vaccine status populations.
What do you think the "death rate of unvaccinated in age groups" shows? Can you give a link to this?
I assume you have already read the latest substack https://wherearethenumbers.substack.com/p/the-latest-ons-data-on-deaths-by , if our NZ numbers are similarly fudged then the data is seriously flawed to begin with and the old saying GIGO stands. The bias I think is the belief in the truth of the "official" numbers and so when you use that data set it can only have an outcome that proves that official outcome. Just like saying NZ has had few deaths to the jab, ignoring the deaths reported and Drs scared to report such deaths. I say this as Mother who's son got Pericarditus straight after the 2nd jab ( needed to work ) All Drs involved with him would not do any adverse event reports.
By the way, I don't assume the official data must be true, if that's what you think my bias is? But it is the only data I have to work with and I don't think it is okay to make data up to serve any agenda.
The widely circulated faulty charts I discuss in the post were because someone processed the offical data wrongly - hopefully an honest mistake. But then nobody seemed to care because it confirmed their bias.
How would you feel when public health incorrectly processes or publishes data. Do you think they should be held to account? Do you think others should mindlessly promote it?
It was reported that NZ had very strict restrictions and there must have been terrible pressure to get vaccinated. Hopefully there will be some reckoning for those responsible.
Thank you for this information from a family in New Zealand!
Luckily there are many people who did not get the jab. The official figure of unvax in NZ only use the data set of all people who interacted with the Drs in the 12 months prior to the rollout. So many like in my small town 75% unvac never complied were not included. Of the 25% who had to comply ,3 deaths "suddenly" in the 5-6 weeks post vax , unexplained. 300 people, 100 odd adults, many cases of heart issues, cancer in those who had to get the jab for work. Almost no one has had the virus here except... those that had the jab, who are now getting it multiple times ( assuming testing is 100% accurate). Thanks for engaging in a good conversation
I also suspect the unvaccinated populations are undercounted in countries all around the world and that of course makes the outcome rates for the unvaccinated seem worse than they actually are.
Again, thanks for this info from NZ. I spent a year there in my twenties climbing, hiking, and working on farms. Beautiful land and lovely people. I was shocked and saddened to see how authoritarian it became during the pandemic. Take care, Wendy.
Need to look at deliberate spreading date in NZ opening border.
Step 1: from 23:59 on 27 February 2022
During this phase, vaccinated New Zealanders and eligible travellers from Australia were able to enter New Zealand without the need to enter Managed Isolation and Quarantine (MIQ). Vaccinated travellers were still required to self-isolate on arrival.
Step 2: from 23:59 on 4 March 2022
Vaccinated New Zealanders and eligible travellers from anywhere in the world can enter New Zealand without the need to enter MIQ.
Like I conceded in the post, I haven't dug very deeply into the NZ big picture, and limited my focus to the charts debacle and a more general philosophical point about cognitive biases.
I don't doubt the NZ is seeing a similar climb in all-cause deaths post vaccinations and the lifting of restrictions. I will try to keep an eye on NZ going forward especially because it offers a unique case study of a highly isolated, highly vaccinated, highly restricted country to compare with Germany and others.
Vaxxed:Unvaxxed Population = 93:13 = ~13.3 (ie. ~93?% population vaxxed)
should read:
Vaxxed:Unvaxxed Population = 93:7 = ~13.3 (ie. ~93?% population vaxxed)
I also think (please correct me if I am wrong) that the formula using the odds ratio is not an approximation but exact, if the distinction is binary (i.e., one can only be vaccinated or unvaccinated, and these two make up the total population). See also the formulas here:
That's so funny =) So I am right that using the Odds Ratio is normally an approximation but as Ben's equations (and the link he provides) show, because I chose binary complimentary categories of vaccinated:unvaccinated, in this instance you are correct and it is indeed exact.
(whereas boosted vs unvaccinated would be an approximation)
I have long used the Odds Ratio when I didn't have accurate population totals to hand and now I finally understand this extra dimension!
I'm not interested in picking personal battles. It's exhausting and generally not productive.
btw. would you be interested sometime in skyping/chatting about your take on the ONS mortality rates? I have the feeling that your take and Prof Fenton's take are more complimentary than opposite but I am struggling to synthesise the two.
"Exhausting and generally not productive" could be the slogan for a monster truck rally. I can, however, confirm that pointing out shoddy work and obvious mistakes on "our side" doesn't bring in new subscribers, regardless of whether it should.
This was excellent. You have very clearly shown the base rate fallacy as well as Simpson's paradox. I would suggest, when looking at observational data, the 3rd item to be added to the Mt. Rushmore of biases to examine is the healthy user bias. I believe you comment on my stack so you are not surprised to know that I think this is a critical component to consider in any look at observational data.
This is why we need RCTs. Until yesterday, I was unaware of any RCTs showing a mortality benefit of the vaccines (I have a full-time, unrelated job... keeping up with all of the literature is not possible for me). Someone on Twitter pointed me to a Cochrane paper which, in turn, seemed to rely on a single RC for VE against COVID deaths for Pfizer and Moderna. Hooray! However, when I looked at the papers involved from that RCT, while it claimed VE against severe disease & death was 97%, it also claimed VE against symptomatic infections was >90%. How can I believe the former when the latter is obviously nonsense?
Commenter Wendy Leonard just pointed out the 4th great unknown of the observational datasets - Natural Immunity. Up to know I have assumed it is evenly distributed in the population although suspect it could be higher in the unvaccinated. How do you address this?
This is a good point. I'm not sure, however, if it will be that simple to include however. Seems to me there might be 3 categories of folks who have had COVID. Those who had COVID who have never been vaccinated. Those who got vaxxed, then got COVID, and those that got COVID and then got vaxxed. I suspect the amount of immunity provided is not consistent across the three groups, with the middle group probably doing the worst.
link to Cochrane paper... https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD015477/full
You might find this paper from the German RKI interesting:
https://www.medrxiv.org/content/10.1101/2022.05.25.22275516v1.full.pdf
Thanks very much!
I fully agree about the Healthy User Bias being a very underappreciated aspect of observational data and I've seen you and also Matthew Crawford often discuss it since 2021.
I know there is a lot of disagreement about this, but my own personal suspicion is that the absolute sickest, weakest, frailest, most immune-compromised are, in fact, disproportionately represented in the unvaccinated. Which obviously plays havoc as a confounder when trying to compare mortality rates.
I'll take a look at the Cochrane Paper.
I think you bring your own confirmation bias to the numbers. You ignore the known death rate of unvaccinated in age groups, and Natural immunity gained from previous infections, so if the vax and boosters actually worked then the deaths should be falling not growing in age groups. Especially as the weakest already succumbed to the more "dangerous" variants. Also as the numbers are fudged a little to say people who die within 14 days of a shot arent vaccinated it puts it own skew on the results. Many more professionals are coming to the same conclusion that these jabs do not help, in fact cause infection rate to go higher.
Hi Wendy, thanks for reading! What exactly to you think my bias is which I thought the numbers confirmed?
I agree, if the vaccines are reputedly so safe and effective (which I do not personally believe) , then why is mortality rising. Note, I pointed out in the sub-section "Graphic (Novel Treatments)" that the base rate fallacy cannot "explain away any trends of increases in totals". It would be great to see an accurate chart of mortality rates per 100K by age group and vaccine status per month to show the trends over time. Unfortunately, that was not the case in the charts widely circulated this week.
Good point about natural immunity - it along with the Healthy User Bias mentioned by T Coddington is another major issue underappreciated when considering observational datasets. Note, it could confound the data either way depending on its prevalence or distribution in the different vaccine status populations.
What do you think the "death rate of unvaccinated in age groups" shows? Can you give a link to this?
I assume you have already read the latest substack https://wherearethenumbers.substack.com/p/the-latest-ons-data-on-deaths-by , if our NZ numbers are similarly fudged then the data is seriously flawed to begin with and the old saying GIGO stands. The bias I think is the belief in the truth of the "official" numbers and so when you use that data set it can only have an outcome that proves that official outcome. Just like saying NZ has had few deaths to the jab, ignoring the deaths reported and Drs scared to report such deaths. I say this as Mother who's son got Pericarditus straight after the 2nd jab ( needed to work ) All Drs involved with him would not do any adverse event reports.
By the way, I don't assume the official data must be true, if that's what you think my bias is? But it is the only data I have to work with and I don't think it is okay to make data up to serve any agenda.
The widely circulated faulty charts I discuss in the post were because someone processed the offical data wrongly - hopefully an honest mistake. But then nobody seemed to care because it confirmed their bias.
How would you feel when public health incorrectly processes or publishes data. Do you think they should be held to account? Do you think others should mindlessly promote it?
I am very, very sorry to read about your son 🙏
It was reported that NZ had very strict restrictions and there must have been terrible pressure to get vaccinated. Hopefully there will be some reckoning for those responsible.
Thank you for this information from a family in New Zealand!
Luckily there are many people who did not get the jab. The official figure of unvax in NZ only use the data set of all people who interacted with the Drs in the 12 months prior to the rollout. So many like in my small town 75% unvac never complied were not included. Of the 25% who had to comply ,3 deaths "suddenly" in the 5-6 weeks post vax , unexplained. 300 people, 100 odd adults, many cases of heart issues, cancer in those who had to get the jab for work. Almost no one has had the virus here except... those that had the jab, who are now getting it multiple times ( assuming testing is 100% accurate). Thanks for engaging in a good conversation
I also suspect the unvaccinated populations are undercounted in countries all around the world and that of course makes the outcome rates for the unvaccinated seem worse than they actually are.
Again, thanks for this info from NZ. I spent a year there in my twenties climbing, hiking, and working on farms. Beautiful land and lovely people. I was shocked and saddened to see how authoritarian it became during the pandemic. Take care, Wendy.
https://nzdsos.com/2023/02/24/excess-death-unprecedented-2022/
Thanks for bringing this up!
Need to look at deliberate spreading date in NZ opening border.
Step 1: from 23:59 on 27 February 2022
During this phase, vaccinated New Zealanders and eligible travellers from Australia were able to enter New Zealand without the need to enter Managed Isolation and Quarantine (MIQ). Vaccinated travellers were still required to self-isolate on arrival.
Step 2: from 23:59 on 4 March 2022
Vaccinated New Zealanders and eligible travellers from anywhere in the world can enter New Zealand without the need to enter MIQ.
That led to the explosion in cases and Deaths.
https://www.worldometers.info/coronavirus/country/new-zealand/
About 4,000 Covid19 dead so far.
Adjust the denominator by age as we go?
https://www.populationpyramid.net/new-zealand/2021/
Thanks for this information, Geoff!
Like I conceded in the post, I haven't dug very deeply into the NZ big picture, and limited my focus to the charts debacle and a more general philosophical point about cognitive biases.
I don't doubt the NZ is seeing a similar climb in all-cause deaths post vaccinations and the lifting of restrictions. I will try to keep an eye on NZ going forward especially because it offers a unique case study of a highly isolated, highly vaccinated, highly restricted country to compare with Germany and others.
Vaxxed:Unvaxxed Population = 93:13 = ~13.3 (ie. ~93?% population vaxxed)
should read:
Vaxxed:Unvaxxed Population = 93:7 = ~13.3 (ie. ~93?% population vaxxed)
I also think (please correct me if I am wrong) that the formula using the odds ratio is not an approximation but exact, if the distinction is binary (i.e., one can only be vaccinated or unvaccinated, and these two make up the total population). See also the formulas here:
https://usmortality.substack.com/p/covid-19-vaccine-efficacy-against
That's so funny =) So I am right that using the Odds Ratio is normally an approximation but as Ben's equations (and the link he provides) show, because I chose binary complimentary categories of vaccinated:unvaccinated, in this instance you are correct and it is indeed exact.
(whereas boosted vs unvaccinated would be an approximation)
I have long used the Odds Ratio when I didn't have accurate population totals to hand and now I finally understand this extra dimension!
The fixed denominator was a pretty obvious problem. No mercy for whatever big-shots fell for it. Name and shame these intellectual toddlers!
I'm not interested in picking personal battles. It's exhausting and generally not productive.
btw. would you be interested sometime in skyping/chatting about your take on the ONS mortality rates? I have the feeling that your take and Prof Fenton's take are more complimentary than opposite but I am struggling to synthesise the two.
"Exhausting and generally not productive" could be the slogan for a monster truck rally. I can, however, confirm that pointing out shoddy work and obvious mistakes on "our side" doesn't bring in new subscribers, regardless of whether it should.