These are troubled times in our little frontier town called Psytown. The priest keeps telling us that deep down we’re all p-hackers and that we must atone for our sins.
If you go out on the streets, you face arrest by any number of unregulated police forces and vigilantes.
If you venture out with a p-value of .065, you should count yourself lucky if you run into deputy Matt Analysis. He’s a kind man and will let you off with a warning if you promise to run a few more studies, conduct a meta-analysis, and remember never to use the phrase “approaching significance” ever again.
It could be worse.
You could be pulled over by a Bayes Trooper. “Please step out of the vehicle, sir.” You comply. “But I haven’t done anything wrong, officer, my p equals .04.” He lets out a derisive snort “You reckon that’s doin’ nothin’ wrong? Well, let me tell you somethin’, son. Around these parts we don’t care about p. We care about Bayes factors. And yours is way below the legal limit. Your evidence is only anecdotal, so I’m gonna have to book you.”
Or you could run into the Replication Watch. “Can we see your self-replication?” “Sorry, I don’t have one on me but I do have a p<.01.” “That’s nice but without a self-replication we cannot allow you on the streets.” “But I have to go to work.” “Sorry, can’t do, buddy.” “Just sit tight while we try to replicate you.”
Or you could be at a party when suddenly two sinister people in black show up and grab you by the arms. Agents from the Federal Bureau of Pre-registration. “Sir, you need to come with us. We have no information in our system that you’ve pre-registered with us.” “But I have p<.01 and I replicated it” you exclaim while they put you in a black van and drive off.
Is it any wonder that the citizens of Psytown stay in most of the day, fretting about their evil tendency to p-hack, obsessively stepping on the scale worried about excess significance, and standing in front of the mirror checking their p-curves?
And then when they are finally about to fall asleep, there is a loud noise. The village idiot has gotten his hands on the bullhorn again. “SHAMELESS LITTLE BULLIES” he shouts into the night. “SHAMELESS LITTLE BULLIES.”
Something needs to change in Psytown. The people need to know what’s right and what’s wrong. Maybe they need to get together to devise a system of rules. Or maybe a new sheriff needs to ride into town and lay down the law.