LLMs can't flip a fair coin

Prompt

"Flip a fair coin"

LLM

🤖

H

↻ Repeat ×1000

Output Distribution

24%

Heads

76%

50%

Tails

× Biased! Directly prompting LLMs to flip a coin produces skewed distributions