We're sunsetting PodQuest on 2025-07-28. Thank you for your support!
Export Podcast Subscriptions
cover of episode [Linkpost] “Histograms are to CDFs as calibration plots are to...” by Optimization Process

[Linkpost] “Histograms are to CDFs as calibration plots are to...” by Optimization Process

2025/6/6
logo of podcast LessWrong (30+ Karma)

LessWrong (30+ Karma)

Shownotes Transcript

This is a link post. As you know, histograms are decent visualizations for PDFs with lots of samples... 10k predictions, 20 bins

...but if there are only a few samples, the histogram-binning choices can matter a lot: 10 predictions, 4 binssame 10 predictions, 7 bins The binning (a) discards information, and worse, (b) is mathematically un-aesthetic.

But a CDF doesn't have this problem! same 10 predictions, every data point precisely represented If you make a bunch of predictions, and you want to know how well they're calibrated, classically you make a graph like this: source: SSC's 2019 prediction grading But, as with a histogram, this depends on how you bin your predictions. 100 predictions, 10 binssame 100 predictions, 30 bins Is there some CDF-like equivalent here? Some visualization with no free parameters?

I asked that question to several people at Arbor Summer Camp. I got three answers:

  • "You get from a PDF to [...]

First published: June 5th, 2025

Source: https://www.lesswrong.com/posts/LFGgwitjertJqch7J/histograms-are-to-cdfs-as-calibration-plots-are-to)

Linkpost URL:https://optimizationprocess.com/calibration-cdf/)

  ---
    

Narrated by TYPE III AUDIO).


Images from the article: 10k predictions, 20 bins)10 predictions, 4 bins)same 10 predictions, 7 bins)same 10 predictions, every data point precisely represented)100 predictions, 10 bins)same 100 predictions, 30 bins).py)Graph showing overprediction curve and standard deviation bands against perfect prediction line.)A calibration graph comparing predicted probabilities versus actual outcomes, with reference line.) Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts), or another podcast app.