“Introducing the WeirdML Benchmark” by Håvard Tveit Ihle
LessWrong (30+ Karma)
Shownotes Transcript
WeirdML website Related posts: How good are LLMs at doing ML on an unknown dataset? o1-preview is pretty good at doing ML on an unknown dataset
** Introduction** How good are Large Language Models (LLMs) at doing machine learning on novel datasets? The WeirdML benchmark presents LLMs with weird and unusual machine learning tasks, designed to require careful thinking and actual understanding to solve, and tests an LLM's ability to:
Actually understand the properties of the data and the problem Come up with an appropriate ML architecture and training setup for the problem, and generate working PyTorch code that implements the solution Debug and improve the solution over 5 iterations based on terminal output and the accuracy on the test set Make good use of limited computational resources and time
Each task comes with a task prompt describing the problem precisely and some example code for loading data and [...]
Outline:
(00:18) Introduction
(01:24) Results
(01:47) Evaluation Setup
(02:28) System Architecture
(03:26) Tasks
(04:04) Shapes (Easy)
(05:53) Shapes (Hard)
(07:42) Image Patch Shuffling (Easy)
(09:46) Image Patch Shuffling (Hard)
(12:17) Chess Game Outcome Prediction
(14:28) Unsupervised Digit Recognition
(15:59) Further Analysis
(16:21) Failure Rate
(17:11) Model Performance by Number of Iterations
(18:13) Maximum of k First Submissions (max@k)
(20:11) Future Directions
First published: January 16th, 2025
Source: https://www.lesswrong.com/posts/LfQCzph7rc2vxpweS/introducing-the-weirdml-benchmark)
---
Narrated by TYPE III AUDIO).
Images from the article:
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts), or another podcast app.