Machine Learning.

LLM-Foundry uses flash_attn_varlen_func by default. BinPackCollator does naive sequence packing.
21:43
LLM-Foundry uses flash_attn_varlen_func by default. BinPackCollator does naive sequence packing.
Understanding Eager Bidirectional Attention via the Attention Mask 🎭
26:09
Understanding Eager Bidirectional Attention via the Attention Mask 🎭
The Evolution of Matrix Multiplication, Part 2: PyTorch and Numba on the GPU | fastai course Part 2
26:59
The Evolution of Matrix Multiplication, Part 2: PyTorch and Numba on the GPU | fastai course Part 2
Understanding ColBERT's ivf.pid.pt: Inspecting Intermediate Artifacts from _build_ivf & optimize_ivf
50:20
Understanding ColBERT's ivf.pid.pt: Inspecting Intermediate Artifacts from _build_ivf & optimize_ivf
Do RAGatouille and ColBERT Produce the Same Index and Retrieval Scores? A Deep Dive Comparison
19:13
Do RAGatouille and ColBERT Produce the Same Index and Retrieval Scores? A Deep Dive Comparison
TIL: Understanding LLM Foundry's BinPackCollator (Sequence Packing for 95% Token Efficiency!)
19:12
TIL: Understanding LLM Foundry's BinPackCollator (Sequence Packing for 95% Token Efficiency!)
Improving LLM Judge Alignment: Enhancing TinyScale Lab Evaluation Agreement to 94%
23:06
Improving LLM Judge Alignment: Enhancing TinyScale Lab Evaluation Agreement to 94%
Technical Report Summary: Nomic Embed
26:38
Technical Report Summary: Nomic Embed
Understanding Sequence Packing: Initial Musings
21:17
Understanding Sequence Packing: Initial Musings
Building an LLM Judge Agreement App: 7 Iterations from Basic to Full Functionality
15:06
Building an LLM Judge Agreement App: 7 Iterations from Basic to Full Functionality
Evaluating First Attempt LLM Judge Scores: Improving Claude Haiku Alignment for Story Scoring
44:00
Evaluating First Attempt LLM Judge Scores: Improving Claude Haiku Alignment for Story Scoring
Manual Scoring Results for TinyStories Models: Grammar, Reasoning, and Emergent Capabilities
36:39
Manual Scoring Results for TinyStories Models: Grammar, Reasoning, and Emergent Capabilities
Look at Your Data: Building an LM Scoring App with FastHTML
40:23
Look at Your Data: Building an LM Scoring App with FastHTML
TSL: Curating Evaluation Prompts, Defining Scoring Criteria + Designing LLM Judge Prompt Template
30:15
TSL: Curating Evaluation Prompts, Defining Scoring Criteria + Designing LLM Judge Prompt Template
TinyScale Lab Update: Setting Eval Targets + Generation Completions for LLM Judge Development
19:41
TinyScale Lab Update: Setting Eval Targets + Generation Completions for LLM Judge Development
TinyScaleLab Project Update: Training Cost Analysis and Evaluation Infrastructure Plans
9:24
TinyScaleLab Project Update: Training Cost Analysis and Evaluation Infrastructure Plans
TinyScale Lab: Exploring the Connection Between Training Dynamics and Model Capabilities
26:51
TinyScale Lab: Exploring the Connection Between Training Dynamics and Model Capabilities
Research Paper Summary: Small-scale proxies for large-scale Transformer training instabilities
1:04:03
Research Paper Summary: Small-scale proxies for large-scale Transformer training instabilities
My Second-Place Winning Tiny Model Hackathon Journey: Pre-Training from Scratch
22:47
My Second-Place Winning Tiny Model Hackathon Journey: Pre-Training from Scratch
LossInspector: A Deep Dive Into LLM-Foundry's Next-Token Prediction with a Custom Composer Callback
21:19
LossInspector: A Deep Dive Into LLM-Foundry's Next-Token Prediction with a Custom Composer Callback
The Evolution of Matrix Multiplication: 12,000x Numba Speedup 🚀 | fastai Course Lesson 11
28:41
The Evolution of Matrix Multiplication: 12,000x Numba Speedup 🚀 | fastai Course Lesson 11
Research Paper Summary: TinyStories
1:23:34
Research Paper Summary: TinyStories
Look at Your Data: Manual Validation of Retrieval Metrics
47:38
Look at Your Data: Manual Validation of Retrieval Metrics
Creating a Custom Composer Callback to Track Data Types in LLM Training | Mixed Precision Deep Dive
46:10
Creating a Custom Composer Callback to Track Data Types in LLM Training | Mixed Precision Deep Dive
Paper Reading: Small-scale proxies for large-scale Transformer training instabilities
1:16:36
Paper Reading: Small-scale proxies for large-scale Transformer training instabilities
Paper Reading: Overtrained Language Models Are Harder to Fine-Tune
1:36:58
Paper Reading: Overtrained Language Models Are Harder to Fine-Tune
Paper Reading: SmolLM2
55:01
Paper Reading: SmolLM2
Exploring Sequential and Merged Linear Layer Forward Passes
20:16
Exploring Sequential and Merged Linear Layer Forward Passes
TIL: Using PyTorch's register_forward_hook to Trace Floating Point Errors
9:21
TIL: Using PyTorch's register_forward_hook to Trace Floating Point Errors
Debugging Un-Merged and Merged LoRA Model Output Differences
26:21
Debugging Un-Merged and Merged LoRA Model Output Differences