Skip to content

Provide better way of recommending a dataset for global distillation for puzzletron algorithm #1850

Description

@danielkorzekwa

Currently, it is unclear which dataset to use for global distillation for different models compressed with Puzzletron and Minitron.

Create a tutorial or a tool to recommend the best dataset for a particular model to compress, e.g. Llama 3.2 3B or Qwen3 8B. Answer those research questions based on the image below:

  • Why wikitext/nemotronv2 datasets help to recover MMLU for Qwen3 8B but are harmful for Llama 3.2 3B
  • Why wikitext helps more than nemotronv2 to improve MMLU for LLama 3.2 3B

Compressed and distilled models with Puzzletron:
Image

Notes:

Metadata

Metadata

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions