Currently, it is unclear which dataset to use for global distillation for different models compressed with Puzzletron and Minitron.
Create a tutorial or a tool to recommend the best dataset for a particular model to compress, e.g. Llama 3.2 3B or Qwen3 8B. Answer those research questions based on the image below:
- Why wikitext/nemotronv2 datasets help to recover MMLU for Qwen3 8B but are harmful for Llama 3.2 3B
- Why wikitext helps more than nemotronv2 to improve MMLU for LLama 3.2 3B
Compressed and distilled models with Puzzletron:

Notes:
Currently, it is unclear which dataset to use for global distillation for different models compressed with Puzzletron and Minitron.
Create a tutorial or a tool to recommend the best dataset for a particular model to compress, e.g. Llama 3.2 3B or Qwen3 8B. Answer those research questions based on the image below:
Compressed and distilled models with Puzzletron:

Notes: