This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models
Source code of paper "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation"
