Knowledge Distillation of Black-Box Large Language Models

View PDF HTML (experimental) Abstract:Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosting the capabilities of smaller models through knowledge distillation (KD) from these powerful yet black-box teachers. While leveraging the high-quality outputs of these teachers is advantageous, the inaccessibility of their internal states often limits effective knowledge transfer. To overcome this limitation, we introduce Proxy-KD, a novel method that uses a proxy model to facilitate the efficient transfer of knowledge from black-box LLMs to smaller models. Our experiments show that Proxy-KD not only enhances the performance of KD from black-box teacher models but also surpasses traditional white-box KD techniques.~This approach presents a compelling new avenue for distilling knowledge from advanced LLMs.

Subjects: Computation and Language (cs.CL) Cite as: arXiv:2401.07013 [cs.CL]   (or arXiv:2401.07013v2 [cs.CL] for this version)   https://doi.org/10.48550/arXiv.2401.07013 arXiv-issued DOI via DataCite Submission history From: Hongzhan Chen [view email] [v1] Sat, 13 Jan 2024 08:43:32 UTC (359 KB) [v2] Sat, 9 Nov 2024 01:35:32 UTC (8,288 KB)