Class-wise knowledge distillation

Fei Li; Yifang Yang

doi:10.1117/12.2667603

22 February 2023 Class-wise knowledge distillation

Fei Li, Yifang Yang

Proceedings Volume 12587, Third International Seminar on Artificial Intelligence, Networking, and Information Technology (AINIT 2022); 125870T (2023) https://doi.org/10.1117/12.2667603
Event: Third International Seminar on Artificial Intelligence, Networking, and Information Technology (AINIT 2022), 2022, Shanghai, China

Abstract

Knowledge distillation (KD) transfers knowledge of a teacher model to improve the performance of a student model which is usually equipped with a lower capacity. The standard KD framework, however, neglects that the DNNs exhibit a wide range of class-wise accuracy and the performance of some classes is even decreased after distillation. Observing the above phenomena, we propose a novel Class-Wise Knowledge Distillation method to find the hard classes with a simple yet effective technique and then make the students take more effort to learn these hard classes. In the experiments on image classification tasks using CIFAR-100 dataset, we demonstrate that the proposed method outperforms the other KD methods and achieves excellent performance enhancement on various networks.

Citation Download Citation

Fei Li and Yifang Yang "Class-wise knowledge distillation", Proc. SPIE 12587, Third International Seminar on Artificial Intelligence, Networking, and Information Technology (AINIT 2022), 125870T (22 February 2023); https://doi.org/10.1117/12.2667603

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
4 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Networks

Neural networks

Performance modeling

Visual process modeling

Visualization

RGB color model

Network architectures

Keywords/Phrases

Search In:

Publication Years