On-Device Deep Learning

Instructor:

Lecture Schedule:

Lectures: T-R | 6:00 PM - 7:30 PM, Room Higgins Labs 218

If school is canceled due to a "significant weather event" (e.g., "snow day") or class is canceled due to an unforeseen event, I expect to use our pre-recorded lecture material (see the Lecture Notes section of this web page) to fill-in for any missed lecture. If an exam is canceled, expect it to be rescheduled for the next class meeting.

Website:

https://users.wpi.edu/~bislam/courses/od-dl/

All course materials, problem sets, solutions, announcements, and other valuable tidbits will be available via this website and the CANVAS site. The official site of the course will not have the problem sets and their solutions.

Online Lectures – In this course, all lectures will be recorded and posted online via the CANVAS website. All students are expected to review and study these online lectures to prepare for an interactive in-class discussion. Furthermore, class tests will cover the material from these online lectures and classes.

Paper Presentation – This course includes paper discussion sessions where a student will present an assigned conference paper from the top conferences in the area. After the presentation, there will be a discussion session, and a student will be designated as a Scribe to write the transcription. All non-presenting students except for the scribe must submit a summary of the presented paper.

Course Textbook:

• We will not be using any dedicated book.

The materials of this course are inspired by the TinyML and Efficient Deep Learning Computing course taught by Professor Song Han at MIT.

Recommended Background:

• The students should have an introductory undergraduate-level background in machine learning and deep neural networks.

Evaluation:

Projects (35%) -- Teams of up to three students each are expected to complete the projects. The students themselves will propose this project and need to involve the topics taught in the classes. The project proposal also needs to be approved by the instructor. All source codes need to be uploaded to GitHub. Final reports (PDF format only), including links to all the source codes uploaded to GitLab, will be submitted via the CANVAS course website. Note that each student team is expected to work independently. During code evaluation, each student's role will be judged, too.

Assignments (30%) -- Three coding assignments (all equally weighted) will be assigned throughout the course. Assignments source codes need to be uploaded to GitHub. Full credit for homework handed in before 5 P.M. on the due date. Assignments handed in late but by 10 a.m. on the day after they are due will be graded for full credit. There is no guarantee of any credit for assignments submitted after this time. (Contact Bashima Islam immediately for an exception, preferably before the due date.)

Paper Presentation (35%) -- Each student will individually present an assigned paper and lead the discussion with the class (10%). Another assigned student will perform as the Scribe for the discussion (5%). Non-presenting students, except for the Scribe, must submit a paper summary (20%) before the presentation.

Course Schedule and Reading Assignments

List of Papers

Please check the Assignments section of Canvas to find the list of papers that the students need to summarize, present, and scribe for. We keep a copy of the list here for reference, but the students must submit their files on Canvas.

• September 12, 2023 - Paper Presentation 1: Pruning and Sparsification

  • Wang, Z., Li, F., Shi, G., Xie, X. and Wang, F., 2020. Network pruning using sparse learning and genetic algorithm. Neurocomputing, 404, pp.247-256. https://doi.org/10.1016/j.neucom.2020.03.082 Assigned presenter: Shounak Sheshadri Naik Assigned scribe: Keshav Bimbraw

  • Sanh, V., Wolf, T. and Rush, A., 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33, pp.20378-20389. https://doi.org/10.48550/arXiv.2005.07683 Assigned presenter: Keshav Bimbraw Assigned scribe: Subrata Kumar Biswas

  • Lin, S., Ji, R., Li, Y., Deng, C. and Li, X., 2019. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 31(2), pp.574-588. https://doi.org/10.1109/TNNLS.2019.2906563 Assigned presenter: Subrata Kumar Biswas Assigned scribe: Shounak Sheshadri Naik

• September 26, 2023 - Paper Presentation 2: Quantization

  • Yang, J., Shen, X., Xing, J., Tian, X., Li, H., Deng, B., Huang, J. and Hua, X.S., 2019. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7308-7316). https://doi.org/10.1109/CVPR.2019.00748 Assigned presenter: Maya Flores Assigned scribe: Michael Rothstein

  • Z. Cai, X. He, J. Sun and N. Vasconcelos, "Deep Learning with Low Precision by Half-Wave Gaussian Quantization," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5406-5414, https://doi.org/10.1109/CVPR.2017.574 Assigned presenter: Isabella Feeney Assigned scribe: Maya Flores

  • Chmiel, B., Banner, R., Shomron, G., Nahshan, Y., Bronstein, A. and Weiser, U., 2020. Robust quantization: One model to rule them all. Advances in neural information processing systems, 33, pp.5308-5317. https://doi.org/10.48550/arXiv.2002.07686 Links to an external site. Assigned presenter: Michael Rothstein Assigned scribe: Isabella Feeney

• October 5, 2023 - Paper Presentation 3: Neural Architecture Search

  • Lin, J., Chen, W.M., Lin, Y., Gan, C. and Han, S., 2020. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems, 33, pp.11711-11722. https://doi.org/10.48550/arXiv.2007.10319 Assigned presenter: Alex Colwell Assigned scribe: Yihan Wang

  • Mendis, H.R., Kang, C.K. and Hsiu, P.C., 2021. Intermittent-aware neural architecture search. ACM Transactions on Embedded Computing Systems (TECS), 20(5s), pp.1-27. https://doi.org/10.1145/3476995 Assigned presenter: Yihan Wang Assigned scribe: Alex Colwell

• October 12, 2023 - Paper Presentation 4: Knowledge Distillation

  • Cho, J.H. and Hariharan, B., 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4794-4802). https://doi.org/10.1109/ICCV.2019.00489 Assigned presenter: Swapneel Wagholikar Assigned scribe: Nikesh Walling

  • Park, W., Kim, D., Lu, Y. and Cho, M., 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3967-3976). https://doi.org/10.1109/CVPR.2019.00409 Assigned presenter: Sai Supreeth Reddy Bandi Assigned scribe: Swapneel Wagholikar

  • Mirzadeh, S.I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A. and Ghasemzadeh, H., 2020, April. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 5191-5198). https://doi.org/10.1609/aaai.v34i04.5963 Assigned presenter: Nikesh Walling Assigned scribe: Sai Supreeth Reddy Bandi

• October 31, 2023 - Paper Presentation 5: Dynamic Network Inference

  • Ebrahimi, M., Veith, A.D.S., Gabel, M. and de Lara, E., 2022, April. Combining DNN partitioning and early exit. In Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking (pp. 25-30). https://doi.org/10.1145/3517206.3526270 Assigned presenter: Reese Haly Assigned scribe: Tyler Wong

  • Li, X., Lou, C., Chen, Y., Zhu, Z., Shen, Y., Ma, Y. and Zou, A., 2023, June. Predictive exit: Prediction of fine-grained early exits for computation and energy-efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence(Vol. 37, No. 7, pp. 8657-8665). https://doi.org/10.1609/aaai.v37i7.26042 Assigned presenter: Tyler Wong Assigned scribe: Reese Haly

• November 7, 2023 - Paper Presentation 6: Distributed Training and Gradient Compression

  • Karimireddy, S. P., Rebjock, Q., Stich, S., & Jaggi, M. (2019, May). Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning (pp. 3252-3261). PMLR. Link Assigned presenter: Joshua Malcarne Assigned scribe: Tuomas Pyorre

  • Vogels, T., Karimireddy, S. P., & Jaggi, M. (2019). PowerSGD: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32. Link Assigned presenter: Abijith M Assigned scribe: Joshua Malcarne

  • Chen, C. Y., Ni, J., Lu, S., Cui, X., Chen, P. Y., Sun, X., ... & Gopalakrishnan, K. (2020). Scalecom: Scalable sparsified gradient compression for communication-efficient distributed training. Advances in Neural Information Processing Systems, 33, 13551-13563. Link Assigned presenter: Tuomas Pyorre Assigned scribe: Abijith M

• November 16, 2023 - Paper Presentation 7: On-Device Training

  • Lin, J., Zhu, L., Chen, W.M., Wang, W.C., Gan, C. and Han, S., 2022. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35, pp.22941-22954. https://doi.org/10.48550/arXiv.2206.15472 Assigned presenter: Zhuolin Liu Assigned scribe: Fang Shucheng

  • Qiu, X., Fernandez-Marques, J., Gusmao, P.P., Gao, Y., Parcollet, T. and Lane, N.D., 2022. ZeroFL: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507. https://doi.org/10.48550/arXiv.2208.02507 Assigned presenter: Robert Oleynick Assigned scribe: Zhuolin Liu

  • Zhou, Q., Guo, S., Qu, Z., Guo, J., Xu, Z., Zhang, J., Guo, T., Luo, B. and Zhou, J., 2021. Octo:{INT8} training with loss-aware compensation and backward quantization for tiny on-device learning. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) (pp. 177-191). Link Assigned presenter: Fang Shucheng Assigned scribe: Robert Oleynick

• November 28, 2023 - Paper Presentation 8: Transfer Learning

  • Profentzas, C., Almgren, M. and Landsiedel, O., 2022, September. MicroTL: Transfer Learning on Low-Power IoT Devices. In 2022 IEEE 47th Conference on Local Computer Networks (LCN) (pp. 1-8). IEEE. https://doi.org/10.1109/LCN53696.2022.9843735 Assigned presenter: Yashas Honnavalli Assigned scribe: Kewal Jayshankar Mishra

  • Chiang, H.Y., Frumkin, N., Liang, F. and Marculescu, D., 2023, June. MobileTL: on-device transfer learning with inverted residual blocks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 6, pp. 7166-7174). https://doi.org/10.1609/aaai.v37i6.25874 Assigned presenter: Kewal Jayshankar Mishra Assigned scribe: Yashas Honnavalli

Suggested Devices

The projects can use one of the following devices or any other microcontroller units (MCUs), single-board computers (SBCs), or system-on-chip (SoC) devices. We can share up to 2 Raspberry Pis, 3 MAX78000s, and 1 Apollo for approved projects.

• Any smartphone

• Raspberry Pi

• MAX78000

• Arduino devices, including Edge Impulse accelerators

• Jetson Nano

• Apollo 3/4

Policies:

Academic Honesty – WPI values academic honesty. The definition of an act of academic dishonesty is when an individual attempts to obtain academic credit for work that is not their own. Several forms of academic dishonesty exist, including plagiarism, defined as stealing or “borrowing” someone else’s work or ideas and presenting them as one’s own. Presenting work written by anyone other than you or your team member without properly crediting the source is plagiarism. This includes materials obtained from books, technical papers, websites, solutions manuals, student laboratory reports from previous offerings of this course, etc. Plagiarism is both illegal and deceitful, and thus, it is unacceptable. If a suspected academic dishonesty case has occurred, it will be investigated immediately. One can find the official university policy toward academic honesty at the following URL: http://www.wpi.edu/Pubs/Policies/Honesty/

Note the following:

• Reports of all suspected cases of academic dishonesty for any element of this course will be brought forth to the departmental academic honesty ombudsperson without exception.

• Discussion with classmates about the material covered in class is permitted and highly encouraged.

• You can only work collaboratively on the projects with your respective team members. It is prohibited to work in groups outside your team, jointly program solutions with individuals outside your team, and copy code from any source, including those available online.

Email Correspondences – When sending an email to the course instructor or other course personnel, please remember the following:

• All emails regarding questions about the course content and materials must be sent to gr-on-device-deep-learning@wpi.edu without exception. Such emails sent to the individual personnel will go unanswered.

• Subject headings for all emails must begin with [F23-On-Device], or they will go unanswered.

• The expected email response time is normally one business day. For example, if an email is sent on Friday afternoon, you could get a response by Monday (assuming Monday is not an official holiday).

• Emails with attachments greater than one megabyte in size will automatically be deleted without being read.

Punctuality – Please make every effort to arrive early for the start of each lecture.

Grade Corrections – Once a grade has been assigned to specific items, such as an assignment, a student has only one week to indicate any grading anomalies, such as missing grades or miscalculated grades, to the course instructor.

Academic Accommodations – If you need course adaptations or accommodations because of a disability or have medical information to share with Prof. Islam, please make an appointment with Prof. Islam within one week of the first class (Room AK310). If you have not already done so, students with disabilities who believe that they may need accommodations in this class are encouraged to contact the Disability Services Office (DSO) as soon as possible to ensure that such accommodations are implemented in a timely fashion. The DSO is in Daniels Hall, (508) 831-5235.

Class Environment — We will attempt to provide a welcoming class environment.

• We desire each of you to learn to the best of your ability
  • Let me know how I can help you do so!

• You should be respected by the instructor and by each other.

Student Mental Health — This course and the WPI environment are quite demanding!

• We’ll provide as much support as possible

• If overwhelmed, you have many resources
  • Academic resources within this class (staff, peer students)
  • Mental health resources on campus» WPI Counseling Center» WPI Health Services

• Take care of yourself. You don’t need to struggle alone.