Research Interests

My research is in machine learning and computer vision, including deep learning, and its applications to affective computing, automatic facial expression recognition, human behavior analysis, and educational data mining. I am also interested in how to make machine learning processes more efficient by devising innovative paradigms for collecting training datasets and annotations.

My work is highly interdisciplinary and frequently intersects cognitive science, psychology, and education.

Grants

• Whitehill, J. CAREER: Developing New Scientific Instruments for Classroom Observation: A Multi-modal Machine Learning Approach (NSF #2046505, $691,980).
• D'Mello, S., Palmer, M., Beveridge, J., Sumner, T., and Puntambekar, S. AI Institute: Institute for Student-AI Teaming (NSF #2019805, $3,997,841). Project Site.
• Liu, S., Van Dessel, S., Whitehill, J., Rodriguez, A., and Farzinmoghadam, M. RAPID: Measuring the Impact of the COVID-19 Pandemic on STEM Student Engagement and Learning in Online Learning Environments (NSF #2028224, $199,943).
• Liu, S., Van Dessel, S., and Whitehill, J. (co-PI). Designing Responsive Physical Learning Environments to Promote Student Engagement and Learning (NSF #1931077, $299,991).
• Calli, B., and Whitehill, J. (co-PI). FW-HTF-RL: Collaborative Research: Shared Autonomy for the Dull, Dirty, and Dangerous: Exploring Division of Labor for Humans and Robots to Transform the Recycling Sorting Industry (NSF #1928506, $604,314).
• Whitehill, J. (PI), Ottmar, E., Harrison, L., and LoCasale-Crouch, J. Teachers are the Learners: Providing Automated Feedback on Classroom Interpersonal Dynamics (NSF #1822768, $749,969). Project Site.
• Botelho, A., Heffernan, N., Whitehill, J. (co-PI), and Ostrow, K. Putting Teachers in the Driver's Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT) (NSF #1822830, $764,317).
• Whitehill, J. (PI) and Ottmar, E. Towards Computer-Assisted Coding of Classroom Observations: A Computer Vision Approach to Measuring Positive Climate (Spencer Foundation Small Research Grant, $49,850)..
• Arroyo, I., and Whitehill, J. (co-PI). INT: Collaborative Research: Detecting, Predicting and Remediating Student Affect and Grit Using Computer Vision (NSF #1551594, $759,887).

Publications

2022

• Whitehill, J., and Erfanian, A. "How to Give Imperfect Automated Guidance to Learners: A Case-Study in Workplace Learning". Artificial Intelligence in Education, 2022. PDF.
• Jiang, H., Li, Z., and Whitehill, J. "Can the Mathematical Correctness of Object Configurations Affect the Accuracy of Their Perception?". 1st CVPR Workshop on Vision Datasets Understanding, 2022. PDF.
• Zhang, Z., Cisneros, E., Lee, H., Vu, J., Chen Q., Benadof, C., Whitehill, J., Rouzbehani, R., Sy, D., Huang, J., Sejnowski, T., Jankovic, J., Factor, S., Goetz, C., Barbano, R., Perlmutter, J., Jinnah, H., Berman, B., Richardson, S., Stebbins, G., Comella, C., and Peterson, D.. "Hold that pose: capturing cervical dystonia's head deviation severity from video". Annals of Clinical and Translational Neurology, 2022. PDF.

2021

• Ruiz, N., Yu, H., Allessio, D., Jalal, M., Joshi, A., Murray, T., Magee, J., Whitehill, J., Ablavsky, V., Arroyo, I., Woolf, B., Sclaroff, S., and Betke, M. "Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System", IEEE Automatic Face & Gesture Recognition, 2021.
• Jiang, H., Serpell, Z., and Whitehill, J. "Measuring the Effect of ITS Feedback Messages on Students' Emotions". AIED Workshop on Multimodal Artificial Intelligence in Education, 2021. PDF.
• Gupta, A., Menon, N., Lee, W., Rebelsky, W., Allesio, D., Murray, T., Woolf, B., Whitehill, J., and Arroyo, I. "Affective Teacher Tools: an Affective Class Report Card and an Affective Dashboard". Artificial Intelligence in Education, 2021.
• Liamthong, P, and Whitehill, J. "Text Representations of Math Tutorial Videos for Clustering, Retrieval, and Learning Gain Prediction". Educational Data Mining, 2021. PDF.
• Li, Z., and Whitehill, J. "Compositional embedding models for speaker identification and diarization with simultaneous speech from 2+ speakers". IEEE International Conference on Speech and Signal Processing (ICASSP), 2021. PDF.
• Kyriacou, H., Ramakrishnan, A., and Whitehill, J. "Learning to Work in a Materials Recovery Facility: Can Humans and Machines Learn from Each Other?". Learning Analytics and Knowledge (LAK), 2021. PDF.
• Li, Z., Mozer, M., and Whitehill, J. "Compositional Embeddings for Multi-Label One-Shot Learning". Winter Conference on Applications of Computer Vision (WACV), 2021. PDF.

2020

• Pham, M., Li, Z., and Whitehill, J. "How Does Label Noise Affect the Quality of Speaker Embeddings?". InterSpeech 2020. PDF.
• Aguerrebere, C., Bulger, M., Cobo, C., Garcia, S., Kaplan, G., and Whitehill, J. "How Teachers Affect Students' Online Participation in EFL Courses in Uruguay". Computer-Based Learning in Context, 2020. PDF.
• Zylich, B., and Whitehill, J. "Noise-Robust Key-Phrase Detectors for Automated Classroom Feedback". IEEE International Conference on Speech and Signal Processing (ICASSP), 2020. PDF.
• Pham, M., Li, Zeqian, and Whitehill, J. "Toward Better Speaker Embeddings: Automated Collection of Speech Samples from Unknown Distinct Speakers". IEEE International Conference on Speech and Signal Processing (ICASSP), 2020. PDF. BookTubeSpeech dataset.

2019

• Weiss, M., Paffenroth, R., Whitehill, J., and Uzarski, J. "Deep Learning with Domain Randomization for Optimal Filtering". IEEE International Conference on Machine Learning and Applications, 2019.
• Whitehill, J., and Ramakrishnan, A. "Automatic Classifiers as Scientific Instruments: One Step Further Away from Ground-Truth". International Conference on Machine Learning, 2019. PDF and Supplementary materials.
• Shirai, S., and Whitehill, J. "Privacy-Preserving Annotation of Face Images through Attribute-Preserving Face Synthesis". CVPR Workshop on the Bright and Dark Sides of Computer Vision: Challenges and Opportunities for Privacy and Security (CV-COPS), 2019. PDF.
• Whitehill, J., Aguerrebere, C., and Hylak, B. "Do Learners Know What's Good for Them? Crowdsourcing Subjective Ratings of OERs to Predict Learning Gains". Educational Data Mining, 2019. PDF.
• Aguerrebere, C., Bulger, M., Cobo, C., Garcia, S., Kaplan, G., and Whitehill, J. "How Should Online English as a Foreign Language Teachers Write their Feedback to Students?". Educational Data Mining, 2019. PDF.
• Jiang, H., Iandoli, M., Van Dessel, S., Liu, S., and Whitehill, J. "Measuring students' thermal comfort and its impact on learning". Educational Data Mining, 2019. PDF.
• Joshi, A., Allessio, D., Magee, J., Whitehill., J., Arroyo, I., Woolf, B., Sclaroff, S., and Betke, M. "Affect-driven Learning Outcomes Prediction in Intelligent Tutoring Systems". IEEE Automatic Face & Gesture Recognition, 2019.
• Whitehill, J. "How Does Knowledge of the AUC Constrain the Set of Possible Ground-truth Labelings?". AAAI, 2019. PDF.

2018

• Aguerrebere, C., Cobo, C., and Whitehill, J. "Estimating the Treatment Effect of New Device Deployment on Uruguayan Students' Online Learning Activity". Educational Data Mining, 2018. PDF.
• Aung, A., Ramakrishnan, A., and Whitehill, J. "Who are they looking at? Automatic Eye Gaze Following for Classroom Observation Video Analysis". Educational Data Mining, 2018. PDF.
• Giancola, M., Paffenroth, R., and Whitehill, J. "Permutation-Invariant Consensus over Crowdsourced Labels". Human Computation (HCOMP), 2018. PDF.
• Aung, A., and Whitehill, J. "Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition". IEEE Automatic Face & Gesture Recognition (FG), 2018. PDF and Supplementary Materials.
• Jiang, H., Dykstra, K., and Whitehill, J. "Predicting When Teachers Look at Their Students in 1-on-1 Tutoring Sessions". IEEE Automatic Face & Gesture Recognition (FG) Workshop on Human Behavior Understanding (HUB). PDF.
• Whitehill, J. "Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle". AAAI Workshop on Engineering Dependable and Secure Machine Learning Systems. PDF.

2017

• Whitehill, J., and Movellan, J. "Approximately Optimal Teaching of Approximately Optimal Learners". Transactions on Learning Technologies (accepted), 2017. PDF. Appendix: PDF.
• Whitehill, J., Mohan, K., Seaton,, D., Rosen, Y., and Tingley, D. "MOOC Dropout Prediction: How to Measure Accuracy?". Learning at Scale 2017. PDF
• Whitehill, J., and Seltzer, M. "Toward Personalized Learning-at-Scale: A Crowdsourcing Approach". Learning at Scale 2017. PDF
• Turkay, S., Eidelman, H., Rosen, Y., Seaton, D., Lopez, G., and Whitehill, J. "Getting to know English language learners in MOOCs: Their motivations, behaviors and outcomes." Learning at Scale 2017.

Older work

• Whitehilll, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. "Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. NIPS 2009. Supplementary materials. Software.

Complete list

See my Google Scholar profile.

Talks

• October 2, 2020: CU Boulder Institute for Cognitive Science Virtual Speaker Series.
• February 20, 2020: Boston University Learning Science Speaker Series, slides.
• July 4, 2019: Educational Data Mining, slides.
• July 16, 2018: Educational Data Mining, slides.
• May 17, 2018: IEEE Automatic Face & Gesture Recognition: Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition. slides.
• April 12, 2018: National Council on Educational Measurement Symposium: Measuring Collaboration and Engagement using Big Data slides.
• February 3, 2018: AAAI Workshop on Engineering Dependable and Secure Machine Learning Systems slides.
• December 14, 2017: IEEE Big Data PSBD Workshop slides.
• March 31, 2017: TU-Delft Data Science Seminar slides.
• October 5, 2016: Boston University IVC Group Talk slides.

Teaching

• Spring 2020, D-term: CS4342: Machine Learning
• Spring 2019, D-term: CS453X: Machine Learning
• Spring 2018, D-term: CS453X: Machine Learning
• Spring 2021: CS/DS 541: Deep Learning
• Spring 2020: CS/DS 541: Deep Learning
• Spring 2019: CS/DS 541: Deep Learning
• Spring 2018: CS525 191D: Deep Neural Networks
• Spring 2017: CS525 191N: Deep Neural Networks
• Fall 2020, B-term: CS2103: Accelerated Object-oriented Design
• Fall 2019, B-term: CS2103: Accelerated Object-oriented Design
• Fall 2018, B-term: CS2103: Accelerated Object-oriented Design
• Fall 2017, B-term: CS210X: Accelerated Object-oriented Design
• Fall 2016, B-term: CS210X: Accelerated Object-oriented Design