{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:05:11Z","timestamp":1760234711821,"version":"build-2065373602"},"reference-count":61,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,6,16]],"date-time":"2021-06-16T00:00:00Z","timestamp":1623801600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation","award":["1757929"],"award-info":[{"award-number":["1757929"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>First impressions make up an integral part of our interactions with other humans by providing an instantaneous judgment of the trustworthiness, dominance and attractiveness of an individual prior to engaging in any other form of interaction. Unfortunately, this can lead to unintentional bias in situations that have serious consequences, whether it be in judicial proceedings, career advancement, or politics. The ability to automatically recognize social traits presents a number of highly useful applications: from minimizing bias in social interactions to providing insight into how our own facial attributes are interpreted by others. However, while first impressions are well-studied in the field of psychology, automated methods for predicting social traits are largely non-existent. In this work, we demonstrate the feasibility of two automated approaches\u2014multi-label classification (MLC) and multi-output regression (MOR)\u2014for first impression recognition from faces. We demonstrate that both approaches are able to predict social traits with better than chance accuracy, but there is still significant room for improvement. We evaluate ethical concerns and detail application areas for future work in this direction.<\/jats:p>","DOI":"10.3390\/s21124127","type":"journal-article","created":{"date-parts":[[2021,6,16]],"date-time":"2021-06-16T21:58:32Z","timestamp":1623880712000},"page":"4127","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["The Effectiveness of Multi-Label Classification and Multi-Output Regression in Social Trait Recognition"],"prefix":"10.3390","volume":"21","author":[{"given":"Will","family":"Farlessyost","sequence":"first","affiliation":[{"name":"Agricultural & Biological Engineering, Purdue University, West Lafayette, IN 47907, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kelsey-Ryan","family":"Grant","sequence":"additional","affiliation":[{"name":"Computer Science, Ithaca College, Ithaca, NY 14850, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9824-792X","authenticated-orcid":false,"given":"Sara R.","family":"Davis","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5502-7513","authenticated-orcid":false,"given":"David","family":"Feil-Seifer","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6990-2833","authenticated-orcid":false,"given":"Emily M.","family":"Hand","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1177\/0022022111411386","article-title":"First impressions from faces among US and culturally isolated Tsimane\u2019people in the Bolivian rainforest","volume":"43","author":"Zebrowitz","year":"2012","journal-title":"J. Cross Cult. Psychol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1111\/j.1467-9280.2006.01750.x","article-title":"First impressions: Making up your mind after a 100-ms exposure to a face","volume":"17","author":"Willis","year":"2006","journal-title":"Psychol. Sci."},{"key":"ref_3","unstructured":"Schaller, M., and Duncan, L.A. (2007). The behavioral immune system: Its evolution and social psychological implications. Sydney Symposium of Social Psychology. Evolution and the Social Mind: Evolutionary Psychology and Social Cognition, Routledge\/Taylor & Francis Group."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1638","DOI":"10.1098\/rstb.2010.0404","article-title":"Facial attractiveness: Evolutionary based research","volume":"366","author":"Little","year":"2011","journal-title":"Philos. Trans. R. Soc. Biol. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Coetzee, V., Greeff, J.M., Stephen, I.D., and Perrett, D.I. (2014). Cross-cultural agreement in facial attractiveness preferences: The role of ethnicity and gender. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0099629"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1037\/h0033731","article-title":"What is beautiful is good","volume":"24","author":"Dion","year":"1972","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1037\/h0026086","article-title":"A multidimensional approach to the structure of personality impressions","volume":"9","author":"Rosenberg","year":"1968","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"E3353","DOI":"10.1073\/pnas.1409860111","article-title":"Modeling first impressions from highly variable facial images","volume":"111","author":"Vernon","year":"2014","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1037\/0033-2909.100.1.3","article-title":"Perceiving character in faces: The impact of age-related craniofacial changes on social perception","volume":"100","author":"Berry","year":"1986","journal-title":"Psychol. Bull."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.evolhumbehav.2004.08.002","article-title":"Natural selection and the regulation of defenses: A signal detection analysis of the smoke detector principle","volume":"26","author":"Nesse","year":"2005","journal-title":"Evol. Hum. Behav."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"11087","DOI":"10.1073\/pnas.0805664105","article-title":"The functional basis of face evaluation","volume":"105","author":"Oosterhof","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"McCurrie, M., Beletti, F., Parzianello, L., Westendorp, A., Anthony, S., and Scheirer, W. (June, January 30). Predicting First Impressions with Deep Learning. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.","DOI":"10.1109\/FG.2017.147"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kumar, N., Belhumeur, P., and Nayar, S.K. (2008, January 12\u201318). Facetracer: A search engine for large collections of images with faces. Proceedings of the European Conference on Computer Vision, Marseille, France.","DOI":"10.1007\/978-3-540-88693-8_25"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kumar, N., Berg, A.C., Belhumeur, P.N., and Nayar, S.K. (October, January 29). Attribute and simile classifiers for face verification. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459250"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Dantone, M., Gall, J., Fanelli, G., and Van Gool, L. (2012, January 16\u201321). Real-time facial feature detection using conditional regression forests. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6247976"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Devries, T., Biswaranjan, K., and Taylor, G.W. (2014, January 6\u20139). Multi-task learning of facial landmarks and expression. Proceedings of the 2014 Canadian Conference on Computer and Robot Vision, Montreal, QC, Canada.","DOI":"10.1109\/CRV.2014.21"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Luo, P., Loy, C.C., and Tang, X. (2014, January 6\u201312). Facial landmark detection by deep multi-task learning. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10599-4_7"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7\u201313). Deep learning face attributes in the wild. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.425"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hand, E., and Chellappa, R. (2017, January 4\u20139). Attributes for improved attributes: A multi-task network utilizing implicit and explicit relationships for facial attribute classification. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11229"},{"key":"ref_20","first-page":"131","article-title":"Active authentication using facial attributes","volume":"3","author":"Samangouei","year":"2017","journal-title":"Mob. Biom."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13640-018-0278-6","article-title":"Age estimation via face images: A survey","volume":"2018","author":"Angulu","year":"2018","journal-title":"EURASIP J. Image Video Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1016\/j.patrec.2011.05.016","article-title":"Learning local binary patterns for gender classification on real-world face images","volume":"33","author":"Shan","year":"2012","journal-title":"Pattern Recognit. Lett."},{"key":"ref_23","unstructured":"Ou, Y., Wu, X., Qian, H., and Xu, Y. (July, January 27). A real time race classification system. Proceedings of the 2005 IEEE International Conference on Information Acquisition, Hong Kong, China."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1962","DOI":"10.1109\/TPAMI.2011.48","article-title":"Describable visual attributes for face verification and image search","volume":"33","author":"Kumar","year":"2011","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yang, J., Fan, J., Wang, Y., Wang, Y., Gan, W., Liu, L., and Wu, W. (2020, January 13\u201319). Hierarchical feature embedding for attribute recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01307"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2781","DOI":"10.1109\/TPAMI.2019.2914680","article-title":"Deep imbalanced learning for face recognition and attribute prediction","volume":"42","author":"Huang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1620","DOI":"10.1109\/TPAMI.2019.2956039","article-title":"On Symbiosis of Attribute Prediction and Semantic Segmentation","volume":"43","author":"Kalayeh","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.patcog.2019.06.006","article-title":"Improving person re-identification by attribute and identity learning","volume":"95","author":"Lin","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Rudd, E.M., G\u00fcnther, M., and Boult, T.E. (2016, January 8\u201316). Moon: A mixed objective optimization network for the recognition of facial attributes. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46454-1_2"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, J., Cheng, Y., and Feris, R.S. (2016, January 27\u201330). Walk and learn: Facial attribute representation learning from egocentric video and contextual data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.252"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hand, E., Castillo, C., and Chellappa, R. (2018, January 2\u20137). Doing the best we can with what we have: Multi-label balancing with selective learning for attribute prediction. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12313"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hand, E.M., Castillo, C.D., and Chellappa, R. (2018, January 12\u201315). Predicting facial attributes in video using temporal coherence and motion-attention. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00017"},{"key":"ref_33","unstructured":"Speth, J., and Hand, E.M. (2019, January 16\u201320). Automated Label Noise Identification for Facial Attribute Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Thom, N., and Hand, E.M. (2020). Facial Attribute Recognition: A Survey. Computer Vision, Springer.","DOI":"10.1007\/978-3-030-03243-2_815-1"},{"key":"ref_35","first-page":"231","article-title":"Personalities in faces: I. An experiment in social perceiving","volume":"49","author":"Secord","year":"1954","journal-title":"Genet. Psychol. Monogr."},{"key":"ref_36","first-page":"710","article-title":"Age similarities in recognizing threat from faces and diagnostic cues","volume":"69","author":"Boshyan","year":"2014","journal-title":"J. Gerontol. Ser. Psychol. Sci. Soc. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1037\/a0031050","article-title":"Accuracy and consensus in judgments of trustworthiness from faces: Behavioral and neural correlates","volume":"104","author":"Rule","year":"2013","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1007\/s10919-009-0082-1","article-title":"Elected in 100 milliseconds: Appearance-based trait inferences and voting","volume":"34","author":"Olivola","year":"2010","journal-title":"J. Nonverbal Behav."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1016\/j.leaqua.2011.05.009","article-title":"Face and fortune: Inferences of personality from Managing Partners\u2019 faces predict their law firms\u2019 financial success","volume":"22","author":"Rule","year":"2011","journal-title":"Leadersh. Q."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1007\/BF01065855","article-title":"The impact of litigants\u2019 baby-facedness and attractiveness on adjudications in small claims courts","volume":"15","author":"Zebrowitz","year":"1991","journal-title":"Law Hum. Behav."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1024\/1421-0185.65.4.237","article-title":"The influence of criminal facial stereotypes on juridic judgments","volume":"65","author":"Dumas","year":"2006","journal-title":"Swiss J. Psychol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1080\/10683160902926141","article-title":"Dangerous decisions: The impact of first impressions of trustworthiness on the evaluation of legal evidence and defendant culpability","volume":"16","author":"Porter","year":"2010","journal-title":"Psychol. Crime Law"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1016\/j.leaqua.2014.06.002","article-title":"The many (distinctive) faces of leadership: Inferring leadership domain from facial appearance","volume":"25","author":"Olivola","year":"2014","journal-title":"Leadersh. Q."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1146\/annurev-psych-113011-143831","article-title":"Social attributions from faces: Determinants, consequences, accuracy, and functional significance","volume":"66","author":"Todorov","year":"2015","journal-title":"Annu. Rev. Psychol."},{"key":"ref_45","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Kaya, H., Gurpinar, F., and Ali Salah, A. (2017, January 21\u201326). Multi-modal score fusion and decision trees for explainable automatic job candidate screening from video cvs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.210"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Sutherland, C., and Andrew Young, G.R. (2016). Facial First Impressions from Another Angle: How Social Judgements are Influenced by Changeable and Invariant Facial Properties. Br. J. Psychol., 108.","DOI":"10.1111\/bjop.12206"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e0193190","DOI":"10.1371\/journal.pone.0193190","article-title":"The Basel Face Database: A Validated Set of Photographs Reflecting Systematic Differences in Big Two and Big Five Personality Dimensions","volume":"13","author":"Walker","year":"2018","journal-title":"PLoS ONE"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Borchani, H., and Gherardo Varando, C.B.P.L. (2015). A Survey on Multi-Output Regression. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 5.","DOI":"10.1002\/widm.1157"},{"key":"ref_50","unstructured":"Bradski, G. (2000). Dr. Dobb\u2019s Journal of Software Tools, OpenCV Team."},{"key":"ref_51","first-page":"1755","article-title":"Dlib-ml: A Machine Learning Toolkit","volume":"10","author":"King","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_52","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Barnes, D.K., Davis, S.R., and Hand, E.M. (2020, January 5\u20137). SAT-CNN: A Small Neural Network for Object Recognition from Satellite Imagery. Proceedings of the International Symposium on Visual Computing, San Diego, CA, USA.","DOI":"10.1007\/978-3-030-64559-5_4"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Sinno Jialin Pan, Q.Y. (2009). A Survey on Transfer Learning. IEEE Trans. Knowl. Data Eng., 22.","DOI":"10.1109\/TKDE.2009.191"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/S1090-5138(99)00014-8","article-title":"Symmetry and human facial attractiveness","volume":"20","author":"Perrett","year":"1999","journal-title":"Evol. Hum. Behav."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Jabbar, H., and Khan, D. (2015). Methods to avoid over-fitting and under-fitting in supervised machine learning (comparative study). Comput. Sci. Commun. Instrum. Devices, 163\u2013172.","DOI":"10.3850\/978-981-09-5247-1_017"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Zebrowitz, L. (2018). Reading Faces: Window to the Soul?, Routledge.","DOI":"10.4324\/9780429493188"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"1623","DOI":"10.1126\/science.1110589","article-title":"Inferences of competence from faces predict election outcomes","volume":"308","author":"Todorov","year":"2005","journal-title":"Science"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1086\/511799","article-title":"Getting a job: Is there a motherhood penalty?","volume":"112","author":"Correll","year":"2007","journal-title":"Am. J. Sociol."},{"key":"ref_60","unstructured":"Kahneman, D. (2011). Thinking, Fast and Slow, Macmillan."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Asch, S.E. (1961). Forming Impressions of Personality, University of California Press.","DOI":"10.1525\/9780520313514-018"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/12\/4127\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:16:49Z","timestamp":1760163409000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/12\/4127"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,16]]},"references-count":61,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["s21124127"],"URL":"https:\/\/doi.org\/10.3390\/s21124127","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,6,16]]}}}