SCRAP Hack Summer '22

Annotating natural social interactions using deep learning

July 30, 2022 by Mark

A two or three times a year, my lab takes some time off from our normal research for a hackathon. I first participated in hackathons as part of the Methods in Neuroscience at Dartmouth (MIND) Summer School, which I attended for three years before I joined the faculty here. MIND features a combination of talks and tutorials in the mornings and early afternoons, but the late afternoons and evenings consist of hackathon time. This is where attendees get to apply the knowledge and techniques they've just been taught to fun, creative projects of their own devising. At the end, everyone shares the results with one another. Every year I was amazed by what folks did in such a short time.

In many ways, I felt like the MIND hackathons represented a idealized version of science: diverse folks coming together to work on projects driven by curiosity and passion; acquiring new knowledge and skills, and freely sharing those they already had; taking huge strides while temporarily freed from the hinderances of bureaucracy and the other vagaries of academic life. When I started as faculty at Dartmouth, I knew that this was something that I wanted to recreate, albeit on a smaller scale, within my lab. From reading management books before I started, I also knew that similar activities are common in many adjacent industry fields, such as tech. For example, many tech companies set aside time for employees to work on personal side projects. Taking a break from one's regular work can help rekindle enthusiasm and creativity, and can also provide an excellent venue for skill development. Moreover, in academia, positive reinforcement can be incredibly sparse. Going from the inception of a reserach project to the publication of a paper can take years. It can be hard to maintain one's morale in the meantime, in the face of often considerable challenges, setbacks, and negative feedback. Hackathons are short and self-contained, making them a great way to get (hopefully positive) feedback on a much faster basis.

Hackathons are not (yet) a big part of academic culture in psychology and neuroscience. The first time we ran one in the lab, I was the only one with any experience of them. That first hack was difficult, not least of all because it was completely remote, since COVID social distancing protocols were still in effect. Over time, we've gotten better and better at them as a team. The brainstorming session at our latest hackathon a couple of weeks ago was an absolute joy. I felt so happy to be in an environment where people were comfortable sharing their ideas freely - even risky ones - and could give feedback, positive or negative, in a friendly, constructive way. Hackathons aren't easy, especially at first, but if you're not already doing them in your lab, I think they are worth a try.

The end result of our latest hackathon you can see in the video below. One of the lab's major new directions is the computational analysis of naturalistic social interactions. Calls for greater naturalism in social psychology go back as far as foundational figures like Kurt Lewin. However, such studies remain the exception, not the rule. I think the reasons are largely methodological. In order to turn recordings of natural social behaviors such as gestures, facial expressions, or speech into numbers that can be analyzed statistically, researchers must typically watch or listen to these recordings, and manually annotating what is happening in them. This is a slow, tedious process. As such, it cannot scale up to the large datasets necessary to disentangle complex social behaviors. Fortunately, recent advances in machine learning now make it possible to automatically annotate many of these features. We are using these tools to understand people's verbal and nonverbal behavior - and the factors which influence this behavior - in naturalistic interactions. Different members of the lab have been working on different aspects of this problem for the last couple of years, but we have finally reached the point where we're ready to put it all together. This hackathon gave us the opportunity to create a fun demonstration of what our lab's tools can do: