Accurately distinguishing friend from foe is a matter of life and death for ants: mistaking an invader for a nest mate—or the reverse—can lead to fatal chaos.

Scientists have long observed ants deftly navigating through crowds, attacking only individuals that might be hostile. New research confirms how smell receptors on the insects' antennae hold the key to this selective violence: without them, ants are socially blind and will not attack.

“The current consensus was aggression between ants follows a simple rule: if [an ant] smells something different from the home colony, attack,” says Laurence Zwiebel, a co-author on the new study and a biologist at Vanderbilt University. But the new research shows it is not that simple. Ants hold off on attacking if they cannot smell anything—or even if they do not recognize a scent. “Rather a precise signal present on the non-nest mate must be correctly decoded for aggression to occur,” Zwiebel says.

He and his colleagues built on previous studies that identified a mix of odors on ants' exoskeletons, as well as odorant receptors that pick up these scents from others. The new study found that if the receptors were compromised, ants could no longer differentiate nest mates from intruders they would normally fight; instead they became docile. The researchers reported their findings in January in the Journal of Experimental Biology.

After designing a miniature dueling arena—a plate with plastic dividers—the scientists chemically manipulated the odorant receptors of Florida carpenter ants from the same and neighboring colonies, either blocking or overexciting the receptors. When the ants were placed into the arena and the dividers lifted, ants with disrupted receptors were meek even when faced with a stranger. “Our study clearly demonstrates that neither the lack of any odor nor the presence of a confusing odor was sufficient to elicit ant aggression,” Zwiebel says.

Ants have more than 400 odorant receptors, and Zwiebel says a next step is to determine which of them must function correctly to decode an enemy's smell. (For this study, researchers dampened or excited all of them.)

Volker Nehring, a biologist at the University of Freiburg in Germany, who was not involved with the study, says this research could also pave the way for other studies of how animals recognize one another. “We hardly understand how the ants know their own nest odor in the first place,” he says, “and temporarily interfering with the receptors might be a good way to address that.”