In this study, we focused on the characteristics of the instigator animal to examine factors that influence social instigation-heightened aggression in male mice. Consistent with previous studies, pre-exposure to a novel male rival caused escalation of aggressive behavior, including shorter attack latency and increased frequencies and durations of aggressive behaviors in the subsequent agonistic encounter (Fish et al. 1999; de Almeida and Miczek 2002; Takahashi et al. 2015, 2022). However, when a familiar intruder male was presented as the instigator, we observed only a reduction in attack latency, with no changes in the overall amount of aggressive behaviors. In addition, when the instigator was an animal with already established dominant-subordinate relationship to the test animal, aggressive behavior did not increase. In contrast, when a familiar male without a prior dominant-subordinate relationship was presented, the test mice showed strong increases in aggressive behaviors comparable to those induced by a novel instigator. These findings suggest that an established dominant-subordinate relationship, rather than mere familiarity, between the test animal and the instigator attenuates the pro-aggressive effect of social instigation.

Our original hypothesis considered the possible involvement of frustration in aggression escalation by the social instigation procedure. Frustration induced by the omission of an expected reward has been shown to increase the frequency and duration of operant responses (e.g. lever pressing), alter running speed, and enhance aggressive behavior (Amsel and Roussel 1952; Azrin et al. 1966; de Almeida and Miczek 2002; Hull 1952; Naik et al. 2024; Vasquez et al. 2021). Increased aggression due to frustration has been observed across various species, including rodents, birds, and humans (Caprara 1982; Dollard et al. 1939; Cherek and Pickens 1970; Thompson and Bloom 1966). During the social instigation procedure, the test animals are unable to directly attack the instigator male due to the protective tube, which may induce a state of frustration. Previous studies have shown that the expressing aggressive behavior is rewarding for male mice (Golden et al. 2019). In this study, we used the same intruder male for each test animal throughout the experiments, and we hypothesized that presenting a familiar intruder as an instigator would trigger a stronger motivation to express the consummatory phase of aggression, leading to greater frustration compared to a novel opponent. However, the results from Experiment 1 indicated that social instigation with a familiar intruder did not increase the duration and frequency of aggressive behavior, whereas a novel instigator did. This suggests that the level of social novelty actively modulates social instigation-heightened aggression. On the other hand, we observed similar reductions in the attack latency following social instigation with both the familiar intruder and the novel male, compared to the RI test without social instigation. These results suggest that the social instigation procedure facilitates the initiation of aggressive behavior (i.e. readiness to attack) regardless of the social characteristics of instigator, and that frustration may be involved in this effect.

In the Intruder-Inst test of Experiment 1, the intruder mice were not only familiar subordinates but also the same animals presented in the subsequent agonistic encounter, whereas in the Novel-Inst test, the instigator and the intruder were different animals. To control for this procedural difference and to examine the effect of the dominant-subordinate relationship between the instigator and the test animal on the pro-aggressive effect of social instigation, we next established dominant-subordinate relationships between pairs of test animals and presented each as an instigator. Consistent with the findings from the intruder-Inst test, instigation by subordinate D/S pair mice inhibited the pro-aggressive effect of social instigation in dominant test animals. Previous studies have shown that in the RI test, resident males exhibited less aggressive behavior toward subordinate males than toward novel males (Parmigiani and Brain 1983), suggesting that subordinate males are less potent stimuli for eliciting aggressive behavior. Interestingly, in our study, subordinate test males in the D/S test also displayed aggressive behavior toward the intruder male in the RI test, and social instigation with a novel male facilitated aggressive behavior compared to baseline RI aggression. Notably, in this study we excluded animals that did not show instigation-heightened aggression in the Novel-Inst test, as individual differences exist in the pro-aggressive effect of social instigation (Fish et al. 1999). However, the proportion of excluded subjects was nearly the same between dominant and subordinate groups, suggesting that D/S subordinate males can show instigation-heightened aggression similar to dominant males. Nevertheless, when dominant D/S pair mice were presented as instigators, the pro-aggressive effect of social instigation was inhibited. Previous studies have reported that resident male mice never show aggressive behavior when dominant males are presented as opponents (Wei et al. 2023). In contrast, in this study, even when dominant mice were presented as instigators in the Dom-Inst test, the test animals showed similar levels of aggression toward subordinate intruder males as observed in the RI test. Taken together, these results suggest that presenting either dominant or subordinate mouse as an instigator does not enhance aggression, indicating that a pre-established dominant-subordinate relationship between the test animal and the instigator plays a modulatory role in social instigation-heightened aggression.

Our results further suggest that the mere social familiarity does not influence the pro-aggressive effect of social instigation. When familiar partner mice without an established dominant-subordinate relationship were presented as instigators, test animals exhibited increased aggressive behavior comparable to that observed during social instigation with a novel male. Thus, the presence of an established dominant-subordinate relationship, rather than familiarity alone, appears to be an important modulating factor in social instigation-heightened aggression. In the field of ethology, a phenomenon known as the ‘dear enemy effect’ has been observed (Fisher 1954). In several animal species living in the wild, individuals in a territory exhibit ‘relatively stronger defensive responses to territorial intrusion by strangers rather than neighbors’ (Christensen and Radford 2018). In this context, “neighbors” refer to familiar individuals encountered around the borders of the territory. The findings of the present study could be interpreted through the lens of the dear enemy effect. In Experiment 3, the effect of social instigation by partner mice was similar to that observed with novel male mice, despite the fact that partner mice had previously intruded into the territory of the test animals during the partition test and could be considered “neighbors”. This suggests that the dear enemy effect in social instigation may require the establishment of a dominant-subordinate relationship, which partner mice lack.

Locomotor activity is considered an indicator of behavioral arousal (Valentinuzzi et al. 2000), with overlapping yet distinct neural mechanisms involved (Lu et al. 2020; Vinck et al. 2015), and heightened arousal has been linked to increased aggression (Jaffe et al. 1974; Zillmann et al. 1974). In Experiment 1, during aggressive encounter, locomotion duration was longer in the Novel-Inst test than in the Intruder-Inst test. Similarly, in Experiment 2, subordinate subjects showed longer locomotion duration in the Novel-Inst test compared to both the RI and Dom-Inst tests, while dominant animals showed a similar trend, though it did not reach statistical significance. These results suggest that arousal levels induced by a familiar instigator with a dominant-subordinate relationship were lower than those induced by a novel instigator, potentially contributing to the reduced pro-aggressive effect. By contrast, In Experiment 3, locomotion duration was longer in the Partner-Inst test than in the other tests, suggesting that a familiar instigator without an established dominant-subordinate relationship still induces arousal, possibly even more than a novel instigator male.

Although both Dom-Inst and Sub-Inst tests induced similar changes in aggressive behaviors and locomotion, differences emerged in other non-aggressive behaviors. In the Dom-Inst test, subordinate subjects exhibited increased self-grooming and social contact during the aggressive encounter compared to the RI test, whereas dominant animals did not show these changes in the Sub-Inst test. Self-grooming is an innate behavior exhibited by various animal species and it is observed in both stressful and comfort conditions, and the interpretation of the change in self-grooming is condition dependent (Fentress 1977; Sachs 1988). Increase of self-grooming in the stressful condition is considered as displacement behavior to reduce arousal (Spruijt et al. 1992). Therefore, the increased self-grooming observed in subordinate subject may indicate a stress-related state caused by the pre-exposure to a dominant animal. On the other hand, during the instigator presentation, dominant males showed longer self-grooming when a subordinate opponent was presented as the instigator compared to the novel instigator. In contrast, subordinate males showed a slight, though non-significant, decrease in self-grooming when a dominant instigator was presented. Since self-grooming is also promoted when mice are in the relaxed condition as self-care behavior (Fentress 1977; Sachs 1988), grooming behavior during social instigation may reflect rather reduced stress. Taken together, the presentation of dominant and subordinate instigators elicited some distinct behavioral responses, suggesting that different mechanisms may underlie their suppressive effects on social instigation-heightened aggression. Further studies should further explore these potential differences.

In addition, which social cues from the instigator contribute to the pro-aggressive effect of social instigation remains to be elucidated. Mice use various sensory modalities in the social context, including tactile, acoustic, visual and olfactory information. Among these, olfactory cues play a particularly important role in the expression of aggressive behavior (Clancy et al. 1984; Denenberg et al. 1973). It has been reported that urine from subordinate males contains fewer aggression-promoting pheromones compared to urine from dominant males (Harvey et al. 1989). However, in our study, neither dominant nor subordinate instigators increased aggressive behavior in subsequent agonistic encounters. This suggests that the modulation of the pro-aggressive effect by a dominant-subordinate relationship may involve modalities other than olfactory cues. Once a dominant-subordinate relationship is established, animals often exhibited behavioral adaptations that deescalate aggression, as such submissive posturing by the subordinate (Ginsburg and Allee, 1942). Although we did not observe clear behavioral differences in the test animals during the social instigation procedure between novel instigators and dominant/subordinate instigators, it remains possible that subtle behavioral cues, undetected in the current study, may modulate the pro-aggressive effect of social instigation. The use of anesthetized instigators in future experiments would allow for the assessment of potential behavioral contributions to this phenomenon.

Sex difference in this phenomenon also need to be explored. Previous studies have shown that social instigation enhances aggressive behavior in female hamsters and postpartum rats (Da Veiga et al. 2011; Potegal and TenBrink 1984). The use of the female mouse rival aggression model (Newman et al. 2019), which shows aggression dynamics similar to those observed in males, will allow us to examine whether the social factors that modulate aggression are comparable between male and female mice.