Sex differences in reward-based operant conditioning performance and neurotransmitter changes following chronic sleep restriction stress in rats

Abstract

Background

Sleep deprivation significantly impairs cognitive function, which disrupts daily life. However, sex differences in these impairments are not well understood, as most preclinical studies primarily use male animals, neglecting potential differences between sexes. This study aims to investigate sex-specific differences in cognitive function under sleep deprivation using reward-based operant conditioning tasks.

Results

Sprague-Dawley rats were pre-trained on a lever-press task and subsequently divided into control and chronic sleep restriction (CSR) groups. The CSR group underwent 14 days of sleep restriction. After CSR modeling, rats were assessed using the open field test, retraining on the lever-pressing task, signal discrimination task, and extinction task to evaluate motor abilities, memory formation, learning, and cognitive flexibility. CSR significantly impaired task performance in both sexes, with rats requiring more time and exhibiting lower accuracy. In the signal discrimination task, male rats showed longer feeding latency and lower accuracy compared to females. CSR also specifically increased the frequency of operant responses in male rats. In the extinction task, CSR enhanced exploration time and frequency in both sexes, with females exhibiting significantly higher exploration frequencies than males. Biochemically, CSR induced sex-specific alterations, including elevated serum MDA and MAO levels in males and increased serotonin, dopamine, and epinephrine in both sexes. Although activation was observed in metabolites of the tryptophan-kynurenine pathway, sex differences were evident in the kynurenic acid metabolism levels in the prefrontal cortex.

Conclusions

CSR impairs cognitive function in both male and female rats, with significant sex differences observed. Male CSR rats exhibited impaired signal discrimination, while CSR impaired extinction learning in female rats. These impairments are accompanied by CSR-induced oxidative stress, neurotransmitter dysregulation, and disturbances in the tryptophan metabolic pathway. These findings underscore the importance of considering sex differences in understanding the effects of sleep deprivation on cognitive function and developing targeted intervention strategies.

Introduction

In contemporary society, characterized by a fast-paced lifestyle, ensuring adequate and high-quality sleep is essential for managing the demanding workloads of individuals. Social studies have shown that modern people experience shorter and poorer-quality sleep than in the past, especially after the outbreak of COVID-19 [1, 2]. The Centers for Disease Control and Prevention (CDC) in Korea has emphasized that individuals with poor sleep quality are more likely to exhibit subjective cognitive decline [3]. A lack of sleep or poor sleep quality can have detrimental effects on the brain, including decreased learning ability, memory impairment, and disruptions to nearly all cognitive processes, even increasing the risk of dementia [4,5,6]. Sleep deprivation-related issues, especially cognitive deficits, have emerged as a critical global concern, posing significant threats to human well-being [7, 8].

Current clinical studies indicate that there are sex differences in vulnerability to the effects of sleep deprivation, including variations in blood pressure, anxiety, and other related factors [9, 10]. However, numerous negative effects, especially regarding cognitive functions, resulting from sleep deprivation are challenging to demonstrate in a sex-dependent context [11]. Recent meta-analyses indicate that women display greater resilience to cognitive dysfunction induced by sleep restriction compared to men [12]. Nonetheless, a subsequent study has shown that sleep deprivation impairs the objective working memory performance of females while males remain unaffected [13]. In an animal experiment, although cognitive impairment was observed in both sexes, only the recognition memory of female rats was affected [14]. These sex differences have largely been neglected in most preclinical research, which may result in preclinical foundational research and drug development being less effective in addressing the impact of sleep deprivation in females [15]. Given these sex differences, studying the behavioral and mechanistic differences between men and women under sleep deprivation is crucial for understanding and targeting strategies to mitigate its negative effects.

In this study, to further elucidate the impact of sleep deprivation on cognitive behavior in male and female rats, we employed chronic sleep restriction (CSR) to induce cognitive impairment and compared the behavioral performance differences between male and female Sprague Dawley rats in three reward-based operant conditioning schedules: lever press operant conditioning, signal discrimination task and extinction test, to comprehensively evaluate the entire process of learning and memory, including acquisition, retention, and extinction. The prefrontal cortex (PFC) and hippocampus are two critical brain regions that play significant roles in cognitive processes such as learning, memory, and decision-making [16, 17]. Therefore, this study focused on the PFC and hippocampus to investigate oxidative stress-induced neuronal damage, dysregulation of the tryptophan-kynurenine pathway, and imbalances in monoamine neurotransmission, all of which have been implicated in cognitive dysfunction. By understanding the differential impacts of sleep deprivation on male and female cognitive functions and the associated neurochemical changes, we hope to contribute to the development of targeted interventions for sleep-related cognitive impairments in both men and women.

Methods

Animals and groups

Forty-two Sprague-Dawley (SD) rats aged at eight weeks on arrival, including twenty-one male and female rats respectively, were purchased from Charles River Laboratories, Beijing, China with Qualified No. SCXK 2021-0006. The animals were housed in stainless steel cages (485 × 350 × 210 mm L × W × H) at a constant temperature (22 ± 2℃) and humidity (55%±10%) environment with a 12:12 h light/dark cycle (lights on at 8:30 a.m.). Water was provided via standard water bottles in the home cage and standard feed were freely supplied during a 7-day adaption. The protocol described in the present study was approved by the Committee for the Care and Use of Laboratory Animals of the Institute of Medicinal Plant Development, Beijing, China.

After pre-training, animals that did not meet the criteria were excluded but they are kept together with experimental animals. Based on the learning situation in the last session, the animals were grouped equally according to completion time so that the average completion time became similar among groups. The final groups were the female control group (♀Control, n = 9); the female CSR group (♀CSR, n = 9), the male control group (♂Control, n = 9), the male CSR group (♂CSR, n = 8). After group assignment, animals were housed separately.

Apparatus

Modeling of CSR was conducted in the automatic rotating sleep disruption device, and the animal behavior test was conducted in operant conditioning chambers, both of which were all jointly developed by the Institute of Medicinal Plant Development of Peking Union Medical College and China Astronaut Center (Chinese patent No.201210356645.X and No. 201210505373.5 respectively). The automated sleep restriction apparatus consisted of stainless steel wheels, with each drum capable of accommodating 1 to 4 rats. The sleep disruption apparatus was cylindrical, with a width of 30 cm and a diameter of 47.5 cm. Water was provided via water bottles within this apparatus. The bottom of the wheel was designed as a platform allowing animals to freely move within the wheel while having unrestricted access to food and water.

The operant conditioning chamber measured 300 × 300 × 550 mm (length × width × height), including a dipper magazine, two retractable levers, and LED signal lights of different colors located 5 cm above the lever [18, 19]. The sidewalls were constructed from PVC panels, while the interior walls and doors were made of aluminum. The chamber floor consisted of stainless steel grids with a spacing of 10 mm. The video analysis system automatically recorded the behavior of animals, and the infrared photobeam sensors assisted in the recognition of the animals’ exploration of the food magazine. A room light, camera, and ventilation fan were installed on the top, and the whole unit is placed in a soundproof box.

Chronic sleep restriction (CSR)

Chronic sleep restriction (CSR) modeling was performed according to published protocols with minor modifications [20, 21]. Before the formal modeling, animals were placed in the experimental environment for 2–6 h (gradually increased) per day to allow them to acclimate and minimize stress when introduced to the CSR procedure for three consecutive days. The stainless-steel wheel rolled slowly at a speed of 1 r/min, with a 3-minute rest for each rotation, allowing animals to be awakened 15 times per hour. The modeling was performed for 14 consecutive days and considering animals’ safety, the sleep restriction procedure was stopped for 2 h a day at a set time (1:00 p.m. ∼ 3:00 p.m.). The control group animals were placed in the sleep restriction device, which remained stationary. During the behavioral test period, the animals in the CSR group were placed back into the sleep restriction device after the daily experiment, which continued until the end of the behavioral test (Fig. 1A).

Fig. 1

Training performance of male and female rats in pre-training. (A)The schematic diagram and experimental protocol. (B) The schematic diagrams of the magazine training section. (C) The schematic diagrams of the lever press training section. (D) Total time spent exploring the food magazine. (E)The correct exploration time. (F) The rate of animals that reached the training criterion in the lever-press training test. (G) The completion time for each group. n = 20 ∼ 21 per group in Fig. 1D ∼ F, n = 8 ∼ 9 per group in Fig. 1G. ⁺p < 0.05, ⁺⁺⁺p < 0.001 significant difference in female groups and ^{^}p < 0.05, ^{^^}p < 0.01, and ^{^^^}p < 0.001 significant difference in male groups

Behavioral test

Diet restriction

Food restriction was conducted following previously established protocols [19, 22], to ensure they had sufficient intrinsic motivation. After habituation, food and water restrictions commenced. Each rat was provided with approximately 17 g of food per day, with daily body weight monitoring to adjust the food amount as needed. The feeding regimen aimed to gradually reduce and maintain body weight at 85–90% of the normal ad libitum feeding weight until the end of the pre-training phase. Regarding water restriction, rats underwent a three-day adaptation period with a two-bottle choice paradigm (sucrose solution and pure water) before experiment. This was implemented to acclimate them to the reward substance and minimize stress responses during testing. The bottle positions were switched daily to prevent location-based preferences for sucrose. Food restriction resumed three days before the end of CSR modeling and continued until the completion of behavioral testing.

Magazine training

Animals underwent 3 days of magazine training, with each session lasting for 20 min. Throughout the Reward-Based Operant Conditioning task, a 20% (w/v) sucrose solution was used as the reward. Each reward consisted of a single drop of sucrose solution, approximately 50 µL in volume. This consistent use of a palatable reward aimed to ensure motivation and engagement of the animals during the behavioral tasks. In this stage, the blue cue light is turned on periodically and lasts for 10 s with 1 drop of liquid reward delivered. An inter-trial-interval (ITI) of 30 ± 10 s was followed after the reward delivery. Magazine training enabled animals to establish a connection between the conditioned stimulus (CS, light cues) and the unconditioned stimulus (US, sucrose water), thereby forming a CS-US relationship.

Lever press training

The protocol for lever-press training had been described before [18, 19]. A fixed ratio = 1 (FR = 1) schedule of reinforcement was used, where each lever press resulted in the delivery of one drop of sucrose solution. In this stage, the lever extended into the operant chamber and only the lever was pressed to initiate the blue cue light and one reward was delivered. The entire experiment lasted for 30 min, or until the animals obtained 45 rewards, whichever came first. Once the animals were able to obtain 45 rewards within 30 min for 3 consecutive days, the training progressed to the next stage. The retraining on the lever-pressing task conducted after CSR is identical to the lever-press training, which is used to test the memory of the learned operation.

Open field test

The open field test (OFT) was performed using the black cylindrical experimental box (Diameter 900 mm × Height 500 mm), and the center area was defined as the inner 50% of the total radius. The animal was gently placed in the center of the detection box, allowing it to freely adapt to the environment for 3 min before starting the experiment. The computer automatically recorded the animal’s average movement speed and movement distance within 10 min to assess the effects of CSR on motor ability.

Signal discrimination task

To further explore behavioral differences between male and female animals following CSR, we conducted a more challenging signal discrimination task (SD task) [23]. A red cue light (S-) was added and these two-color cue lights alternately light up for 120 s each time. The blue cue light (S+) that appeared during the magazine training and lever press stage was considered a correct signal and the red was a false signal (S-). In other words, when the blue light (S+) was on, the animal could get a reward through lever-press operation as before; while when the red light (S-) was on, no reward would be triggered for any action. The experiment lasted for 6 days, with each session lasting 28 min.

Extinction

Extinction was conducted at the end of all experiments for 3 continuous days. The process was the same as the SD task, except that no sucrose water would be delivered regardless of any operation during the whole task.

Biological sample collection

All rats were sacrificed the day after the last behavioral tests to collect the biological samples. Rats were anesthetized with urethane (intraperitoneal, 1 g/kg). Blood samples were collected through the abdominal aorta and placed at 4 °C for 12 h to obtain the serum. After the brain was removed from the skull and washed with ice-cold saline, the prefrontal cortex (PFC) and hippocampus were dissected as whole regions without further subdivision. The entire dissection process was performed on ice to preserve tissue integrity. All biological samples were stored at -80 °C until analysis.

Biochemical assays

The levels of superoxide dismutase (SOD), monoamine oxidase (MAO), and malondialdehyde (MAD) in serum were measured by commercial kits (Jiancheng Biology, Nanjing, China) according to the manufacturer’s instructions.

Neurotransmitter detection

The neurotransmitter analysis method was performed as previously described [24]. The prefrontal cortex and hippocampus samples were weighted and homogenized with ultrapure water, using trifluoroacetic acid to precipitate protein. After being centrifuged, the supernatant was taken for liquid chromatography-tandem mass spectrometry (LC-MS/MS) analysis. Neurotransmitters were detected by prominence ultrafast liquid chromatography (UFLC) (Shimadzu, Kyoto, Japan) coupled with a QTRAP 5500 mass spectrometer (AB SCIEX, Framingham, MA, USA). The metabolites were separated using the Restek Ultra Aqueous C18 column (100 mm × 2.1 mm, 3 μm, Bellefonte, PA, USA).

Data analysis

In the magazine training phase, a two-way repeated measures ANOVA was performed to assess the effects of sex and days on the measured variables. For significant main effects or interactions, post hoc comparisons were conducted using Bonferroni correction.

For the grouping data on the last day of the pre-training phase and the open field test, a two-way ANOVA was conducted, with sex and treatment as the main factors.

In the lever-pressing task retraining experiment following CSR, a two-way ANOVA was used for analysis, except for the comparison of completion times before and after CSR, where a paired t-test was applied.

For the signal discrimination task and extinction task, a three-way repeated measures ANOVA were performed, with sex, treatment, and days as the main factors. Post hoc comparisons were conducted using Bonferroni correction for significant main effects or interactions.

In the mechanism detection experiments, a two-way ANOVA was used to analyze the main effects of sex and treatment.

To ensure data accuracy and reliability, outlier detection was performed using SPSS boxplot analysis. Statistical analyses were conducted using SPSS 26.0, GraphPad Prism 8.0, and ImageJ software. A p-value < 0.05 was considered statistically significant. All data are presented as mean ± S.E.M.

Results

Female and male rats exhibited similar learning performance during the pre-training phase

Magazine training (Fig. 1B) enabled the animals to establish a CS-US relation, associating the signal light with sucrose water delivery. We refer to the time spent by animals exploring the food magazing during the signal light phase as the correct exploration time. A two-way repeated measures ANOVA was conducted to examine the effects of sex (male vs. female) and day (Day 1–3) on exploratory behavior in the magazine training. There was no significant main effect of sex (for total time, F_(1,38) = 0.28, p = 0.60; for correct time, F_(1,38) = 0.05, p = 0.83) or day × sex interaction (for total time, F_(2,76) = 1.41, p = 0.24; for correct time, F_(2,76) = 0.34, p = 0.72). However, a significant main effect of day was observed, with both total exploration time and correct exploration time increasing as training progressed (F_(2,76) = 23.80, p < 0.001; F_(2,76) = 7.69, p < 0.001). Bonferroni post hoc tests further revealed that, compared to Day 1, exploration time in the food magazine was significantly higher on Days 2 and 3 for both male and female rats (p < 0.01), and the correct exploration time was also significantly prolonged (p < 0.05). These results indicate that both male and female animals displayed similar motivation for rewards and established an association between the signal and the reward location.

In the lever press training (Fig. 1C), over 80% of both female and male rats achieved the standard (Fig. 1F). Based on the completion time on the final day, there was no difference between the male and female groups. Animals that did not meet the standard were excluded from further analysis but continued to be housed with the others. The remaining animals were divided into control and CSR groups based on the completion time of training on the last day, ensuring each group had comparable levels of reward-directed instrumental learning. A two-way ANOVA was conducted to confirm that there were no significant differences and the analysis revealed no significant main effects of sex (F_(1,31) = 0.61, p = 0.44) or treatment (F_(1,31) = 0.03, p = 0.87), nor a significant sex × treatment interaction (F_(1,31) = 0.10, p = 0.75). Each group consisted of at least 8 animals.

CSR modeling does not affect spontaneous locomotor activity in rats

After a 14-day chronic sleep restriction (CSR) stress, the open field test (OFT) was first conducted to determine the animal’s autonomous activity. The CSR did not induce fatigue or alter motor function, as evidenced by the same level of movement distance, average speed, and the percentage of movement distance in the central area shown in Fig. 2. A two-way ANOVA revealed that for total movement distance, there were no significant main effects of sex (F_(1,31) = 0.00, p = 1.00) or treatment (F_(1,31) = 2.08, p = 0.16), and no significant sex × treatment interaction (F_(1,31) = 0.32, p = 0.57). Similarly, for average speed, there were no significant effects of sex (F_(1,31) = 0.00, p = 1.00) or treatment (F_(1,31) = 2.09, p = 0.16), and no significant interaction (F_(1,31) = 0.33, p = 0.57). In addition, the percentage of movement distance in the central area also showed no significant main effects of sex (F_(1,31) = 0.00, p = 0.99) or treatment (F_(1,31) = 0.02, p = 0.89), with no significant sex × treatment interaction (F_(1,31) = 0.24, p = 0.63). This also demonstrates that in the subsequent operant conditioning training, the animals’ level of autonomous activity is consistent.

Fig. 2

Spontaneous locomotion after 14 days of CSR modeling. (A) Schematic diagram (left) and trajectory (right) of OFT. (B) Movement distance. (C) Average speed. (D) the ratio of movement distance in the central zone. (n = 8 ∼ 9)

Differences in performance emerged between female and male rats in the retraining on the lever-pressing task after CSR

To investigate whether CSR affects memory consolidation of the learned association between the cue light, lever-press, and reward, we performed retraining on the lever-pressing task after modeling in rats (Fig. 3A). Figure 3B visually illustrates the animals’ movement trajectories during the experiment. For an animal that has established operant conditioning memory through training, its trajectory is more focused on the food magazine to get the reward most efficiently. We did not find differences in the average reward latency, including main effects of sex (F_(1,30) = 0.34, p = 0.86) or treatment (F_(1,30) = 0.39, p = 0.54), and no significant sex × treatment interaction (F_(1,30) = 1.71, p = 0.20), which reflects the animals’ responsiveness to the signal and reward, indicating that the animals’ CS-US relation was not impaired. The LP/NP ratio was calculated as the total number of LP divided by the total number of NP during the session. The analysis revealed no significant differences in the LP/NP ratio (Fig. 3D) (for main effect of sex, F_(1,31) = 0.42, p = 0.52; for main effect of treatment, F_(1,31) = 1.14, p = 0.29; for interaction (F_(1,31) = 3.38, p = 0.08), indicating that the association between the operant response and the reward was not impaired in the animals. Although there was no significant difference in completion time among groups (Fig. 3E) (for main effect of sex, F_(1,30) = 3.06, p = 0.09; for main effect of treatment, F_(1,30) = 1.72, p = 0.20; for interaction (F_(1,30) = 1.73, p = 0.20), a repeated measures analysis revealed that, compared to the 14-day baseline or pre-modeling phase, the completion time significantly increased in all other groups (Fig. 3F) (p < 0.05).

Fig. 3

Retraining the lever-pressing task after 14 days of CSR modeling. (A) Schematic diagram. (B) Representative trajectory heatmaps during retraining on the lever-pressing task after modeling. (C) Average reward latency. (D) The ratio of LP/NP. (E) Completion time of retraining on the lever-pressing task. (F) The comparison of completion times before and after modeling was analyzed using repeated measures. (n = 8 ∼ 9)⁺p < 0.05 significant difference in female groups and ^{^}p < 0.05 significant difference in male groups

CSR-induced sex differences in behavioral performance in the signal discrimination task

During the signal discrimination task (SD task), the difficulty was increased by adding additional signals to be distinguished (Fig. 4A). The exploration time (Fig. 4B) increased significantly over the 6 days across all groups, with a significant main effect of days was observed (F_{(1, 150)} = 34.13, p < 0.001). However, there were no significant main effects of sex (F_{(1, 30)} = 0.04, p = 0.85) or treatment (F_{(1, 30)} = 0.54, p = 0.47), nor a significant sex × treatment interaction (F_{(1, 30)} = 0.20, p = 0.66). These results suggest that the animals exhibited comparable levels of engagement with the reward-associated cue in each session Fig. 4C shows that during the 6-day SD experiment, the latency to the first lever press (F_{(5, 150)} = 6.43, p < 0.001), which reflects the animal’s familiarity with the lever-pressing task, gradually decreased for all animals. The between-subjects analysis revealed no significant main effect of sex (F_{(1, 30)} = 5.54, p < 0.05), and no significant effects of treatment (F_{(1, 30)} = 0.06, p = 0.82) or sex × treatment interaction (F_{(1, 30)} = 0.012, p = 0.74). Further analysis revealed that the latency to operant behavior in females on Day 1 was significantly lower than that of males (p < 0.05). The total number of nose pokes (NP) changed significantly throughout training (F_{(5, 150)} = 3.41, p < 0.05). Within-subjects contrast tests showed a significant quadratic relationship over time (F_{(1, 30)} = 10.33, p < 0.01), indicating that the number of nose pokes peaked during the middle of the experiment and then gradually declined (Fig. 4D). This suggests that the animals exhibited a gradual adaptation behavior pattern during the early stages of the task. We defined the animals’ nose-poke (NP) responses during the blue cue light (S+) phase as the correct NP response. The number of correct NP (Fig. 4E) responses increased significantly (F_{(5, 150)} = 34.22, p < 0.001). A significant main effect of treatment was observed in the between-subjects analysis (F_{(1, 30)} = 5.10, p < 0.05), and further post-hoc tests revealed that the CSR group had significantly fewer correct NP responses than the control group on the first, second, and fifth days (p < 0.05).

The daily lever press (LP) counts, which reflect the animal’s learning efficiency, are shown in Fig. 4F. The results revealed a significant three-way interaction of time × sex × treatment (F_{(5, 150)} = 3.43, p < 0.05). Further analysis showed that in the male group, the model group had significantly higher total LP counts than the control group on the fourth, fifth, and sixth days (p < 0.05). The number of correct LP directly reflects the animals’ learning ability during the operant task. The results in Fig. 4G showed a significant main effect of days (F_{(5, 150)} = 82.73, p < 0.001) and a significant three-way interaction of time × sex × treatment (F_{(5, 150)} = 4.13, p < 0.01). Additionally, significant main effects of sex (F_{(1, 30)} = 15.13, p < 0.001) and treatment (F_{(1, 30)} = 16.22, p < 0.001) were observed in the between-subjects analysis. These findings suggest that learning performance varied significantly over time, and this variation was influenced by both sex and treatment. Further analysis of sex differences revealed that starting from the third day, the overall performance of the female group was significantly better than that of the male group (p < 0.05). Additionally, the LP accuracy in the female control group was significantly higher than that in the male control group on the first and third days (p < 0.05), suggesting that female animals may have stronger adaptability. Analysis of treatment differences showed that significant differences between the Control group and the CSR group emerged starting from the second day (p < 0.05). Further within-group comparisons revealed that the correct lever press rate in the female CSR group was significantly lower than that in the female model group on the first and second days (p < 0.05), while the correct lever press rate in the male CSR group was significantly lower than that in the male model group from the fourth to the sixth day (p < 0.05).

Fig. 4

Performance discrepancy between male and female animals in the signal discrimination task. (A) Schematic diagram of visual signal discrimination task. (B) The total exploration time animals spent at the food magazine. (C) The latency to the first lever press. (D) The total NP times (E) The correct NP times. (F)The total lever press times (LP) times. (G) the percentage of correct LP responses, including comparison of all groups (left), comparison between female groups (middle) and male groups (right). (n = 8 ∼ 9) *p < 0.05 indicates a significant comparison between main effects or control groups. ⁺p < 0.05 significant difference in female groups and ^{^}p < 0.05, ^{^^}p < 0.01, ^{^^^}p < 0.001 significant difference in male groups

The female group was more affected by CSR in the extinction experiment

During the extinction process (Fig. 5A), the exploration time of the food-cue light significantly decreased (Fig. 5B) (F_{(2, 60)} = 21.32, p < 0.001). A significant time × treatment interaction was also observed (F_{(2, 60)} = 3.52, p < 0.05), suggesting that the extinction trends differed between treatment groups. Between-subjects analysis revealed a significant main effect of treatment (F_{(1, 30)} = 11.16, p < 0.01), indicating that the overall extinction performance of the CSR group was significantly different from that of the Control group. Further simple effects analysis showed that the CSR group had significantly longer exploration times of the food-cue light on the first and second days compared to the Control group (p < 0.05). Additionally, within the female group, exploration times on the first and second days were significantly higher in the CSR group compared to the control group (p < 0.01), suggesting that CSR treatment may have inhibited the extinction process. In the latency to the first lever press (Fig. 5C), significant main effects of time and treatment were observed (F_{(2, 60)} = 4.95, p < 0.05; F_{(1, 30)} = 5.59, p < 0.05), indicating that CSR treatment shortened the latency to the first lever press.

Analysis of nose-poke (NP) exploration counts revealed significant main effects of time, sex, and treatment (Fig. 5D) (F_{(2, 60)} = 70.56, p < 0.001; F_{(1, 30)} = 7.75, p < 0.01; F_{(1, 30)} = 15.04, p < 0.001). CSR treatment led to significantly higher NP counts on the first and second days (p < 0.05). Female rats also exhibited significantly higher NP counts on the first and third days compared to male rats (p < 0.05). Further analysis revealed that, within the female group, the CSR group had significantly more NP counts on day 1 compared to the control group (p < 0.01). Lever-press (LP) results showed that the number of active lever presses significantly decreased throughout training (Fig. 5E) (F_{(2, 60)} = 53.91, p < 0.001), with no significant main effects of sex or treatment (F_{(1, 30)} = 0.21, p = 0.65; F_{(1, 30)} = 2.95, p = 0.10).

Fig. 5

Performance discrepancy between male and female animals in extinction task. (A) Schematic diagram of extinction task. (B) The time spent exploring the food magazine. (C) The latency to the first lever press. (D) The nose-poke (NP) times and (E) the lever press (LP) times. (n = 8 ∼ 9) ^*p < 0.05, ^**p < 0.01, ^***p < 0.001 compared with the data between main effect; ⁺p < 0.05 and ⁺⁺p < 0.01 significant difference in female groups

CSR causes an imbalance of oxidative factors in the serum

Sleep deprivation can lead to oxidative stress, which is closely related to an increased risk of cognitive impairment [25]. Accordingly, we tested the levels of oxidative factors in serum. The results showed (Fig. 6A-C) that CSR treatment significantly increased the oxidative markers SOD, MDA and MAO in serum (for SOD, F_(1,30) = 4.74, p < 0.05; for MDA, F_(1,29) = 10.67, p < 0.01; for MAO, F_(1,28) = 16.53, p < 0.001). Further analysis revealed that, within the male group, CSR treatment caused significant adverse changes in SOD, MDA, and MAO (p < 0.05).

Fig. 6

The effect of CSR on the metabolites in serum and the neurotransmitters in the brain. (A-C) The level of SOD, MDA, and MAO in serum. (D-F) The effect of CSR on the levels of 5-HT, DA, and Epi in PFC, and (G-I) The effect of CSR on the neurotransmitter levels in hippocampus. (n = 8–9). ⁺p < 0.05, ⁺⁺p < 0.01significant difference in female groups and ^{^}p < 0.05, ^{^^}p < 0.01, ^{^^^}p < 0.001 significant difference in male groups

CSR-induced changes in neurotransmitters in the brains of male and female rats

The central nervous system is highly susceptible to the influence of sleep deprivation, leading to complex alterations in neurotransmitter balance [26]. Therefore, we separately investigated the changes in neurotransmitters within the prefrontal cortex (PFC) and hippocampus of rats. The results showed that CSR treatment significantly affected the levels of neurotransmitters in the prefrontal cortex of animals (Fig. 6D-F), including serotonin (5-HT), dopamine (DA), and epinephrine (Epi) (for 5-HT, F_(1,29) = 11.57, p < 0.01; for DA, F_(1,30) = 25.06, p < 0.001; for E, F_(1,31) = 8.77, p < 0.01). Further analysis revealed that within the female group, CSR treatment resulted in significantly higher levels of 5-HT and DA compared to the control group (p < 0.01, p < 0.05). In the male group, CSR treatment significantly increased all three neurotransmitters compared to the control group (p < 0.001, p < 0.05, and p < 0.01, respectively).

In the hippocampus (Fig. 6G-I), the levels of 5-HT and DA were significantly affected by the CSR treatment (for 5-HT, F_(1,29) = 13.51, P < 0.001; for DA, F_(1,31) = 9.16, p < 0.01). Between groups, 5-HT level in the male CSR group was significantly higher than in the male control group, while the DA level in the female CSR group was significantly higher than in female control groups. Although the CSR main effect on Epi was not significant, within the male groups, the CSR group had significantly higher levels of Epi compared to the control group (p < 0.05).

CSR-induced changes in tryptophan metabolic pathway in the brains of male and female rats

Increasing evidence suggests that dysregulation of tryptophan metabolism in the brain is closely related to cognitive impairment and neurodegenerative diseases [27, 28], which may be associated with the cognitive dysfunction induced by CSR in rats. Figure 7A-D illustrates the alterations in the tryptophan metabolic pathway in the PFC. The main effect of CSR treatment was significant for TRP, KYN, and KYNA (F_(1,29) = 10.54, p < 0.001; F_(1,29) = 4.59, p < 0.05; F_(1,31) = 17.79, p < 0.01, respectively), while no significant changes were observed in 3-HK. Further comparisons between groups revealed that in the female group, CSR treatment significantly increased TRP and KYN levels compared to the control group (p < 0.05, p < 0.05); in the male group, CSR treatment significantly increased TRP and KYNA levels compared to the control group (p < 0.05, p < 0.001).

However, in the hippocampus (Fig. 7E-H), there were no significant differences in TRP levels. Its metabolites, including KYN, KYNA, and 3-HK, showed significant main effects of CSR treatment (F_(1,30) = 6.37, p < 0.05; F_(1,30) = 18.54, p < 0.001; F_(1,30) = 17.21, p < 0.001, respectively). Further comparisons revealed that in the female CSR group, KYN levels were significantly higher than in the female control group (p < 0.01); while CSR treatment resulted in significantly higher KYNA and 3-HK levels compared to the control group (in females, p < 0.01; in males, p < 0.05).

Fig. 7

The effect of CSR on the tryptophan metabolism in the brain. (A-D) The effect of CSR on the level of TRP, KYN, 3-HK, and KYNA in PFC. (E-H) The effect of CSR on the level of TRP, KYN, 3-HK, and KYNA in hippocampus. (n = 8–9). ⁺p < 0.05, ⁺⁺p < 0.01significant difference in female groups and ^{^}p < 0.05, ^{^^}p < 0.01, ^{^^^}p < 0.001 significant difference in male groups

Discussion

Since the pioneering study by Patrick et al. in 1896 on the effects of sleep deprivation on humans, the critical role of sleep in human health and cognitive function has gained significant attention from the scientific community [29]. Chronic sleep restriction (CSR) serves as a modeling method capable of inducing sustained sleep deprivation, offering a more accurate simulation of the prolonged sleep disturbances commonly reported in clinical settings, with broad applications in preclinical animal research and pharmacodynamic studies [30,31,32]. In this study, we adopted optimized mild modeling parameters, waking the animals at a slow speed of one rotation per minute, and achieving gentle awakening every two minutes [33]. The results of the open field test and food magazine exploration time demonstrated that this method effectively eliminates the impact of forced wheel movement on the animals’ spontaneous motor abilities and exploratory behavior.

In reward-based operant conditioning tasks, animals develop a cognitive understanding of behaviors that lead to positive or negative outcomes, allowing them to adjust their actions to achieve desired results and the advanced cognitive ability enables them to modify their behavior in changing environments, ensuring survival advantage [34, 35]. In recent years, this task has been used to study gender differences in cognitive impairment caused by pathophysiological factors such as Huntington’s disease models, alcohol consumption, high-fat diets, and so on [36,37,38]. A systematic review has reported that sleep deprivation leads to cognitive impairments in animals during reward-based operant conditioning tasks, but the specific effects on male and female rats remain unclear [39]. In our study, both male and female animals underwent standardized training before the CSR procedure and exhibited the same level of proficiency in establishing the CS-US association, indicating that both sexes are equally suited for this type of training.

After chronic sleep deprivation, retraining on the lever-pressing task can assess the animals’ memory of the learned behavior. Our results showed that although CSR did not affect the CS-US association in animals, it significantly prolonged the time required to complete the task. We subsequently conducted a more challenging signal discrimination task (SD task). The added signal identification condition makes the SD task more complicated than the lever press test while mobilizing the discrimination, decision-making as well as learning abilities of the animals, which amplify the behavioral differences resulting from the modeling, making the detection more sensitive [40]. In this study, the indicators directly reflecting the animals’ learning ability, such as the correct nose-poke (NP) times and correct lever-press (LP) rate, both showed that CSR led to a decrease in the learning ability of rats, which is consistent with the findings of other behavioral studies [35, 36]. Further analysis revealed that not only did female control rats exhibit a significantly higher correct lever-press (LP) rate compared to male control rats, but female rats also outperformed male rats overall. These results suggest that female rats demonstrated faster learning speed in the SD task, while also indicating that the impact of CSR on female rats may be less pronounced compared to its effects on male rats. Additionally, CSR resulted in abnormally elevated lever-press behavior in male rats. Combined with the decreased correct lever-press rate, this indicates impaired discriminative ability, suggesting that male rats exposed to CSR engaged in indiscriminate lever-pressing behavior, potentially due to an inability to accurately interpret the task’s cue signals. Studies have shown that sleep-deprived male rats are more likely to exhibit deficits in behavioral regulation, with shorter latency of choice and increased impulsivity under reduced-risk conditions [41, 42]. While these findings align with the impaired performance observed in male CSR animals, it is important to note that our signal discrimination task primarily assesses cognitive behavioral inhibition rather than direct risk-taking tendencies. Similarly, clinical studies have reported that sleep deprivation increases risky decision-making in men while reducing risk-taking behavior in healthy women [43, 44]. Despite task-specific differences, these studies collectively suggest that sleep deprivation modulates decision-making processes in males, potentially leading to deficits in adaptive response selection.

Extinction involves new learning about the association between new outcomes and behaviors after observing the disappearance of the reward and is often used to measure cognitive flexibility [45, 46]. Learning to cease existing behaviors is as crucial as acquiring new ones, which allows behavior to continuously adapt to dynamic environments, thereby mitigating or redirecting stress [47, 48]. Our results show that as the experiment progressed, both male and female rats exhibited significant declines in their operant and exploratory behaviors. However, CSR hindered the extinction process, with rats showing a more persistent exploratory behavior towards the food magazine and a shorter latency to operate. Notably, female rats had significantly higher nose poke counts compared to male rats. Our results revealed that in reward-based extinction tasks, female rats showed a lower ability to discriminate the signals of reward disappearance compared to males, manifested as significantly longer exploration time and frequency of exploring the food magazine. In contrast, other studies have shown that female SD rats exhibit faster extinction in punishment-based conditioning, suggesting that female SD rats are more sensitive to punishment or frustration [49].

Reports focusing on sex differences in cognitive impairment caused by sleep deprivation are rare. According to existing reports, female animals exhibit more pronounced cognitive impairments in tasks involving punishment environments compared to males, such as the Morris water maze task [50], passive avoidance paradigm [14], and plus-maze discriminative avoidance task [51], while these differences in performance potentially being attributed to stress induced by the task environment [52, 53]. Clinical research has also shown that females are more sensitive to stress, and the incidence of stress-related mental disorders is higher in women [54]. Reward-based operant conditioning method allows rats to complete tasks actively by stimulating the animals’ intrinsic motivation, thus avoiding the interference of intense environmental stress [22, 55]. For the first time, our study explores the sex differences in cognitive function decline caused by sleep deprivation from a perspective that more closely reflects human daily life environments. We hope to provide new insights and methodological strategies for studying sex differences.

Our study indicates that CSR elevates the risk of oxidative stress damage in rats, which aligns with previous research [56]. However, under our experimental conditions, these changes were less pronounced in female rats compared to males. We speculate that this might be due to the protective effects of sex hormones, as estrogen in females has antioxidant properties that can partially mitigate oxidative stress [57]. Studies have shown that estrogen can upregulate the activity of antioxidant enzymes and inhibit or eliminate free radicals [58, 59]. Similarly, this protective effect is lost in ovariectomized female rats, further supporting our findings [60], though the underlying mechanisms require further investigation.

The central nervous system is highly susceptible to the influence of sleep deprivation, leading to complex alterations in neurotransmitter balance [26]. Studies have shown that the prefrontal cortex (PFC) is particularly vulnerable to sleep deprivation, exhibiting widespread reductions in activity [43], and neuroimaging research indicates that the PFC is pivotal for executing many higher-order and complex cognitive processes during periods of sleep deprivation [61]. The hippocampus is the most vulnerable brain region under stress or other pathological conditions [62], and both long-term and short-term sleep deprivation can cause hippocampal neuronal damage [63, 64]. Therefore, this study focused on the prefrontal cortex (PFC) and hippocampus to investigate changes in tryptophan metabolism and neurotransmitter levels, aiming to uncover the potential mechanisms by which sleep deprivation leads to cognitive impairment. Neurotransmitters play a crucial role in signal transmission in the central nervous system, and abnormal changes in neurotransmitters can lead to neuronal dysfunction, resulting in cognitive impairment [65]. Reduced sleep time triggers a stress response, and long-term sleep deprivation can activate the HPA axis, leading to elevated epinephrine levels, which keeps the body in an overexcited state, causing central nervous system hyperactivity and disorder, ultimately resulting in cognitive impairment [66]. 5-HT, DA, and Epi are monoamine neurotransmitters that play complex roles in regulating arousal and behavior. Several studies have indicated that increased levels of these neurotransmitters can lead to sleep fragmentation and promote arousal [67, 68]. Previous studies have reported that sleep deprivation leads to elevated levels of 5-HT in the hippocampus of mice, accompanied by impaired performance in the MWMT and NORT [69]. Additionally, sleep deprivation has been shown to increase DA levels in the prefrontal cortex, potentially inducing a manic-like state in animals [70]. In our study, chronic sleep restriction was associated with increased levels of 5-HT, DA, and Epi in the PFC and hippocampus. These findings suggest that chronic sleep restriction may lead to dysregulation of monoamine neurotransmission, contributing to central nervous system hyperactivity and cognitive impairment. However, the specific effects of these changes depend on the balance of receptor activation and downstream signaling pathways. Future studies are necessary to investigate the causal role of these alterations in mediating behavioral and cognitive outcomes.

TRP is an essential amino acid, and the TRP-kynurenine (TRP-KYN) pathway is the predominant metabolic route, accounting for 95% of tryptophan metabolism, the metabolic state of which is closely related to the pathogenesis of memory decline and various neurodegenerative diseases [71]. Kynurenine (KYN) can be further metabolized into products such as kynurenic acid (KYNA) and 3-hydroxykynurenine (3-HK) [72]. Several reports have shown that increased activity of the TRP-KYN pathway has been observed in both acute and chronic sleep deprivation in male rats [73, 74]. In our study, CSR was associated with varying degrees of activation of the TRP-KYN metabolism in the PFC of both male and female rats, with a notable increase in KYNA observed in male CSR animals. KYNA, an endogenous N-methyl-D-aspartate receptor antagonist, has long been considered a neuromodulator. Elevated levels of KYNA have been observed in various neuropsychiatric disorders, and exogenous administration of KYNA has been shown to induce cognitive deficits in animals [75, 76]. Similar sex differences in KYNA levels have also been reported during acute sleep deprivation [14].

The hippocampus is a critical region responsible for learning and memory, closely linked to cognitive flexibility and reversal learning [77]. In this study, CSR was associated with significant activation of the TRP-KYN pathway in the hippocampus. Additionally, CSR was observed to increase the levels of the neurotoxic metabolite 3-HK, which has been implicated in inflammation, neurotoxicity, and other related pathologies [78]. These findings are consistent with prior research suggesting that abnormal activation of the TRP-KYN pathway and increased 3-HK levels may contribute to hippocampal dysfunction [28]. While this study characterizes the neurobiological changes associated with CSR, further research is needed to investigate the causal role of these changes in mediating sleep deprivation-induced cognitive impairments. This may provide deeper insights into the mechanisms underlying cognitive deficits observed in sleep-deprived animals.

Limitation

This study utilized Sprague-Dawley (SD) rats, a widely used strain in reward-related research, facilitating direct comparisons with previous studies. Although SD rats, as an albino strain, exhibit lower visual acuity thresholds compared to pigmented strains, they have demonstrated comparable performance in visual discrimination tasks [79, 80]. Our preliminary laboratory studies further confirmed that SD rats successfully acquired the signal discrimination task employed in this experiment. However, the absence of a comprehensive strain comparison limits the generalizability of our findings to other rat strains.

In our study, the lever-press task is centered on reward-driven actions, while the signal discrimination task requires animals to differentiate between signals for reinforcement, with both tasks necessitating active engagement in operant behavior to obtain rewards. Previous research has suggested that individual differences in conditioned Pavlovian associations, such as Goal Tracking and Sign Tracking, may influence task performance. While we did not specifically categorize animals based on these behavioral phenotypes, our findings reflect a primarily reward-driven learning process. Future studies may consider assessing Goal Tracking and Sign Tracking tendencies to further explore their potential interaction with CSR effects.

While this study identified distinct behavioral and mechanistic profiles between experimental groups, the relationships between these domains remain exploratory. Future studies integrating real-time bio-sampling, pharmacological manipulations, and larger cohorts will be critical to disentangle the causal pathways linking sleep disruption, neurochemical adaptations, and maladaptive reward-seeking behaviors.

Conclusion

In summary, CSR significantly impacted memory and learning abilities in both male and female rats during reward-based operant conditioning tasks. Male CSR rats showed lower learning efficiency in signal discrimination tasks, while CSR impaired extinction learning in female rats. These outcomes may be associated with CSR-induced oxidative stress imbalance and dysregulation of monoamine neurotransmitters. In tryptophan metabolism, the extent of changes in key metabolites differed between male and female rats, but the trends were similar, indicating abnormal activation of the TRP metabolic pathway in both sexes. Our study proposes a behavioral research approach with the potential to enhance clinical relevance and translational potential in sleep studies. Further exploration of sex differences in stress responses is essential to understanding the effects of sleep deprivation on cognitive function, laying a scientific foundation for developing more targeted and effective sex-specific prevention and intervention strategies.