Deep Learning-Based Pig Behavior Captioning for Smart Livestock Farming

Abstract

Precise monitoring of pig behavior has become pivotal for enhancing animal welfare and breeding efficiency. However, existing studies predominantly focus on behavior recognition while neglecting environmental influences, and lack specialized image captioning models and datasets tailored for farm scenarios, hindering textual analysis of behavior-environment interactions. In this study, a multimodal image captioning model was proposed to generate semantic textual descriptions of pig behavior, thereby supporting smart decision-making in farm management. The model employs a ResNet-18 encoder to extract pig visual biometric features from RGB and depth images, coupled with an innovative decoder integrating an enhanced Long Short-Term Memory (LSTM) network and Graph Convolutional Network (GCN) for pig behavior textual description, effectively resolving the input inconsistency between training and inference phases in traditional Encoder-Decoder architectures. Additionally, a dedicated pig behavior dataset comprising 9,052 annotated images was constructed, covering four behavioral categories: standing, sitting, lying, and eating. The experimental results show that the proposed approach achieves a METEOR score of 88.25%, which is outperformed baseline models by up to 21.58%. By recognizing pig behavior and interpreting environmental context, the proposed approach introduces a practical methodology for analyzing behavior-environment interactions and facilitates the integration of LLM-embedded robotic systems into smart livestock farming.