K-ON: Stacking Knowledge on the Head Layer of Large Language Model
Lingbing Guo, Yichi Zhang, Zhongpu Bo, Zhuo Chen, Mengshu Sun, Zhiqiang Zhang, Wen Zhang, Huajun Chen
Abstract
Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.
BibTeX
@article{Guo_Zhang_Bo_Chen_Sun_Zhang_Zhang_Chen_2025, title={K-ON: Stacking Knowledge on the Head Layer of Large Language Model}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/33278}, DOI={10.1609/aaai.v39i11.33278}, abstractNote={Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.}, number={11}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Guo, Lingbing and Zhang, Yichi and Bo, Zhongpu and Chen, Zhuo and Sun, Mengshu and Zhang, Zhiqiang and Zhang, Wen and Chen, Huajun}, year={2025}, month={Apr.}, pages={11745-11753} }