基于改进Transformer的复杂逻辑查询模型

doi:10.3969/j.issn.0255-8297.2026.01.003

摘要/Abstract

摘要： 随着知识图谱在智能问答和推荐系统等场景中的广泛应用，回答不完整知识图谱上的复杂逻辑查询成为当前研究的重点与难点。针对基于普通嵌入的方法需要在复杂逻辑查询上进行训练，无法很好地泛化到分布外的查询结构的问题，提出了一种融合动态可组合的多头注意力（dynamically composable multi-head attention,DCMHA）机制与混合专家（mixture-of-experts,MoE）网络的Transformer改进模型DCMHA-MoE。该模型利用三元组变换与双向路径编码技术，将复杂查询图表示为序列输入，并动态建模其中的结构依赖与语义交互，从而实现复杂逻辑查询。DCMHA实现注意力头的自适应组合，增强语义表达能力；MoE网络引入稀疏激活机制，提高对不同查询结构的适应性并降低计算成本。在FB15K-237与NELL-995数据集上的实验结果表明，与基线模型DiffCLR相比，DCMHA-MoE模型在存在正一阶逻辑（existential positive first-order logic,EPFO）查询($\wedge$,$\vee$)中的平均倒数排名（mean reciprocal rank,MRR）平均指标分别提升了10.4%和7.2%，验证了其在复杂逻辑推理任务中的有效性和优越性。

关键词: 复杂逻辑查询, 知识图谱, Transformer, 动态多头注意力机制, 混合专家网络

Abstract: With the widespread application of knowledge graphs in scenarios such as intelligent question answering and recommendation systems, answering complex logical queries on incomplete knowledge graphs has become the focus and difficulty of current research. In view of the fact that ordinary embedding-based methods need to be trained on complex logical queries and cannot be well generalized to query structures outside the distribution, this paper proposed an improved-Transformer-based model DCMHA-MoE that integrated the dynamically composable multi-head attention (DCMHA) mechanism and the mixture-of-experts (MoE) network. This model represented complex query graphs as sequence inputs through triple transformation and bidirectional path encoding technology, and dynamically modeled the structural dependencies and semantic interactions therein, so that complex logical queries can be realized. The DCMHA realized the adaptive combination of attention heads to enhance the semantic expression ability. The MoE network introduced a sparse activation mechanism to improve the adaptability to different query structures and reduce the computational cost. Experiments were conducted on the FB15K-237 and NELL-995 datasets. The results show that compared with the baseline model DiffCLR, the DCMHA-MoE model improves the mean reciprocal rank (MRR) in existential positive first-order logic (EPFO) query $(\wedge, \vee)$ by 10.4% and 7.2%, respectively, which verifies the effectiveness and superiority of DCMHA-MoE in complex logical query tasks.

Key words: complex logical query, knowledge graph, Transformer, dynamic multi-head attention mechanism, mixture-of-experts network

中图分类号:

P751.1

陈昱胤, 李贯峰, 秦晶, 肖毓航. 基于改进Transformer的复杂逻辑查询模型[J]. 应用科学学报, 2026, 44(1): 34-49.

CHEN Yuyin, LI Guanfeng, QIN Jing, XIAO Yuhang. Complex Logical Query Model Based on Improved Transformer[J]. Journal of Applied Sciences, 2026, 44(1): 34-49.

参考文献

[1] Bonatti P A, Sauro L. On the logical properties of the nonmonotonic description logic DLN [J]. Artificial Intelligence, 2017, 248: 85-111.
[2] Hasan S M S, Rivera D, Wu X C, et al. Knowledge graph-enabled cancer data analytics [J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(7): 1952-1967.
[3] Han J L, Cheng B, Wang X. Two-phase hypergraph based reasoning with dynamic relations for multi-hop KBQA [C]//29th International Conference on Artificial Intelligence, 2021: 3615- 3621.
[4] 李继演. 基于知识推理的自然语言数据探索关键技术研究[D]. 成都: 电子科技大学, 2022.
[5] Hamilton W, Bajaj P, Zitnik M, et al. Embedding logical queries on knowledge graphs [C]//32nd Conference on Neural Information Processing Systems (NIPS), 2018: 2030-2041.
[6] Ren H Y, Hu W H, Leskovec J. Query2box: reasoning over knowledge graphs in vector space using box embeddings [DB/OL]. (2020-02-29) [2025-08-08]. https://arxiv.org/abs/2002.05969.
[7] Ren H Y, Leskovec J. Beta embeddings for multi-hop logical reasoning in knowledge graphs [DB/OL]. (2020-10-22) [2025-08-08]. https://arxiv.org/abs/2010.11465.
[8] Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multirelational data [C]//Annual Conference on Neural Information Processing Systems, 2013: 2799- 2807.
[9] Allauzen A, Grefenstette E, Hermann K M, et al. Joint semantic relevance learning with text data and graph knowledge [C]//3rd Workshop on Continuous Vector Space Models and their Compositionality, 2015: 32-40.
[10] Xiao D, Meng Q Y, Li S P, et al. Improving transformers with dynamically composable multi-head attention [DB/OL]. (2024-06-04) [2025-08-08]. https://arxiv.org/abs/2405.08553.
[11] Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer [DB/OL]. (2017-01-23) [2025-08-08]. https://arxiv.org/abs/1701.06538.
[12] Dasgupta S S, Boratko M, Zhang D X, et al. Improving local identifiability in probabilistic box embeddings [DB/OL]. (2020-10-29) [2025-08-08]. https://arxiv.org/abs/2010.04831.
[13] Choudhary N, Rao N, Katariya S, et al. Probabilistic entity representation model for reasoning over knowledge graphs [DB/OL]. (2021-10-30) [2025-08-08]. https://arxiv.org/abs/2110.13522.
[14] Liu X, Zhao S Y, Su K, et al. Mask and reason: pre-training knowledge graph transformers for complex logical queries [C]//28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022: 1120-1130.
[15] Xu Z Z, Zhang W, Ye P, et al. Neural-symbolic entangled framework for complex query answering [DB/OL]. (2022-09-19) [2025-08-08]. https://arxiv.org/abs/2209.08779.
[16] Chen X L, Hu Z U, Sun Y Z. Fuzzy logic based logical query answering on knowledge graphs [DB/OL]. (2022-06-16) [2025-08-08]. https://arxiv.org/abs/2108.02390.
[17] Kim J, Kwon Y, Jo Y, et al. KG-GPT: a general framework for reasoning on knowledge graphs using large language models [C]//Conference on Empirical Methods in Natural Language Processing, 2023: 9410-9421.
[18] Koroteev M V. BERT: a review of applications in natural language processing and understanding [DB/OL]. (2021-03-22) [2025-08-08]. https://arxiv.org/abs/2103.11943.
[19] Choudhary N, Reddy C K. Complex logical reasoning over knowledge graphs using large language models [DB/OL]. (2024-03-31) [2025-08-08]. https://arxiv.org/abs/2305.01157.
[20] Ying C X, Cai T L, Luo S J, et al. Do transformers really perform badly for graph representation? [C]//35th Annual Conference on Neural Information Processing Systems, 2021: 28877-28888.
[21] Lepikhin D, Lee H J, Xu Y Z, et al. GShard: scaling giant models with conditional computation and automatic sharding [DB/OL]. (2020-06-30) [2025-08-08]. https://arxiv.org/abs/2006.16668.
[22] 方萍, 徐宁. 基于BERT双向预训练的图模型摘要抽取算法[J]. 计算机应用研究, 2021, 38(9): 2657-2661. Fang P, Xu N. Graph model summary extraction algorithm based on BERT bidirectional pretraining [J]. Application Research of Computers, 2021, 38(9): 657-2661. (in Chinese)
[23] Kersting W, Carr W, Kerestes R. The grounded wye-delta transformer with a zig-zag transformer [C]//2022 IEEE Rural Electric Power Conference (REPC), 2022: 71-78.
[24] Rokach L. Pattern classification using ensemble methods [M]. Hackensack: World Scientific Publishing Co., 2009.
[25] Devlin J, Chang M W, Lee K, et al. Bert: pre-training of deep bidirectional transformers for language understanding [C]//Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019: 4171-4186.
[26] 白文超, 韩希先, 王金宝. 基于条件生成模型的高效近似查询处理框架[J]. 浙江大学学报(工学版), 2022, 56(5): 995-1005. Bai W C, Han X X, Wang J B. Efficient approximate query processing framework based on conditional generative model [J]. Journal of Zhejiang University (Engineering Science), 2022, 56(5): 995-1005. (in Chinese)
[27] Koutris P, Wijsen J. Consistent query answering for self-join-free conjunctive queries under primary key constraints [J]. ACM Transactions on Database Systems, 2017, 42(2): 9.
[28] Toutanova K, Chen D Q. Observed versus latent features for knowledge base and text inference [C]//Workshop on Continuous Vector Space Models and Their Compositionality, 2015: 57-66.
[29] Xiong W H, Hoang T, Wang W Y. DeepPath: a reinforcement learning method for knowledge graph reasoning [DB/OL]. (2018-07-07) [2025-08-08]. https://arxiv.org/abs/1707.06690.
[30] Zhang Z Q, Wang J, Chen J J, et al. ConE: cone embeddings for multi-hop reasoning over knowledge graphs [DB/OL]. (2021-12-22) [2025-08-08]. https://arxiv.org/abs/2110.13715.
[31] Wang D M, Chen Y Y, Grau B C. Efficient embeddings of logical variables for query answering over incomplete knowledge graphs [C]//37th AAAI Conference on Artificial Intelligence (AAAI)/35th Conference on Innovative Applications of Artificial Intelligence/13th Symposium on Educational Advances in Artificial Intelligence, 2023: 4652-4659.
[32] Liu Y, Cao Y N, Wang S, et al. Generative models for complex logical reasoning over knowledge graphs [C]//17th ACM International Conference on Web Search and Data Mining, 2024: 492-500.
[33] Yin H, Wang Z H, Song Y Q. Rethinking complex queries on knowledge graphs with neural link predictors [DB/OL]. (2024-10-22) [2025-08-08]. https://arxiv.org/abs/2304.07063.
[34] Voorhees E M. The TREC-8 question answering track report [EB/OL]. (2017-02-17) [2025- 08-08]. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=151495.