With the widespread application of knowledge graphs in scenarios such as intelligent question answering and recommendation systems, answering complex logical queries on incomplete knowledge graphs has become the focus and difficulty of current research. In view of the fact that ordinary embedding-based methods need to be trained on complex logical queries and cannot be well generalized to query structures outside the distribution, this paper proposed an improved-Transformer-based model DCMHA-MoE that integrated the dynamically composable multi-head attention (DCMHA) mechanism and the mixture-of-experts (MoE) network. This model represented complex query graphs as sequence inputs through triple transformation and bidirectional path encoding technology, and dynamically modeled the structural dependencies and semantic interactions therein, so that complex logical queries can be realized. The DCMHA realized the adaptive combination of attention heads to enhance the semantic expression ability. The MoE network introduced a sparse activation mechanism to improve the adaptability to different query structures and reduce the computational cost. Experiments were conducted on the FB15K-237 and NELL-995 datasets. The results show that compared with the baseline model DiffCLR, the DCMHA-MoE model improves the mean reciprocal rank (MRR) in existential positive first-order logic (EPFO) query $(\wedge, \vee)$ by 10.4% and 7.2%, respectively, which verifies the effectiveness and superiority of DCMHA-MoE in complex logical query tasks.
CHEN Yuyin
,
LI Guanfeng
,
QIN Jing
,
XIAO Yuhang
. Complex Logical Query Model Based on Improved Transformer[J]. Journal of Applied Sciences, 2026
, 44(1)
: 34
-49
.
DOI: 10.3969/j.issn.0255-8297.2026.01.003
[1] Bonatti P A, Sauro L. On the logical properties of the nonmonotonic description logic DLN [J]. Artificial Intelligence, 2017, 248: 85-111.
[2] Hasan S M S, Rivera D, Wu X C, et al. Knowledge graph-enabled cancer data analytics [J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(7): 1952-1967.
[3] Han J L, Cheng B, Wang X. Two-phase hypergraph based reasoning with dynamic relations for multi-hop KBQA [C]//29th International Conference on Artificial Intelligence, 2021: 3615- 3621.
[4] 李继演. 基于知识推理的自然语言数据探索关键技术研究[D]. 成都: 电子科技大学, 2022.
[5] Hamilton W, Bajaj P, Zitnik M, et al. Embedding logical queries on knowledge graphs [C]//32nd Conference on Neural Information Processing Systems (NIPS), 2018: 2030-2041.
[6] Ren H Y, Hu W H, Leskovec J. Query2box: reasoning over knowledge graphs in vector space using box embeddings [DB/OL]. (2020-02-29) [2025-08-08]. https://arxiv.org/abs/2002.05969.
[7] Ren H Y, Leskovec J. Beta embeddings for multi-hop logical reasoning in knowledge graphs [DB/OL]. (2020-10-22) [2025-08-08]. https://arxiv.org/abs/2010.11465.
[8] Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multirelational data [C]//Annual Conference on Neural Information Processing Systems, 2013: 2799- 2807.
[9] Allauzen A, Grefenstette E, Hermann K M, et al. Joint semantic relevance learning with text data and graph knowledge [C]//3rd Workshop on Continuous Vector Space Models and their Compositionality, 2015: 32-40.
[10] Xiao D, Meng Q Y, Li S P, et al. Improving transformers with dynamically composable multi-head attention [DB/OL]. (2024-06-04) [2025-08-08]. https://arxiv.org/abs/2405.08553.
[11] Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously large neural networks: the sparsely-gated mixture-of-experts layer [DB/OL]. (2017-01-23) [2025-08-08]. https://arxiv.org/abs/1701.06538.
[12] Dasgupta S S, Boratko M, Zhang D X, et al. Improving local identifiability in probabilistic box embeddings [DB/OL]. (2020-10-29) [2025-08-08]. https://arxiv.org/abs/2010.04831.
[13] Choudhary N, Rao N, Katariya S, et al. Probabilistic entity representation model for reasoning over knowledge graphs [DB/OL]. (2021-10-30) [2025-08-08]. https://arxiv.org/abs/2110.13522.
[14] Liu X, Zhao S Y, Su K, et al. Mask and reason: pre-training knowledge graph transformers for complex logical queries [C]//28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022: 1120-1130.
[15] Xu Z Z, Zhang W, Ye P, et al. Neural-symbolic entangled framework for complex query answering [DB/OL]. (2022-09-19) [2025-08-08]. https://arxiv.org/abs/2209.08779.
[16] Chen X L, Hu Z U, Sun Y Z. Fuzzy logic based logical query answering on knowledge graphs [DB/OL]. (2022-06-16) [2025-08-08]. https://arxiv.org/abs/2108.02390.
[17] Kim J, Kwon Y, Jo Y, et al. KG-GPT: a general framework for reasoning on knowledge graphs using large language models [C]//Conference on Empirical Methods in Natural Language Processing, 2023: 9410-9421.
[18] Koroteev M V. BERT: a review of applications in natural language processing and understanding [DB/OL]. (2021-03-22) [2025-08-08]. https://arxiv.org/abs/2103.11943.
[19] Choudhary N, Reddy C K. Complex logical reasoning over knowledge graphs using large language models [DB/OL]. (2024-03-31) [2025-08-08]. https://arxiv.org/abs/2305.01157.
[20] Ying C X, Cai T L, Luo S J, et al. Do transformers really perform badly for graph representation? [C]//35th Annual Conference on Neural Information Processing Systems, 2021: 28877-28888.
[21] Lepikhin D, Lee H J, Xu Y Z, et al. GShard: scaling giant models with conditional computation and automatic sharding [DB/OL]. (2020-06-30) [2025-08-08]. https://arxiv.org/abs/2006.16668.
[22] 方萍, 徐宁. 基于BERT双向预训练的图模型摘要抽取算法[J]. 计算机应用研究, 2021, 38(9): 2657-2661. Fang P, Xu N. Graph model summary extraction algorithm based on BERT bidirectional pretraining [J]. Application Research of Computers, 2021, 38(9): 657-2661. (in Chinese)
[23] Kersting W, Carr W, Kerestes R. The grounded wye-delta transformer with a zig-zag transformer [C]//2022 IEEE Rural Electric Power Conference (REPC), 2022: 71-78.
[24] Rokach L. Pattern classification using ensemble methods [M]. Hackensack: World Scientific Publishing Co., 2009.
[25] Devlin J, Chang M W, Lee K, et al. Bert: pre-training of deep bidirectional transformers for language understanding [C]//Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019: 4171-4186.
[26] 白文超, 韩希先, 王金宝. 基于条件生成模型的高效近似查询处理框架[J]. 浙江大学学报(工学版), 2022, 56(5): 995-1005. Bai W C, Han X X, Wang J B. Efficient approximate query processing framework based on conditional generative model [J]. Journal of Zhejiang University (Engineering Science), 2022, 56(5): 995-1005. (in Chinese)
[27] Koutris P, Wijsen J. Consistent query answering for self-join-free conjunctive queries under primary key constraints [J]. ACM Transactions on Database Systems, 2017, 42(2): 9.
[28] Toutanova K, Chen D Q. Observed versus latent features for knowledge base and text inference [C]//Workshop on Continuous Vector Space Models and Their Compositionality, 2015: 57-66.
[29] Xiong W H, Hoang T, Wang W Y. DeepPath: a reinforcement learning method for knowledge graph reasoning [DB/OL]. (2018-07-07) [2025-08-08]. https://arxiv.org/abs/1707.06690.
[30] Zhang Z Q, Wang J, Chen J J, et al. ConE: cone embeddings for multi-hop reasoning over knowledge graphs [DB/OL]. (2021-12-22) [2025-08-08]. https://arxiv.org/abs/2110.13715.
[31] Wang D M, Chen Y Y, Grau B C. Efficient embeddings of logical variables for query answering over incomplete knowledge graphs [C]//37th AAAI Conference on Artificial Intelligence (AAAI)/35th Conference on Innovative Applications of Artificial Intelligence/13th Symposium on Educational Advances in Artificial Intelligence, 2023: 4652-4659.
[32] Liu Y, Cao Y N, Wang S, et al. Generative models for complex logical reasoning over knowledge graphs [C]//17th ACM International Conference on Web Search and Data Mining, 2024: 492-500.
[33] Yin H, Wang Z H, Song Y Q. Rethinking complex queries on knowledge graphs with neural link predictors [DB/OL]. (2024-10-22) [2025-08-08]. https://arxiv.org/abs/2304.07063.
[34] Voorhees E M. The TREC-8 question answering track report [EB/OL]. (2017-02-17) [2025- 08-08]. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=151495.
