【论文整理】知识表征学习必读论文 Must-read papers on KRL/KE. KRL: knowledge representation learning. KE: knowledge embedding. Survey papers: 1. Representation Learning: A Review and New Perspectives. Yoshua Bengio, Aaron Courville, and Pascal Vincent. TPAMI 2013. paper 2. Knowledge Representation Learning: A Review. (In Chinese) Zhiyuan Liu, Maosong Sun, Yankai Lin, Ruobing Xie. 计算机研究与发展 2016. paper 3. A Review of Relational Machine Learning for Knowledge Graphs. Maximilian Nickel, Kevin Murphy, Volker Tresp, Evgeniy Gabrilovich. Proceedings of the IEEE 2016. paper 4. Knowledge Graph Embedding: A Survey of Approaches and Applications. Quan Wang, Zhendong Mao, Bin Wang, Li Guo. TKDE 2017. paper Journal and Conference papers: 1. RESCAL: A Three-Way Model for Collective Learning on Multi-Relational Data. Maximilian Nickel, Volker Tresp, Hans-Peter Kriegel. ICML 2011. paper code
RESCAL is a tensor factorization approach to knowledge representation learning, which is able to perform collective learning via the latent components of the factorization. 2. SE: Learning Structured Embeddings of Knowledge Bases. Antoine Bordes, Jason Weston, Ronan Collobert, Yoshua Bengio. AAAI 2011. paper
SE assumes that the head and tail entities are similar in a relation-dependent subspace, where each relation is represented by two different matrices. 3. LFM: A Latent Factor Model for Highly Multi-relational Data. Rodolphe Jenatton, Nicolas L. Roux, Antoine Bordes, Guillaume R. Obozinski. NIPS 2012. paper
LFM is based on a bilinear structure, which captures variouts orders of interaction of the data, and also shares sparse latent factors across different relations. 4. NTN: Reasoning With Neural Tensor Networks for Knowledge Base Completion. Richard Socher, Danqi Chen, Christopher D. Manning, Andrew Ng. NIPS 2013. paper
NTN is a neural network which allows mediated interaction of entity vectors via a tensor. NTN might be the most expressive model to date, but it is not sufficiently simple and efficient to handle large-scale KGs. 5. TransE: Translating Embeddings for Modeling Multi-relational Data. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko. NIPS 2013. paper code
TransE is the first model to introduce translation-based embedding, which interprets relations as the translations operating on entities. 6. TransH: Knowledge Graph Embedding by Translating on Hyperplanes. Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen. AAAI 2014. paper code
To preserve the mapping propertities of 1-N/N-1/N-N relations, TransH inperprets a relation as a translating operation on a hyperplane. In addition, TransH proposes “bern.”, a strategy of constructing negative labels. 7. TransR & CTransR: Learning Entity and Relation Embeddings for Knowledge Graph Completion. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, Xuan Zhu. AAAI 2015. paper KB2E OpenKE
An entity may have multiple aspects and various relations may focus on different aspects of entites. TransR first projects entities from entity space to corresponding relation space and then builds translations between projected entities. CTransR extends TransR by clustering diverse head-tail entity pairs into groups and learning distinct relation vectors for each group, which is the initial exploration for modeling internal correlations within each relation type. 8. TransD: Knowledge Graph Embedding via Dynamic Mapping Matrix. Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, Jun Zhao. ACL 2015. paper KB2E OpenKE
TransD constructs a dynamic mapping matrix for each entity-relation pair by considering the diversity of entities and relations simultaneously. Compared with TransR/CTransR, TransD has fewer parameters and has no matrix vector multiplication. 9. TransA: An Adaptive Approach for Knowledge Graph Embedding. Han Xiao, Minlie Huang, Hao Yu, Xiaoyan Zhu. arXiv 2015. paper
Applying elliptial equipotential hypersurfaces and weighting specific feature dimensions for a relation, TransA can model complex entities and relations. 10. KG2E: Learning to Represent Knowledge Graphs with Gaussian Embedding. Shizhu He, Kang Liu, Guoliang Ji and Jun Zhao. CIKM 2015. paper code
Different entities and relations may contain different certainties, which represent the confidence for indicating the semantic when scoring a triple. KG2E represents each entity/relation by a Gaussion distribution, where the mean denotes its position and the covariance presents its certainty. 11. DistMult: Embedding Entities and Relations for Learning and Inference in Knowledge Bases.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, Li Deng. ICLR 2015. paper code
DistMult is based on the bilinear model where each relation is represented by a diagonal rather than a full matrix. DistMult enjoys the same scalable property as TransE and it achieves superior performance over TransE. 12. PTransE: Modeling Relation Paths for Representation Learning of Knowledge Bases. Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, Song Liu. EMNLP 2015. paper code
Multi-step relation paths contain rich inference patterns between entities. PTransE considers relation paths as translations between entities and designs an excellent algorithm to measure the reliablity of relation paths. Experiment shows PTransE achieves outstanding improvements on KBC and RE tasks. 13. RTransE: Composing Relationships with Translations. Alberto García-Durán, Antoine Bordes, Nicolas Usunier. EMNLP 2015. paper
RTransE learns to explicitly model composition of relationships via the addition of their corresponding translations vectors. In addition, the experiments include a new evaluation protocal, in which the model answers questions related to compositions of relations directly. 14. ManifoldE: From One Point to A Manifold: Knowledge Graph Embedding For Precise Link Prediction. Han Xiao, Minlie Huang and Xiaoyan Zhu. IJCAI 2016. paper
ManifoldE expands point-wise modeling in the translation-based principle to manifold-wise modeling, thus overcoming the issue of over-strict geometric form and achieving remarkable improvements for precise link prediction. 15. TransG: A Generative Mixture Model for Knowledge Graph Embedding. Han Xiao, Minlie Huang, Xiaoyan Zhu. ACL 2016. paper code
A relation in knowledge graph may have different meanings revealed by the associated entity pairs. TransG generates multiple translation components for a relation via a Bayesian non-parametric infinite mixture model. 16. ComplEx: Complex Embeddings for Simple Link Prediction. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier and Guillaume Bouchard. ICML 2016. paper code OpenKE
ComplEx extends DistMult by introducing complex-valued embeddings so as to better model asymmetric relations. It is proved that HolE is subsumed by ComplEx as a special case. 17. ComplEx extension: Knowledge Graph Completion via Complex Tensor Factorization. Théo Trouillon, Christopher R. Dance, Johannes Welbl, Sebastian Riedel, Éric Gaussier, Guillaume Bouchard. JMLR 2017. paper code OpenKE 18. HolE: Holographic Embeddings of Knowledge Graphs. Maximilian Nickel, Lorenzo Rosasco, Tomaso A. Poggio. AAAI 2016. paper code OpenKE
HolE employs circular correlations to create compositional representations. HolE can capture rich interactions but simultaneously remains efficient to compute. 19. KR-EAR: Knowledge Representation Learning with Entities, Attributes and Relations. Yankai Lin, Zhiyuan Liu, Maosong Sun. IJCAI 2016. paper code
Existing KG-relations can be divided into attributes and relations, which exhibit rather distinct characteristics. KG-EAR is a KR model with entities, attributes and relations, which encodes the correlations between entity descriptions. 20. TranSparse: Knowledge Graph Completion with Adaptive Sparse Transfer Matrix. Guoliang Ji, Kang Liu, Shizhu He, Jun Zhao. AAAI 2016. paper code
The entities and relations in a knowledge graph are heterogeneous and unbalanced. To overcome the heterogeneity, TranSparse uses sparse matrices to model the relations. To deal with the problem of imbalance of relations, each relation has two separate sparse transfer matrices for head and tail entity. 21. TKRL: Representation Learning of Knowledge Graphs with Hierarchical Types. Ruobing Xie, Zhiyuan Liu, Maosong Sun. IJCAI 2016. paper code
Entities should have multiple representations in different types. TKRL is the first attempt to capture the hierarchical types information, which is significant to KRL. 22. TEKE: Text-Enhanced Representation Learning for Knowledge Graph. Zhigang Wang, Juan-Zi Li. IJCAI 2016. paper
TEKE incorporates the rich textual content information to expand the semantic structure of the knowledge graph. Thus, each relation is enabled to own different representations for different head and tail entities to better handle 1-N/N-1/N-N relations. TEKE handle the problems of low performance on 1-N/N-1/N-N1 relations and KG sparseness. 23. STransE: A Novel Embedding Model of Entities and Relationships in Knowledge Bases. Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu and Mark Johnson. NAACL-HLT 2016. paper code
STransE is a simple combination of the SE and TransE model, using two projection matrices and one translation vector to represent each relation. STransE produces competitive results on link prediction evaluations. 24. GAKE: Graph Aware Knowledge Embedding. Jun Feng, Minlie Huang, Yang Yang, Xiaoyan Zhu. COLING 2016. paper code
Regarding a knowledge base as a directed graph rather than independent triples, GAKE utilizes graph context (neighbor/path/edge context) to learn knowledge representions. Furthermore, GAKE designs an attention mechanism to learn representitive powers of different subjects. 25. DKRL: Representation Learning of Knowledge Graphs with Entity Descriptions. Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, Maosong Sun. AAAI 2016. paper code
DKRL takes advantages of entity descriptions to learn knowledge representations. Outstanding performances under the zero-shot setting indicate that DKRL is capable of building representations for novel entities according to their descriptions. 26. ProPPR: Learning First-Order Logic Embeddings via Matrix Factorization. William Yang Wang, William W. Cohen. IJCAI 2016. paper
ProPPR is the first foraml study to investigate the problem of learning low-dimensional first-order logic embeddings from scratch, while scaling formula embeddings based probabilistic logic reasoning to large knowledge graphs. 27. SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions. Han Xiao, Minlie Huang, Lian Meng, Xiaoyan Zhu. AAAI 2017. paper
SSP models the strong correlatons between triples and the textual correlations by performing the embedding process in a sementic improvements against the state-of-the-art baselines. 28. ProjE: Embedding Projection for Knowledge Graph Completion. Baoxu Shi, Tim Weninger. AAAI 2017. paper code
ProjE views the KGC task as a ranking problem and projects candidate-entities onto a vector representing a combined embedding of the known parts of an input triple. Besides, ProjE optimizes a ranking loss of the list of candidate-entities collectively. ProjE can be viewed as a simplified version of NTN. 29. ANALOGY: Analogical Inference for Multi-relational Embeddings. Hanxiao Liu, Yuexin Wu, Yiming Yang. ICML 2017. paper code
Analogical inference is of greate use to knowledge base completion. ANALOGY models analogical structure in knowledge embedding. In addition, it is proved that DistMult, HolE and ComplEx are special cases of ANALOGY. 30. IKRL: Image-embodied Knowledge Representation Learning. Ruobing Xie, Zhiyuan Liu, Tat-Seng Chua, Huan-Bo Luan, Maosong Sun. IJCAI 2017. paper code
IKRL is the first attemp to combine images with knowledge graphs for KRL. Its promising performances indicate the significance of visual information for KRL. 31. ITransF: An Interpretable Knowledge Transfer Model for Knowledge Base Completion. Qizhe Xie, Xuezhe Ma, Zihang Dai, Eduard Hovy. ACL 2017. paper
Equipped with a sparse attention mechanism, ITransF discovers hidden concepts of relations and transfer statistical strength through the sharing of concepts. Moreover, the learned associations between relations and concepts, which are represented by sparse attention vectors, can be interpreted easily. 32. RUGE: Knowledge Graph Embedding with Iterative Guidance from Soft Rules. Shu Guo, Quan Wang, Lihong Wang, Bin Wang, Li Guo. AAAI 2018. paper code
RUGE is the first work that models interactions between embedding learning and logical inference in a principled framework. It enables an embedding model to learn simultaneously from labeled triples, unlabeled triples and soft rules in an iterative manner. 33. ConMask: Open-World Knowledge Graph Completion. Baoxu Shi, Tim Weninger. AAAI 2018. paper
ConMask is a novel open-world Knowledge Graph Completion model that uses relationship-dependent content masking, fully convolutional neural networks, and semantic averaging to extract relationship-dependent embeddings from the textual features of entities and relationships in KGs. 34. TorusE: Knowledge Graph Embedding on a Lie Group. Takuma Ebisu, Ryutaro Ichise. AAAI 2018. paper
TorusE defines the principle of TransE on Lie group. A torus, which is one of the compact Lie groups, can be chosen for the embedding space to avoid regularization. TorusE is the first model that embeds objects on other than a real or complex vector space, and this paper is the first to formally discuss the problem of regularization of TransE. 35. On Multi-Relational Link Prediction with Bilinear Mels. Yanjie Wang, Rainer Gemulla, Hui Li. AAAI 2018. paper code
The main goal of this paper is to explore the expressiveness of and the connections between various bilinear models for knowledge graph embedding proposed in the literature. This paper also provides evidence that relation-level ensembles of multiple bilinear models can achieve state-of-the art prediction performance. 36. Convolutional 2D Knowledge Graph Embeddings. Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel. AAAI 2018. paper code
ConvE is a multi-layer convolutional network model for link prediction of KGs, and it reports state-of-the-art results for several established datasets. Unlike previous work which has focused on shallow, fast models that can scale to large knowledge graphs, ConvE uses 2D convolution over embeddings and multiple layers of nonlinear features to model KGs. 37. Accurate Text-Enhanced Knowledge Graph Representation Learning. Bo An, Bo Chen, Xianpei Han, Le Sun. NAACL-HLT 2018. paper
This paper proposes an accurate text-enhanced knowledge graph representation framework, which can utilize accurate textual information to enhance the knowledge representations of a triple, and can effectively handle the ambiguity of relations and entities through a mutual attention model between relation mentions and entity descriptions. 38. KBGAN: Adversarial Learning for Knowledge Graph Embeddings. Liwei Cai, William Yang Wang. NAACL-HLT 2018. paper code
KBGAN employs adversarial learning to generate useful negative training examples to improve knowledge graph embedding. This framework can be applied to a wide range of KGE models. 39. ConvKB: A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network. Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, Dinh Phung. NAACL-HLT 2018. paper code
ConvKB applies the global relationships among same dimensional entries of the entity and relation embeddings, so that ConvKB generalizes the transitional characteristics in the transition-based embedding models. In addition, ConvKB is evaluated on WN18RR and FB15K237. 40. Modeling Relational Data with Graph Convolutional Networks. Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling. ESWC 2018. paper code1 code2
R-GCN applies Graph Convolutional Networks to relational Knowledge Bases creating a new encoder for the link predicion and entity classification tasks. 41. Improving Knowledge Graph Embedding Using Simple Constraints. Boyang Ding, Quan Wang, Bin Wang, Li Guo. ACL 2018. paper code
This paper investigates the potential of using very simple constraints to improve KG embedding. It examines non-negativity constraints on entity representations and approximate entailment constraints on relation representations. 42. Differentiating Concepts and Instances for Knowledge Graph Embedding. Xin Lv, Lei Hou, Juanzi Li, Zhiyuan Liu. EMNLP 2018. paper code
TransC proposes a novel knowledge graph embedding model by differentiating concepts and instances. Specifically, TransC encodes each concept in knowledge graph as a sphere and each instance as a vector in the same semantic space. This model can also handle the transitivity of isA relations much better than previous models. 43. SimplE Embedding for Link Prediction in Knowledge Graphs. Seyed Mehran Kazemi, David Poole. NeurIPS 2018. paper code
SimplE is a simple enhancement of CP (Canonical Polyadic) to allow the two embeddings of each entity to be learned dependently. The complexity of SimplE grows linearly with the size of embeddings. The embeddings learned through SimplE are interpretable, and certain types of background knowledge can be incorporated into these embeddings through weight tying. 44. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. Zhiqing Sun, Zhi Hong Deng, Jian Yun Nie, Jian Tang. ICLR 2019. paper code
RotatE infers various relation patterns including: symmetry/antisymmetry, inversion, and composition. Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space. 45. TuckER: Tensor Factorization for Knowledge Graph Completion. Ivana Balazˇevic ́, Carl Allen, Timothy M. Hospedales. arxiv 2019. paper code
TuckER is a relatively simple but powerful linear model based on Tucker decomposition of the binary tensor representation of knowledge graph triples. TuckER is a fully expressive model, deriving the bound on its entity and relation embedding dimensionality for full expressiveness which is several orders of magnitude smaller than the bound of previous models ComplEx and SimplE. Besides, TuckER achieves the state-of-the-art performance. 46. CrossE: Interaction Embeddings for Prediction and Explanation in Knowledge Graphs. Wen Zhang, Bibek Paudel, Wei Zhang. WSDM 2019. paper
CrossE, a novel knowledge graph embedding which explicitly simulates crossover interactions. It not only learns one general embedding for each entity and relation as most previous methods do, but also generates multiple triple specific embeddings for both of them, named interaction embeddings. 47. Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs. Deepak Nathani, Jatin Chauhan, Charu Sharma, Manohar Kaul. ACL 2019. paper code blog
This is a novel attention-based feature embedding model that captures both entity and relation features in any given entity’s neighborhood. This architecture is an encoder-decoder model where the generalized graph attention model and ConvKB play the roles of encoder and decoder respectively. 48. RSN: Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs. Lingbing Guo, Zequn Sun, Wei Hu. ICML 2019. paper supplementary code
RSNs integrate recurrent neural networks with residual learning to efficiently capture the long-term relational dependencies of entities within and between KGs. We also design an end-to-end framework to support RSNs on two tasks link prediction and entity alignment. 49. DihEdral: Relation Embedding with Dihedral Group in Knowledge Graph. Canran Xu, Ruijiang Li. ACL 2019. paper
DihEdral models the relation in knowledge graphs with the representation of dihedral group. It is a bilinear model and supports relation symmetry, skew-symmetry, inversion, abelian composition and non-abelian composition due to the properties of dihedral group. 50. CapsE:A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization. Dai Quoc Nguyen, Thanh Vu, Tu Dinh Nguyen, Dat Quoc Nguyen, Dinh Q. Phung. NAACL-HIT 2019. paper code
CapsE employs capsule network to model the entries in the triple at the same dimension. The high-level hypothesis is that each capsule accounts for capture variants of a relation-specific attribute of the entities. The length of the final vector serves as the plausibility score of the triple. 51. CaRe: Open Knowledge Graph Embeddings Swapnil Gupta, Sreyash Kenkre, Partha Talukdar. EMNLP-IJCNLP 2019.paper code
CaRe focuses on canonicalization of OpenKGs. The model infuses canonicalization in-formation combined with the neighborhood graph structure to learn rich representations of NPs. And it captures the semantic similarity of RPs.