This table contains an overview of 123 QA systems, selected for the leaderboard. For each system the following information is provided:
- System name (if available),
- Reported by (authors, who report or introduce the systems)
- Reported in the paper (link to the paper, where system is mentioned or presented by the authors)
- Demo/Repo/API available (yes, no, not working)
- Link to Demo/Repo/API
- Original paper (if it is system mention in col.3)
- System description (short description retrieved from the original system's paper)
Note: please horizontally slide.
| System Name | Reported by | Reported in the paper | Demo/Repo/API available (yes or no or not working) | Link to Demo/Repo/API | Original paper (if it is system mention in col.3) | System description (short description retrieved from the original system's paper) | Reference | |
|---|---|---|---|---|---|---|---|---|
| Alexandria | Lopez et al. | Link | not working | Link | Link | Alexandria is a German question answering system over a domain ontology that was built primarily with data from Freebase, Authors propose a new formal query building approach that consists of two stages. In the first stage, they predict the query structure of the question and leverage the structure to constrain the generation of the candidate queries and propose a novel graph generation framework to handle the structure prediction task and design an encoder-decoder model to predict the argument of the predetermined operation in each generative step. In the second stage, they follow the previous methods to rank the candidate queries. predict the query structure of the question and leverage the structure to constrain the generation of the candidate queries and propose a novel graph generation framework to handle the structure prediction task and design an encoder-decoder model to predict the argument of the predetermined operation in each generative step. In the second stage, they follow the previous methods to rank the candidate queries. | Chen et al. nly one triple, provide a modular, easy-toextend QA pipeline and evaluate it on the SimpleQuestionsWikidata benchmark. Ranking is learned from the training set. | |
| Aqqu Wikidata (rules) | Thomas Goette | Link | no | same as reporting paper | Author focus on simple questions which means thatthe corresponding SPARQL query contains only one triple, provide a modular, easy-toextend QA pipeline and evaluate it on the SimpleQuestionsWikidata benchmark. Ranking with a set of weighted features. | Thomas Goette | ||
| AskNow | Diefenbach et al. | Link | yes | Link | Link | Authors propose a framework, called AskNow, where users can pose queries in English to a target RDF knowledge base (e.g. DBpedia), which are first normalized into an intermediary canonical syntactic form, called Normalized Query Structure (NQS), and then translated into SPARQL queries. NQS facilitates the identification of the desire (or expected output information) and the user-provided input information, and establishing their mutual semantic relationship. At the same time, it is sufficiently adaptive to query paraphrasing. We have empirically evaluated the framework with respect to the syntactic robustness of NQS and semantic accuracy of the SPARQL translator on standard benchmark datasets. | Dubey et al. | |
| BART | Chen et al. | Link | yes | Link | Link | BART is a strong pre-trained sequence-tosequence model, that treats the problem of KGQA as a conventional machine translation task from NLQ to SPARQL. | Lewis et al. | |
| BELA | Walter et al. | Link | no | same as reporting paper | Authors present a question answering system architecture whichprocesses natural language questions in a pipeline consisting of five steps:i) question parsing and query template generation, ii) lookup in an inverted index, iii) string similarity computation, iv) lookup in a lexicaldatabase in order to find synonyms, and v) semantic similarity computation | Walter et al. | ||
| CASIA | He et al. | Link | no | same as reporting paper | CASIA implements a basic pipeline framework which consists three main components, including question analysis, resource mapping and SPARQL generation. Inspecific, authors first employ shallow and deep linguistic analysis to transform NL-queriesinto a set of Query Triples with <subject, predict, object> format. Second, they mapeach phrase in Query Triple to the corresponding resource (class, entity, or property)in DBpedia. As a result, Ontology Triples are generated. Thirdly, the SPARQL querieswill be constructed based on Ontology Triple and question type. At last, the generatedSPARQL queries is used to search on the Linked Data, and the best answer can bepicked out through validating and ranking. | He et al. | ||
| DAM | Chen et al. | Link | no | Link | Authors propose a transformer-based deep attentive semantic matching model (DAM), to identify the KB relations corresponding to the questions. The DAM is completely based on the attention mechanism and applies the fine-grained word-level attention to enhance the matching of questions and relations. On the basis of the DAM, we build a three-stage KBQA pipeline system | Chen et al. | ||
| DEANNA | Diefenbach et al. | Link | yes | Link | Link | The method is based on an integer linear program to solve several disambiguation tasks jointly: the segmentation of questions into phrases; the mapping of phrases to semantic entities, classes, and relations; and the construction of SPARQL triple patterns. Our solution harnesses the rich type system provided by knowledge bases in the web of linked data, to constrain our semantic-coherence objective function. | Yahya et al. | |
| DeepPavlov KBQA | Evseev D. A. et al | Link | yes | Link | same as reporting paper | For translation of a question to a SPARQL query, authors first define the type of the query template. Then they fill the empty slots in the template with entities, relations from Wikidata and constraints. For entity detection BERT sequence labeling model is used. Relation ranking is performed by BiLSTM, path ranking—by BERT-based ranking model.This KBQA system is capable of answering complex questions with logical or comparative reasoning and was released as a component of open-source DeepPavlov library. | Evseev D. A. et al. | |
| DTQA | Omar et al. | Link | no | Link | Authors demonstrate Deep Thinking Question Answering (DTQA), a semantic parsing and reasoning-based KBQA system. DTQA (1) integrates multiple, reusable modules that are trained specifically for their individual tasks (e.g. semantic parsing, entity linking, and relationship linking), eliminating the need for end-to-end KBQA training data; (2) leverages semantic parsing and a reasoner for improved question understanding. | Abdelaziz et al. | ||
| ElNeuQA-ConvS2S | Diomedi, Hogan | Link | yes | Link | same as reporting paper | Authors propose an approach, called ElNeuQA, that combines EL with NMT. Specifically, an EL system is used to identify entity mentions in the question and link them to the knowledge graph. They combine this with an NMT model that is trained and used to generate template queries with placeholders for entities. Also the model is strengthened with ConvS2S (Convolutional Sequence-to-Sequence): a CNN-based architecture, featuring gated linear units, residual connections, and attention. | Diomedi, Hogan | |
| Elon | Zheng et. al. | Link | yes | Link, demo not working Link | same as reporting paper | Elon by Szab´o Bence et al. from Paderborn University in Germany stemsfrom a student project and is available at Link is based on an own dictionary and not yet published. | Zheng et. al. | |
| Frankenstein | Liang et al. | Link | yes | Link | Link | Modern question answering (QA) systems need to flexibly integrate a number of components specialised to fulfil specific tasks in a QA pipeline. Since a number of different software components exist that implement different strategies for each of these tasks, it is a major challenge to select and combine the most suitable components into a QA system, given the characteristics of a question. The authors study this optimisation problem and train classifiers, which take features of a question as input and have the goal of optimising the selection of QA components based on those features and devise a greedy algorithm to identify the pipelines that include the suitable components and can effectively answer the given question. We implement this model within Frankenstein, a QA framework able to select QA components and compose QA pipelines. Evaluation results not only suggest that Frankenstein precisely solves the QA optimisation problem but also enables the automatic composition of optimised QA pipelines, which outperform the static Baseline QA pipeline. | Singh et al. | |
| FREyA | Lopez et al. | Link | no | Link | FREyA combines syntactic parsing with the knowledge encoded in ontologies in order to reduce the customisation effort. If the system fails to automatically derive an answer, it will generate clarification dialogs for the user. The user’s selections are saved and used for training the system in order to improve its performance over time. | Damljanovic et al. | ||
| gAnswer | Omar et al. | Link | not working | Link, Link | Link | Authors propose a semantic query graph to model the query intention in the natural language question in a structural way, based on which, RDF Q/A is reduced to subgraph matching problem. They resolve the ambiguity of natural language questions at the time when matches of query are found. The cost of disambiguation is saved if there are no matching found. Two different frameworks to build the semantic query graph are proposed, gAnswer is relation (edge)-first. | Hu et al. | |
| gAnswer2 | Zheng et. al. | Link | no | Link | Authors propose a semantic query graph to model the query intention in the natural language question in a structural way, based on which, RDF Q/A is reduced to subgraph matching problem. They resolve the ambiguity of natural language questions at the time when matches of query are found. The cost of disambiguation is saved if there are no matching found. Two different frameworks to build the semantic query graph are proposed, gAnswer2 is node-first. | Hu et al. | ||
| gGCN | Wu et al. | Link | no | same as reporting paper | Authors present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism.The gGCN model is obtained from gRGCN by replacing RGCN with Graph Convolutional Network (GCN) | Wu et al. | ||
| GGNN | Sorokin and Gurevych | Link | yes | Link | same as reporting paper | Authors address the problem of learning vector representations for complex semantic parses that consist of multiple entities and relations. For each input question, they construct an explicit structural semantic parse (semantic graph). Semantic parses can be deterministically converted to a query to extract the answers from the KB. To investigate ways to encode the structure of a semantic parse and to improve the performance for more complex questions, authors adapt Gated Graph Neural Networks (GGNNs), described in Li et al. (2016), to process and score semantic parses. | Sorokin and Gurevych | |
| GRAFT-Net | Y Feng et al. | Link | yes | Link | Link | Authors propose a novel graph convolution based neural network, called GRAFT-Net (Graphs of Relations Among Facts and Text Networks), specifically designed to operate over heterogeneous graphs of KB facts and text sentences. First, they propose heterogeneous update rulesthat handle KB nodes differently from the textnodes: for instance, LSTM-based updates are usedto propagate information into and out of text nodes. Second, authors introduce a directed propagation method, inspired by personalized Pagerankin IR (Haveliwala, 2002). | Sun et al. | |
| GRAFT-Net + Clocq | Christmann P. et al. | Link | yes | Link (demo is available for further work on CLOCQ) | same as reporting paper | This work presents CLOCQ, an efficient method that prunes irrelevant parts of the search space using KB-aware signals. CLOCQ uses a top-𝑘 query processor over score-ordered lists of KB items that combine signals about lexical matching, relevance to the question, coherence among candidate items, and connectivity in the KB graph. | Christmann P. et al . | |
| gRGCN | Wu et al. | Link | no | same as reporting paper | Authors present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism. | Wu et al. | ||
| Hakimov | Diefenbach et al. | Link | no | Link | Authors investigate how much lexical knowledge would need to be added so that a semantic parsing approach can perform well on unseen data. We manually add a set of lexical entries on the basis of analyzing the test portion of the QALD-4 dataset. Further, we analyze if a state-of-the-art tool for inducing ontology lexica from corpora can derive these lexical entries automatically. | Hakimov et al. | ||
| HGNet | Chen et al. | Link | yes | Link | Link | Hierarchical Graph Generation Network (HGNet) focues on generating search query by proposing a new unified query graph grammar to adapt to SPARQL's syntax. FIrstly, the autho ranks top k entity, relation, and value by ralation ranking and pattern matching. Secondly, HGNet is used to encde and decode the natural questions to generate query graph. The project is open-sourced on github. | Chen et al. | |
| HR-BiLSTM | Chen et al. | Link | no | Link | Authors propose a hierarchical recurrent neural network enhanced by residual learning which detects KB relations given an input question. The method uses deep residual bidirectional LSTMs to compare questions and relation names via different levels of abstraction. Additionally, they propose a simple KBQA system that integrates entity linking and our proposed relation detector to make the two components enhance each other | Yu et al. | ||
| Intui2 | Diefenbach et al. | Link | no | Link | The system takes as input a natural language question formulated in English and generates an equivalent SPARQL query. The mapping is based on the analysis of the syntactic patterns present in the input question. | Corina Dima | ||
| Intui3 | Diefenbach et al. | Link | no | Link | The system accepts as input a question formulated in natural language (in English), and uses syntactic and semantic information to construct its interpretation with respect to a given database of RDF triples (in this case DBpedia 3.9). The interpretation is mapped to the corresponding SPARQL query, which is then run against a SPARQL endpoint to retrieve the answers to the initial question. | Corina Dima | ||
| ISOFT | Diefenbach et al. | Link | no | Link | Authors use natural language processing tools to extract slots and SPARQL templates from the question and semantic similarity to map a natural language question to a SPARQL query. | Park et al. | ||
| KBQA-Adapter | Oliya A et al | Link | yes | Link | Link | In this paper, we propose a simple mapping method, named representation adapter, to learn the representation mapping for both seen and unseen relations based on previously learned relation embedding. The authors employ the adversarial objective and the reconstruction objective to improve the mapping performance | Wu et al. | |
| KGQAn | Omar et al. | Link | yes | Link | same as reporting paper | KGQAn transforms a question into semantically equivalent SPARQL queries via a novel three-phase strategy based on natural language models trained generally for understanding and leveraging short English text. Without preprocessing or annotated questions on KGs, KGQAn outperformed the existing systems in KG question answering by an improvement of at least 33% in F1-measure and 61% in precision | Omar et al. | |
| KrantikariQA (Pairwise) | G Maheshwari et. al. | Link | yes | Link | same as reporting paper | Authors conduct an empirical investigation of neural query graph ranking approaches for the task of complex question answering over knowledge graphs. They experiment with six different ranking models and propose a novel self-attention based slot matching model which exploits the inherent structure of query graphs. Pairwise counterparts perform worse. | G Maheshwari et. al. | |
| KrantikariQA (Pointwise) | G Maheshwari et. al. | Link | yes | Link | same as reporting paper | Authors conduct an empirical investigation of neural query graph ranking approaches for the task of complex question answering over knowledge graphs. They experiment with six different ranking models and propose a novel self-attention based slot matching model which exploits the inherent structure of query graphs. Pointwise models generally outperform their pairwise counterparts when trained on small datasets but have a comparable performance otherwise. | G Maheshwari et. al. | |
| LAMA | Radoev et. al. | Link | no | same as reporting paper | The proposed method is based on transforming natural language questions into SPARQL queries by leveraging the syntactic information of questions. Authors describe a set of lexico-syntactic patterns used to automatically generate triple patterns and SPARQL queries. | Radoev et. al. | ||
| Liang et al. | Liang et al. | Link | yes | Link | same as reporting paper | Authors propose a new QA system for translating natural language questions into SPARQL queries. The key idea is to break up the translation process into 5 smaller, more manageable sub-tasks and use ensemble machine learning methods as well as Tree-LSTM-based neural network models to automatically learn and translate a natural language question into a SPARQL query. | Liang et al. | |
| LingTeQA | D. Nhuan et al. | Link | no | same as reporting paper | Authors introduce a Question-Answering (QA) system that allows users to ask questions in English. The uniqueness of this system is its ability to answer questions containing linguistic terms, i.e., concepts such as SMALL, LARGE, or TALL. Those concepts are defined via membership functions drawn by users using a dedicated software designed for entering ‘shapes’ of these functions. The system is built based on an analogical problem solving approach, and is suitable for providing users with comprehensive answers. | D. Nhuan et al. | ||
| Luo et al. | Wu et al. | Link | no | Link | Authors propose a neural network based approach to improve the performance of semantic similarity measurement in complex question answering. Given candidate query graphs generated from one question, their model embeds the question surface and predicate sequences into a uniform vector space. The main difference between their approach and previous methods is that the authors integrate hidden vectors of various semantic components and encode their interaction as the hidden semantics of the entire query graph. In addition, to cope with different semantic components of a query graph, dependency parsing information is leveraged as a complementary of sentential information for question encoding, which makes the model better align each component to the question. | Luo et al. | ||
| mBERT | Zhou Y. et al. | Link | no | same as reporting paper | A KGQA baseline, proposed in Zhou et al. for multilingual QA , implemented with fine-tuning pre-trained multilingual models (e.g. mBERT) in source language and directly perform inference in target language. | Zhou Y. et al. | ||
| MemNN | Oliya A et al | Link | no | Link | Authors present an embedding-based QA system developed under the framework of Memory Networks (MemNNs) (Weston et al., 2015; Sukhbaatar et al., 2015). Memory Networks are learning systems centered around a memory component that can be read and written to, with a particular focus on cases where the relationship between the input and response languages (here natural language) and the storage language (here, the facts from KBs) is performed by embedding all of them in the same vector space. The setting of the simple QA corresponds to the elementary operation of performing a single lookup in the memory. | Bordes et al. | ||
| MHE | Lopez et al. | Link | no | no paper submitted, mentioned only in the organizers report | MHE is a method for retrieving entities from an entity graph given an input query in natural language. It was developed by Marek Ciglan at the Institute of Informatics at the Slovak Academy of Sciences. The method relies on query annotation, where parts of the query are labeled with possible mappings to the given knowledge base. The annotations comprise entities and relations, and were generated by means of a gazetteer, in order to expand relations with synonyms, and a Wikifier tool, in order to annotate entities. From those annotations, MHE constructs possible sub-graphs as query interpretation hypotheses and matches them against the entity graph of DBpedia. MHE was the onlyQALD-2 participant that provided answers to alltypes of questions, performing best on string anddate questions. | Only in organizers' report. | ||
| Multi-hop QGG | Zou et al. | Link | no | same as reporting paper | Authors propose an end-to-end text-to-SPARQL baseline, which can effectively answer multitype complex questions, such as fact questions, dual-intent questions, boolean questions and counting questions, with Wikidata as the background knowledge base. The baseline's is implemented as relation-aware attention encoder and pointer network decoder. | Zou et al. | ||
| NHGG | Chen et al. | Link | no | same as reporting paper | Non-hierarchical Graph Generation (NHGG) integrates Outlining and Filling into one procedure. For AddVertex and AddEdge, the model directly predicts instances instead of classes. In this way, the query graph can be completed by only one decoding process without Filling operations. | Chen et al. | ||
| NSM | Y Feng et al. | Link | yes | Link | Link | Authors propose a novel teacher-student approach for the multi-hop KBQA task. In their approach, the student network aims to find the correct answer to the query, while the teacher network tries to learn intermediate supervision signals for improving the reasoning capacity of the student network. The major novelty lies in the design of the teacher network, where we utilize both forward and backward reasoning to enhance the learning of intermediate entity distributions. By considering bidirectional reasoning, the teacher network can produce more reliable intermediate supervision signals, which can alleviate the issue of spurious reasoning | He et al. | |
| NSQA | P.Kapanipathi et al. | Link | no | same as reporting paper | Authors propose Neuro-Symbolic Question Answering (NSQA), a modular KBQA system, that leverages (1) Abstract Meaning Representation (AMR) parses for task-independent question understanding; (2) a simple yet effective graph transformation approach to convert AMR parses into candidate logical queries that are aligned to the KB; (3) a pipeline-based approach which integrates multiple, reusable modules that are trained specifically for their individual tasks (semantic parser, entity and relationship linkers, and neuro-symbolic reasoner) and do not require end-to-end training data. | P.Kapanipathi et al. | ||
| NT-GRAFT-Net | Y Feng et al. | Link | no | same as reporting paper | Extension of NSM: authors replace NSM with GRAFT-Net in NT-NSMto create NT-GRAFT-Net and obtain 6.5-12.6%Hits@1 improvement on GRAFT-Net. GRAFT-Net is a novel graph convolution based neural network,called GRAFT-Net (Graphs of Relations AmongFacts and Text Networks), specifically designedto operate over heterogeneous graphs of KB factsand text sentences, proposed by Sun et al., 2018. | Y Feng et al. | ||
| NT-NSM | Y Feng et al. | Link | no | same as reporting paper | Authors present NumericalTransformer on top of NSM, a state-of-the-art embedding-based KBQA model, to create NT-NSM. To enable better training, they propose two pre-training tasks with explicit numerical-oriented loss functions on two generated training datasets and a template-based data augmentation method for enriching ordinal constrained QA dataset. | Y Feng et al. | ||
| O-Ranking | Chen et al. | Link | no | same as reporting paper | Outlining+Ranking (O-Rank) is an approach proposed by Chen et al. to generate AQG (Abstract Query Graph) by Outlining and subsequently produces the candidate graphs by enumerating the combination of instances to fill the AQG. Thereafter, the candidates are also ranked with CompQA. | Chen et al. | ||
| openQA | Marx et al. | Link | yes | Link | same as reporting paper | Authors present a modular and extensible open-source question answering framework and demonstrate how the framework can be used by integrating two state-of-the-art question answering systems. | Marx et al. | |
| Platypus | Orogat et al. | Link | yes | only find result on qald-7&8 demo available | Link | same as reporting paper | Platypus is an question answering platform which has been stoped maintaining after 2018. Platypus supports multilingual question answering by processing question in three steps: 1. convert natural question into internal logical representations. 2. rank the representations by their closeness to the correct interpretation of the question. 3. convert the representation into SPARQL query. | Orogat et al. |
| POMELO | Zhang et. al. | Link | no | Link | Authors design a four-step method which pre-process the question, generation an abstraction of the question, then build a representation of the SPARQL query and finally generate the query. | Hamon et al. | ||
| PowerAqua | Lopez et al. | Link | yes | only find result on qald-1 | Link (not working), demo: Link | Link | This QA system is built to fix searching and managing massive scale and heterogeneous content in knowledge base. It applys an ontology basedapproach to locate and integrate information. | Lopez et al. |
| QAKiS | Zheng et. al. | Link | not working | Link, demo: Link | Link | QAKiS exploits the alignment between properties carried out by DBpedia contributors as a mapping from Wikipedia terms to a common ontology, to exploit information coming from DBpedia multilingual chapters, broadening therefore its coverage. | Cabrio et al. | |
| QAmp | Kapanipathi et al. | Link | yes | Link | Link | QAmp is an approach to complex KGQA that uses unsupervised message passing, which propagates confidence scores obtained by parsing an input question and matching terms in the knowledge graph to a set of possible answers. First, we identify entity, relationship, and class names mentioned in a natural language question, and map these to their counterparts in the graph. Then, the confidence scores of these mappings propagate through the graph structure to locate the answer entities. Finally, these are aggregated depending on the identified question type. | Vakulenko et al. | |
| Qanary(TM+DP+QB) | Orogat et al. | Link | yes | Link | Link, Link | Qanary is a methodology for open question answering systems with the following attributes (requirements): interoperability, i.e., an abstraction layer for communication needs to be established, exchangeability and reusability, i.e., a component within a question answering system might be exchanged by another one with the same purpose, flexible granularity, i.e., the approach needs to be agnostic the processing steps implemented by a question answering system, isolation, i.e., each component within a QA system is decoupled from any other component in the QA system. In the cited pipeline, the following components were used: NED-tagme (for the entity recognition module), Diambiguation-Property-OKBQA (for the relationmapping module) and Query Builder (for the query generationmodule). | ||
| QAnswer | Diefenbach et al. | Link | yes | Link, Link | Link | QAnswer is a question answering system developed by Resuti S et al, developed for qald 5 challenge. Now they offer a frontend to type in question, also a API to load your own RDF file to build personalized system. The QAnswer aims at improving the match between entities, relations and natural language text. They adopt a DBpedia and Wikipedia- based approacd. | Singh et al. | |
| QASparql | Orogat et al. | Link | yes | Link | Link | a new QA system for translating natural language questions into SPARQL queries. The key idea is to break up the translation process into 5 smaller, more manageable sub-tasks and use ensemble machine learning methods as well as Tree-LSTM-based neural network models to automatically learn and translate a natural language question into a SPARQL query | Ruseti et al. | |
| qaSQP | Zheng et. al. | Link | no | same as reporting paper | Authors propose an approach powered by a notion of structural query pattern, in this paper. Given an input question, they first generate its query sketch that is compatible with the underlying structure of the knowledge graph. Then, the query graph is completed by labeling the nodes and edges under the guidance of the structural query pattern. Finally, answers can be retrieved by executing the constructed query graph over the knowledge graph. | Zheng et. al. | ||
| QASystem | Zheng et. al. | Link | yes | Link | Link (system mentioned in the report, no paper submitted) | QASystem by Lukas Bl¨ubaum and Nick D¨usterhus is also a student project from Paderborn University Germany and available at Link Their system is able to cope with comparatives and superlatives in questions via hand-crafted rules. | Link (system mentioned in the report, no paper submitted) | |
| RealTextasg | Perera and Nand | Link | not working | only tested on qald-2 | Link | same as reporting paper | Authors propose a typed dependency based approach to generate an answer sentence where linguistic structure of the question is transformed and realized into a sentence containing the answer. They employ the factoid questions from QALD-2 training question set to extract typed dependency patterns based on the root of the parse tree. Using identified patterns the authors generate a rule set which is used to generate a natural language sentence containing the answer extracted from a knowledge source, realized into a linguistically correct sentence. | Perera and Nand |
| Rigel-Baseline | Oliya A et al. | Link | no | same as reporting paper | Rigel is an end-to-end model for KGQA which includes a entity resolution module. The training data only includes questions and corresponding SPARQL, with no need for a seperate entity resolution data. The model learns to jointly perform entity resolution and inference. The model has three variantes, namely Rigel-Baseline, Rigel-ER and Rigel-E2E. In Rigel-baseline, the model has the golen entity as input. | Oliya A et al. | ||
| Rigel-E2E | Oliya A et al. | Link | no | same as reporting paper | Rigel is an end-to-end model for KGQA which includes a entity resolution module. The training data only includes questions and corresponding SPARQL, with no need for a seperate entity resolution data. The model learns to jointly perform entity resolution and inference. The model has three variantes, namely Rigel-Baseline, Rigel-ER and Rigel-E2E. In Rigel-E2E, the model has the natural question as input. | Oliya A et al. | ||
| Rigel-ER | Oliya A et al. | Link | no | same as reporting paper | Rigel is an end-to-end model for KGQA which includes a entity resolution module. The training data only includes questions and corresponding SPARQL, with no need for a seperate entity resolution data. The model learns to jointly perform entity resolution and inference. The model has three variantes, namely Rigel-Baseline, Rigel-ER and Rigel-E2E. In Rigel-ER, the model has the golden entity span. | Oliya A et al. | ||
| RO FII | Zhang et. al. | Link | no | only on qald 4 | Link (organizers report) | The Faculty of Computer Science at Alexandru Ioan Cuza University of Iasi, Romania, participated with two systems (RO FII), one tackling question answering over DBpedia and one tackling interlinked biomedical datasets. The former builds on Quepy, a Python tool for transforming natural language questions into SPARQL or MQL queries. The latter comprises three components, based on Service Oriented Architecture principles: a text annotator that receives the question in plain text and returns a list of compound words annotated with POS tags and lemmas (using Standford Core NLP), the triple builder that builds a list of triples given a list of keywords and URIs. | Unger et al. | |
| robustQA | Yahya et al. | Link | no | only on qald 2 | same as reporting paper | This paper advocates a new approach that allows questions to be partially translated into relaxed queries, covering the essential but not necessarily all aspects of the user's input. To compensate for the omissions, we exploit textual sources associated with entities and relational facts. The system translates user questions into an extended form of structured SPARQL queries, with text predicates attached to triple patterns. robustQA is based on a novel optimization model, cast into an integer linear program, for joint decomposition and disambiguation of the user question. | Yahya et al. | |
| RTV | Diefenbach et al. | Link | no | only on qald 3 | Link | The system integrates lexical semantic modeling and statistical inference within a complex architecture that decomposes the NL interpretation task into a cascade of three different stages: (1) The selection of key ontological information from the question (i.e. predicate, arguments and properties), (2) the location of such salient information in the ontology through the joint disambiguation of the different candidates and (3) the compilation of the final SPARQL query. This architecture characterizes a novel approach for the task and exploits a graphical model (i.e. an Hidden Markov Model) to select the proper ontological triples according to the graph nature of RDF | Giannone et al. | |
| S-Ranking | Chen et al. | Link | no | same as reporting paper | S-Ranking is a baseline model designed by author by combining STAGG (Link) and CompQA (Link). STAGG is used to generate query candidates and ComQA to rank the candidates. | Chen et al. | ||
| semanticQA | Hakimov et al. | Link | no | only on qald 2 | same as reporting paper | Authors present a method for mapping natural language questions to ontology-based structured queries to retrieve direct answers from open knowledge bases (linked data). It is based on translating natural language questions into RDF triple patterns using the dependency tree of the question text. In addition, the method uses relational patterns extracted from the Web. | Hakimov et al. | |
| SemGraphQA | Diefenbach et al. | Link | no | only on qald 5 | Link | Authors proposed an unsupervised method for the semantic analysis of questions, that generates queries, based on graph transformations, in two steps. First step is independent of the knowledge base schema and makes use of very general constraints on the query structure that allows us to maintain semantic ambiguities in different graphs. Ambiguities are then solved globally at the final step when querying the knowledge base. | Beaumont et al. | |
| SemSeK | Lopez et al. | Link | no | only on qald 2 | same as reporting paper | The authors implemented a series of evaluation challenges for question answering over linked data. The main goal of the challenge was to get insight into the strengths, capabilities, and current shortcomings of question answering systems as interfaces to query linked data sources, as well as benchmarking how these interaction paradigms can deal with the fact that the amount of RDF data available on the web is very large and heterogeneous with respect to the vocabularies and schemas used. | Lopez et al. | |
| SenseAware | Elbedweihy et al. | Link | no | only on qald 2 | same as reporting paper | Authors present a free-NL semantic search approach that bridges the gap between the sense of the user query terms and the underlying ontology’s concepts and properties. They use an extended-Lesk WSD approach and a NE recogniser together with a set of advanced string similarity algorithms and ontology-based heuristics to match query terms to ontology concepts and properties. | Elbedweihy et al. | |
| SINA | Diefenbach et al. | Link | not working | only on qald-3&4 | Link, Link | Link | SINA is a scalable keyword search system that can answer user queries by transforming user-supplied keywords or natural-languages queries into conjunctive SPARQL queries over a set of interlinked data sources. SINA uses a hidden Markov model to determine the most suitable resources for a user-supplied query from different datasets. The framework is able to construct federated queries by using the disambiguated resources and leveraging the link structure underlying the datasets to query. | Shekarpour et al. |
| Slot-Matching | Chen et al. | Link | yes | Link | Link | This model is a neural network based QA system, which exploits the inherent structure of query graphs. The recurrent neural networt, convolutional neural network and attention structure are tested and compared in ranking and finding the optimal semantic graph. The code is open-sourced on github as well. | Maheshwari et al. | |
| SPARQL Silhouette Stage-I Full Noise | Purkayastha et al. | Link | no | same as reporting paper | This is a modular two-stage neural architecture KGQA system which focuses on out of vocabulary problem. The first stage is called SPARQL silhouette which adapts a seq2seq model to generate SPARQL for the input natural question. The second stage is a neural graph search session whicch distill the output from the first stage by linking the relations through a BERT based architecture. In stage-I, there are three model training strategies, where noise in entity and relation linking is not injected, partly injected and fully injected. In stage three, there are two variant w/o type and w type, they are connected to type ontology classifier which has differnet strategy on dbpedia and yago ontology, however, the author is unclear in the article. Current setting is full noise setting in stage-I, without stage-II | Purkayastha et al. | ||
| SPARQL Silhouette Stage-I No Noise | Purkayastha et al. | Link | no | same as reporting paper | This is a modular two-stage neural architecture KGQA system which focuses on out of vocabulary problem. The first stage is called SPARQL silhouette which adapts a seq2seq model to generate SPARQL for the input natural question. The second stage is a neural graph search session whicch distill the output from the first stage by linking the relations through a BERT based architecture. In stage-I, there are three model training strategies, where noise in entity and relation linking is not injected, partly injected and fully injected. In stage three, there are two variant w/o type and w type, they are connected to type ontology classifier which has differnet strategy on dbpedia and yago ontology, however, the author is unclear in the article. Current setting is no noise setting in stage-I, without stage-II | Purkayastha et al. | ||
| SPARQL Silhouette Stage-I Part Noise | Purkayastha et al. | Link | no | same as reporting paper | This is a modular two-stage neural architecture KGQA system which focuses on out of vocabulary problem. The first stage is called SPARQL silhouette which adapts a seq2seq model to generate SPARQL for the input natural question. The second stage is a neural graph search session whicch distill the output from the first stage by linking the relations through a BERT based architecture. In stage-I, there are three model training strategies, where noise in entity and relation linking is not injected, partly injected and fully injected. In stage three, there are two variant w/o type and w type, they are connected to type ontology classifier which has differnet strategy on dbpedia and yago ontology, however, the author is unclear in the article. Current setting is part noise setting in stage-I and w/ type. | Purkayastha et al. | ||
| SPARQL Silhouette Stage-II w/ type | Purkayastha et al. | Link | no | same as reporting paper | This is a modular two-stage neural architecture KGQA system which focuses on out of vocabulary problem. The first stage is called SPARQL silhouette which adapts a seq2seq model to generate SPARQL for the input natural question. The second stage is a neural graph search session whicch distill the output from the first stage by linking the relations through a BERT based architecture. In stage-I, there are three model training strategies, where noise in entity and relation linking is not injected, partly injected and fully injected. In stage three, there are two variant w/o type and w type, they are connected to type ontology classifier which has differnet strategy on dbpedia and yago ontology, however, the author is unclear in the article. Current setting is full noise setting in stage-I and w/ type. | Purkayastha et al. | ||
| SPARQL Silhouette Stage-II w/o type | Purkayastha et al. | Link | no | same as reporting paper | This is a modular two-stage neural architecture KGQA system which focuses on out of vocabulary problem. The first stage is called SPARQL silhouette which adapts a seq2seq model to generate SPARQL for the input natural question. The second stage is a neural graph search session whicch distill the output from the first stage by linking the relations through a BERT based architecture. In stage-I, there are three model training strategies, where noise in entity and relation linking is not injected, partly injected and fully injected. In stage three, there are two variant w/o type and w type, they are connected to type ontology classifier which has differnet strategy on dbpedia and yago ontology, however, the author is unclear in the article. Current setting is full noise setting in stage-I and w/o type. | Purkayastha et al. | ||
| sparql-qa | M. Borroto et al. | Link | no | same as reporting paper | Sparql-qa aims at fixing the word out of vocabulary issue in semantic parsing module of KGQA. They design a special format to represent natural language to SPARQL, called QQT format. This format is used in the semantic parsing/translation module, which composes of neural machine translation and name entity recognition. Both are based on seq2seq structure. | M. Borroto et al. | ||
| sparql-qa | M. Borroto et al. | Link | no | same as reporting paper | To reduce the impact of the WOOV and improve the training time of the entire process, the authors introduce in sparql-qa some remedies, including a new format to represent an NL to SPARQL datasets. In particular, sparql-qa implements a neural-network-based architecture for question answering that accomplishes the objective by resorting to a novel combination of tools. The architecture is composed of three main modules: Input preparation, Translation, and Assembling. | M. Borroto et al. | ||
| STaG-QA | Ravishankar et al. | Link | no | same as reporting paper | Semantic parsing for Transfer and Generalization (STaG-QA) aims to facilitate generalization across knowledge bases. It supports working on multiple KGs and is easy to transfer on various datasets. The model is ran in two stages: 1. Softly-tied Query Sketch and 2. KG Interaction. The model can both be pre-trained on big KGQA dataset and trained on target dataset. This variant is trained on target dataset. | Ravishankar et al. | ||
| STaG-QA_pre | Ravishankar et al. | Link | no | same as reporting paper | Semantic parsing for Transfer and Generalization (STaG-QA) aims to facilitate generalization across knowledge bases. It supports working on multiple KGs and is easy to transfer on various datasets. The model is ran in two stages: 1. Softly-tied Query Sketch and 2. KG Interaction. The model can both be pre-trained on big KGQA dataset and trained on target dataset. This variant is trained on Lc-Quad2 dataset. | Ravishankar et al. | ||
| STAGG | Wu et al. | Link | no | Link | Staged query graph generation (STAGG) is a novel semantic parsing framework. The framework consists of three steps: 1. link topic entity from question to knowledge base 2. Identify Core Inferential Chain connected to the topic entity, where relation is matched using a convolutional neural network 3. augment constraint to the chain. Therefore STAGG is able to extend the query graph to represent more complicated graphs. | Yih et al. | ||
| SWIP | Diefenbach et al. | Link | no | only qald -3 | Link | In the SWIP system, the query interpretation process is made of two main steps: the translation of the NL user query into a pivot query, and the formalization of this pivot query. | Pradel et al. | |
| SYGMA | Neelam S et al. | Link | no | same as reporting paper | System for Generalizable and Modular question Answering over knowledge bases (SYGMA is built on a frame-work adaptable to different KB representations and reasoning types. The SYGMA achives the functionality by three modules: question understanding, question mapping and question mapping, among which the first is kb-agnostic and the rest are kb-specific. | Neelam S et al. | ||
| TeBaQA | L Siciliani et al. | Link | yes | Link | Link | TeBaQA learns to answer questions based on graph isomorphisms from basic graph patterns of SPARQL queries. Learning basic graph patterns is efficient due to the small number of possible patterns. This novel paradigm reduces the amount of training data necessary to achieve state-of-the-art performance. TeBaQA also speeds up the domain adaption process by transforming the QA system development task into a much smaller and easier data compilation task. | Vollmers et al . | |
| TeBaQA RNN | Athreya et. al | Link | yes | Link | Link | TeBaQA RNN is a recursive neural network based QA system. The RNN is used in template classification to replace the traditional query building process. Therefore, this model can generalize to any QA dataset and KG. The model is open source on github. | Athreya et al. | |
| TLDRet | Rahoman and Ichise | Link | no | only on QALD-2 | same as reporting paper | Authors propose a keyword-based linked data information retrieval framework that can incorporate temporal features and give more concise results. | Rahoman and Ichise | |
| UTQA | Diefenbach et al. | Link | no | only tested on qald-5 | Link | Authors introduce a new cross-lingual approach using a unified semantic space among languages. After keyword extraction, entity linking and answer type detection, they use cross lingual semantic similarity to extract the answer from knowledge base via relation selection and type matching. Evaluation is performed on Persian and Spanish which are typologically different languages. | Veyseh et al. | |
| virtual player | Molino et al. | Link | no | only tested on qald-3 | same as reporting paper | This paper describes the techniques used to build a virtual player for the popular TV game “Who Wants to Be a Millionaire?”. The player must answer a series of multiple-choice questions posed in natural language by selecting the correct answer among four different choices. The architecture of the virtual player consists of 1) a Question Answering (QA) module, which leverages Wikipedia and DBpedia datasources to retrieve the most relevant passages of text useful to identify the correct answer to a question, 2) an Answer Scoring (AS) module, which assigns a score to each candidate answer according to different criteria based on the passages of text retrieved by the Question Answering module, and 3) a Decision Making (DM) module, which chooses the strategy for playing the game according to specific rules as well as to the scores assigned to the candidate answers. | Molino et al. | |
| WDAqua-core0 | Ravishankar et al. | Link | not working | Link | Link | WDAqua-core0 supports answering English questions on DBpedia and 4 different language over Wikidata, namely English, French, German and Italian. The detail of the system is emitted since the autho states 'The full details will be disclosed in an upcoming publication as this is only a challenge submission.'. Since the link to this model's demo has the same link as WDAqua-core1, there might be a chance that this system is actually the same as core-1. However, the demo link is not working, making it hard to investigate. | Diefenbach et al. | |
| WDAqua-core1 | Omar et al. | Link | not working | Link, demo: Link | Link | WDQqua-core1 is one of the few QA system which are running as web-services. The model is a pipeline model which aims to convert question to SPARQL query. The model includes following sessions: Query Expansion, Query Con- struction, Query Ranking and Answer Decision. The service suppoprts multilingual question, natural question as well as key word questions and is integrated into Qanary framwork. | Diefenbach et al. | |
| WolframAlpha | Walter et al. | Link | yes | only tested on qald-2 | Link | Link | WolframAlpha is an engine for computing answers and providing knowledge. It combines curated knowledge, linguistic analysis and dynamic computation. | Only website available m |
| Xser | Diefenbach et al. | Link | no | only tested on qald 4 & 5 | Link | Authors present a question answering system (Xser) over Linked Data(DBpedia), converting users’ natural language questions into structured queries. There are two challenges involved: recognizing users’ query intention and mapping the involved semantic items against a given knowledge base (KB), which will be in turn assembled into a structured query. In this paper, we propose an efficient pipeline framework to model a user’s query intention as a phrase level dependency DAG which is then instantiated according to a given KB to construct the final structured query. | Xu et al. | |
| YodaQA | Diefenbach et al. | Link | not working | only tested on qald-5 | Link, webservice Link not working | Link | YodaQA is an pipeline factoid question answering system that can produce answer both from knowledge base and unstructured text. The model answers question in following steps: Question Analysis, Answer Production, Answer Analysis and Answer Merging and Scoring. In the answer production part, the model has different strategy on knolwegde base and on corpora. | Petr Baudiˇs and Jan Sˇedivy ́ |
| Yu et al. | Wu et al. | Link | no | same as reporting paper | This model is semantic parsing based KBQA system, where the parsing section is a relational graph convolutional network (RGCN) called gRGCN. gRGCN combines RGCN and hierarchical relation attention mechanism. Therefore, it is capable of exracting both structure semantics and relational semantics from the question and generate corresponsing seach queries. | Wu et al. | ||
| Zhang et. al. | Zhang et. al. | Link | no | same as reporting paper | This paper presents a joint model based on integer linear programming (ILP), uniting alignment construction and query construction into a uniform framework. As a result, the model is able to outperform pipeline model with train the two sectors seperately and be robot to noise propogation. | Zhang et. al. | ||
| Zhu et al. | Zhu et al. | Link | no | no | same as reporting paper | Focusing on solving the non-aggregation questions, in this paper, authors construct a subgraph of the knowledge base from the detected entities and propose a graph traversal method to solve both the semantic item mapping problem and the disambiguation problem in a joint way. Compared with existing work, they simplify the process of query intention understanding and pay more attention to the answer path ranking. | Zhu et al. | |
| Zou et al. + Bert | Zou et al. | Link | no | same as reporting paper | This model shares the structure with the one above, however the input for encoder is initialized on BERT (Link). | Zou et al. | ||
| Zou et al. + Tencent Word | Zou et al. | Link | no | same as reporting paper | This work presents an end-to-end text-to-SPARQL baseline model, which is composed of a relation-aware attention encoder and multi-types pointer network decoder. The encoder takes natural question, entities and relation as input, which is initialized with a 200-dimensional Chinese word embedding trained by Link The decoder is a Long Short Term Memory (LSTM) with attention to generate SPARQL queries by incorporating the representation of entities, relations and SPARQL keywords. Therefore the model is believed to be effective in answering multi-type complex questions, such as factual questions, dual intent ques- tions, boolean questions, etc. | Zou et al. | ||
| SSKGQA | Mingchen Li and Jonathan Shihao Ji | Link | yes | Link | same as reporting paper | This work presents SSKGQA, a semantic structure based method to predict query graphs from natural language questions. Beside, the paper also develops a novel Structure-BERT to predict the semantic structure of each question, and a BERT-based ranking model with a triplet loss to identify the top-1 query graph candidate. And the model achieves outstanding performance WSP and MetaQA. | Mingchen Li and Jonathan Shihao Ji | |
| T5-Base | Banerjee et al. | Link | no | no | same as reporting paper | This work applys pre-trained langage models (BART, T5 and PGNs (Pointer Generator Networks)) in semantic parsing task, for KGQA. More sepcifically, T5-Small, T5-Base, BART, PGN-BERT-BERT and PGN-BERT are tested on two benchmark datasets. T5 requires special input tokenisation, but produces state of the art performance on LC-QuAD 1.0 and LC-QuAD 2.0 datasets, and outperforms task-specific models from previous works. | Banerjee et al. | |
| T5-Small | Banerjee et al. | Link | no | no | same as reporting paper | This work applys pre-trained langage models (BART, T5 and PGNs (Pointer Generator Networks)) in semantic parsing task, for KGQA. More sepcifically, T5-Small, T5-Base, BART, PGN-BERT-BERT and PGN-BERT are tested on two benchmark datasets. T5 requires special input tokenisation, but produces state of the art performance on LC-QuAD 1.0 and LC-QuAD 2.0 datasets, and outperforms task-specific models from previous works. | Banerjee et al. | |
| BART | Banerjee et al. | Link | no | no | same as reporting paper | This work applys pre-trained langage models (BART, T5 and PGNs (Pointer Generator Networks)) in semantic parsing task, for KGQA. More sepcifically, T5-Small, T5-Base, BART, PGN-BERT-BERT and PGN-BERT are tested on two benchmark datasets. The Bidirectional and Auto-Regressive Transformer or BART is a Transformer that combines the Bidirectional Encoder with an Autoregressive decoder into one Seq2Seq model. The model achieves outstanding performance on two benchmark datasets. | Banerjee et al. | |
| PGN-BERT | Banerjee et al. | Link | no | no | same as reporting paper | This work applys pre-trained langage models (BART, T5 and PGNs (Pointer Generator Networks)) in semantic parsing task, for KGQA. More sepcifically, T5-Small, T5-Base, BART, PGN-BERT-BERT and PGN-BERT are tested on two benchmark datasets. PGN is a seq2seq network with an encoder and a decoder block. And this work experiments exclusively with LSTM-based PGNs. Under PGN-BERT, author uses the setting where BERT embeddings is fed to PGN. Also there is no re-ranker to the output beams, and the author consider the first query in the ranked beams to produce a valid response from the KG as the output of the system. | Banerjee et al. | |
| PGN-BERT-BERT | Banerjee et al. | Link | no | no | same as reporting paper | This work applys pre-trained langage models (BART, T5 and PGNs (Pointer Generator Networks)) in semantic parsing task, for KGQA. More sepcifically, T5-Small, T5-Base, BART, PGN-BERT-BERT and PGN-BERT are tested on two benchmark datasets. PGN is a seq2seq network with an encoder and a decoder block. And this work experiments exclusively with LSTM-based PGNs. PGN-BERT-BERT refers to the setting where the beams are further re-ranked with the BERT-based re-ranker described previously. | Banerjee et al. | |
| Reranking | Yonghui Jia and Wenliang Chen | Link | no | no | same as reporting paper | This paper proposes a BERT-based ranking-reranking strategy for query graph selection for KGQA. After ranking the candidate query with BERT-based semantic matching model, this work utilize answer types and query candidates that are produced in the ranking stage to re-rank the queries. The test result on CompQuestion and WebQuestion proves the effect of the additional re-ranking module. | Yonghui Jia and Wenliang Chen | |
| SGM | Ma L et al. | Link | no | no | same as reporting paper | This paper aims at enhancing semantic parsing by combining syntax information. They propose a syntax-based graph matching method which model both questions and logic graph, and match on both word and structure level. The model is tested on two benchmark dataset and outperform former models. | Ma L et al. | |
| Text2Graph + MEKER Wikidata5m | Chekalina et al. | Link | no | no | same as reporting paper | This paper creates a Text2Graph system for, which integrates the knowledge graph embedding (KGE) trained on a novel model MEKER, Memory Efficient Knowledge Embedding Representation for Link Prediction and Question Answering. The KGE model is trained on Wikidata4m and Wikidata5m respectively for RuBQ 2.0 benchmark and SimpleQuestions benchmark dataset. The test results on two benchmarks outperform the former knowledge graph question answering models. | Chekalina et al. | |
| GSM | Liu et al. | Link | no | no | same as reporting paper | This paper proposes GSM, an end-to-end question answering model to learn both structural and relational representations of query graphs. The authors design a global semantic model to explicitly encode the structural and relational semantics of query graphs and a question-guiding mechanism to enhance the understanding of question semantics in query graph representations. The model achieves high performance on LC-QuAD 1 and QALD-7. | Liu et al. | |
| FOFE-net | Jiang et al. | Link | no | no | based on two papers: Zhang et al., 2015 and Xu et al., 2017 | FOFE-net is a model proposed in the paper which publishes feebaseQA, as a baseline system. The system is based on two papers: Zhang et al., 2015 and Xu et al., 2017 andconsists of subject detection, entity linking and relation detection in the pipeline. The model achieves 37.0% accuracy in freebaseQA dataset. | Jiang et al. | |
| XLM-R Large Translate Train | Longpre et al. | Link | yes | https://github.com/apple/ml-mkqa | https://arxiv.org/pdf/1911.02116.pdf | XLM-R is a model proposed in the paper which publishes MKQA, as a baseline system. The system is a cross-lingual language model incorporating multilingual knowledge in pre-training. XLM-K augments existing multilingual pre-training with two knowledge tasks, namely Masked Entity Prediction Task and Object Entailment Task. | Longpre et al. | |
| AlAgha, 2022 | AlAgha, 2022 | Link | no | no | https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9834917 | This question answering system is proposed by answering multi-hop questions through leveraging sequential matching and backtracing algorithm to identiy minumum path while minimizing accumulated error. The model achieves outstanding performnace in a few datasets. | AlAgha, 2022 | |
| SPARQL Generator | Lin and Lu | Link | no | no | https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9834917 | SPARQL Generator is a generative neural machine translation based method, which utilizes transformer model. The model is evaluated on LQ-Quad 1, qald 7 and qald 8, on BLEU and accurcy metrics. The model shows good performance on above datasets. | AlAgha, 2022 | |
| Entity Type Tags Modified | Lin and Lu | Link | no | no | https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9834917 | Entity Type Tags Modified is based on SPARQL Generator model which is proposed in the same paper. The modified model adds entity type information which is recognized by a entity type tagger on input question. The model is evaluated on LQ-Quad 1, qald 7 and qald 8, on BLEU and accurcy metrics. The modified version shows increase on test score compared to SPARQL Generator model. | AlAgha, 2022 | |
| KGQA Based on Query Path Generation | Yang et al. | Link | no | no | https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12#Sec6 | KGQA Based on Query Path Generation is a staged query path generation method. The approach firstly takes the predicate sequence of the question in knowledge graphs as the breakthrough and constructs the core path, with the constraint learnt on the question. The final answer to the question is determined through the query path. The model is tested on webquestionSP and MetaQA dataset and achieve competitive performance on some multi-hop subsets. | Yang et al. | |
| KBIGER | Du et al. | Link | no | no | https://arxiv.org/pdf/2209.03005.pdf | KBIGER (Knowledge Base Iterative Instruction GEnerating and Reasoning) generates the instructions dynamically with the help of reasoning graph. The model firstly encode the question and aggregated question graph, then send the embedded information to a graph reasoning modele. The outcome is used to instruct retrieval of the target entity from the knowledge graph. The model is tested on wo multi-hop KBQA benchmarks and achives outstanding performances. | Du et al. | |
| QAWizard | Chen et al. | Link | no | no | https://ieeexplore.ieee.org/abstract/document/9627128/ | QAWizard answers input questions through two stages of Maximum Entropy Markov Model (MEMM) training, including an entity type tagging stage and an RDF type tagging stage. Seven predefined templates are used to generate SPARQL queries based on the results of the two stages. | Chen et al. | |
| Light-QAWizard | Chen et al. | Link | no | no | https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9893129 | Light-QAWizard is a low-cost SPARQL generator, which integrates multi-label classification into a recurrent neural network (RNN), builds a template classifier, and generates corresponding query gramars based on the results of template classifier. | Chen et al. | |
| NRQA | Guo et al. | Link | no | https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf | NRQA is an effective reasoning model that fuses neighbor interaction and a relation recognition model for multi-hop QA. | Guo et al. | ||
| DECAF | Yu et al. | Link | no | no | Link | DECAF is a framework which aims to combine semantic parsing based and direct answer extraction based kgqa method tofether. The model firstly linearlize the kg into passages, retrieve the most relevant passages based on the question and fuse the question with the chosen passages with a fuison-encoder and decoder. The output from direct answer based decoder and logic form based decoder are ranked with a special hyperparameter. This work achieves SOTA result on a few benchmark datasets. | Yu et al. | |
| TwiRGCN | Sharma et al. | Link | no | no | Link | TwiRGCN is a framework which applys relational graph convolutional networks to answer temporal questions over knowledge graph. A novelty about the model is the gated information which decides whether the answer is a KG entity or a time. The model achives state-of-the-art performance on a complex temporal question answering dataset. | Sharma et al. | |
| TERP | Qiao et al. | Link | no | no | Link | TERP is a method that exploits hybrid features of the off-the-shelf relation paths for KGQA. It fuses explicit textual information and implicit structural features via a question-aware manner. By projecting a natural language question as well as the obtained hybrid features of candidate relation paths into a novel rotate-and-scale entity link prediction framework, question and relation paths are effectively coordinated to select the answer entity. | Qiao et al. | |
| Edge-aware GNN | Zhang et al. | Link | no | no | Link | Edge-aware GNN is dedsigned for multi-hop path questions. The systems firstly extract sub-graph from knowledge graph and add nodes (which contains the information from question) to the sub-graph. An attention mechanism is then applied to to weight the information from the neighbouring nodes before finding the right path. The model is evaluated on two datasets, PQL and metaQA and achieves leading scores on two benchmark datasets. | Zhang et al. | |
| REAREV | Costas Mavromatis and George Karypis | Link | yes | https://github.com/cmavro/ReaRev_KGQA | Link | REAREV decodes the question into instruction, which is applied to a GNN for reasoning. This step is repeated to enhance the decoding of the instruction before retrieving the correct question. The model is tested on 2 benchmark datasets, Web question SP and complex web questions. The performance excels former state-of-the-art models. | Costas Mavromatis and George Karypis | |
| GMT-KBQA | Hu et al. | Link | yes | https://github.com/HXX97/GMT-KBQA | Link | GMT-KBQA is a generation based KGQA system which utilizes multi-task learning method. After rretrieving the entities and relations from the question, the model disambiguates the entities, classifies the relation and generate the logical form in a multi-task learning manner. GMT-KGQA is tested against two benchmark datasets on freebase, complex web questions and wen question sq and shows outstanding yet not state-of-the-art performance. | Hu et al. | |
| MRP- QA | Wang et al. | Link | no | - | Link | Mutliple Reasoning Paths KBQA System (MRP- QA) aims to utilize the multiple possible paths in kb to enhance reasoning capacity of the knowlegde graph question answering model. The model fuses the topic entity at different step, question vector based on attention weight, information from last step to predict the relation. This process is repeated until the model predicts a terminate sign. The model is tested on 3 popular kgqa datasets on freebase and achieves outstanding results. | Wang et al. | |
| ARN | Cui et al. | Link | no | - | Link | ARN is an interpretable model based on RL for multi-hop KGQA with weak supervisionwhich incorporates KG embeddings as anticipation information into RL framework to avoid aimlessly traversing in KG. | Cui et al. | |
| ValueNet4SPARQL | Kosten et al. | Link | no | no | same as reporting paper | ValueNet4SPARQL is an end-to-end framework to convert natural language questions into SPARQL. The speciality of this model is three-fold: 1. theu utilize large scale pre-trained language models without further training samples. 2. the model acess the KG layer based on past examples and work flow. 3. A special mechanism to map keywords in natural language to that in SPARQL. The system achieves good performance on LC-QuAD 1.0 dataset. | Kosten et al. | |
| TMS | Liu et al. | Link | no | no | same as reporting paper | Time-aware Multiway Adaptive (TMA) fusion network is inspired by step-by-step reasoning process of human. TMA firstly extracts the relevant entity from the given question, link the entity to the temporal KG, and then feeds them into a multiway adaptive module to produce a temporal-specific representation of the question. The system achieves good performance on CronQuestions dataset. | Liu et al. | |
| GETT-QA | Banerjee et al. | Link | no | no | same as reporting paper | Architecture of GETT-QA: T5 generates a skeleton SPARQL query with entity and relation labels and a truncated KG embedding string. The entity labels are grounded using label based search and embedding based sorting, while the relations are grounded using BERT embedding based sorting. The final query is executed against a KG to fetch the answer. | Banerjee et al. | |
| Templet | Suárez and Hogan | Link | yes | Link | same as reporting paper | Templet is an online QA system for Wikidata. Templet is based on the collaboratively-edited repository QAWiki, which collects questions in multiple natural languages along with their corresponding structured queries. Templet generates templates from question–query pairs on QAWiki by replacing key entities with identifiers. Using autocompletion, the user can type a question in natural language, select a template, and again using autocompletion, select the entities they wish to insert into the template’s placeholders, generating a concrete question, query and results. | Suárez and Hogan | |
| KGQAn | Omar et al. | Link | no | no | same as reporting paper | KGQAn introduces a novel formalization of question understanding as a text generation problem to convert a question into an intermediate abstract representation via a neural sequence-to-sequence model. We also develop a just-in-time linker that maps at query time the abstract representation to a SPARQL query for a specific KG, using only the publicly accessible APIs and the existing indices of the RDF store, without requiring any pre-processing. | Omar et al. | |
| chatGPT | Tan et al. | Link | yes | https://github.com/tan92hl/Complex-Question-Answering- Evaluation-of-ChatGPT | official page paper | ChatGPT is a variant of GPT-3.5, similar to the GPT-3.5 v3, which is fine-tuned through instruction tuning with reinforcement learning from human feedback (RLHF) based on human feedback. ChatGPT is tested on a few benchmark KGQA datasets and show outstanding performance in most of them. | Tan et al. | |
| GPT-3 | Tan et al. | Link | yes | https://github.com/tan92hl/Complex-Question-Answering- Evaluation-of-ChatGPT | official page paper | GPT-3 is a family of autoregressive Transformer language models trained on 570GB of Internet text (WebText). GPT-3 is tested on a few benchmark KGQA datasets and show outstanding performance in most of them. | Tan et al. | |
| GPT-3.5 | Tan et al. | Link | yes | https://github.com/tan92hl/Complex-Question-Answering- Evaluation-of-ChatGPT | official page paper | GPT-3.5 v2 is trained usng both text and code, which enhances its capacity to understand coding language and zero-shot capacity. In comparison, GPT-3.5 v3 is a variant that employs instruction tuning with reinforcement learning from human feedback, similar to ChatGPT, but with potentially stronger contextual learning ability. Both models are tested on a few benchmark KGQA datasets and show outstanding performance in most of them. | Tan et al. | |
| FLAN-T5 | Tan et al. | Link | yes | https://github.com/tan92hl/Complex-Question-Answering- Evaluation-of-ChatGPT | official page paper | (Text-to-Text Transfer Transformer, 11B)is Transformer-based language model built on encoder-decoder architecture. It is trained on a filtered variant of CommonCrawl (C4). It is trained on a mix of supervised and unsupervised tasks, which are all changed to a prompt-based input. | Tan et al. | |
| MuHeQA | Badenes-Olmedo and Corcho | Link | yes | Link | same as reporting paper | The MuHeQA (Multiple and Heterogeneous Question-Answering) system creates natural language answers from natural language questions using knowledge base from both structured (KG) and unstructured (documents) data sources. | Badenes-Olmedo and Corcho | |
| Ensemble BR framework | Chen et al. | Link | no | no | same as reporting paper | Two multi-label learning methods are used for question transformation: Binary Relevance (BR) and Classifier Chains (CC). To predict all the labels that match the query intentions, we use Recurrent Neural Networks (RNNs) to build a multilabel classifier for generating RDF triples. To better consider the relationship between RDF triples, the Binary Relevance is integrated into an ensemble learning approach to propose an Ensemble BR. | Chen et al. | |
| TIARA | Shu et al. | Link | yes | https://github.com/microsoft/KC/tree/main/papers/TIARA | same as reporting paper | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | |
| MACRE | Xu et al. | Link | no | same as reporting paper | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | ||
| KGQAcl/rr | Hu et al. | Link | yes | Link | same as reporting paper | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | |
| AE-TQ | Long et al. | Link | no | same as reporting paper | AE-TQ leverages abstract meaning representation (AMR) for understanding complex questions, and produces question-oriented semantic information for explicit and effective temporal reasoning. A question is parsed into a semantic information structure (SISs), and for each SIS with unknown elements, the TKG is queried to obtain candidate answers, which are then reasoned together with the temporal constraints in pursuit of the final answer. | Long et al. |