From c156a6eb651fac60de18ef2ce467d9479322a35e Mon Sep 17 00:00:00 2001 From: David Rath Date: Sun, 29 Oct 2023 15:47:02 +0100 Subject: [PATCH] Added FRED to systems.md and WQSP --- freebase/WebQuestionsSP.md | 189 +++++++++++++++++++------------------ systems.md | 1 + 2 files changed, 96 insertions(+), 94 deletions(-) diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index 62f3b19f..adfc4211 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -3,97 +3,98 @@ datasetUrl: https://www.microsoft.com/en-us/download/details.aspx?id=52763 --- -| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | -| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------------: | -| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| DECAF (BM25 + FiD-large) | 2022 | 74.9 ± 0.3 | 79.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | -| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | -| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | -| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | -| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | -| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | -| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | -| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | -| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | -| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | +| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :---------------------------------------------------------------------------------------: | +| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| FRED | 2023 | 0.86 ± 0.05| - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | +| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| DECAF (BM25 + FiD-large) | 2022 | 74.9 ± 0.3 | 79.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | +| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | +| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | +| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | +| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | +| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | +| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | +| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | +| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | +| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | diff --git a/systems.md b/systems.md index 39eed033..0176ace0 100644 --- a/systems.md +++ b/systems.md @@ -139,3 +139,4 @@ | TIARA | Shu et al. | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | yes | [Link](https://github.com/microsoft/KC/tree/main/papers/TIARA) | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | +| FRED | Lamott et al. | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | no | - | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | Fred combines pattern-based entity retrieval with a transformer-based question encoder. The method uses an evolutionary approach to learn SPARQL patterns, which retrieve candidate entities from a knowledge base. The transformer-based regressor is then trained to estimate each pattern’s expected F1 score for answering the question, resulting in a ranking of candidate entities. Unlike other approaches, FRED can attribute results to learned SPARQL patterns, making them more interpretable. | Lamott et al. |