diff --git a/dbpedia/QALD-9-Plus-DBpedia.md b/dbpedia/QALD-9-Plus-DBpedia.md index 6b7b5a1c..c423e5d0 100644 --- a/dbpedia/QALD-9-Plus-DBpedia.md +++ b/dbpedia/QALD-9-Plus-DBpedia.md @@ -4,21 +4,24 @@ --- -| Model / System | Year |Precision|Recall| F1 |Language| Reported by | -|:--------------:|:----:|:-------:|:----:|:---:|:------:|:------------------------------------------------------------------------------:| -| QAnswer | 2022 | - |- |30.39 (Macro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271) | -| QAnswer | 2022 | - |- |19.98 (Macro F1)| DE | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| QAnswer | 2022 | - |- |15.06 (Macro F1)| FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| QAnswer | 2022 | - |- |9.57 (Macro F1)| RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| QAnswer | 2022 | - |- |5.27 (Micro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271) | -| QAnswer | 2022 | - |- |2.19 (Micro F1)| DE | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| QAnswer | 2022 | - |- |4.06 (Micro F1)| FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| QAnswer | 2022 | - |- |1.53 (Micro F1)| RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| DeepPavlov | 2022 | - |- |12.40 (Macro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| DeepPavlov | 2022 | - |- |0.13 (Micro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| Platypus | 2022 | - |- |15.03 (Macro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| Platypus | 2022 | - |- |1.26 (Micro F1)| EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| DeepPavlov | 2022 | - |- |8.70 (Macro F1)| RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| DeepPavlov | 2022 | - |- |0.05 (Micro F1)| RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| Platypus | 2022 | - |- |4.17 (Macro F1)| FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| -| Platypus | 2022 | - |- |0.00 (Micro F1)| FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Model / System | Year |Precision|Recall| F1 | Language | Reported by | +|:------------------------:|:----:|:-------:|:----:|:----------------:|:--------:|:------------------------------------------------------------------------------:| +| QAnswer | 2022 | - |- | 30.39 (Macro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271) | +| QAnswer | 2022 | - |- | 19.98 (Macro F1) | DE | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| QAnswer | 2022 | - |- | 15.06 (Macro F1) | FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| QAnswer | 2022 | - |- | 9.57 (Macro F1) | RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| QAnswer | 2022 | - |- | 5.27 (Micro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271) | +| QAnswer | 2022 | - |- | 2.19 (Micro F1) | DE | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| QAnswer | 2022 | - |- | 4.06 (Micro F1) | FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| QAnswer | 2022 | - |- | 1.53 (Micro F1) | RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| DeepPavlov | 2022 | - |- | 12.40 (Macro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| DeepPavlov | 2022 | - |- | 0.13 (Micro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Platypus | 2022 | - |- | 15.03 (Macro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Platypus | 2022 | - |- | 1.26 (Micro F1) | EN | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| DeepPavlov | 2022 | - |- | 8.70 (Macro F1) | RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| DeepPavlov | 2022 | - |- | 0.05 (Micro F1) | RU | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Platypus | 2022 | - |- | 4.17 (Macro F1) | FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Platypus | 2022 | - |- | 0.00 (Micro F1) | FR | [Perevalov et. al.](https://ieeexplore.ieee.org/document/9736271)| +| Lingua Franca (LIBRE MT) | 2023 | - |- | 27.1 (Macro F1) | DE | [Srivastava et. al.](https://dl.acm.org/doi/pdf/10.1145/3587259.3627567)| +| Lingua Franca (LIBRE MT) | 2023 | - |- | 38.6 (Macro F1) | FR | [Srivastava et. al.](https://dl.acm.org/doi/pdf/10.1145/3587259.3627567)| +| Lingua Franca (OPUS MT) | 2023 | - |- | 26.0 (Macro F1) | EN | [Srivastava et. al.](https://dl.acm.org/doi/pdf/10.1145/3587259.3627567)| diff --git a/freebase/WebQuestions.md b/freebase/WebQuestions.md index b1d640fc..442054fd 100644 --- a/freebase/WebQuestions.md +++ b/freebase/WebQuestions.md @@ -3,22 +3,25 @@ datasetUrl: https://worksheets.codalab.org/worksheets/0xba659fe363cb46e7a505c5b6a774dc8a --- -| Model / System | Year |Precision| Recall | F1 |Language| Reported by | -|:-----------------:|:----:|:-------:|:------:|:----:|:------:|:---------------------------------------------------------------------:| -| Reranking | 2022 | - | - | 56.0 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Jain [1] | 2016 | - | - | 55.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Xu et al. [2] | 2019 | - | - | 54.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Hu et al. [3] | 2018 | - | - | 53.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Wang et al. | 2022 | - | - | 53.2 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Luo et al. [4] | 2018 | - | - | 52.7 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Yih et al. [5] | 2015 | - | - | 52.5 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Bao et al. [6] | 2016 | - | - | 52.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Ranking | 2022 | - | - | 52.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Zhu et al. | 2022 | - | - | 52.1 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Chen et al. [7] | 2015 | - | - | 51.8 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | -| Abujabal et al. | 2022 | - | - | 51.0 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Berant et al. | 2022 | - | - | 49.7 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Yao et al. | 2022 | - | - | 44.3 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Dong et al. | 2022 | - | - | 40.8 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | -| Berant et al. [8] | 2013 | - | - | 36.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Model / System | Year | Precision | Recall | F1 |Language| Reported by | +|:-------------------------------:|:----:|:---------:|:------:|:-----:|:------:|:-----------------------------------------------------------------------------:| +| Reranking | 2022 | - | - | 56.0 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Jain [1] | 2016 | - | - | 55.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Xu et al. [2] | 2019 | - | - | 54.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Hu et al. [3] | 2018 | - | - | 53.6 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Wang et al. | 2022 | - | - | 53.2 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Luo et al. [4] | 2018 | - | - | 52.7 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Yih et al. [5] | 2015 | - | - | 52.5 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Bao et al. [6] | 2016 | - | - | 52.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Ranking | 2022 | - | - | 52.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Zhu et al. | 2022 | - | - | 52.1 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Chen et al. [7] | 2015 | - | - | 51.8 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Abujabal et al. | 2022 | - | - | 51.0 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Berant et al. | 2022 | - | - | 49.7 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Yao et al. | 2022 | - | - | 44.3 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Dong et al. | 2022 | - | - | 40.8 | EN | [Wang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_15) | +| Berant et al. [8] | 2013 | - | - | 36.4 | EN | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | +| Sen et al. (Flan-T5 11B ) | 2023 | - | - | 59.79 | EN | [Flan et al.](https://aclanthology.org/2023.nlrse-1.1.pdf) | +| Retrieve-Rewrite-Answer (T0 3B) | 2023 | - | - | 74.02 | EN | [Wu et al.](https://arxiv.org/pdf/2309.11206) | + diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index 0b99b378..899d8eb1 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -3,99 +3,104 @@ datasetUrl: https://www.microsoft.com/en-us/download/details.aspx?id=52763 --- -| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | -| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------------: | -| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | +|:-----------------------------------:|:----:|:---------:| :--------: |:--------:|:--------:|:-------------------------------------------------------------------------------------:| +| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | | DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| TERP | 2022 | - | 76.8 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | -| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| TERP | 2022 | - | 76.8 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | +| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | DECAF (BM25 + FiD-large) | 2022 | 74.9 ± 0.3 | 79.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| W. Han et al. | 2023 | - | 75.2 | - | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | -| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | -| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | -| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | -| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | -| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | -| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | -| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | -| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | -| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | -| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| W. Han et al. | 2023 | - | 75.2 | - | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | +| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | +| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | +| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | +| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | +| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | +| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | +| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | +| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | +| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | +| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KAPING (GPT-3 175B) | 2023 | | | 73.89 | EN | [Baek et al.](https://arxiv.org/pdf/2306.04136) | +| KAPING (AlexaTM 20B) | 2023 | | | 67.67 | EN | [Baek et al.](https://arxiv.org/pdf/2306.04136) | +| KAPING (T5 11B) | 2023 | | | 24.91 | EN | [Baek et al.](https://arxiv.org/pdf/2306.04136) | +| Retrieve-Rewrite-Answer (T5 11B) | 2023 | | | 79.36 | EN | [Wu et al.](https://arxiv.org/pdf/2309.11206) | + diff --git a/frontend/static/entries.txt b/frontend/static/entries.txt index 142566ff..429ff747 100644 --- a/frontend/static/entries.txt +++ b/frontend/static/entries.txt @@ -52,6 +52,8 @@ /datasets/wikidata/TimeQuestions - Ordinal /datasets/wikidata/TimeQuestions - Overall /datasets/wikidata/TKGQA +/datasets/wikidata/MarkQA +/datasets/wikidata/WebQuestionsSP /datasets/other/ /datasets/other/COVID-QA /datasets/other/CrunchQA @@ -64,4 +66,5 @@ /datasets/other/WC2014QA - 2 Hop /datasets/other/WC2014QA - Total /datasets/other/Spider4SPARQL +/datasets/other/ZJQA /systems/ diff --git a/other/MetaQA - 2 Hop.md b/other/MetaQA - 2 Hop.md index ebc5fa1d..bdd20856 100644 --- a/other/MetaQA - 2 Hop.md +++ b/other/MetaQA - 2 Hop.md @@ -3,45 +3,46 @@ datasetUrl: https://github.com/yuyuz/MetaQA --- -| Model / System | Year | Hits@1 | F1 | Exact Match | Language | Reported by | -|:-----------------------------------:|:----:|:------:|:----:|:-----------:|:--------:|:----------------------------------------------------------------------------------:| -| KGQA Based on Query Path Generation | 2022 | 99.9 | - | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | -| PullNet | 2019 | 99.9 | - | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | -| DCRN | 2022 | 99.9 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| NSM | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| NSM+p | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| NSM+h | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| QAGCN | 2022 | 99.9 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | -| QNRKGQA | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| QNRKGQA+h | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| SSKGQA | 2022 | 99.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| TERP | 2022 | 99.4 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | -| EmbedKGQA | 2020 | 98.8 | - | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | -| TransferNet | 2022 | 98.5 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| NRQA | 2022 | 97.5 | - | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | -| Edge-aware GNN | 2022 | 96.8 | 93.7 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| GlobalGraph | 2022 | 95.5 | 83.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| 2HR-DR | 2022 | 93.7 | 81.4 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| SGReader | 2022 | 80.7 | 79.8 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| GraftNet | 2022 | 94.8 | 72.7 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| ReifKB+mask | 2020 | 95.4 | - | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | -| SRN | 2020 | 95.1 | - | - | EN | [Qiu et al.](https://dl.acm.org/doi/10.1145/3336191.3371812) | -| ARN_ComplEx | 2023 | 94.92 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| GraftNet | 2018 | 94.8 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| AlAgha, 2022 | 2022 | 94.1 | - | - | EN | [AlAgha, 2022](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9834917) | -| ARN_ConvE | 2023 | 93.59 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| ARN_TuckER | 2023 | 93.20 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| MINERVA | 2022 | 92.9 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | -| ARN_DistMult | 2023 | 92.54 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| VRN | 2017 | 89.9 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | -| VRN | 2022 | 89.2 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| KV-MemNN | 2022 | 82.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| Borders et al. 's QA system | 2017 | 81.8 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | -| ReifKB | 2020 | 81.1 | - | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | -| KV-MemNN | 2022 | 76.0 | - | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | -| IRN | 2022 | 71.3 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | -| Supervised embedding | 2017 | 29.1 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | -| KV-MemNN | 2017 | 25.1 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | -| T5+KG | 2022 | - | - | 33.57 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | -| T5+C4 | 2022 | - | - | 32.78 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | -| T5 | 2022 | - | - | 32.65 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | +| Model / System | Year | Hits@1 | F1 | Exact Match | Language | Reported by | +|:----------------------------------------------:|:----:|:-------:|:-------:|:-----------:|:--------:|:----------------------------------------------------------------------------------:| +| KGQA Based on Query Path Generation | 2022 | 99.9 | - | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | +| PullNet | 2019 | 99.9 | - | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | +| DCRN | 2022 | 99.9 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| NSM | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| NSM+p | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| NSM+h | 2021 | 99.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| QAGCN | 2022 | 99.9 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | +| QNRKGQA | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| QNRKGQA+h | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| SSKGQA | 2022 | 99.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| TERP | 2022 | 99.4 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | +| EmbedKGQA | 2020 | 98.8 | - | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | +| TransferNet | 2022 | 98.5 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| NRQA | 2022 | 97.5 | - | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | +| Edge-aware GNN | 2022 | 96.8 | 93.7 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| GlobalGraph | 2022 | 95.5 | 83.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| 2HR-DR | 2022 | 93.7 | 81.4 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| SGReader | 2022 | 80.7 | 79.8 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| GraftNet | 2022 | 94.8 | 72.7 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| ReifKB+mask | 2020 | 95.4 | - | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | +| SRN | 2020 | 95.1 | - | - | EN | [Qiu et al.](https://dl.acm.org/doi/10.1145/3336191.3371812) | +| ARN_ComplEx | 2023 | 94.92 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| GraftNet | 2018 | 94.8 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| AlAgha, 2022 | 2022 | 94.1 | - | - | EN | [AlAgha, 2022](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9834917) | +| ARN_ConvE | 2023 | 93.59 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| ARN_TuckER | 2023 | 93.20 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| MINERVA | 2022 | 92.9 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | +| ARN_DistMult | 2023 | 92.54 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| VRN | 2017 | 89.9 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | +| VRN | 2022 | 89.2 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| KV-MemNN | 2022 | 82.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| Borders et al. 's QA system | 2017 | 81.8 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | +| ReifKB | 2020 | 81.1 | - | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | +| KV-MemNN | 2022 | 76.0 | - | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| IRN | 2022 | 71.3 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | +| Supervised embedding | 2017 | 29.1 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | +| KV-MemNN | 2017 | 25.1 | - | - | EN | [Zhang et al.](https://arxiv.org/pdf/1709.04071.pdf) | +| T5+KG | 2022 | - | - | 33.57 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | +| T5+C4 | 2022 | - | - | 32.78 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | +| T5 | 2022 | - | - | 32.65 | EN | [Moiseev et al.](https://arxiv.org/pdf/2205.08184.pdf) | +| Retrieve-Rewrite-Answer (LLama-2-chat 13B/13B) | 2023 | - | - | 99.07 | EN | [Wu et al.](https://arxiv.org/pdf/2309.11206) | diff --git a/other/ZJQA.md b/other/ZJQA.md new file mode 100644 index 00000000..651a829e --- /dev/null +++ b/other/ZJQA.md @@ -0,0 +1,8 @@ +--- + name: ZhejiangQA +--- + +| Model / System | Year | Hits@1 | Language | Reported by | +|:--------------------------:|:----:|:------:|:--------:|:---------------------------------------------------------------------:| +| Retrieve-Rewrite-Answer (LLama-2-chat 7B/13B) | 2023 | 93.21 | Chinese | [Wu et al.](https://arxiv.org/pdf/2309.11206) | + diff --git a/other/^ZJQA.md b/other/^ZJQA.md new file mode 100644 index 00000000..eab142b5 --- /dev/null +++ b/other/^ZJQA.md @@ -0,0 +1,3 @@ +# ZhejiangQA + +**ZhejiangQA** is a Chinese KGQA dataset of 20,491 questions provided by Zhejiang Lab. diff --git a/systems.md b/systems.md index bda7b77e..5cd07e48 100644 --- a/systems.md +++ b/systems.md @@ -1,5 +1,5 @@ | System Name | Reported by | Reported in the paper | Demo/Repo/API available | Link to Demo/Repo/API | Original paper | System description | Reference | -| ------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Alexandria | Lopez et al. | [Link](https://www.sciencedirect.com/science/article/pii/S157082681300022X?casa_token=NBVj-I48uxAAAAAA:izoYV-LubTYApUYRCtnZFPuvdACyWHHNnwVBjo1S1K24AiXYmMde9vdEBsCxdpAvlfNvPswrzr8#br000150) | not working | [Link](http://alexandria.neofonie.de/) | [Link](https://link.springer.com/chapter/10.1007/978-3-662-46641-4_8) | Alexandria is a German question answering system over a domain ontology that was built primarily with data from Freebase, Authors propose a new formal query building approach that consists of two stages. In the first stage, they predict the query structure of the question and leverage the structure to constrain the generation of the candidate queries and propose a novel graph generation framework to handle the structure prediction task and design an encoder-decoder model to predict the argument of the predetermined operation in each generative step. In the second stage, they follow the previous methods to rank the candidate queries. predict the query structure of the question and leverage the structure to constrain the generation of the candidate queries and propose a novel graph generation framework to handle the structure prediction task and design an encoder-decoder model to predict the argument of the predetermined operation in each generative step. In the second stage, they follow the previous methods to rank the candidate queries. | Chen et al. only one triple, provide a modular, easy-toextend QA pipeline and evaluate it on the SimpleQuestionsWikidata benchmark. Ranking is learned from the training set. | | Aqqu Wikidata (rules) | Thomas Goette | [Link](https://ad-publications.cs.uni-freiburg.de/theses/Master_Thomas_Götte_2021.pdf) | no | - | [Link](https://ad-publications.cs.uni-freiburg.de/theses/Master_Thomas_Götte_2021.pdf) | Author focus on simple questions which means that the corresponding SPARQL query contains only one triple, provide a modular, easy-toextend QA pipeline and evaluate it on the SimpleQuestionsWikidata benchmark. Ranking with a set of weighted features. | Thomas Goette | | AskNow | Diefenbach et al. | [Link](http://www.semantic-web-journal.net/system/files/swj2038.pdf) | yes | [Link](https://github.com/AskNowQA) | [Link](https://www.springerprofessional.de/en/asknow-a-framework-for-natural-language-query-formalization-in-s/10191942) | Authors propose a framework, called AskNow, where users can pose queries in English to a target RDF knowledge base (e.g. DBpedia), which are first normalized into an intermediary canonical syntactic form, called Normalized Query Structure (NQS), and then translated into SPARQL queries. NQS facilitates the identification of the desire (or expected output information) and the user-provided input information, and establishing their mutual semantic relationship. At the same time, it is sufficiently adaptive to query paraphrasing. We have empirically evaluated the framework with respect to the syntactic robustness of NQS and semantic accuracy of the SPARQL translator on standard benchmark datasets. | Dubey et al. | @@ -19,7 +19,7 @@ | gGCN | Wu et al. | [Link](https://arxiv.org/pdf/2101.01510.pdf) | no | - | [Link](https://arxiv.org/pdf/2101.01510.pdf) | Authors present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism.The gGCN model is obtained from gRGCN by replacing RGCN with Graph Convolutional Network (GCN) | Wu et al. | | GGNN | Sorokin and Gurevych | [Link](https://aclanthology.org/C18-1280.pdf) | yes | [Link](https://github.com/UKPLab/coling2018-graph-neural-networks-question-answering) | [Link](https://aclanthology.org/C18-1280.pdf) | Authors address the problem of learning vector representations for complex semantic parses that consist of multiple entities and relations. For each input question, they construct an explicit structural semantic parse (semantic graph). Semantic parses can be deterministically converted to a query to extract the answers from the KB. To investigate ways to encode the structure of a semantic parse and to improve the performance for more complex questions, authors adapt Gated Graph Neural Networks (GGNNs), described in Li et al. (2016), to process and score semantic parses. | Sorokin and Gurevych | | GRAFT-Net | Y Feng et al. | [Link](https://arxiv.org/pdf/2112.06109.pdf) | yes | [Link](https://github.com/haitian-sun/GraftNet) | [Link](https://arxiv.org/abs/1809.00782) | Authors propose a novel graph convolution based neural network, called GRAFT-Net (Graphs of Relations Among Facts and Text Networks), specifically designed to operate over heterogeneous graphs of KB facts and text sentences. First, they propose heterogeneous update rulesthat handle KB nodes differently from the textnodes: for instance, LSTM-based updates are usedto propagate information into and out of text nodes. Second, authors introduce a directed propagation method, inspired by personalized Pagerankin IR (Haveliwala, 2002). | Sun et al. | -| GRAFT-Net + Clocq | Christmann P. et al. | [Link](https://arxiv.org/pdf/2108.08597.pdf) | yes | [Link](https://github.com/PhilippChr/CLOCQ) (demo is available for further work on CLOCQ) | [Link](https://arxiv.org/pdf/2108.08597.pdf) | This work presents CLOCQ, an efficient method that prunes irrelevant parts of the search space using KB-aware signals. CLOCQ uses a top-𝑘 query processor over score-ordered lists of KB items that combine signals about lexical matching, relevance to the question, coherence among candidate items, and connectivity in the KB graph. | Christmann P. et al . | +| GRAFT-Net + Clocq | Christmann P. et al. | [Link](https://arxiv.org/pdf/2108.08597.pdf) | yes | [Link](https://github.com/PhilippChr/CLOCQ) (demo is available for further work on CLOCQ) | [Link](https://arxiv.org/pdf/2108.08597.pdf) | This work presents CLOCQ, an efficient method that prunes irrelevant parts of the search space using KB-aware signals. CLOCQ uses a top-𝑘 query processor over score-ordered lists of KB items that combine signals about lexical matching, relevance to the question, coherence among candidate items, and connectivity in the KB graph. | Christmann P. et al . | | gRGCN | Wu et al. | [Link](https://arxiv.org/pdf/2101.01510.pdf) | no | - | [Link](https://arxiv.org/pdf/2101.01510.pdf) | Authors present a relational graph convolutional network (RGCN)-based model gRGCN for semantic parsing in KBQA. gRGCN extracts the global semantics of questions and their corresponding query graphs, including structure semantics via RGCN and relational semantics (label representation of relations between entities) via a hierarchical relation attention mechanism. | Wu et al. | | Hakimov | Diefenbach et al. | [Link](http://www.semantic-web-journal.net/system/files/swj2038.pdf) | no | - | [Link](https://www.semanticscholar.org/paper/Applying-Semantic-Parsing-to-Question-Answering-the-Hakimov-Unger/126ee532d48302b31f899ab392c51ad982ee5cad) | Authors investigate how much lexical knowledge would need to be added so that a semantic parsing approach can perform well on unseen data. We manually add a set of lexical entries on the basis of analyzing the test portion of the QALD-4 dataset. Further, we analyze if a state-of-the-art tool for inducing ontology lexica from corpora can derive these lexical entries automatically. | Hakimov et al. | | HGNet | Chen et al. | [Link](https://arxiv.org/pdf/2111.00732.pdf) | yes | [Link](https://github.com/Bahuia/HGNet) | [Link](https://arxiv.org/pdf/2111.00732.pdf) | Hierarchical Graph Generation Network (HGNet) focues on generating search query by proposing a new unified query graph grammar to adapt to SPARQL's syntax. FIrstly, the autho ranks top k entity, relation, and value by ralation ranking and pattern matching. Secondly, HGNet is used to encde and decode the natural questions to generate query graph. The project is open-sourced on github. | Chen et al. | @@ -91,7 +91,7 @@ | WDAqua-core1 | Omar et al. | [Link](http://ceur-ws.org/Vol-2980/paper312.pdf) | not working | [Link](https://github.com/WDAqua), demo: [Link](http://wdaqua.eu/qa) | [Link](https://dl.acm.org/doi/pdf/10.1145/3184558.3191541) | WDQqua-core1 is one of the few QA system which are running as web-services. The model is a pipeline model which aims to convert question to SPARQL query. The model includes following sessions: Query Expansion, Query Con- struction, Query Ranking and Answer Decision. The service suppoprts multilingual question, natural question as well as key word questions and is integrated into Qanary framwork. | Diefenbach et al. | | WolframAlpha | Walter et al. | [Link](https://download.hrz.tu-darmstadt.de/pub/FB20/Dekanat/Publikationen/UKP/76500354.pdf) | yes | [Link](https://www.wolframalpha.com/) - only tested on QALD-2 | [Link](https://www.wolframalpha.com/) | WolframAlpha is an engine for computing answers and providing knowledge. It combines curated knowledge, linguistic analysis and dynamic computation. | Only website available m | | Xser | Diefenbach et al. | [Link](http://www.semantic-web-journal.net/system/files/swj2038.pdf) | no | only tested on QALD 4 & 5 | [Link](http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-XuEt2014.pdf) | Authors present a question answering system (Xser) over Linked Data(DBpedia), converting users’ natural language questions into structured queries. There are two challenges involved: recognizing users’ query intention and mapping the involved semantic items against a given knowledge base (KB), which will be in turn assembled into a structured query. In this paper, we propose an efficient pipeline framework to model a user’s query intention as a phrase level dependency DAG which is then instantiated according to a given KB to construct the final structured query. | Xu et al. | -| YodaQA | Diefenbach et al. | [Link](http://www.semantic-web-journal.net/system/files/swj2038.pdf) | not working | [Link](https://github.com/brmson/yodaqa), webservice [Link](http://live.ailao.eu/) not working - only tested on QALD-5 | [Link](https://pasky.or.cz/dev/brmson/yodaqa-clef2015-QALD.pdf) | YodaQA is an pipeline factoid question answering system that can produce answer both from knowledge base and unstructured text. The model answers question in following steps: Question Analysis, Answer Production, Answer Analysis and Answer Merging and Scoring. In the answer production part, the model has different strategy on knolwegde base and on corpora. | Petr Baudiˇs and Jan Sˇedivy ́ | +| YodaQA | Diefenbach et al. | [Link](http://www.semantic-web-journal.net/system/files/swj2038.pdf) | not working | [Link](https://github.com/brmson/yodaqa), webservice [Link](http://live.ailao.eu/) not working - only tested on QALD-5 | [Link](https://pasky.or.cz/dev/brmson/yodaqa-clef2015-QALD.pdf) | YodaQA is an pipeline factoid question answering system that can produce answer both from knowledge base and unstructured text. The model answers question in following steps: Question Analysis, Answer Production, Answer Analysis and Answer Merging and Scoring. In the answer production part, the model has different strategy on knolwegde base and on corpora. | Petr Baudiˇs and Jan Sˇedivy ́ | | Yu et al. | Wu et al. | [Link](https://arxiv.org/pdf/2101.01510.pdf) | no | - | [Link](https://arxiv.org/pdf/2101.01510.pdf) | This model is semantic parsing based KBQA system, where the parsing section is a relational graph convolutional network (RGCN) called gRGCN. gRGCN combines RGCN and hierarchical relation attention mechanism. Therefore, it is capable of exracting both structure semantics and relational semantics from the question and generate corresponsing seach queries. | Wu et al. | | Zhang et. al. | Zhang et. al. | [Link](https://ojs.aaai.org/index.php/AAAI/article/view/10381) | no | - | [Link](https://ojs.aaai.org/index.php/AAAI/article/view/10381) | This paper presents a joint model based on integer linear programming (ILP), uniting alignment construction and query construction into a uniform framework. As a result, the model is able to outperform pipeline model with train the two sectors seperately and be robot to noise propogation. | Zhang et. al. | | Zhu et al. | Zhu et al. | [Link](https://arxiv.org/abs/1510.04780) | no | - | [Link](https://arxiv.org/abs/1510.04780) | Focusing on solving the non-aggregation questions, in this paper, authors construct a subgraph of the knowledge base from the detected entities and propose a graph traversal method to solve both the semantic item mapping problem and the disambiguation problem in a joint way. Compared with existing work, they simplify the process of query intention understanding and pay more attention to the answer path ranking. | Zhu et al. | @@ -140,4 +140,12 @@ | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | | W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. | -| GAIN | Shu et al. | [Link](https://arxiv.org/pdf/2309.08345.pdf) | no | - | [Link](https://arxiv.org/pdf/2309.08345.pdf) | GAIN is not a KGQA system, but a data augmentation method named Graph seArch and questIon generatioN (GAIN). GAIN applies to KBQA corresponding to logical forms or triples, and scales data volume and distribution through four steps: 1) Graph search: Sampling logical forms or triples from arbitrary domains in the KB, without being restricted to any particular KBQA dataset. 2) Training question generator on existing KBQA datasets, i.e., learning to convert logical forms or triples into natural language questions. 3) Verbalization: Using the question generator from step 2 to verbalize sampled logical forms or triples from step 1, thus creating synthetic questions. 4) Training data expansion: Before fine-tuning any neural models on KBQA datasets, GAIN-synthetic data can be used to train these models or to expand the corpus of in-context samples for LLMs. That is, as a data augmentation method, GAIN is not a KBQA model, but it is used to augment a base KBQA model. | Shu et al. | \ No newline at end of file +| GAIN | Shu et al. | [Link](https://arxiv.org/pdf/2309.08345.pdf) | no | - | [Link](https://arxiv.org/pdf/2309.08345.pdf) | GAIN is not a KGQA system, but a data augmentation method named Graph seArch and questIon generatioN (GAIN). GAIN applies to KBQA corresponding to logical forms or triples, and scales data volume and distribution through four steps: 1) Graph search: Sampling logical forms or triples from arbitrary domains in the KB, without being restricted to any particular KBQA dataset. 2) Training question generator on existing KBQA datasets, i.e., learning to convert logical forms or triples into natural language questions. 3) Verbalization: Using the question generator from step 2 to verbalize sampled logical forms or triples from step 1, thus creating synthetic questions. 4) Training data expansion: Before fine-tuning any neural models on KBQA datasets, GAIN-synthetic data can be used to train these models or to expand the corpus of in-context samples for LLMs. That is, as a data augmentation method, GAIN is not a KBQA model, but it is used to augment a base KBQA model. | Shu et al. | +| PER-KBQA | Guo et al. | [Link](https://arxiv.org/pdf/2305.03356) | no | - | [Link](https://arxiv.org/pdf/2305.03356) | PER-KBQA is a system designed to enhance the reasoning capabilities of semantic parsers for complex KBQA tasks. It builds on the KoPL parser, which models reasoning processes explicitly. PER-KBQA follows a parse-execute-refine paradigm to improve complex multi-step reasoning. The system comprises three stages: parsing, execution, and refinement. In the parsing stage, the KoPL parser generates transparent logical forms. During the execution stage, these logical forms are aligned and executed over the knowledge base to produce intermediate reasoning steps. In the refinement stage, these intermediate reasoning processes are demonstrated to the KBQA model, enhancing its ability to handle complex questions. Experiments on benchmark datasets show that PER-KBQA significantly outperforms state-of-the-art baselines in complex KBQA tasks. | Guo et al. | +| KAPING | Baek et al. | [Link](https://arxiv.org/pdf/2306.04136) | no | - | [Link](https://arxiv.org/pdf/2306.04136) | KAPING (Knowledge-Augmented language model PromptING) is a system designed to enhance zero-shot closed-book question answering by large language models (LLMs) without the need for fine-tuning. Instead of relying solely on the internalized knowledge within LLMs, which can be insufficient or incorrect, KAPING augments this knowledge by incorporating relevant facts directly into the input. The system retrieves pertinent facts from a knowledge graph based on semantic similarities between the question and the facts. These facts are then prepended to the input question, forming a prompt that is fed to the LLM to generate an answer. This approach requires no model training, making it entirely zero-shot. KAPING has demonstrated superior performance on the knowledge graph question answering task, outperforming relevant zero-shot baselines by up to 48% on average across multiple LLMs of various sizes. | Baek et al. | +| Retrieve-Rewrite-Answer | Wu et al. | [Link](https://arxiv.org/pdf/2309.11206) | yes | [Link](https://github.com/wuyike2000/Retrieve-Rewrite-Answer) | [Link](https://arxiv.org/pdf/2309.11206) | Retrieve-Rewrite-Answer is a system designed to enhance large language models (LLMs) for Knowledge Graph Question Answering (KGQA) by addressing their limitations in retaining extensive world knowledge, especially long-tail knowledge. This system employs an answer-sensitive KG-to-Text approach to convert knowledge graph (KG) information into well-structured textual statements. By doing so, it bridges the gap between KG representations and textual representations, making the knowledge more accessible and useful for KGQA. Experimental results show that Retrieve-Rewrite-Answer significantly improves answer accuracy and the effectiveness of knowledge statements compared to previous KG-augmented LLM approaches. | Wu et al. | +| Lingua Franca | Srivastava et al. | [Link](https://dl.acm.org/doi/pdf/10.1145/3587259.3627567) | yes | [Link](https://github.com/dice-group/LFQA) | [Link](https://dl.acm.org/doi/pdf/10.1145/3587259.3627567) | Lingua Franca is an approach to enhancing machine translation quality by accurately translating named entities through the use of a knowledge graph. The technique involves preserving named entities during translation and subsequently replacing them with their corresponding labels from the knowledge graph in the target language. This method is particularly beneficial for entity-oriented search and Knowledge Graph Question Answering systems. The paper also presents an open-source, modular framework for researchers to create named entity-aware machine translation pipelines. Experimental results indicate that Lingua Franca significantly improves translation quality compared to baseline models, enhancing the performance of multiple Knowledge Graph Question Answering systems across various datasets. | Srivastava et al. | +| Sen et al. | Sen et al. | [Link](https://aclanthology.org/2023.nlrse-1.1.pdf) | no | - | [Link](https://aclanthology.org/2023.nlrse-1.1.pdf) | This method combines a knowledge graph retriever based on an end-to-end KGQA model with a language model that reasons over the retrieved facts to return an answer. | Sen et al. | + + + diff --git a/wikidata/KQA Pro.md b/wikidata/KQA Pro.md index bf7c396b..b0b56655 100644 --- a/wikidata/KQA Pro.md +++ b/wikidata/KQA Pro.md @@ -3,3 +3,13 @@ datasetUrl: http://thukeg.gitee.io/kqa-pro/ --- + +| Model | Year | Accuracy | Reported By | +|:--------:|:----:|:--------:|:-----------------------------------------------------:| +| KVMemNet | 2023 | 0.1661 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| EmbedKGQA | 2023 | 0.2836 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| RGCN | 2023 | 0.3507 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| BART SPARQL | 2023 | 0.8968 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| BART KoPL | 2023 | 0.9055 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| GraphQ IR | 2023 | 0.9170 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | +| PER-KBQA | 2023 | 0.9382 | [Guo et al., 2023](https://arxiv.org/pdf/2305.03356) | diff --git a/wikidata/MarkQA.md b/wikidata/MarkQA.md new file mode 100644 index 00000000..87411119 --- /dev/null +++ b/wikidata/MarkQA.md @@ -0,0 +1,9 @@ +--- + name: MarkQA + datasetUrl: https://github.com/cdhx/MarkQA/ +--- + + +| Model | Year | Accuracy | Reported By | +|:------------:|:----:|:--------:|:------------------------------------------------------:| +| QDTQA (PyQL) | 2023 | 44.12 | [Huang et al., 2023](https://arxiv.org/pdf/2310.15517) | diff --git a/wikidata/Mintaka.md b/wikidata/Mintaka.md index 52658491..038b4c5a 100644 --- a/wikidata/Mintaka.md +++ b/wikidata/Mintaka.md @@ -4,16 +4,18 @@ --- -|Model | Year | Hits@1 | Language | Reported By | -|:-------------:|:-------------:|:----------:|:-------------:|:--------------:| -| XL-T5 (fine-tuned) | 2022 | 0.38 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| DPR (trained) |2022 | 0.31 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| T5 | 2022 | 0.28 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| T5 for CBQA (zero-shot) | 2022 | 0.20 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -|Rigel | 2022 | 0.20 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| EmbedKGQA | 2022 | 0.18 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| DPR (zero-shot) |2022 | 0.15 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| KVMemNet | 2022 | 0.12 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| T5 for CBQA (translated)|2022 | 0.31 | Multilingually | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| Rigel | 2022 | 0.19 | Multilingually |[Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| -| MT5|2022 | 0.16 | Multilingually | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf)| +|Model | Year | Hits@1 | Language | Reported By | +|:-------------:|:-------------:|:------:|:--------:|:------------------------------------------------------------------:| +| XL-T5 (fine-tuned) | 2022 | 0.38 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| DPR (trained) |2022 | 0.31 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| T5 | 2022 | 0.28 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| T5 for CBQA (zero-shot) | 2022 | 0.20 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +|Rigel | 2022 | 0.20 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| EmbedKGQA | 2022 | 0.18 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| DPR (zero-shot) |2022 | 0.15 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| KVMemNet | 2022 | 0.12 | English | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| T5 for CBQA (translated)|2022 | 0.31 | Multilingually | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| Rigel | 2022 | 0.19 | Multilingually | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| MT5|2022 | 0.16 | Multilingually | [Sen et al., 2022](https://aclanthology.org/2022.coling-1.138.pdf) | +| KAPING (GPT-3 175B) | 2023 | 0.4908 | English | [Baek et al.](https://arxiv.org/pdf/2306.04136) | +| KAPING (AlexaTM 20B) | 2023 | 0.5686 | English | [Baek et al.](https://arxiv.org/pdf/2306.04136) | diff --git a/wikidata/WebQuestionsSP.md b/wikidata/WebQuestionsSP.md new file mode 100644 index 00000000..f3b085ad --- /dev/null +++ b/wikidata/WebQuestionsSP.md @@ -0,0 +1,10 @@ +--- + name: WebQuestionsSP + datasetUrl: https://www.microsoft.com/en-us/download/details.aspx?id=52763 +--- + +| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | +|:-----------------------------------:|:----:|:---------:| :--------: |:--------:|:--------:|:-------------------------------------------------------------------------------------:| +| KAPING (GPT-3 175B) | 2023 | | | 69.58 | EN | [Baek et al.](https://arxiv.org/pdf/2306.04136) | +| KAPING (AlexaTM 20B) | 2023 | | | 65.04 | EN | [Baek et al.](https://arxiv.org/pdf/2306.04136) | + diff --git a/wikidata/^MarkQA.md b/wikidata/^MarkQA.md new file mode 100644 index 00000000..38bddf7d --- /dev/null +++ b/wikidata/^MarkQA.md @@ -0,0 +1,7 @@ +# MarkQA + +**MarkQA**[[1]](#myfootnote1) MarkQA is dataset to facilitate the development of NR-KBQA (multi-hop reasoning and numerical reasoning), which is automatically constructed from a small set of seeds. Each question in MarkQA is equipped with its corresponding SPARQL query, alongside the step-by-step rea- soning process in the QDMR format and PyQL program. + + +## References +[1] Xiang Huang, Sitao Cheng, Yuheng Bao, Shanshan Huang, Yuzhong Qu: MarkQA: A large scale KBQA dataset with numerical reasoning . CoRR abs/2310.15517 (2023) \ No newline at end of file diff --git a/wikidata/^WebQuestionsSP.md b/wikidata/^WebQuestionsSP.md new file mode 100644 index 00000000..b8a69403 --- /dev/null +++ b/wikidata/^WebQuestionsSP.md @@ -0,0 +1,7 @@ +# WebQuestionsSP + +**WebQuestionsSP**[[1]](#myfootnote1) is the annotated version of WebQuestions, which contains questions that require a 1 or 2-hop relation path to arrive at the answer entity. More specifically, about 40% of the questions require a 2-hop relation to reach the answer. Freebase is its backgound KB. Originally, it splits into 3,298 questions as train set and 1,639 questions as test set. + + +## References +[1] Yih, Wen-tau, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. [The value of semantic parse labeling for knowledge base question answering.](http://anthology.aclweb.org/P16-2033) In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 201-206. 2016. \ No newline at end of file