From bf726de15e38b27facba18acfb3de7f1bbd14e27 Mon Sep 17 00:00:00 2001 From: Niels Rogge Date: Tue, 15 Dec 2020 17:54:06 +0000 Subject: [PATCH] Add wikisql dataset --- README.md | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..468bbb4 --- /dev/null +++ b/README.md @@ -0,0 +1,114 @@ +--- +language: en +tags: +- tapas +- question-answering +license: apache-2.0 +datasets: +- wikisql +--- + +# TAPAS base model fine-tuned on WikiSQL (in a supervised fashion) + +This model has 4 versions which can be used. The latest version, which is the default one, corresponds to the `tapas_wikisql_sqa_inter_masklm_base_reset` checkpoint of the [original Github repository](https://github.com/google-research/tapas). +This model was pre-trained on MLM and an additional step which the authors call intermediate pre-training, and then fine-tuned on [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) and +[WikiSQL](https://github.com/salesforce/WikiSQL). It uses relative position embeddings by default (i.e. resetting the position index at every cell of the table). + +The other (non-default) versions which can be used are: +- `revision="v3"`, which corresponds to `tapas_wikisql_sqa_inter_masklm_base` (intermediate pre-training, absolute position embeddings) +- `revision="v2"`, which corresponds to `tapas_wikisql_sqa_masklm_base_reset` (no intermediate pre-training, relative position embeddings) +- `revision="v1"`, which corresponds to `tapas_wikisql_sqa_masklm_base` (no intermediate pre-training, absolute position embeddings) + +Disclaimer: The team releasing TAPAS did not write a model card for this model so this model card has been written by +the Hugging Face team and contributors. + +## Model description + +TAPAS is a BERT-like transformers model pretrained on a large corpus of English data from Wikipedia in a self-supervised fashion. +This means it was pretrained on the raw tables and associated texts only, with no humans labelling them in any way (which is why it +can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it +was pretrained with two objectives: + +- Masked language modeling (MLM): taking a (flattened) table and associated context, the model randomly masks 15% of the words in + the input, then runs the entire (partially masked) sequence through the model. The model then has to predict the masked words. + This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, + or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional + representation of a table and associated text. +- Intermediate pre-training: to encourage numerical reasoning on tables, the authors additionally pre-trained the model by creating + a balanced dataset of millions of syntactically created training examples. Here, the model must predict (classify) whether a sentence + is supported or refuted by the contents of a table. The training examples are created based on synthetic as well as counterfactual statements. + +This way, the model learns an inner representation of the English language used in tables and associated texts, which can then be used +to extract features useful for downstream tasks such as answering questions about a table, or determining whether a sentence is entailed +or refuted by the contents of a table. Fine-tuning is done by adding a cell selection head and aggregation head on top of the pre-trained model, and then jointly train these randomly initialized classification heads with the base model on SQA and WikiSQL. + + +## Intended uses & limitations + +You can use this model for answering questions related to a table. + +For code examples, we refer to the documentation of TAPAS on the HuggingFace website. + + +## Training procedure + +### Preprocessing + +The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are +then of the form: + +``` +[CLS] Question [SEP] Flattened table [SEP] +``` + +The authors did first convert the WikiSQL dataset into the format of SQA using automatic conversion scripts. + +### Fine-tuning + +The model was fine-tuned on 32 Cloud TPU v3 cores for 50,000 steps with maximum sequence length 512 and batch size of 512. +In this setup, fine-tuning takes around 10 hours. The optimizer used is Adam with a learning rate of 6.17164e-5, and a warmup +ratio of 0.1424. See the [paper](https://arxiv.org/abs/2004.02349) for more details (tables 11 and 12). + + +### BibTeX entry and citation info + +```bibtex +@misc{herzig2020tapas, + title={TAPAS: Weakly Supervised Table Parsing via Pre-training}, + author={Jonathan Herzig and Paweł Krzysztof Nowak and Thomas Müller and Francesco Piccinno and Julian Martin Eisenschlos}, + year={2020}, + eprint={2004.02349}, + archivePrefix={arXiv}, + primaryClass={cs.IR} +} +``` + +```bibtex +@misc{eisenschlos2020understanding, + title={Understanding tables with intermediate pre-training}, + author={Julian Martin Eisenschlos and Syrine Krichene and Thomas Müller}, + year={2020}, + eprint={2010.00571}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +```bibtex +@article{DBLP:journals/corr/abs-1709-00103, + author = {Victor Zhong and + Caiming Xiong and + Richard Socher}, + title = {Seq2SQL: Generating Structured Queries from Natural Language using + Reinforcement Learning}, + journal = {CoRR}, + volume = {abs/1709.00103}, + year = {2017}, + url = {http://arxiv.org/abs/1709.00103}, + archivePrefix = {arXiv}, + eprint = {1709.00103}, + timestamp = {Mon, 13 Aug 2018 16:48:41 +0200}, + biburl = {https://dblp.org/rec/journals/corr/abs-1709-00103.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} +``` \ No newline at end of file