Transferencia de Tareas basada en Implicación Textual para la Clasificación de Textos en Catalán en Escenarios de Pocos Datos

  1. Villegas Montserrat, Marta
  2. López de Lacalle Lekuona, Oier
  3. Baucells de la Peña, Irene
  4. Calvo Figueras, Blanca
Revista:
Procesamiento del lenguaje natural

ISSN: 1135-5948

Año de publicación: 2023

Número: 71

Páginas: 165-177

Tipo: Artículo

DIALNET GOOGLE SCHOLAR lock_openRUA editor

Otras publicaciones en: Procesamiento del lenguaje natural

Resumen

This study investigates the application of a state-of-the-art zero-shot and few-shot natural language processing (NLP) technique for text classification tasks in Catalan, a moderately under-resourced language. The approach involves reformulating the downstream task as textual entailment, which is then solved by an entailment model. However, unlike English, where entailment models can be trained on huge Natural Language Inference (NLI) datasets, the lack of such large resources in Catalan poses a challenge. In this context, we comparatively explore training on monolingual and (larger) multilingual resources, and identify the strengths and weaknesses of monolingual and multilingual individual components of entailment models: pre-trained language model and NLI training dataset. Furthermore, we propose and implement a simple task transfer strategy using open Wikipedia resources that demonstrates significant performance improvements, providing a practical and effective alternative for languages with limited or no NLI datasets.

Referencias bibliográficas

  • Agerri, R., I. S. Vicente, J. A. Campos, A. Barrena, X. Saralegi, A. Soroa, and E. Agirre. 2020. Give your text representation models some love: the case for basque.
  • Armengol-Estapé, J., C. P. Carrino, C. Rodriguez-Penagos, O. de Gibert Bonet, C. Armentano-Oller, A. Gonzalez- Agirre, M. Melero, and M. Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946, 7.
  • Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. Mc- Candlish, A. Radford, I. Sutskever, and D. Amodei. 2020. Language models are few-shot learners. 5.
  • Chang, M.-W., L. Ratinov, D. Roth, and V. Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, page 830–835. AAAI Press.
  • Gao, T., A. Fisch, and D. Chen. 2021. Making pre-trained language models better few-shot learners. ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3816–3830. Lee, C., K. Cho, and W. Kang. 2019. Mixout: Effective regularization to finetune largescale pretrained language models. 9.
  • Liu, H., D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. 5.
  • Liu, P., W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
  • Ma, T., J. G. Yao, C. Y. Lin, and T. Zhao. 2021. Issues with entailment-based zeroshot text classification. ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 2:786–796.
  • Mahabadi, R. K., L. Zettlemoyer, J. Henderson, M. Saeidi, L. Mathias, V. Stoyanov, M. Yazdani, and M. Ai. 2022. Perfect: Prompt-free and efficient few-shot learning with language models.
  • Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings.
  • Obamuyide, A. and A. Vlachos. 2018. Zeroshot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 72–78, Brussels, Belgium, November. Association for Computational Linguistics.
  • Pàmies, M., J. Llop, F. Multari, N. Duran- Silva, C. Parra-Rojas, A. Gonzalez-Agirre, F. A. Massucci, and M. Villegas. 2023. A weakly supervised textual entailment approach to zero-shot text classification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 286–296, Dubrovnik, Croatia, May. Association for Computational Linguistics.
  • Phang, J., T. Févry, and S. R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeleddata tasks.
  • Sainz, O., O. L. de Lacalle, G. Labaka, A. Barrena, and E. Agirre. 2021. Label verbalization and entailment for effective zeroand few-shot relation extraction.
  • Sainz, O., I. Gonzalez-Dios, O. Lopez de Lacalle, B. Min, and E. Agirre. 2022a. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2439–2455, Seattle, United States, July. Association for Computational Linguistics.
  • Sainz, O., H. Qiu, O. L. de Lacalle, E. Agirre, and B. Min. 2022b. Zs4ie: A toolkit for zero-shot information extraction with simple verbalizations. NAACL 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Demonstrations Session, pages 27–38.
  • Sainz, O. and G. Rigau. 2021. Ask2transformers: Zero-shot domain labelling with pre-trained language models. GWC 2021 - Proceedings of the 11th Global Wordnet Conference, pages 44–52, 1.
  • Schick, T. and H. Schütze. 2021a. Exploiting cloze questions for few shot text classification and natural language inference. EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, pages 255–269.
  • Schick, T. and H. Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. NAACL-HLT 2021 - 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pages 2339–2352.
  • Tam, D., R. R. Menon, M. Bansal, S. Srivastava, and C. Raffel. 2021. Improving and simplifying pattern exploiting training. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 4980–4991.
  • Tunstall, L., N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, O. Pereg, and H. Face. 2022. Efficient few-shot learning without prompts.
  • Wang, S., H. Fang, M. Khabsa, H. Mao, and H. Ma. 2021. Entailment as few-shot learner. 4.
  • Xie, Q., Z. Dai, E. Hovy, M. T. Luong, and Q. V. Le. 2019. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 2020-December, 4.
  • Yin, W., J. Hay, and D. Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. EMNLP-IJCNLP 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3914–3923, 8.
  • Zhang, T., F.Wu, A. Katiyar, K. Q.Weinberger, and Y. Artzi. 2020. Revisiting fewsample BERT fine-tuning.
Fuente de los datos: Dialnet