Generación automática de meta-resúmenes para la evaluación del manejo de estructuras discursivas y coherencia en el alumnado

  1. Unai Atutxa
  2. Alejandro Molina Villegas
  3. Mikel Iruskieta Quintian
Aldizkaria:
Procesamiento del lenguaje natural

ISSN: 1135-5948

Argitalpen urtea: 2021

Zenbakia: 66

Orrialdeak: 165-175

Mota: Artikulua

DIALNET GOOGLE SCHOLAR lock_openRUA editor

Beste argitalpen batzuk: Procesamiento del lenguaje natural

Laburpena

Crowd-sourcing can help teachers to evaluate student summaries and give them feedback to improve their summarization skills. In this paper, we propose an approach for meta-summaries generation, to design and develop the automatic evaluation of extractive summaries for the Basque language. We propose a novel algorithm that allows to use the generated meta-summaries to i) compare students meta-summaries at different ages and education stages (elementary and undergraduates), ii) evaluate classroom meta-summaries (classroom evaluation) and iii) evaluate each student (individual evaluation). The results show that our proposed method, based on qualitative (coherence discourse structure) and quantitative (Fleis kappa and Hamming distance) measures, is accurate to compare both: groups and individuals.

Erreferentzia bibliografikoak

  • Alvarez, I. A. 2004. Evaluación y calificación de resúmenes de textos expositivos en el aula de ile/ife: la guía”babar”. Ib´erica: Revista de la Asociación Europea de Lenguas para Fines Específicos (AELFE), 1(8):81–99.
  • Atutxa, U. 2018. Ikasleen laburpencorpusa eta laburpen-gaitasunaren ebaluazioa: oinarri metodologikoak. Master’s thesis, University of the Basque Country (UPV/EHU), Donostia.
  • Atutxa, U., M. Iruskieta, O. Ansa, y A. Molina. 2017. Compress-eus: I (ra) kasleen laburpenak lortzeko tresna. EUUnai Atutxa,
  • Cabrera-Diego, L. A. y J.-M. Torres-Moreno. 2018. Summtriver: A new trivergent model to evaluate summaries automatically without human references. Data & Knowledge Engineering, 113:184 – 197.
  • CY, L. 2004. Rouge: a package for automatic evaluation of summaries. En Proceedings of the Workshop on Text Summarization Branches Out. Barcelona, Spain, páginas 56–60.
  • Fleiss, J. L., B. Levin, y M. C. Paik. 2013. Statistical methods for rates and proportions. john wiley & sons.
  • Iruskieta, M., A. D. Diaz de Ilarraza, y M. Lersundi. 2014. The annotation of the central unit in rhetorical structure trees: A key step in annotating rhetorical relations. En Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, páginas 466–475.
  • Louis, A. y A. Nenkova. 2009. Automatically evaluating content selection in summarization without human models. En Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, páginas 306–314. Association for Computational Linguistics.
  • Mann, W. C. y S. A. Thompson. 1987. Rhetorical structure theory: A theory of text organization. University of Southern California, Information Sciences Institute Los Angeles.
  • Molina, A. y J.-M. Torres. 2015. El test de turing para la evaluación de resumen automático de texto. Linguamática, 7(2):45– 55.
  • Molina-Villegas, A. 2013a. Compresión automática de frases: un estudio hacia la generación de resúmenes en espa˜nol. Inteligencia Artificial, 16(51):41–62.
  • Molina-Villegas, A. 2013b. Sistemas web colaborativos para la recopilación de datos bajo el paradigma de ciencia ciudadana. Komputer Sapiens, 1(5):6–18.
  • Nenkova, A. y R. J. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. En Proceedings of the human language technology conference of the north american chapter of the association for computational linguistics: Hlt-naacl 2004, páginas 145–152.
  • Papineni, K., S. Roukos, T. Ward, y W.-J. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. En Proceedings of the 40th annual meeting on association for computational linguistics, páginas 311–318. Association for Computational Linguistics.
  • Radev, D. R., H. Jing, M. Sty´s, y D. Tam. 2004. Centroid-based summarization of multiple documents. Information Processing & Management, 40(6):919–938.
  • Radev, D. R. y D. Tam. 2003. Summarization evaluation using relative utility. En Proceedings of the twelfth international conference on Information and knowledge management, páginas 508–511.
  • Saggion, H. 2008. A robust and adaptable summarization tool. Traitement Automatique des Langues, 49(2).
  • Saggion, H. y T. Poibeau. 2013. Automatic text summarization: Past, present and future. En Multi-source, multilingual information extraction and summarization. Springer, páginas 3–21.
  • Saggion, H., J.-M. Torres-Moreno, I. d. Cunha, y E. SanJuan. 2010. Multilingual summarization evaluation without human models. En Proceedings of the 23rd International Conference on Computational Linguistics: Posters, páginas 1059–1067.
  • Sanz, A. 2005. Irakurmena lantzeko jarduerak nola prestatu: Lehen hezkuntzako 3. zikloa eta dbhko 1. zikloa. Nafarroako Gobernua.
  • Zipitria, I., A. Arruarte, y J. A. Elorriaga. 2008. Lea: A summarization web environment based on human instructors’ behaviour. En 2008 Eighth IEEE International Conference on Advanced Learning Technologies, páginas 564–568. IEEE.
  • Zipitria, I., P. Larra˜naga, R. Arma˜nanzas, A. Arruarte, y J. A. Elorriaga. 2008. What is behind a summary-evaluation decision? Behavior Research Methods, 40(2):597–612
Datuen iturria: Dialnet