Identifying Problem Types in Automated Question Generation
- Automated question generation holds great promise in many fields, such as education, to reduce the workload and automate an otherwise tedious task. However, major challenges remain regarding the quality of generated questions. To identify and address these challenges generated questions are evaluated either automatically or manually. While several automated metrics, mostly based on the comparison with a gold standard, exist, their usefulness is limited and human evaluation is often used for more accurate assessment. Our research generates questions using several models and methods, including fine-tuning, zero-shot and few-shot. We compare model performance by classifying the generated questions using a multi-label approach. This approach evaluates by sorting generated questions into zero or more binary problem classes and attempting to identify different problems with the generated questions. Our results show that different models tend to generate questions that fit into different problem classes. Additionally, the problem classification evaluation is capable of recognizing these differences and weighing the classes for the models accordingly, creating model-specific distribution characteristics.
Author: | Joshua BergerORCiD, Markos StamatakisORCiD, Anett HoppeORCiD, Ralph EwerthORCiD, Christian WartenaORCiDGND |
---|---|
URN: | urn:nbn:de:bsz:960-opus4-34511 |
DOI: | https://doi.org/10.25968/opus-3451 |
ISBN: | 978-3-69018-002-3 |
Parent Title (English): | KI-Forum 2024: AI 4 Students – AI in Research – AI Showroom |
Publisher: | HsH Applied Academics |
Place of publication: | Hannover |
Document Type: | Conference Proceeding |
Language: | English |
Year of Completion: | 2024 |
Publishing Institution: | Hochschule Hannover |
Release Date: | 2025/01/24 |
Tag: | AQG; Automated Question Generation; NLP; Problem Categorization; Transformers |
GND Keyword: | Künstliche IntelligenzGND; Automatische SprachanalyseGND |
Page Number: | 4 |
First Page: | 34 |
Last Page: | 37 |
Institutes: | Fakultät III - Medien, Information und Design |
DDC classes: | 004 Informatik |
Licence (German): | ![]() |