Optimizing the Effectiveness of Chat GPT’s Feedback on ESL Student's Written Productions: An approach from the programmer teacher
(es) Optimizando la Efectividad de la Retroalimentación de Chat GPT’s de las producciones Escritas de Estudiantes ESL: Un enfoque desde el docente programador
(port) Optimizando la Efectividad de la Retroalimentación de GPT’s de las producciones Escritas de Estudiantes ESL: Uma abordagem do professor programador
Roque Jacinto Hernández Bustos
Universidad Casa Grande
roque.hernandez@casagrande.edu.ec
https://orcid.org/0009-0004-8260-500X
Hernández, R. J. (2024). Optimizing the Effectiveness of Chat GPT’s Feedback on ESL Student's Written Productions: An approach from the programmer teacher. YUYAY: Estrategias, Metodologías & Didácticas Educativas, 3(2), 50–61. https://doi.org/10.59343/yuyay.v3i2.69
Recepción: 06-04-2024 / Aceptación: 12-07-2024 / Publicación: 30-07-2024
C.net Magister
Abstract
This essay explores the integration of AI technologies, specifically ChatGPT, into ESL education to enhance the feedback process. It argues for a rubric-based framework to ensure the feedback aligns with pedagogical objectives and effectively meets student needs. The discussion includes various studies highlighting the importance of feedback in language learning and the potential of AI to offer timely, personalized feedback. By employing a systematic evaluation of ChatGPT’s responses through a well-defined rubric, educators can refine the feedback to be more supportive and effective. This approach not only optimizes AI's utility in ESL education but also promotes a deeper understanding of effective teaching and learning strategies. The essay underscores the transformative potential of AI in education, advocating for a balanced integration that enhances rather than replaces traditional educational methods.
Keywords: Artificial Intelligence; ESL Feedback; Rubric Evaluation; Prompt Engineering; Language Learning
Resumen
Este ensayo explora la integración de tecnologías de IA, específicamente ChatGPT, en la educación de ESL para mejorar el proceso de retroalimentación. Argumenta a favor de un marco basado en rúbricas para asegurar que la retroalimentación se alinee con los objetivos pedagógicos y satisfaga efectivamente las necesidades de los estudiantes. La discusión incluye varios estudios que destacan la importancia de la retroalimentación en el aprendizaje de idiomas y el potencial de la IA para ofrecer retroalimentación oportuna y personalizada. Al emplear una evaluación sistemática de las respuestas de ChatGPT a través de una rúbrica bien definida, los educadores pueden refinar la retroalimentación para que sea más efectiva y de apoyo. Este enfoque no solo optimiza la utilidad de la IA en la educación de ESL, sino que también promueve una comprensión más profunda de las estrategias efectivas de enseñanza y aprendizaje. El ensayo subraya el potencial transformador de la IA en la educación, abogando por una integración equilibrada que mejore, en lugar de reemplazar, los métodos educativos tradicionales.
Palabras clave: Inteligencia Artificial; Retroalimentación ESL; Evaluación por Rúbricas; Ingeniería de Prompts; Aprendizaje de Idiomas
Resumo:
Este ensaio explora a integração de tecnologias de IA, especificamente ChatGPT, na educação ESL para aprimorar o processo de feedback. Ele defende uma estrutura baseada em rubricas para garantir que o feedback esteja alinhado com os objetivos pedagógicos e atenda efetivamente às necessidades dos alunos. A discussão inclui vários estudos destacando a importância do feedback no aprendizado de idiomas e o potencial da IA para oferecer feedback oportuno e personalizado. Ao empregar uma avaliação sistemática das respostas do ChatGPT por meio de uma rubrica bem definida, os educadores podem refinar o feedback para ser mais favorável e eficaz. Essa abordagem não apenas otimiza a utilidade da IA na educação ESL, mas também promove uma compreensão mais profunda de estratégias eficazes de ensino e aprendizagem. O ensaio sublinha o potencial transformador da IA na educação, defendendo uma integração equilibrada que melhore, em vez de substituir, os métodos educativos tradicionais.
Palavras-chave: Inteligência Artificial; Feedback ESL; Avaliação de Rubrica; Engenharia de Prompt; Aprendizagem de idiomas
Thesis
In the evolving landscape of education, feedback serves as a crucial element, particularly in ESL education, where it transcends mere correction to include guidance and support (Yu & Yang, 2021). AI technologies like ChatGPT introduce innovative ways to deliver feedback, promising immediacy and adaptability that traditional methods lack. This essay argues for leveraging a rubric-based framework to enhance ChatGPT's feedback, ensuring it aligns with pedagogical objectives and student needs by focusing on clarity, constructiveness, balance, consistency, encouragement, and timeliness.
The necessity for effective feedback in ESL education is evident as it not only supports but catalyzes learning and skill development. Ismail, Maulan, & Hasan (2008) found that various types of feedback foster self-revision, critical for ongoing learning. Razali & Jupri (2014) show that specific feedback types like criticism can drive substantial student revisions, enhancing the learning process. This specificity of feedback types could be achieved through precise prompt engineering.
Further insights by Hyland and Hyland (2006) advocate for feedback that is both corrective and constructive, facilitating improvement pathways. This is complemented by López Casoli (2023) who highlights how ESL students' perceptions of feedback significantly influence their engagement and motivation, reinforcing the need for feedback practices to be adaptive and supportive (Agbayahoun, 2016; Yu & Yang, 2021).
The integration of AI in ESL teaching, explored by Wang and Brown (2007), has evolved, with recent studies like those by Kostka and Toncelli (2023) suggesting a shift towards more integrated feedback systems. Yoon, Miszoglad, and Pierce (2023) note the capabilities of AI-generated feedback in addressing writing proficiency, particularly coherence and cohesion, yet underscore the need for feedback that is nuanced and aligned with students' objectives.
AI-driven feedback, as explored by Marvin et al. (2024) and Jacobsen and Weber (2023), relies heavily on the sophistication of prompt engineering to be effective. Their research into the dynamics of prompt engineering shows that well-crafted prompts can significantly enhance the quality of feedback, making it more relevant and supportive for ESL learners.
The sucessfull use of AI technology, particularly ChatGPT, into ESL education marks a significant advancement in how feedback is utilized to enhance learning. By adopting a systematic approach to prompt engineering, as advocated by Schmidt et al. (2023), and ensuring feedback adheres to a well-defined rubric, the potential of AI to enrich the educational experience through personalized and effective feedback can be fully realized. This not only supports diverse learning pathways but also fosters a deeper understanding of effective teaching and learning principles, paving the way for future integrations of technology in education.
Development
Adopting a rubric-based approach to evaluate and refine ChatGPT's feedback represents a strategic melding of technology and pedagogical insight. The proposed rubric encompasses six key dimensions reflective of effective feedback: clarity, constructiveness, balance, consistency, encouragement, and timeliness. Each dimension is articulated through specific measurable criteria, allowing for a nuanced assessment of feedback quality.
· Clarity involves the feedback's ability to be understood by ESL learners, avoiding technical jargon and complex linguistic structures that may obscure the message.
· Constructiveness focuses on feedback's capacity to guide improvement without demoralizing the learner, emphasizing positive reinforcement alongside constructive critique.
· Balance ensures that feedback provides a holistic view of the learner's work, highlighting strengths as well as areas for improvement.
· Consistency relates to the uniform application of evaluative criteria across different instances of feedback, fostering a reliable learning environment.
· Encouragement seeks to bolster learner confidence and motivation through positive reinforcement and recognition of progress.
· Timeliness underscores the importance of providing feedback in a timely manner, aligning with the learners' immediate needs and facilitating prompt revision and improvement.
By applying this rubric to ChatGPT-generated feedback, educators can identify specific areas where adjustments are necessary, whether in the formulation of prompts given to ChatGPT or in the interpretation and application of the feedback provided. For instance, if feedback scores low on clarity, educators might simplify the prompts to generate more straightforward responses. Conversely, if feedback lacks constructiveness, prompts can be tailored to elicit responses that offer more actionable suggestions for improvement.
Implementing this rubric-based evaluation involves a cyclic process of assessment, adjustment, and reassessment. Initially, educators would collect a sample of ChatGPT-generated feedback on ESL students' writing assignments. This feedback would then be evaluated against the rubric, with scores assigned to each dimension. Based on these scores, educators could adjust the prompts given to ChatGPT, seeking to enhance the feedback's alignment with the rubric's criteria. Subsequent rounds of feedback and evaluation would iteratively refine the process, ideally leading to a progressive improvement in feedback quality.
Consider, for example, a scenario where ChatGPT's feedback on a student's essay is evaluated as highly constructive but lacking in clarity. The educator could then modify the prompt to ChatGPT, requesting feedback that not only highlights areas for improvement but also explains these points in simpler terms. This adjustment could lead to feedback that is both constructive and clear, thereby enhancing the student's understanding and ability to act on the feedback.
Creating a rubric to evaluate the quality and effectiveness of feedback involves a multi-step process that starts with defining the key dimensions of effective feedback. Drawing from educational research and best practices in ESL instruction, these dimensions include clarity, constructiveness, balance, consistency, encouragement, and timeliness. The next step involves operationalizing these dimensions into specific, observable behaviors or characteristics, enabling educators to assess feedback with objectivity and precision.
The rubric is structured to offer five levels of performance for each dimension, ranging from "Excellent" to "Needs Improvement." This gradation allows educators to identify not just the presence of effective feedback elements but also their degree of effectiveness. Below is a simplified version of the rubric with the six dimensions mentioned:
Table 1
|
Dimension |
Excellent
(5) |
Good
(4) |
Satisfactory
(3) |
Fair
(2) |
Needs Improvement (1) |
|
Clarity |
Feedback is exceptionally clear and concise, using simple language that is easily understood by ESL students. |
Feedback is clear with minor ambiguities that do not impede understanding. |
Feedback is generally clear but may include some jargon or complex language. |
Feedback occasionally lacks clarity, making it difficult for students to understand without assistance. |
Feedback is often unclear, using complex language or terminology unfamiliar to ESL students. |
|
Constructiveness |
Feedback provides specific, actionable suggestions for improvement and highlights strengths effectively. |
Feedback is helpful and provides actionable suggestions but may lack balance. |
Feedback provides general suggestions for improvement with some mention of strengths. |
Feedback offers limited actionable suggestions and focuses more on weaknesses. |
Feedback is vague or generic, with little to no actionable advice or recognition of strengths. |
|
Balance |
Feedback is well-balanced, offering a thorough assessment of strengths and areas for improvement. |
Feedback provides a good balance but may emphasize one aspect slightly more than the other. |
Feedback is somewhat balanced but needs more equal representation of positives and negatives. |
Feedback tends to focus more on either strengths or weaknesses, lacking balance. |
Feedback focuses almost exclusively on either strengths or weaknesses. |
|
Consistency |
Feedback consistently applies criteria across all evaluated aspects, showing no variance in evaluative standards. |
Feedback is mostly consistent with slight variations in criteria application. |
Feedback shows occasional inconsistency in the application of evaluative criteria. |
Feedback is inconsistent, applying different standards at different times. |
Feedback lacks consistency, with significant variances in how criteria are applied. |
|
Encouragement |
Feedback is highly encouraging, motivating the student with positive reinforcement and recognition of effort. |
Feedback is encouraging and supportive, but could be more personalized. |
Feedback provides general encouragement but lacks specificity. |
Feedback offers minimal encouragement, with a focus on critique over motivation. |
Feedback is discouraging or lacks any positive reinforcement or motivation. |
|
Timeliness |
Feedback is provided promptly, allowing for immediate reflection and application by the student. |
Feedback is provided soon after submission, with minimal delay. |
Feedback is provided in a reasonable timeframe but could be faster. |
Feedback is delayed, hindering timely reflection and improvement by the student. |
Feedback is significantly delayed, offering little value to the student's learning process. |
Source: ChatGPT 4.0 (2024)
In operationalizing the rubric's
dimensions, the evidence suggests a multifaceted approach to feedback can
stimulate a broader range of student revisions. The findings from Razali &
Jupri (2014), indicating that criticism, suggestions, and praise each play a
role in encouraging student revisions, inform a nuanced approach to feedback
that encompasses a spectrum of responses to student writing .
This approach aligns with the development of a rubric that values feedback's
multifaceted nature, suggesting that educators should vary their feedback to
address different aspects of student writing comprehensively.
Incorporating Ismail, Maulan, & Hasan's (2008) insights into the experimental effects of teacher feedback on ESL students' writing performance further enriches this discussion. Their study underscores the transformative potential of feedback on students' ability to engage in self-revision, highlighting the importance of feedback that prepares students for future writing endeavors. This resonates with the development section's emphasis on feedback that not only addresses immediate writing issues but also fosters an environment conducive to long-term learning and improvement.
To create prompts that guide ChatGPT in generating feedback aligned with the established rubric, educators must navigate the delicate balance between specificity and flexibility. This balance is crucial to ensure that the feedback is not only relevant and actionable but also adaptable to the varied needs and proficiency levels of ESL students. Prompt engineering, as discussed by Marvin et al. (2024), involves crafting prompts that precisely communicate the educator's goals to the AI, enabling it to generate responses that meet specific pedagogical criteria.
The process starts with the identification of key dimensions of effective feedback as outlined in the rubric—clarity, constructiveness, balance, consistency, encouragement, and timeliness. For each dimension, prompts must be engineered to elicit responses from ChatGPT that exemplify these qualities. For instance, to ensure clarity, a prompt might specifically ask ChatGPT to "provide feedback using simple language suitable for an ESL learner at the B1 proficiency level, avoiding complex grammar constructions and vocabulary."
Jacobsen and Weber (2023) offer insights into the potential of ChatGPT as a feedback tool, suggesting that effective prompt engineering can mitigate some of the AI's limitations in understanding the nuances of human learning processes. This underscores the importance of incorporating explicit instructions in the prompts regarding the desired structure and tone of the feedback, ensuring it is both encouraging and balanced.
Following the guidance from Schmidt et al. (2023), educators could develop a catalog of prompt patterns that have been effective in generating the desired types of feedback. This catalog serves as a dynamic resource, evolving based on the continuous assessment of ChatGPT's feedback against the rubric and the specific needs of the ESL students. For example, if feedback generated after a prompt consistently scores high on constructiveness but low on encouragement, the educator can refine the prompt pattern to include explicit requests for positive reinforcement and recognition of effort.
Implementing these refined prompts involves an iterative process where feedback from ChatGPT is continually evaluated against the rubric. This feedback loop not only fine-tunes the prompt engineering process but also enhances the overall quality of feedback provided to students. Educators can leverage this approach to tailor ChatGPT's feedback to the diverse and evolving needs of ESL learners, making the feedback process more aligned with educational objectives and responsive to student progress.
To further personalize and refine the feedback process, incorporating a mechanism where students can rate the helpfulness of the feedback could prove invaluable. Inspired by findings in the study on 'Rubrics and Corrective Feedback in ESL Writing,' which highlighted a learner's preference for feedback that addresses more than just form, a feature could be added to ChatGPT allowing students to provide immediate responses to the feedback they receive (Ene & Kosobucki, 2016). This direct input from students would enable the AI to regenerate feedback that better aligns with their individual needs and preferences, thereby enhancing learner satisfaction and the educational impact of the feedback.
By systematically employing principles of prompt engineering and incorporating direct student responses to refine AI interactions, educators can significantly enhance the efficacy of AI-generated feedback. This approach of integrating ChatGPT in educational settings is particularly effective in the context of ESL education, where it aligns feedback with structured evaluation rubrics and adapts based on student-rated feedback. Such dynamic adjustments ensure that the feedback not only meets educational objectives but also resonates with individual student needs, making the learning experience more personalized and effective. This forward-thinking methodology underlines the transformative potential of AI tools in education, offering a tailored educational journey that is responsive to the evolving landscapes of student engagement and pedagogical demands.
Conclusions
The integration of AI technologies like ChatGPT into ESL education offers a profound potential to enhance the learning experience. However, the effectiveness of AI-generated feedback depends heavily on its alignment with pedagogical objectives and its resonance with student needs. By employing a rubric-based evaluation framework, educators can systematically assess and refine ChatGPT's feedback, ensuring it adheres to the principles of effective feedback. This approach not only optimizes AI's utility in language learning but also highlights the importance of integrating technology in education thoughtfully and pedagogically (Cui, 2021).
Enhancing ChatGPT's feedback through rubric-based evaluation marks a significant shift in educational practices, where AI becomes instrumental in shaping the learning experience. As noted by Marvin et al. (2024), prompt engineering is crucial in ensuring that interactions with AI are pedagogically meaningful and aligned with instructional goals. This requires a deep understanding of both the AI's capabilities and the educational context, highlighting the art and science behind effective prompt engineering.
Jacobsen and Weber (2023) discuss the dual nature of AI-driven feedback, underscoring the challenges and opportunities it presents. Effective prompt engineering enables educators to transform potential pitfalls into avenues for enhanced student engagement and learning. By integrating effective feedback principles into the prompts, guided by the developed rubric, educators can tailor ChatGPT's feedback to meet the unique needs of each ESL learner.
Furthermore, Stevenson and Phakiti (2019) emphasize the supportive role of automated feedback systems in language learning. Implementing ChatGPT as a supplementary tool allows for enriched educational interactions, where students can share AI-generated feedback with teachers. This collaborative approach enhances the educational process, providing a bridge between AI capabilities and the nuanced understanding offered by human instructors, thus improving both student involvement and pedagogical effectiveness.
The continuous evaluation and refinement of ChatGPT's feedback using a catalog of prompt patterns (Schmidt et al., 2023) represent a dynamic approach to improving AI's integration into feedback mechanisms. This process not only optimizes the AI's output but also deepens educators' understanding of effective feedback and its impact on language learning.
In conclusion, the journey toward refining AI feedback for ESL education demonstrates the critical interplay between technology and pedagogy. It underscores the essential role of prompt engineering in maximizing the pedagogical benefits of AI feedback systems. As we look ahead, the insights from this exploration advocate for a balanced approach to integrating AI in education, ensuring that the technology enhances rather than replaces traditional educational values. This balanced integration promises to significantly enrich the educational experience, setting the stage for a future where technology and pedagogy merge to foster deeper and more effective learning.
Reference
Agbayahoun, J. P. (2016). Teacher Written Feedback on Student Writing: Teachers' and Learners' Perspectives. Theory & Practice in Language Studies. Theory and Practice in Language Studies, 6(10), pp. 1895-1904. http://dx.doi.org/10.17507/tpls.0610.01
Cui Y, Schunn CD, Gai X, Jiang Y and Wang Z (2021) Effects of Trained Peer vs. Teacher Feedback on EFL Students’ Writing Performance, Self-Efficacy, and Internalization of Motivation. Front. Psychol. 12:788474.
Ene, E., & Kosobucki, V. (2016). Rubrics and corrective feedback in ESL writing: A longitudinal case study of an L2 writer. Assessing writing, 30, 3-20. https://doi.org/10.1016/j.asw.2016.06.003
Hyland, K., & Hyland, F. (2006). Feedback on second language students' writing. Language teaching, 39(2), 83-101. https://doi.org/10.1017/S0261444806003399
Ismail, N., Maulan, S., & Hasan, N. H. (2008). The impact of teacher feedback on ESL students’ writing performance. Academic Journal of Social Studies, 8(1), 45-54.
Jacobsen, L. J., & Weber, K. E. (2023, September 29). The Promises and Pitfalls of ChatGPT as a Feedback Provider in Higher Education: An Exploratory Study of Prompt Engineering and the Quality of AI-Driven Feedback. https://doi.org/10.31219/osf.io/cr257
Kostka, I., & Toncelli, R. (2023). Exploring applications of ChatGPT to English language teaching: Opportunities, challenges, and recommendations. Tesl-Ej, 27(3), n3.
López Casoli, M. (2023). The Effectiveness of Teacher Feedback on Writing: Teaching English As A Foreign Language Students' Perceptions. UCMaule-Revista Académica de la Universidad Católica del Maule, (65)., ISO 690.
Marvin, G., Hellen, N., Jjingo, D., Nakatumba-Nabende, J. (2024). Prompt Engineering in Large Language Models. In: Jacob, I.J., Piramuthu, S., Falkowski-Gilski, P. (eds) Data Intelligence and Cognitive Informatics. ICDICI 2023. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-99-7962-2_30
Razali, R., & Jupri, R. (2014). Exploring teacher written feedback and student revisions on ESL students’ writing. IOSR Journal of Humanities and Social Science (JHSS), 19(5), 63-70.
Schmidt, D. C., Spencer-Smith, J., Fu, Q., & White, J. (2024). Towards a catalog of prompt patterns to enhance the discipline of prompt engineering. ACM SIGAda Ada Letters, 43(2), 43-51.
Stevenson, M., & Phakiti, A. (2019). Automated feedback and second language writing. Feedback in second language writing: Contexts and issues, 125-142.
Wang, J., & Brown, M. S. (2007). Automated essay scoring versus human scoring: A comparative study. Journal of technology, Learning, and assessment, 6(2), n2. https://eric.ed.gov/?id=EJ838612
Yoon, S. Y., Miszoglad, E., & Pierce, L. R. (2023). Evaluation of ChatGPT Feedback on ELL Writers' Coherence and Cohesion. arXiv preprint arXiv:2310.06505.
Yu, R., & Yang, L. (2021). ESL/EFL Learners' Responses to Teacher Written Feedback: Reviewing a Recent Decade of Empirical Studies. Frontiers in Psychology, 12, 735101.
