Helge Pfeiffer will give a talk about his work on On the Effects of a Local LLM in an Introductory Programming Exam. Details below.
SPEAKER
Helge Pfeiffer, Prof., ITU.
TITLE
Faster, More Correct, Higher Scores: The Effects of a Local LLM in an Introductory Programming Exam
ABSTRACT
Contribution: In this presentation, I demonstrate that
a local large language model (LLM) significantly improves
results of students in an Introductory Programming (IP) exam.
Consequently, only restricting internet access in IP exams cannot
ensure integrity of exams in which students have access to
computers.
Background: Since the increase of popularity of LLMs, edu-
cators are concerned about integrity of exams. Some educators
discuss the possibility of conducting IP exams on computers that
are not connected to the internet to ensure integrity of exams,
since students often rely on web-clients like ChatGPT to access
LLMs that are hosted on powerful data centers. However, many
LLMs can run directly on consumer grade hardware, like student
laptops, which can be used in exams at our institution.
Research Questions: RQ1 : How does a local LLM affect lead
time for solving an IP exam? RQ2 : How does a local LLM affect
the correctness of source code in an IP exam? RQ3 : How does a
local LLM affect scores in an IP exam? RQ4 : How does the way
students prompt a local LLM affect their grade?
Methodology: We conduct a within-subject counter-balanced
controlled experiment with 24 experimental subjects who are
drawn from the student population of an IP course.
Findings: Experimental subjects with access to a local LLM
complete exam tasks faster (median decreases by ca. 25%), create
more correct solutions (median increases by ca. 32%), receive
more points (median increases by ca. 30%), and they score better
grades (median increases by. 41%).