LLM-Powered Auto-Grader

All Courses: Pedagogy and Best Practices
, ,
1

Hi everyone!

After years of completion-grading CSD projects, I decided to see if I could offload some of the more mindless parts to an LLM, and the auto-grader that emerged was totally worth the development time. It works with both Game Lab and App Lab. I only use it for simple, completion-based assessment levels (e.g. move the square on top of the circles and make it purple), not for original creative projects. It ships with criteria for CSD Unit 3a, but you can write your own criteria for any course or unit. It requires a Gemini API key, but those can be created for free under a personal Google account (in my experience, the free tier is enough to cover a single teacher’s grading needs). There’s a link to a YouTube video in the Help / Setup Guide dropdown that shows how to create a Google API key.

What the Student Experience Looks Like

When finished with an assessment level, students:

  1. Click Share → Copy link to project in Code.org
  2. Open a Google Form link that you post for them (the autograder will generate the form for you, all you have to do is share the link)
  3. Enter their name, select their class period, select the assignment they’re submitting (e.g. Lesson-03-Level-08), paste their project link, and click Submit
  4. Within a minute or so, they receive an email from you containing the auto-graded results, their score with a precise breakdown of which requirements they met, which they didn’t, and why

What the Teacher Sees

Everything comes together in an easy-to-use Google Sheet with these tabs:

  • Form Responses 1: This is where the Google Form “dumps” entries into the sheet. You don’t have to touch this.
  • Submissions: Student submissions are automatically pulled from Form Responses into this sheet, where you can see each one on its own row with its auto-graded score and the time the student was emailed with the result.
  • Criteria: This sheet lets you set the specific grading criteria, in natural language, for each assessment level (e.g. “The score variable is increased when the bunny touches the coin”). I’ve included my battle-tested set of criteria for CSD 2025 Unit 3a Lessons 3–25 assessment levels, but you can change them or replace them entirely. I’ve verified that this works with both Game Lab and App Lab, but most of my testing has been with Game Lab. You can give levels whatever LevelID you want, and when you click Create Submission Form it will automatically give the form a dropdown with all LevelIDs that are present in your Criteria tab.
  • Grade View P# — There’s a separate tab for every class period you teach (auto-generated when you first run the setup — you can choose up to 8 periods). On these grade view sheets you can see exclusively the submissions for that class, grouped by assignment and sorted by last name for easy transfer into your gradebook software.

Getting Started

The easiest way is to make a copy of the Starter Sheet:

:backhand_index_pointing_right: Make a copy of the Code.org Autograder Starter Sheet

After making a copy, wait a few seconds for the Autograder menu to appear at the top, then:

  1. Click Autograder → Initial Setup…, select the class periods you teach, and click Create Sheets
  2. Add your API key: go to Extensions → Apps Script → :gear: gear icon → Script Properties → paste your key as the value for GEMINI_API_KEY
  3. Click Autograder → Test API Connection to make sure it’s working
  4. Peruse the Criteria tab and make any changes you like
  5. Click Autograder → Create Submission Form — it’ll give you the form link to share with your students

You’re good to go!

If you’re slightly more technical, here’s the GitHub repo where you can look things over and follow the instructions in the README.

A Note on Permissions

Because this tool runs a script that pulls in code from an external source (code.org) and sends emails to students from your account (the auto-graded results), Google will show a scary-looking warning and ask you to grant permissions. It’s safe, but you can verify for yourself (a good habit anyway!) by pasting the JavaScript source code into an LLM of your choice and having it do a security audit.

Contributing

Criteria sheets that you create could be highly valuable to the teacher community! If you’d like to share, download your Criteria sheet as a CSV file and share it either through GitHub (pull request instructions are in the repo) or just email it to me: fotoloca@gmail.com

3 Likes