In this video, we’ll explore OpenAI’s function calling feature and show you how it can automatically grade student submissions on Canvas using MongoDB data. This feature allows large language models like GPT to decide whether to invoke specific functions, such as retrieving database information or calling external APIs. The model determines the appropriate function to call based on user input, but keep in mind that OpenAI doesn't run the functions. You, the developer, must write the code to handle function execution.
Demo notebook: https://github.com/xbwei/data-analysis-with-generative-ai/blob/main/Automating-Grading-with-OpenAI-Function-Calling.ipynb
Step 1: Set Up Your Canvas and OpenAI API Keys
To get started, you need a Canvas API key, which can be easily generated from your Canvas account. Here’s how:
Go to your Canvas profile and click on Settings.
Scroll down to find Access Tokens, click to generate a new token, and specify its purpose and expiration date.
Copy the token and save it in a secure location like AWS Secrets Manager.
Additionally, you’ll need an OpenAI API key, which can be purchased from OpenAI’s website. These two keys will allow you to access Canvas data and use OpenAI models.
Step 2: Install the Required Libraries
You’ll need three libraries for this task:
Canvas API: To retrieve student submissions and post grades on Canvas.
OpenAI Library: To interact with OpenAI’s GPT models and use function calling.
PyMongo: To query the MongoDB database where students' collected data is stored.
You will also use AWS Secrets Manager to securely store and retrieve your credentials.
Step 3: Retrieve Student Submissions from Canvas
Before you can grade students, you need to retrieve their submissions. To do this, you need:
Canvas API URL: This is the URL provided by your university (e.g., canvas.jmu.edu).
Course ID: You can find this in the URL when you access your course in Canvas.
Assignment ID: This corresponds to the specific assignment you want to grade.
Student ID: You’ll need the student’s ID to retrieve their specific submission.
In this example, we’ll use a test student account to demonstrate the process.
Step 4: Define Grading Logic
In the grading scenario, you ask students to answer five questions based on data they collected in MongoDB. These questions include:
What is the highest salary from the data?
Which location has the most job postings?
Which organization posted the most jobs?
How many jobs started in October 2024?
How many jobs mention AI in the summary?
You’ll use OpenAI to compare students’ answers to the correct answers and calculate their scores. If a student doesn’t provide a valid MongoDB connection string, they receive zero points for the assignment.
Step 5: Function Calling for Automated Grading
The next step is to automate the grading process using OpenAI’s function calling. With function calling, you define the logic in a function, and the large language model will decide whether to call that function based on the input it receives.
You’ll create a function that checks the correctness of student responses by:
Querying MongoDB to calculate the answers to the five questions.
Comparing these answers to the student’s submission.
Generating a grade and providing feedback in a structured format (JSON).
For example, if students answer all the questions correctly, they’ll receive full points. If the answer is incorrect, the model will deduct points and explain why.
Step 6: Posting Grades and Feedback
Once OpenAI has graded the submission, the final step is to post the grade and feedback to Canvas. Based on the students' answers, OpenAI will generate a score and a comment for each student. You can review these grades before posting them to Canvas. Once satisfied, use the Canvas API to submit the grade and feedback.
Important Reminder: Before implementing LLMs for automating grading, check with your institution to ensure that the use of AI for grading aligns with their policies and guidelines.
Comentarios