2 sigma problem solving using GPT4
How Khan Academy is using GPT4 to create a personalised AI tutor on their platform.
I came across an interesting TED talk by Sal Khan, the founder and CEO of Khan Academy (educational organization), where he describes their usage of GPT-4. They gained access to GPT-4 in August 2022 and spent 6-7 months developing their solution on top of it. They call it Khanmigo - an educational chatbot, an AI tutor that follows the principles of the Socratic method (a philosophical approach to inquiry and learning that involves asking questions to stimulate critical thinking).
Khanmigo can ask questions about the text, provide leading questions instead of direct answers, and act as a writing coach. If you write, say, an essay, you can ask it “Does my evidence support my claim?” It will then highlight parts of your text and tell you which parts do not support your claim, and ask you to explain why. There is also a teacher's mode - the bot acts as a teacher's guide. It not only explains the answer but also explains how you might want to teach it.
That what Khan says about development process:
And I could go through the whole list of everything we've been working on, many, many people for over six, seven months to make it feel magical. But perhaps the most intellectually interesting one is we realized, and this was an idea from an OpenAI researcher, that we could dramatically improve its ability in math and its ability in tutoring if we allow the AI to think before it speaks. So if you're tutoring someone and you immediately just start talking before you assess their math, you might not get it right. But if you construct thoughts for yourself, and what you see on the right there is an actual AI thought, something that it generates for itself but it does not share with the student, then its accuracy went up dramatically, and its ability to be a world-class tutor went up dramatically
So what is 2 Sigma problem?
I like how Khan explains the main value of such augmented AI chatbots. He talks about the 2 sigma problem - a study by educational psychologist Benjamin Bloom. Bloom found that students who received one-to-one tutoring with a skilled teacher performed two standard deviations (2 sigmas) better than students in a conventional classroom setting. This suggests that while conventional classroom teaching is effective, personalized one-to-one tutoring can significantly improve learning outcomes. Here is an image from the study - you can see that 1-to-1 teacher-student ratio curve is way to the right and more narrow than 1-to-30 curve. In simple words, average student in conventional approach scores as the worst student from individual approach, and the student from 1-to-1 is way better than in 1-to-30.
The problem, as Khan points out, is that the student-to-teacher ratio in the modern world is 300 to 1. Now, I don't know the source of this information, and it is a bit hard to give one number once you start diving into it. In the Netherlands, for example, it is 20-to-1 in universities (closer to 1-to-30 as in the paper). Nevertheless, the idea is that an AI chatbot like Khanmigo can give us personalized tutoring, which is a super-efficient way of teaching and learning.
I am eager to see how my kids will experience learning something in 20 years.
Overall, that’s one of the best and inspiring examples of GPT4 usage that I’ve seen so far. Hope to see more of that in coming years!