Making our App learn from the student: AI machine learning embedded in EdTech.
At Citizen Literacy we are primarily focussed on improving the literacy (reading and writing) of adults who are already fluent speakers of English and developing a ground breaking adult literacy training course and investigating how we can use technology to assist learners and tutors.
Part of our solution (the app) uses voice and handwriting recognition to help learners practice between their face to face sessions. We are also exploring other options such as the use of chatbots and AI to help absolute beginners gain confidence to join a face to face class.
Thinking about our App.
Apps can no longer be one size fits all: we need to make our App learn from the student, so it can teach the student more effectively. How to do this requires some sort of AI machine learning embedded in our EdTech.
Who else is doing this out there in EdTech Universe?
The obvious choice is one of the world leading companies: our good friends in Silicon Valley at Roybi Robot (https://roybirobot.com/) featured in TIME and named “the best invention”. Their Roybi Robot is an AI powered educational learning toy robot for kids in language and STEM. Recently we were talking with Roybi Robot about developing effective lessons for a wide range of people and the challenges when you are doing this using technology instead of a human. Teachers treat each student as individuals: dealing with accents, physical abilities; understanding of concepts, facial expressions and responding to student emotions and so on. Teachers are never going to be replaced by technology, let’s make that clear!
Teachers can adapt lessons to each student: how can technology mimic this?
Can technology adapt to accents? Can technology adapt the lessons to individuals, so the student journey through a lesson is unique to themselves? These are the sorts of questions and challenges we discussed with Roybi Robot. Making an online course where “the technology is invisible: it just works” is a goal. Making a course where the technology can learn from the student and modify the course journey to help support the student- the way a teacher would, is another goal. One current area is the question of whether our “invisible” technology can adapt to a student’s accent?
USE CASE EXAMPLE
We have student working their way through the course. They come across some reading tasks and they need to read out the words and be assessed on whether they have said the word correctly.
They see a Pin and say the word “pin”. the technology thinks they have said pen*. they try again. get the same feedback. this could go on and on. the student gets no-where; becomes disheartened and quits after a few other tasks.
* pin /pen Minimal pairs are pairs of words that only have one sound different. Minimal pairs are a useful way to highlight a sound in a meaningful context and to show the learner how important correct pronunciation of the sound is.
Currently technology in APPs just can’t adapt to the student; the way a teacher could. Speech recognition can give you the top ten guesses of the word: but if the accent is quite strong; or the student has any speech impediments such as dyspraxia, then the technology is just not able enough to support a student the way a teacher can.
Solution
After a couple of attempts of saying a word; we can trigger AI machine learning within the APP; such that the APP can ask the user to say maybe one or two words, the App then trains itself to understand the difference. So, instead of simple speech recognition failing the student the machine learning can overrule the standard speech recognition and the student can continue their studies. In a similar way a teacher can adapt to a student and better understand their accent or pronunciation. It takes just a short time for a human to adapt, so too must the technologies we use in EdTech.
How to do this
We’ll assume the app and student are working along fine until the student hits a barrier with pronunciation. After a few attempts the app qualifies the need for it to learn the student’s pronunciation for particular words and asks the user to say the word. Alternatively, we can train the app at the start of a session and get the student to repeat a word so the app can learn to recognise it in the future
Techy bit
The APP will fine tune itself by building a new model using transfer learning with TensorFlow. The app can then use this functionality as and when required; it will then use the rest of the app functionality as is and only call in the learning functionality when the app needs to adapt.
This proof of concept here demonstrates that machine learning can work even with very little input from the student.
A bit more on the TensorFlow settings
Here we are using TensorFlow with a Chrome browser on Windows 10 laptop.The student is using a headphone with microphone on Laptop windows 10 with Chrome
Main TensorFlow settings
// Array of words that the recognizer is trained to recognize. predictWord() with {probabilityThreshold: 0.99});
Listening for new labels (transfer learning) with overlap of 0.5
recognizer.listen(async ({spectrogram: {frameSize, data}}) overlapFactor: 0.5, includeSpectrogram: true, invokeCallbackOnNoiseAndUnknown: true
So far so good. Will let you know how this goes. Until then: here at Citizen Literacy we just keep on keeping on!
Colin