ManuAI
VI
All 12 lessons

Lesson 6 of 12 · How machines learn

Learning from rewards

A warmer or colder clapping game and a browser lab where students design a reward for a walking robot show that a machine learns exactly what you reward.

Or print one part

In the print dialog choose A4, scale 100% (or Default) and turn on Background graphics for the coloured parts. Choose Save as PDF to keep a file.

Lesson 6 of 12 · How machines learn · Teacher's plan

Learning from rewards

A warmer or colder clapping game and a browser lab where students design a reward for a walking robot show that a machine learns exactly what you reward.

Students will be able to

  • describe learning from rewards as trying, getting feedback and adjusting
  • explain that a machine learns what the reward measures, which may differ from what we meant
  • spot a loophole in a reward and suggest a better one
  • compare learning from rewards with learning from labelled examples

Prepare

  • A hidden task written on a folded slip, such as "touch the window" or "pick up the red pen"
  • A projector and one teacher device with internet, or a few devices for pairs
  • The ManuAI page /play/lab/kit-learns-to-walk loaded in advance (free, no account)
  • One worksheet per student

Words to know

Reward:
A signal that says "that was good", given after an action.
Feedback:
Information about how well you did, which you use to adjust what you do next.
Loophole:
A way to get the reward without doing what was really wanted.

Main activity · In the browser: ManuAI

Warmer or colder, then teach a robot to walk

One student finds a hidden task only from the class clapping louder or softer. Then, on one projected screen or in pairs, the class designs a reward in a ManuAI lab where a program teaches a fox character to walk, and watches it find loopholes when the reward is careless.

ManuAI pages for this lesson

The 45-minute plan · 45 minutes

Warm-up8 min

Warmer or colder clapping

  • Send one volunteer out of the room. Show the class the hidden task on the slip.
  • Bring the volunteer back. The class claps louder as they get closer to the task and softer as they move away. Nobody speaks.
  • When they succeed, ask the volunteer how they worked it out. Ask the class what they did as the teacher.
  • Name it: trying, feedback, adjusting. This is learning from rewards.
Activity17 min

Kit learns to walk

  • Open /play/lab/kit-learns-to-walk on the projector. Explain that the program tries many ways of moving and keeps the ones that score best.
  • Ask the class to choose a reward, for example "distance moved forward". Run it and watch several rounds.
  • Look for odd tricks, such as falling forward, wriggling or hopping. Ask: "Did it do what we asked, or what we meant?"
  • Change the reward together, for example add points for staying upright, and run again. Compare.
  • If pairs have devices, let them try one reward each and report the strangest result.
Practice15 min

Design a reward on paper

  • Students complete the worksheet, writing rewards for everyday tasks and looking for loopholes.
  • Pairs swap worksheets and try to find a loophole in each other's rewards.
  • Optional: show /play/lab/teach-kit-a-beat for two minutes as an example of learning a pattern from a recording, for comparison.
Wrap-up5 min

You get what you reward

  • Write on the board: a machine learns exactly what you reward.
  • Compare with lesson 5: there we gave labelled answers, here we only gave scores.
  • Hand out the exit ticket.

Teacher tips

  • Devices: one projected screen is enough for this lesson. The ManuAI labs are free and need no account. Never ask students to create accounts on other sites.
  • Clapping game in a large class: let one row clap at a time, or use a single clapper with a drum or desk tap to keep the noise manageable. Choose a task that is safe and stays inside the room.
  • Misconception: "the robot is being naughty" or "cheating on purpose". It has no intentions. It repeats whatever raised its score, so the loophole is a sign that the reward was written loosely.

Exit ticket answers

  1. A walking robot was rewarded only for distance moved forward. It learned to throw itself forward and fall. Why? B. Falling forward earned the reward, and the reward did not mention staying upright
  2. What is a loophole in a reward? Give an example. A good answer: A way to get the points without doing what was really wanted. For example, a robot rewarded for picking up rubbish could drop it and pick it up again.
  3. In learning from rewards, what does the learner get after each try? D. A score or signal saying how good the try was
Free to print and share in classrooms and at home.ManuAI · free AI lessons for everyone · ai.manusignal.com

Worksheet · Lesson 6 of 12 · How machines learn

Design the reward

NameClassDate

A machine will do whatever earns the most points. Write your rewards carefully.

  1. 1In the clapping game, what was the reward and how did the learner use it?

  2. 2For each task, write a reward, then a loophole a clever machine could use.

    TaskReward (points for...)Possible loophole
    Robot cleans the classroom floor
    Robot waters the plants
    Robot walks to the door

  3. 3A cleaning robot gets a point every time it picks up rubbish. What loophole might it find?

    • It stops working at night
    • It tips rubbish out and picks it up again to earn more points
    • It cleans more slowly
    • It asks a person for help

  4. 4Pick one reward from your table and improve it to close the loophole.

  5. 5How is learning from rewards different from learning from labelled examples (lesson 5)?

Free to print and share in classrooms and at home.ManuAI · free AI lessons for everyone · ai.manusignal.com

Cut along the dotted lines: two tickets per page.

Exit ticket · Lesson 6ManuAI

Learning from rewards

NameClassDate

  1. 1A walking robot was rewarded only for distance moved forward. It learned to throw itself forward and fall. Why?

    • AThe computer was broken
    • BFalling forward earned the reward, and the reward did not mention staying upright
    • CRobots like falling
    • DNobody gave it a reward

  2. 2What is a loophole in a reward? Give an example.

  3. 3In learning from rewards, what does the learner get after each try?

    • AThe full correct answer
    • BA new computer
    • CNothing at all
    • DA score or signal saying how good the try was