forked from ecraft2learn/ai
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
308 lines (305 loc) · 15.6 KB
/
index.html
File metadata and controls
308 lines (305 loc) · 15.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
<!doctype html>
<html>
<head>
<title>A guide to AI extensions to Snap!</title>
<meta charset='utf-8'>
<link href="css/ai-teacher-guide.css" rel="stylesheet">
<link rel="icon" type="image/png" href="images/eCraft2Learn-Favicon.png" />
<script src="js/ai-guide.js"></script>
</head>
<body>
<script src="js/translate.js"></script>
<h2>Enabling children and beginning programmers to build AI programs</h2>
<p>
<a href="?translate=1">Click here if you wish to read this and the AI guide in a language other than English</a>.
</p>
<h3>Introduction</h3>
<p>
The <a href="https://project.ecraft2learn.eu/">eCraft2Learn project</a> developed a set of extensions to
<a href="https://snap.berkeley.edu">the Snap! programming language</a> to enable children
(and non-expert programmers) to build AI programs.
The blocks are available as projects with examples of using the blocks as well as libaries to download and then import
into <a href="snap/snap.html" target="_blank">Snap!</a> or
<a href="http://snap4arduino.rocks/" target="_blank">Snap4Arduino</a>.
It is possible to <a href="./localhost.html">download the files</a> needed to run most of the blocks and projects
described here without an Internet connection.
</p>
<p>
Important updates are
<a href="https://docs.google.com/document/d/1Vf7c9e4l4SECOXAelYcr02bH3qnoVeykN_TThbJXQAw/edit?usp=sharing" target="_blank">documented here</a>.
</p>
<ol>
<li>
Enabling your sprites to speak in over a hundred languages.
<a href="snap/snap.html?project=speaking&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/speaking blocks.xml" download target="_blank">library</a>.
</li>
<li>
Enabling your sprites to listen to speech in over a hundred languages. And to recognize sounds.
<a href="snap/snap.html?project=listening&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/listening blocks.xml" download target="_blank">library</a>.
</li>
<li>
Enabling your sprites to see using the camera.
<a href="snap/snap.html?project=seeing&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/seeing blocks.xml" download target="_blank">library</a>.
</li>
<li>
Enabling your projects to do arithmetic on words.
<a href="snap/snap.html?project=words&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/words blocks.xml" download target="_blank">library</a>.
</li>
<li>
Enabling your projects to create, train, and use deep learning neural networks.
<a href="snap/snap.html?project=neural nets&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/neural nets blocks.xml" download target="_blank">library</a>.
</li>
<li>
Miscellaneous AI blocks (style transfer, image embedding, and using Wikipedia and Yahoo Weather).
<a href="snap/snap.html?project=misc AI&editMode" target="_blank">Project</a> or
<a href="AI-Teacher-Guide/misc AI blocks.xml" download target="_blank">library</a>.
</li>
</ol>
<h3 id="programming-guides">AI programming guides</h3>
<p>
A guide currently consisting of six chapters describes
the new blocks, possible projects, sample programs, background information,
and the larger context about AI and machine learning:
</p>
<table>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-1.html"><b>Adding speaking to your programs</b></a></td>
<td>Speech synthesis requires only speakers or earphones</td>
</tr>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-2.html"><b>Adding listening to your programs</b></a></td>
<td>Speech recognition requires a microphone (builtin or connected to a USB port)</td>
</tr>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-3.html"><b>Adding image recognition to your programs</b></a></td>
<td>Requires a camera (builtin or connected to a USB port) and registering to get <a href="AI-Teacher-Guide/chapter-3.html#api-keys">API keys</a></td>
</tr>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-4.html"><b>Adding pre-trained machine learning models to your programs</b></a></td>
<td>Many examples rely upon a camera. It is very slow unless your device has a <a href="#gpu">GPU</a>.</td>
</tr>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-5.html"><b>Working with words and language</b></a></td>
<td>No special hardware requirements</td>
</tr>
<tr>
<td><a class="guide-link" href="AI-Teacher-Guide/chapter-6.html"><b>Making machine learning neural nets</b>
</a></td>
<td>It is very slow unless your device has a <a href="#gpu">GPU</a>.</td>
</tr>
</table>
<h3 id="publications">Publications</h3>
<p>
The paper <a href="resources/Ai_programming_Constructionism_2018 v2.pdf" download>AI Programming by Children</a>
by Ken Kahn and Niall Winters describes this work and was published
in the Proceedings of the Constructionism 2018 Conference.
<a href="resources/EC-TEL_2018_source-files_48 kk edits changes accepted.pdf" download>AI Programming by Children using Snap! Block Programming in a Developing Country</a>
by Ken Kahn, Rani Megasari, Erna Piantari and Enjun Junaeti was published at the EC-TEL 2018 conference and
describes workshops with high school students in Indonesia using these AI blocks.
An early version of the AI extension to Snap! is described in the paper
<a href="https://ora.ox.ac.uk/objects/uuid:12124254-acce-4c11-a540-19e74530798d" download>
Child-friendly Programming Interfaces to AI Cloud Services</a>
also by Ken Kahn and Niall Winters.
</p>
<h3 id="presentations">Presentations</h3>
<p>
This work has been presented at several conferences and universities.
The <a href="https://docs.google.com/presentation/d/1QZMyeC0wa6xJ1KIFrQXgzCzO4qKKmb68clNM_nvHFPs" target="_blank">
slides can be found here</a>.
There are also <a href="https://docs.google.com/presentation/d/1xG1qpBohHSXwKoK-E1l05kO3O8P8dR1KCAeW4tjMK-g/edit?usp=sharing" target+"_blank">
slides</a> from a talk about word embeddings and mathematics education.
</p>
<h3 id="sample-programs">Sample programs</h3>
<p>
The following demonstration programs use the AI Snap! blocks.
They run best in the Chrome browser.
</p>
<div class="flow-row">
<!-- Added &noRun to all projects that require speech output since must be triggered by a user action -->
<figure>
<img src="images/speak randomly.png">
<!-- doesn't need &noRun since must click or touch to activate -->
<figcaption><a href="snap/snap.html?project=speak%20randomly" target="_blank">
<b>Listen to generated speech with random pitch, rate, voice, and language</b></a><br><span class="requirements">(requires speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/command broadcast 2.png">
<figcaption><a href="snap/snap.html?project=command%20broadcast%202" target="_blank">
<b>Speak single word commands to a sprite (with synonym support)</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/command sentences.png">
<figcaption><a href="snap/snap.html?project=command%20sentences&noRun" target="_blank">
<b>Speak full sentence commands to a sprite (with keyword search)</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/part of speech broadcast.png">
<figcaption><a href="snap/snap.html?project=part%20of%20speech%20broadcast&noRun" target="_blank">
<b>Create funny sentences by verbally answering questions</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/story generator.png">
<figcaption><a href="snap/snap.html?project=story%20generator&noRun" target="_blank">
<b>Customise stories by verbally answering questions</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/Wikipedia answers.png">
<figcaption><a href="snap/snap.html?project=Wikipedia%20answers&noRun" target="_blank">
<b>Ask questions of Wikipedia</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/weather.png">
<figcaption><a href="snap/snap.html?project=weather&noRun" target="_blank">
<b>Ask about the weather anywhere in the world</b></a><br><span class="requirements">(requires microphone and speaker/headphones)</span></figcaption>
</figure>
<figure>
<img src="images/camera and say no keys.png">
<figcaption><a href="snap/snap.html?project=camera%20and%20say%20no%20keys&noRun" target="_blank">
<b>Listen to a description of what is in front of the camera</b></a><br><span class="requirements"> (speaker/headphones, webcam, and API key(s))</span></figcaption>
</figure>
<figure>
<img src="images/camera, listen, and speak no keys.png">
<figcaption><a href="snap/snap.html?project=camera%2C%20listen%2C%20and%20speak%20no%20keys&noRun" target="_blank">
<b>Listen to a description of what is in front of the camera in response to you speaking</b></a><br><span class="requirements">(requires microphone, speaker/headphones, webcam, and API key(s))</span></figcaption>
</figure>
<figure>
<img src="images/follow finger.png">
<figcaption><a href="snap/snap.html?project=follow%20finger" target="_blank">
<b>Train the turtle to move depending on which way your finger is pointed</b></a><br><span class="requirements">(requires webcam and <a href="#gpu">a GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/train and speak commands.png">
<figcaption><a href="snap/snap.html?project=train%20and%20speak%20commands" target="_blank">
<b>Train the the turtle to move depending on the voice commands you give</b></a><br>
<span class="requirements">(requires microphone)</span></figcaption>
</figure>
<figure>
<img src="images/follow lean.png">
<figcaption><a href="snap/snap.html?project=follow%20lean" target="_blank">
<b>Train the turtle to move left or right depending on which way you lean</b></a><br>
<span class="requirements">(requires webcam and a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/rock paper scissors.png">
<figcaption><a href="snap/snap.html?project=rock%20paper%20scissors&noRun" target="_blank">
<b>Play <i>Rock Paper Scissors</i> using machine learning</b></a><br>
<span class="requirements">(requires webcam, speaker/headphones and a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/simon says.png">
<figcaption><a href="snap/snap.html?project=simon%20says&noRun" target="_blank">
<b>Play <i>Simon Says</i> using machine learning</b></a><br>
<span class="requirements">(requires webcam, speaker/headphones and a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/guess my word.png">
<figcaption><a href="snap/snap.html?project=guess%20my%20word" target="_blank">
<b>Play <i>Guess My Word</i> that works by turning words into 300 numbers</b></a><br>
<span class="requirements">(requires speaker/headphones and optional microphone)</span></figcaption>
</figure>
<figure>
<img src="images/style transfer.png">
<figcaption><a href="snap/snap.html?project=style%20transfer" target="_blank">
<b>Turn a photo into 8 different paintings</b></a><br>
<span class="requirements">(requires webcam (but easily modified to use any image instead) and <a href="#gpu">a GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/AI Questions.png">
<figcaption><a href="snap/snap.html?project=AI Questions" target="_blank">
<b>Answers questions about this Snap! AI library using the<a href="https://research.google/pubs/pub46808/" target="_blank">
Universal Sentence Encoder</a></b><br>
<span class="requirements">(might be slow without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/question answer.png">
<figcaption><a href="snap/snap.html?project=question answer" target="_blank">
<b>Answers questions about Snap! using the Snap! Wikipedia article using
<a href="https://blog.tensorflow.org/2020/03/exploring-helpful-uses-for-bert-in-your-browser-tensorflow-js.html"
target="_blank">BERT</a></b><br>
<span class="requirements">(might be slow without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/codenames.png">
<figcaption><a href="snap/snap.html?project=codenames" target="_blank">
<b>Suggest spymaster clues for the <a href="chapter-5.html#codenames">Codenames</a> game.</b></a><br>
<span class="requirements">(might be slow without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/translate.png">
<figcaption><a href="snap/snap.html?project=translate" target="_blank">
<b>Translation using 300 numbers and a word list with 2.5% coverage</b></a><br>
<span class="requirements">(no special requirements)</span></figcaption>
</figure>
<figure>
<img src="images/predict function.png">
<figcaption><a href="snap/snap.html?project=predict function" target="_blank">
<b>Guesses the relationship between two lists of numbers</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="AI-Teacher-Guide/images/noisy polygon.png">
<figcaption><a href="snap/snap.html?project=noisy polygon" target="_blank">
<b>Learns to predict your ratings of "Abstract Art"</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/tic tac toe machine learning.png">
<figcaption><a href="snap/snap.html?project=tic tac toe machine learning" target="_blank">
<b>Learns to play Tic Tac Toe</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/naming-colors.png">
<figcaption><a href="snap/snap.html?project=naming colors" target="_blank">
<b>Names colors badly - Improve it!</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/confidence guesser.png">
<figcaption><a href="snap/snap.html?project=confidence guesser" target="_blank">
<b>Guesses the level of confidence indicated by a sentence</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
<figure>
<img src="images/image analogy.png">
<figcaption><a href="snap/snap.html?project=image embedding" target="_blank">
<b>Tries to solve analogy problems with photos (partially successfully - can you improve it?)</b></a><br>
<span class="requirements">(runs very slowly without a <a href="#gpu">GPU</a>)</span></figcaption>
</figure>
</div>
<p>
We have collected <a href="student-projects/">some projects by students</a> who have used these AI blocks.
</p>
<h3>Source code</h3>
<p>
The Snap! AI blocks load <a href="https://github.com/ecraft2learn/ai/blob/master/ecraft2learn.js" target="_blank">
this JavaScript file on Github</a> and then calls the functions defined therein.
</p>
<h3>Contact</h3>
<p>
Send email to Ken Kahn (toontalk@gmail.com) if you encounter problems or have questions.
</p>
<h3 id="gpu">GPU required for chapter 4 image recognition, chapter 6 machine learning, and some demos</h3>
<p>
Nearly all modern computers have a <a href="http://www.nvidia.com/object/what-is-gpu-computing.html" target="_blank">GPU</a>.
Without one or with a very primitive one programs using machine learning may run very slowly.
For example, the image machine learning commands run very slowly on Raspberry Pis.
One version of the audio training primitives, however, do run well on Raspberry Pis.
</p>
<h3>Using the AI blocks in your projects</h3>
<p>
You can import
<a href="AI-Teacher-Guide/eCraft2Learn%20blocks.xml" download>this file</a> into your existing or new projects.
The AI blocks can then be found in the Snap! palette under
<img src="images/sound-palette.png">,
<img src="images/sensing-palette.png">, or
<img src="images/other-palette.png">.
</p>
<script src="js/bottom-of-page.js"></script>
</body>
</html>