-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathevaluation.html
More file actions
333 lines (303 loc) · 20.4 KB
/
evaluation.html
File metadata and controls
333 lines (303 loc) · 20.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Evaluation - TrueLearn</title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/logos/TL-favicon.png" rel="icon">
<link href="assets/img/logos/TL-logo.png" rel="apple-touch-icon">
<!-- Google Fonts -->
<link
href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,700,700i|Raleway:300,400,500,700,800|Montserrat:300,400,700"
rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/aos/aos.css" rel="stylesheet">
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<!-- =======================================================
* Template Name: Reveal - v4.10.0
* Template URL: https://bootstrapmade.com/reveal-bootstrap-corporate-template/
* Author: BootstrapMade.com
* License: https://bootstrapmade.com/license/
======================================================== -->
</head>
<body>
<!-- ======= Header/top bar ======= -->
<header id="header" class="d-flex align-items-center">
<div class="container d-flex justify-content-between">
<div id="logo">
<h1><a href="index.html">True<span>Learn</span></a></h1>
<!-- Uncomment below if you prefer to use an image logo -->
<!-- <a href="index.html"><img src="assets/img/logo.png" alt=""></a>-->
</div>
<nav id="navbar" class="navbar">
<ul>
<li class="dropdown"><a href="index.html"><span>Home</span> <i class="bi bi-chevron-down"></i></a>
<ul>
<li><a class="nav-link scrollto" href="index.html#about">Abstract</a></li>
<li><a class="nav-link scrollto" href="index.html#portfolio">Portfolio Video</a></li>
<li><a class="nav-link scrollto" href="index.html#timeline">Project Timeline</a></li>
<li><a class="nav-link scrollto" href="index.html#team">Team</a></li>
</ul>
</li>
<li><a class="nav-link" href="requirements.html">Requirements</a></li>
<li><a class="nav-link" href="research.html">Research</a></li>
<li><a class="nav-link" href="design.html">System Design</a></li>
<li><a class="nav-link" href="implementation.html">Implementation</a></li>
<li><a class="nav-link" href="testing.html">Testing</a></li>
<li><a class="nav-link" href="evaluation.html">Evaluation</a></li>
<li class="dropdown"><a href="appendices.html"><span>Appendices</span><i class="bi bi-chevron-down"></i></a>
<ul>
<li><a class="nav-link scrollto" href="appendices.html#documentation">Documentation</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#development-blog">Development Blog</a></li>
<li><a class="nav-link scrollto" href="appendices.html#development-videos">Monthly Videos</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#license">License</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#privacy-policy">Privacy policy</a> </li>
</ul>
</li>
</ul>
<i class="bi bi-list mobile-nav-toggle"></i>
</nav><!-- .navbar -->
</div>
</header><!-- End Header -->
<!-- ======= Banner ======= -->
<section id="banner">
<div class="banner-content" data-aos="fade-up">
<h2>Evaluation</h2>
</div>
</section>
<!-- End Banner -->
<main>
<section id="evaluation">
<div class="container" data-aos="fade-up">
<div class="section-header">
<h2>MoSCoW Achievement Table</h2>
</div>
<div class="row">
<div class="col-lg-2"></div>
<div class="col-lg-8" style="margin: auto;">
<img src="assets/img/evaluation/moscow-completion.png" class="img-fluid" alt="">
</div>
<div class="col-lg-2"></div>
<br>
<div>
<br>
<p>Key Functionalities (<b>Must Haves</b> and <b>Should Haves</b>): 100% (39/39)</p>
<p>Optional Functionalities (<b>Could Haves</b>): 80% (4/5) </p>
<p>Total: 98% (43/44)</p>
</div>
</div>
<div class="section-header">
<h2>List of Known Bugs</h2>
<p>None.</p>
</div>
<div class="section-header">
<h2>Individual Contribution Distribution Table</h2>
</div>
<div class="row">
<div class="col-lg-2"></div>
<div class="col-lg-8" style="margin: auto;">
<img src="assets/img/evaluation/contributions.png" class="img-fluid" alt="">
</div>
<div class="col-lg-2"></div>
</div>
<div class="section-header">
<h2>Critical Evaluation of the Project</h2>
</div>
<div class="row">
<div>
<h3>User Experience</h3>
<p>During the development of this project, we based our decision-making upon what users of our
library are looking for.</p>
<p>On the learner side, we focus on producing interpretable and insightful visualizations to
track and guide their learning journey.</p>
<ol>
<li>We studied research papers (“What Learning Visualisations to Offer Students” by Susan
Bull and “Individual and Peer Comparison Open Learner Model Visualisations to Identify
What to Work On Next” by Susan Bull) on effective visualisations that students find the
most helpful and understandable and finalised on nine different types of visualisations:
Bar Charts, Line Charts, Dot Plots, Pie Charts, Rose Charts, Bubble Charts, Tree Maps,
Radar Charts and Word Clouds. These visualisations inform users on their skill level
across different subjects, the level of certainty of our prediction as well as how this
has developed over time.
</li>
<li>We used a universal colour-scale “Greens” (which is a shade rather than several
different colours) due to feedback from our client that this is this is an easily
understandable way, to depict the skill level of users and its certainty level.</li>
</ol>
<p>On the developer side, we focused on adhering to design principles mentioned in our
literature review, such as "Consistency", “Inspections” and “Sensible Defaults.”</p>
<ol>
<li><b>Consistency</b>: All the classifiers in truelearn library has the same and consistent
interface. For training, all the classifiers have <code>fit</code> method, and for
prediction, all the classifiers have <code>predict</code> and <code>predict_proba</code>
function.</li>
<li><b>Inspection</b>: All the classes in <code>truelearn.models</code> and
<code>truelearn.learning</code> are inspectable, which means developers can inspect the
internal states of these classes via either getter/setter or exposed
<code>@property</code>.
</li>
<li><b>Sensible Defaults</b>: Parameters are key parts of all the classifiers. To reduce the
time cost of researching on different parameters for users, we provide an appropriate
default value to each parameter. The default value is not necessarily the best in a
custom use case but provides sensible results in general. Based on our evaluation of the
default values chosen in our tests, it is reasonable to say that our classifier has
sensible default values.
</li>
</ol>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
<div>
<h3>Functionality</h3>
<p>We have always focused on the key design target, which is about modelling learners'
knowledge, predicting their engagement with educational resources and providing insightful
visualisations to help learners reflect on their study. We delivered 100% (36 out of 36) of
the key functionalities we intended to deliver and 50% (3 out of 6) of the optional
functionalities on our MoSCoW list. Without diving too deep, these include but not limited
to:</p>
<ul>
<li>Build a model of a learner's knowledge from their learning history</li>
<li>Predict learners' engagement based on their past learning events</li>
<li>Create various kinds of visualizations that highlight changes in users' knowledge over
time and users' understanding of different knowledge</li>
</ul>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
<div>
<h3>Stability</h3>
<p>Throughout the development process, we followed an intensive testing procedure. We achieved
100% on code coverage of core functionalities of truelearn library (models, learning,
datasets, preprocessing, utils.metrics). We also deploy a list of linters that statically
analyse our codebase based on different perspectives (code quality, code style,
documentation style, security, type checking). All of these are checked on every commit by
using CI/CD.</p>
<p>Overall, we would say we did <b>Good</b> in this aspect.</p>
</div>
<div>
<h3>Efficiency</h3>
<p>Throughout our development process, we continuously looked for ways to do optimizations and
do timing profiling by using <code>hyperfine</code> to study how our refactoring affects the
wall clock
running time of our application. As stated in the “Testing/Performance Testing” section, we
improved the running time of the <code>NoveltyClassifier</code> by about 20%, significantly
reduced the
number of memory allocations (by more 50%) and slightly decreased the overall memory
consumption.</p>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
<div>
<h3>Compatibility</h3>
<p>Building on top of Python, our library is naturally compatible with different operating
systems (OSX, Linux, Windows). In terms of Python version support, we choose to support any
Python version between 3.7 (the earliest version that has not ended its lifecycle yet) and
3.11(the latest release). To ensure that these compatibility guarantees were not broken, we
deploy ci checks on these 3 operating systems and these 5 Python versions, which is 3*5=15
checks in total. </p>
<p>For dependencies, we choose them carefully by considering their API stability, minimal Python
version support (which must be Python 3.7+), and license (which must be a permissive
license). This allows us to balance functionality support with our compatibility guarantees
and licensing option (MIT license).</p>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
<div>
<h3>Maintainability</h3>
<p>Another key aspect we prioritised was maintainability.</p>
<p>We made sure that any future development would be seamless process by including comprehensive
type annotations, explanations and examples for every method and class in our documentation.
We also provide contributors with detailed guidelines on how to set up a Truelearn
development environment, design new components, create new tests and submit PRs.</p>
<p>In terms of codebase, we utilised multiple design patterns to make sure our code is easily
extensible and loosely coupled. Each public method and class are tested thoroughly by using
doc tests and unit tests. At the same time, we established a consistent code style in the
codebase based on PEP8, Google Python Style Guide, and Black Code Style, and implemented
strict ci checks (prospector-pydocstyle, typos, black), which further reduced maintenance
costs.</p>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
<div>
<h3>Project management</h3>
<p>The team worked well together. Throughout the 15 weeks, we made sure to produce a to-do list
on a weekly basis. Each task was assigned to team members alongside a deadline. We stuck to
our deadlines well. We organised weekly meetings with our clients and ourselves outside of
labs. Some meetings were cancelled due to train strikes or if it were not needed. Every
member played their part to ensure the success of the project.</p>
<p>Overall, we would say we did <b>Very Good</b> in this aspect.</p>
</div>
</div>
<div class="section-header">
<h2>Future Work</h2>
</div>
<div>
<ol>
<li><b>Explore real-world application</b>: we intend to bring the exciting functionalities and
visualisations of truelearn library to x5Learn, an open-education platform. In terms of
data, x5learn has divided all videos into 5-minute segments, extracted the Wikipedia topics
within each segment and was able to record user engagement through their algorithm. Thus, to
integrate TrueLearn into their platform, we only need to study how to integrate our model
into their backend API and database and embed visualisations into their React-based WebApp.
</li>
<li><b>Extend Classifiers</b>: currently, the developers can only run a simple
version of TrueLearn experiments by using our library. To allow them to easily replicate the
entire TrueLearn experiment, we can add supports for classifiers that are based on knowledge
tracing and methods that can perform hyperparameter training.
</li>
<li><b>Improve Documentation</b>: we can improve the user experience of our documentation by
introducing interactive code snippets that allow users to execute examples in our
documentation in their browsers. To achieve this, we need to research on WASM, which allows
us to run Python in browser.
</li>
<li><b>Explore Semantic Information</b>: currently, classifiers in truelearn library does not
utilize any information between different knowledge components. For example, machine
learning and linear regression are treated as different unrelated topics (but there are
connections between them). In theory, the relationships between different topics give us
more information about the learner and can therefore, boost the performance (accuracy,
precision, recall, and f1) of the truelearn classifiers. These semantic relationships also
allow us to design new visualisations. For example, we can group related topics into
categories and create category-based visualisations. To achieve this, we need to integrate
semantic TrueLearn https://discovery.ucl.ac.uk/id/eprint/10141501/ into our library, which
entails extending the current model sub-package (storing Semantic Relatedness (SR)
annotations) and implementing the semantics algorithm logic in the learning
sub-package.
</ol>
</div>
</div>
</section>
</main>
<!-- ======= Footer ======= -->
<footer id="footer">
<div class="container">
<div class="copyright">
© Copyright <strong>TrueLearn</strong>. All Rights Reserved
</div>
<div class="credits">
<!--
All the links in the footer should remain intact.
You can delete the links only if you purchased the pro version.
Licensing information: https://bootstrapmade.com/license/
Purchase the pro version with working PHP/AJAX contact form: https://bootstrapmade.com/buy/?theme=Reveal
-->
<div class="text-center">Designed by Karim Djemili</div>
Theme from <a href="https://bootstrapmade.com/">BootstrapMade</a>
</div>
</div>
</footer><!-- End Footer -->
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i
class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/aos/aos.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
</body>
</html>