-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathtesting.html
More file actions
386 lines (331 loc) · 33.5 KB
/
testing.html
File metadata and controls
386 lines (331 loc) · 33.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Testing - TrueLearn</title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/logos/TL-favicon.png" rel="icon">
<link href="assets/img/logos/TL-logo.png" rel="apple-touch-icon">
<!-- Google Fonts -->
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,700,700i|Raleway:300,400,500,700,800|Montserrat:300,400,700" rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/aos/aos.css" rel="stylesheet">
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<!-- =======================================================
* Template Name: Reveal - v4.10.0
* Template URL: https://bootstrapmade.com/reveal-bootstrap-corporate-template/
* Author: BootstrapMade.com
* License: https://bootstrapmade.com/license/
======================================================== -->
</head>
<body>
<!-- ======= Header/top bar ======= -->
<header id="header" class="d-flex align-items-center">
<div class="container d-flex justify-content-between">
<div id="logo">
<h1><a href="index.html">True<span>Learn</span></a></h1>
<!-- Uncomment below if you prefer to use an image logo -->
<!-- <a href="index.html"><img src="assets/img/logo.png" alt=""></a>-->
</div>
<nav id="navbar" class="navbar">
<ul>
<li class="dropdown"><a href="index.html"><span>Home</span> <i class="bi bi-chevron-down"></i></a>
<ul>
<li><a class="nav-link scrollto" href="index.html#about">Abstract</a></li>
<li><a class="nav-link scrollto" href="index.html#portfolio">Portfolio Video</a></li>
<li><a class="nav-link scrollto" href="index.html#timeline">Project Timeline</a></li>
<li><a class="nav-link scrollto" href="index.html#team">Team</a></li>
</ul>
</li>
<li><a class="nav-link" href="requirements.html">Requirements</a></li>
<li><a class="nav-link" href="research.html">Research</a></li>
<li><a class="nav-link" href="design.html">System Design</a></li>
<li><a class="nav-link" href="implementation.html">Implementation</a></li>
<li><a class="nav-link" href="testing.html">Testing</a></li>
<li><a class="nav-link" href="evaluation.html">Evaluation</a></li>
<li class="dropdown"><a href="appendices.html"><span>Appendices</span><i class="bi bi-chevron-down"></i></a>
<ul>
<li><a class="nav-link scrollto" href="appendices.html#documentation">Documentation</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#development-blog">Development Blog</a></li>
<li><a class="nav-link scrollto" href="appendices.html#development-videos">Monthly Videos</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#license">License</a> </li>
<li><a class="nav-link scrollto" href="appendices.html#privacy-policy">Privacy policy</a> </li>
</ul>
</li>
</ul>
<i class="bi bi-list mobile-nav-toggle"></i>
</nav><!-- .navbar -->
</div>
</header><!-- End Header -->
<section id="banner">
<div class="banner-content center" data-aos="fade-up">
<h2>Testing</h2>
</div>
</section>
<main>
<section id="testing">
<div class="container" data-aos="fade-up">
<div class = "section-header">
<h2 id="strategy">Testing Strategy</h2>
<p>Since truelearn is a Python library targeting developers, researchers and learners, we must conduct comprehensive tests to ensure that its functionality, usability and stability meet the desired standards outlined in our MoSCoW list.</p> </div>
<div class="row">
<div>
<h3>Test Methodology</h3>
<p>Besides the <code>truelearn.utils.visualisations</code> sub-package, the library should be tested by unit and integration tests. We will use PyTest with <code>pytest-socket</code> (control access to the Internet), <code>pytest-cov</code> (generate coverage report) and <code>doctest</code> (documentation tests). We will run these tests on three operating systems (OSX, Linux, Windows) and five Python versions (3.7-3.11) to ensure the compatibility of our library.</p>
<p>For visualisations, we will conduct user acceptance tests by consulting our users, who are developers and learners, setting up different test cases for them and asking them to provide feedback.</p>
</div>
<div>
<h3>Continuous integration</h3>
<p>We have also deployed several continuous integrations (CI) tools in GitHub to automate the testing process. These tools include:</p>
<ul>
<li><code>PyTest</code>: run all the unit and integration tests</li>
<li><code>prospector</code>: check for security and style issues and violations of linting rules</li>
<li><code>black</code>: check for code formatting issues</li>
<li><code>typos</code>: check for typos</li>
<li><code>coverage</code>: check for test coverage, which is used to guide us to improve tests</li>
</ul>
</div>
</div>
<div class="section-header">
<h2>Unit and integration testing</h2>
<p>Unit tests are the first type of test added to the truelearn library. In truelearn, there are two forms of unit tests.</p>
<p>The first form is based on doctest. For most of the classes in <code>truelearn.models</code> and <code>truelearn.learning</code>, we have added examples of how to use itself in its docstring and provided some expected output. When the doctest test framework runs the doctest, it will run these examples and compare the actual output with the expected one.</p>
</div>
<div class="row">
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/doctests.png" class="img-fluid" alt="">
</div>
<div class="col-lg-3"></div>
<div>
<p>The second form is the traditional unit test where we test all the subpackages and modules in our library.</p> <p>Simultaneously, instead of purely relying on unit testing, we apply integration testing on testing classifiers (e.g. <code>KnowledgeClassifier</code>, <code>NoveltyClassifier</code>, <code>InterestClassifier</code> and <code>INKClassifier</code>) and visualisations.</p>
<p>For classifiers, some of their methods, like <code>fit</code> and <code>predict</code>, read or update with learner and event models, which complicates testing. In order to successfully test them, we have carefully crafted some test cases based on the mathematical principles behind them.</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/knowledge-classifier.png" class="img-fluid" alt="">
</div>
<div class="col-lg-3"></div>
<div>
<p>Since we have the original algorithm logic for the classifier, we also generated some random inputs and compared the output of the existing algorithm with that of the original one.</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/novelty-classifier.png" class="img-fluid" alt="">
</div>
<div class="col-lg-3"></div>
<div>
<p>For visualisation, we are primarily concerned with the charts generated. So, we selected some sample <code>Knowledge</code> and used the visualisation sub-package to generate some baseline files, which are placed in <code>truelearn/tests/baseline_files</code>. The integration tests for our visualisations sub-package are mostly to test whether we can produce images that are identical or similar to the baseline files across platforms and versions.</p>
<p>For matplotlib-based plotters, we focus on generating similar png files. For plotly-based plotter class, we focus on generating html and json files. The reasons for such choice are</p>
<ul>
<li>matplotlib: matplotlib itself has tests to ensure that images in different format can be correctly rendered and exported. So, we only need to ensure that one of the images types work correctly. At the same time, testing different file extensions requires the installation of special external dependencies (GhostScript and Inkspace). These make the testing process pointlessly complex and make it difficult for developers to run our tests in a simple manner. Thus, we only choose to test png file.</li>
<li>plotly: plotly supports the export of figures to static (png, jpg, pdf...) and interactive (html, json - which can be used by plotly.js on the front-end) formats. The reason we only test to html and json output is because of the limitations of its backend. The backend it uses is <code>Kaleido</code>, which uses Chromium internally to export the images. Based on our testing, we found that due to chromium's font rendering policy, there was no way to generate the same/similar images on different operating systems, even if we used the same font, which made it impossible to replicate our testing for images. So we decided to use only reproducible (with a little setup) html and json formats for our tests.</li>
</ul>
<p>By using these tests, TrueLearn achieves 100% test coverage.</p>
</div>
</div>
<div class="section-header">
<h2>Evaluating Default Values</h2>
</div>
<div class="row">
<div>
<p>As stated in our research, we wanted to choose default values for our classifier so that it performs reasonably. To quantitatively assess what are reasonable results, we used the classifier performance as a metric and conducted several tests based on the PEEK dataset.</p>
<p>Since we wanted to investigate how sensible the results generated by the default values are, we ignored the <code>train.csv</code> file in the dataset, which is used for hyperparameter training. We only utilized the learning events in the <code>test.csv</code> file and conducted experiments on all learners and the 20 most active learners in the training set.</p>
<p>The following three tables show the weighted average test performance for accuracy, precision, recall and F1. The first two tables are the results we collected in our experiments using the default values we eventually chose. The third table shows the results of the classifier after hyperparameter training, presented in “TrueLearn: A Bayesian Algorithm Series for Matching Lifelong Learners with Open Educational Resources" by Bulathwela, et al.</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/performance-all-learners.png" class="img-fluid" alt="">
<p style="text-align: center">The performance of classifiers on all learners</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/performance-20-learners.png" class="img-fluid" alt="">
<p style="text-align: center">The performance of classifiers on the 20 most active learners</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/performance-in-paper.png" class="img-fluid" alt="">
<p style="text-align: center">The performance of classifiers, presented in the TrueLearn paper</p>
</div>
<div class="col-lg-3"></div>
<div>
<p>As can be seen from these experiments, the default values we chose for the classifiers are sensible: for the <code>KnowledgeClassifier</code> and the <code>NoveltyClassifier</code>, they perform close to or even better than the hyperparameter-trained classifiers (TrueLearn fixed-depth and TrueLearn Novelty) in terms of recall and f1 score. Also, we can see that the <code>INKClassifier</code>, which utilizes <code>NoveltyClassifier</code> and <code>InterestClassifier</code> internally, is ahead of most of the classifiers in terms of accuracy and precision.</p>
<p>Therefore, it is reasonable to conclude that by default, the classifier performs sensibly, which makes our library user-friendly.</p>
</div>
</div>
<div class="section-header">
<h2>Performance Tests</h2>
<p>As stated in our MoSCoW list, we want the performance of our library to be comparable to that of the original code. Thus, we conducted performance tests by benchmarking our library against the original code <a href="https://github.com/sahanbull/TrueLearn">https://github.com/sahanbull/TrueLearn</a>.</p>
</div>
<div class="row">
<div>
<h3>Experiment Setup</h3>
</div>
<div>
<p><strong>Outline</strong>: We chose the <code>KnowledgeClassifier</code> and <code>NoveltyClassifier</code> (with their default parameters) for benchmarking, as each represents different assumptions in TrueLearn (so, different fitting and prediction methods), and they are publicly available in the original code. The functions we chose to benchmark are <code>fit</code> and <code>predict</code>, as they are the core of each classifier and are responsible for training and providing predictions.</p>
</div>
<div>
<p><strong>Input</strong>: We used the PEEK dataset with the 20, 40 and 80 most active learners as input to our program. Below is an example of the input array.</p>
</div>
<div>
<pre class="language-python" style='font-family: Consolas, Monaco, "Andale Mono", "Ubuntu Mono", monospace; font-size: 16px; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.5; background-color: rgb(246, 248, 250); border-radius: 6px; color: rgb(0, 0, 0); text-shadow: rgb(255, 255, 255) 0px 1px; text-align: left; white-space: pre; word-spacing: 0px; word-break: normal; tab-size: 4; hyphens: none; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;' tabindex="0"><code class="language-python" style='font-family: Consolas, Monaco, "Andale Mono", "Ubuntu Mono", monospace; font-size: 16px; padding: 0px; margin: 0px; white-space: pre; background: 0px 0px transparent; border-radius: 6px; color: rgb(0, 0, 0); text-shadow: rgb(255, 255, 255) 0px 1px; text-align: left; word-spacing: normal; word-break: normal; overflow-wrap: normal; line-height: inherit; tab-size: 4; hyphens: none; border: 0px; display: inline; overflow: visible;'><span class="token punctuation" style="color: rgb(153, 153, 153);">[</span>
<span class="token punctuation" style="color: rgb(153, 153, 153);">(</span>learner_id<span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> <span class="token punctuation" style="color: rgb(153, 153, 153);">[</span><span class="token punctuation" style="color: rgb(153, 153, 153);">(</span>event1<span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> label1<span class="token punctuation" style="color: rgb(153, 153, 153);">)</span><span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> <span class="token punctuation" style="color: rgb(153, 153, 153);">(</span>event2<span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> label2<span class="token punctuation" style="color: rgb(153, 153, 153);">)</span><span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> <span class="token punctuation" style="color: rgb(153, 153, 153);">.</span><span class="token punctuation" style="color: rgb(153, 153, 153);">.</span><span class="token punctuation" style="color: rgb(153, 153, 153);">.</span><span class="token punctuation" style="color: rgb(153, 153, 153);">]</span><span class="token punctuation" style="color: rgb(153, 153, 153);">)</span><span class="token punctuation" style="color: rgb(153, 153, 153);">,</span> <span class="token comment" style="color: rgb(112, 128, 144);"># first learner</span>
<span class="token punctuation" style="color: rgb(153, 153, 153);">.</span><span class="token punctuation" style="color: rgb(153, 153, 153);">.</span><span class="token punctuation" style="color: rgb(153, 153, 153);">.</span> <span class="token comment" style="color: rgb(112, 128, 144);"># second, third, ... learner</span>
<br><span class="token punctuation" style="color: rgb(153, 153, 153);">]</span></code></pre>
</div>
<div>
<p>We focused on comparing the wall clock time and peak memory size under these different input sizes:</p>
<ul>
<li>The wall clock time takes into account the effects of external factors such as IO, context switching, etc. It gives us a more accurate picture of performance compared to CPU time.</li>
<li>Peak memory size measures the maximum amount of memory used by a program during execution. Total memory allocated tells us the total amount of memory allocated during execution. We are more concerned about peak memory size in this experiment because this tells us how much memory the user will actually use when running the library/original code.</li>
</ul>
<p><strong>Method</strong>: We iterated over the input of all learners. For each learner, we iterated over its learning events. For each learning event, we first call the <code>predict</code> method of the classifier and store the returned result in an array. Then, we call the <code>fit</code> method of the classifier with <code>event</code> and <code>label</code> as parameters. Below is a high-level overview of the structure of our benchmark.</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/benchmark.png" class="img-fluid" alt="">
<p style="text-align: center"><strong>High-level overview of the structure of our benchmark</strong></p>
</div>
<div class="col-lg-3"></div>
<div>
<p><strong>Tools</strong>: For time-based benchmarking, we use <code>hyperfine</code>, a state-of-the-art command line benchmarking tool that allows us to benchmark the wall clock time of our program without making any modifications. For memory profiling, we use <code>memray</code>, a memory profiler for Python that can “keep track of allocations in Python code, in native extension modules, and in the Python interpreter itself.” <a href="https://github.com/bloomberg/memray">https://github.com/bloomberg/memray</a>.</p>
<p>We ran the benchmarks described above with a single thread on a MacBook M1 and Python 3.7 (x86-64) and tweaked the original code slightly.</p>
</div>
<div>
<h3>Results</h3>
</div>
<div>
<p>For timing, we get the following results:</p>
</div>
<div class="col-lg-12">
<img src="assets/img/testing/timing.PNG" class="img-fluid" alt="">
</div>
<div>
<p>From the table, we can see that:</p>
<ul>
<li><code>KnowledgeClassifier</code>: On average, the original code is 1.04 times faster than the library implementation. This is reasonable and acceptable as we track additional fields in the knowledge component, learner model and event model.</li>
<li><code>NoveltyClassifier</code>: On average, the library implementation is 1.17 times faster than the original code. This is because we have optimised an operation that calculates the average value of an array.</li>
<li>In summary, our knowledge classifier has comparable performance to the original code, and our novelty classifier is even 20% faster than the original code due to our optimisation.</li>
</ul>
</div>
<div>
<p>For memory consumption, we get the following results:</p>
</div>
<div class="col-lg-3"></div>
<div class="col-lg-6">
<img src="assets/img/testing/peak-memory-size.svg" class="img-fluid" alt="">
</div>
<div class="col-lg-3"></div>
<div>
<ul>
<li>
<p><code>KnowledgeClassifier</code>: For different sizes of inputs, the raw code always consumes slightly more memory than the library implementation. As the size of the input increases, the difference between the memory consumed by the original code and the memory consumed by the library becomes larger and larger.</p>
</li>
<li>
<p><code>NoveltyClassifier</code>: When the input size is 20, the peak memory consumption of the original code and the library implementation is very similar. But, like <code>KnowledgeClassifier</code>, as the input size increases, the difference between the memory consumed by the original code and the memory consumed by the library becomes larger and larger.</p>
</li>
</ul>
<p>Based on the comparison of peak memory size, we can see that the peak memory consumption of the library does not exceed that of the original code.</p>
<p>In summary, based on our benchmarking results, it is reasonable to conclude that our library is comparable to or better than the original code presented in the TrueLearn paper.</p>
</div>
</div>
<div class="section-header">
<h2>User Acceptance Testing</h2>
<p>To ensure our library performs its desired functionalities to the satisfaction of users, we gathered four users to use our library and complete three different surveys, for each test case. These results were then recorded and are displayed on a table which you can find below. </p>
</div>
<div class="row">
<div>
<h3>Testers</h3>
<p><strong><span data-contrast="auto">Denis L.</span></strong><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335551550":1,"335551620":1,"335559685":0,"335559737":0,"335559738":0,"335559739":0,"335559740":259}"> </span></p>
<ul>
<li><span data-contrast="auto">Denis is a student studying Computer Science at the University College London. </span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="auto">He has a strong desire for learning and is curious about what is around him.</span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="auto">He often makes use of different Open Courseware websites to complement his learning and to discover new courses that spark her interest.</span><span data-ccp-props="{}"> </span></li>
</ul>
<p><strong><span data-contrast="auto">Steven M.</span></strong><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335551550":1,"335551620":1,"335559685":0,"335559737":0,"335559738":0,"335559739":0,"335559740":259}"> </span></p>
<ul>
<li><span data-contrast="auto">Steven is a student studying Finance at the University of Oxford.</span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="auto">He is looking to gain insights into what he learns and identify his weaknesses which is important to him in refining his knowledge.</span><span data-ccp-props="{}"> </span></li>
</ul>
<p><strong><span data-contrast="auto">Jack G.</span><span data-ccp-props="{}"> </span></strong></p>
<ul>
<li><span data-contrast="none"><span data-ccp-charstyle="normaltextrun" data-ccp-charstyle-defn="{"ObjectId":"0e0fa2d8-6448-415a-9093-8e7cc1120496|89","ClassId":1073872969,"Properties":[469775450,"normaltextrun",201340122,"1",134233614,"true",469778129,"normaltextrun",335572020,"1",469778324,"Default Paragraph Font"]}">Jack </span><span data-ccp-charstyle="normaltextrun">is a software developer working </span><span data-ccp-charstyle="normaltextrun">for</span><span data-ccp-charstyle="normaltextrun"> LinkedIn.</span></span><span data-ccp-props="{"134233117":false,"134233118":false,"335559738":0,"335559739":0}"> </span></li>
<li><span data-contrast="none"><span data-ccp-charstyle="normaltextrun">He</span><span data-ccp-charstyle="normaltextrun"> has several years of experience in developing educational platforms that </span><span data-ccp-charstyle="normaltextrun">provide</span><span data-ccp-charstyle="normaltextrun"> online learning.</span><span data-ccp-charstyle="eop" data-ccp-charstyle-defn="{"ObjectId":"0e0fa2d8-6448-415a-9093-8e7cc1120496|90","ClassId":1073872969,"Properties":[469775450,"eop",201340122,"1",134233614,"true",469778129,"eop",335572020,"1",469778324,"Default Paragraph Font"]}"></span></span><span data-ccp-props="{"134233117":false,"134233118":false,"335559738":0,"335559739":0}"> </span></li>
<li><span data-contrast="none"><span data-ccp-charstyle="normaltextrun">He focuses on developing software that is well designed and scalable to masses of students.</span></span><span data-ccp-props="{"134233117":false,"134233118":false,"335559738":0,"335559739":0}"> </span></li>
</ul>
<p><strong><span data-contrast="auto">Sarah S.</span></strong><span data-ccp-props="{}"> </span></p>
<ul>
<li><span data-contrast="auto">Sarah is a software developer working as a part-time lecturer at the University of Cambridge.</span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="none"><span data-ccp-charstyle="normaltextrun">She</span><span data-ccp-charstyle="normaltextrun"> wishes to effectively model learners to </span><span data-ccp-charstyle="normaltextrun">provide</span><span data-ccp-charstyle="normaltextrun"> relevant videos that keep his learners engaged and learning.</span><span data-ccp-charstyle="eop"></span></span><span data-ccp-props="{"134233117":false,"134233118":false,"335559738":0,"335559739":0}"> </span></li>
</ul>
</div>
<div>
<h3>Test cases</h3>
<p><span data-contrast="auto">We divided the tests into three cases with each user completing a survey for each one. The survey questions were based on different objectives with an additional option for the user to leave comments.</span><span data-ccp-props="{}"> </span></p>
<p><strong><span data-contrast="auto">Case 1:</span></strong><span data-contrast="auto"> One of our main objectives for our visualisations was to make them understandable for learners. Therefore, we showed the user our bar charts, bubble charts and rose charts in 4 different color scales. In the survey questions, we asked them which color scale they found the clearest. Each tester could vote only once.</span><span data-ccp-props="{}"> </span></p>
<p><strong><span data-contrast="auto">Case 2:</span></strong><span data-contrast="auto"> Another visualisation objective was to enable learners to evaluate their skills as well as areas for improvement. So, we gathered relevant information that learners would find useful by asking what data could be provided through interactions namely in the hover text. </span><span data-ccp-props="{"134233117":false,"134233118":false,"201341983":0,"335551550":1,"335551620":1,"335559685":0,"335559737":0,"335559738":0,"335559739":0,"335559740":259}"> </span></p>
<p><strong><span data-contrast="auto">Case 3:</span></strong><span data-contrast="auto"> We developed a web application to display the visualisations to mimic the behavior of an educational platform that uses TrueLearn. We presented it to our surveyors and asked them to rate the web application in terms of ease of use, understanding, and usefulness.</span><span data-ccp-props="{}"> </span></p>
</div>
<div>
<h3>Feedback</h3>
</div>
<div class="col-lg-2"></div>
<div class="col-lg-8">
<img src="assets/img/testing/user-acceptance.png" class="img-fluid">
</div>
<div class="col-lg-2"></div>
<div>
<h3>Conclusion</h3>
<p><span data-contrast="auto">After the results mentioned above, we made the following changes to our python library:</span><span data-ccp-props="{}"> </span></p>
<ol style="list-style-type: lower-alpha;">
<li><span data-contrast="auto">Based on case 1, we used the Greens color scale where we made darker shades represent greater confidence in our classifier's prediction.</span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="auto">Based on case 2, we decided to include the topic, mean (skill level) and variance (confidence/certainty) in the hover text and inserted an additional parameter into the plotting function so users can choose how many videos to include the number of videos watched for that given topic and the date of the last video watched.</span><span data-ccp-props="{}"> </span></li>
<li><span data-contrast="auto">Based on case 3, we simplified the transitions involved in selecting the type of visualisation and adding subjects to be visualised by allowing for the user to click on them respectively.</span><span data-ccp-props="{}"> </span></li>
</ol>
<p><span data-contrast="auto">Overall, the user acceptance tests have allowed for improved visualisations whilst also increasing the user friendliness of our web application.</span><span data-ccp-props="{"335559685":0}"> </span></p>
</div>
</div>
</div>
</section>
</main>
<!-- ======= Footer ======= -->
<footer id="footer">
<div class="container">
<div class="copyright">
© Copyright <strong>TrueLearn</strong>. All Rights Reserved
</div>
<div class="credits">
<!--
All the links in the footer should remain intact.
You can delete the links only if you purchased the pro version.
Licensing information: https://bootstrapmade.com/license/
Purchase the pro version with working PHP/AJAX contact form: https://bootstrapmade.com/buy/?theme=Reveal
-->
<div class = "text-center">Designed by Karim Djemili</div>
Theme from <a href="https://bootstrapmade.com/">BootstrapMade</a>
</div>
</div>
</footer><!-- End Footer -->
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/aos/aos.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
</body>
</html>