-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathbeat-slicer.html
More file actions
239 lines (230 loc) · 14.5 KB
/
beat-slicer.html
File metadata and controls
239 lines (230 loc) · 14.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
<!DOCTYPE html>
<html>
<head>
<title>Andrew Zhao - Beat Slicer</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
<link rel="stylesheet" href="assets/css/main.css" />
<noscript><link rel="stylesheet" href="assets/css/noscript.css" /></noscript>
<script src="https://kit.fontawesome.com/b244fa50d8.js" crossorigin="anonymous"></script>
</head>
<div header>
<a class="back-button" href="index.html">
<i class="fas fa-chevron-left"></i>
<span>   back</span>
</a>
<div class="project-title-wrapper">
<span class="title project-title">Beat Slicer</span>
<a href="https://github.com/47hao/Beat-Slicer" target="_blank" class="project-github-icon">
<i class="fab fa-github"></i>
</a>
</div>
<span class="title subtitle">Webcam-based 3D motion game</span>
</div>
<body class="project-body">
<div class="video-container">
<iframe class="project-video"
src="https://www.youtube.com/embed/RjHvfmKpxdg" title="Beat Slicer Video" frameborder="0"
allow="encrypted-media; gyroscope; picture-in-picture"
allowfullscreen></iframe>
</div>
<div class="text-block">
<p>
<b><i>Beat Slicer</i> is a motion-tracking rhythm game that I wrote entirely in Python, that was voted
1st place out of 450+ student projects at CMU.</b>
</p>
<ul class="half-width-list">
<li class="list-item">I created an original OpenCV algorithm to read player movements through a webcam, so it requires
no specialized hardware and is accessible and convenient to play.</li>
<li class="list-item">
The game features a self-written 3D engine, and splits cubes into 3D fragments
based on the exact spot where they're hit.
</li>
<li class="list-item">
Keeping game video and audio synchronized in base Python was
a huge challenge, which I solved by playing the music and running the game as parallel threads.
</li>
</ul>
<p>
Ultimately, I'm really proud that my project inspired many other CMU students to push the
limits of their coding skills and create their own awesome projects they could be excited about.
</p>
<span class="title subtitle">The Process</span>
<p>
When I began building Beat Slicer, I had no clue what the end product might look like.
I did have a set of requirements for myself: it had to be a really cool experience, be accessible to
most people, and be elegant in a way that's easy to learn but hard to master. The result is a fusion of design, programming,
art, and problem solving that I'm really proud to share.
</p>
</div>
<div class="text-block">
<button type="button" class="collapsible title subtitle">
<i class="fas fa-angle-right coll-arrow"></i>
<span class="coll-label">Design</span>
</button>
<div class="collapsible-content" id="coll-design">
<p>With a set of goals laid out, I started thinking about features for my game, with three priorities in mind:</p>
<img src="images/beat-slicer-1.jpg" class="project-image">
<p>
The idea of a 3D music-based motion game reminded me of <i>Beat Saber</i>, a widely popular virtual reality game.
Unfortunately, <i>Beat Saber</i> requires a VR headset, which means most people
can't play it conveniently. Thus, I was inspired to create a PC-based game inspired by <i>Beat Saber</i> with
an emphasis on accessibility to the average person.
</p>
<p>Motion tracking would be the biggest design challenge for my project.</p>
<img src="images/beat-slicer-2.jpg" class="project-image">
<p>
My first two ideas proved unreliable or unstable, and would not produce a working game. They also
heavily compromised accessibility.
</p>
<p> After thinking for a bit about how cameras work, I realized that <i>reflective</i> objects
blur into the background when moving, but <b>emissive</b> light sources create bright streaks
that are very visible.
</p>
<img src="images/beat-slicer-3.jpg" class="project-image">
<p>
With a little openCV code, I was able to get the computer to track the light stick! However, I was pretty
unhappy with this setup as it meants players would have to buy parts and build a light stick to play.
Not cool. While pondering my options, I realized that <i>any light source</i> worked
with the tracking algorithm, and was having some fun:
</p>
<img src="images/beat-slicer-lights.png" class="project-image">
<p>
Wait. <i>Almost everyone has a phone!</i> I had accidentally designed a tracking system that had
low system requirements, was robust in varying environments, and used just one convenient
item that everyone carries around.
</p>
<p>
At the same time, I remembered that VR systems track player motions via <i>infrared</i> lights mounted on the controllers.
My idea of using visible light, in comparison, traded some of the performance and reliability offered by custom
controllers for the greater accessibility of less hardware.
</p>
<p>
I've always believed that if something is designed <i>well enough</i>, it shouldn't need instructions to be understood.
I was very happy with how simple and intuitive the system was, and was excited to take the project further.
</p>
</div>
<button type="button" class="collapsible title subtitle">
<i class="fas fa-angle-right coll-arrow"></i>
<span class="coll-label">Computer Vision</span>
</button>
<div class="collapsible-content" id="coll-cv">
<p>
The choice to track a flashlight offered several benefits to the project. Traditional OpenCV algorithms are often confused
by noisy and cluttered backgrounds, but since such distractions usually aren't too bright, my algorithm simply ignores
them. Since webcams tend to have narrow dynamic ranges,
the rest of the image becomes naturally darker as the camera adjusts for exposure.
</p>
<img src="images/beat-slicer-gray-1.JPG" width=300em class="center-image">
<p>
When processing inputs, I (1) turned the image grayscale, (2) maximized image contrast and sharpness,
and (3) picked out the largest light spot among those available, in the event that there were multiple.
The result was a robust algorithm that would track accurately in virtually any typical indoor setting.
</p>
<img src="images/beat-slicer-algo.png" width=600em class="center-image">
<p>
Unfortunately, the cycle of reading frames, applying filters, and extracting coordinates greatly slowed the
game's framerate as it ran alongside the other processes.
My solution was to <b>multithread</b> the application and run the camera input, filtering, coordinate extrapolation,
blade construction, and cube slice check on a separate thread, allowing the graphics and movements to both run smoothly
</p>
<img src="images/beat-slicer-5.jpg" width=500em class="center-image">
</div>
<button type="button" class="collapsible title subtitle">
<i class="fas fa-angle-right coll-arrow"></i>
<span class="coll-label">3D Slicing and Graphics</span>
</button>
<div class="collapsible-content" id="coll-3d">
<p>
Writing a 3D graphics engine meant translating xyz coordinates to 2D screen space coordinates. Since farther away
objects appear visually smaller, I knew that x and y would scale based on z. After a bit of trigonometry,
I got decent results.
</p>
<img src="images/beat-slicer-6.png" width=500em class="center-image">
<img src="images/beat-slicer-7.jpg" width=300em class="center-image">
<p>
Next was the biggest challenge of the whole project - <b>3D Slicing</b>. Mathematically, I needed to split a cube
into two fragments with a bisecting plane.
</p>
<img src="images/beat-slicer-slicing-1.png" class="column-image">
<img src="images/beat-slicer-slicing-2.png" class="column-image">
<img src="images/beat-slicer-slicing-3.png" class="column-image">
<p>
Writing an algorithm to create convex polyhedra from 3D coordinates was a bit beyond my time budget,
so I borrowed help from scipy's ConvexHull package to create faces from the sets of points. This still
left me with a major problem:
</p>
<img src="images/beat-slicer-cube-split.png" width=300em class="center-image">
<p>
It produced a mesh of triangles, which makes sense <i>mathematically</i>, but causes ugly extra lines
when drawn out for my purposes. To remedy this, I created an algorithm to merge coplanar faces, and wrote a suite
of functions to accomplish it.
</p>
<img src="images/beat-slicer-merge.jpg" width=500em class="center-image">
<p>
Finally, the cubes and their fragments were being drawn correctly. I began experimenting with
how the cubes should look:
</p>
<img src="images/cube-design.png" width=350em class="center-image">
<p>
They were all a bit too complex, and I wanted something simple. I decided to try the simplest possible
representation of a cube, with just a square on top of a hexagon:
</p>
<img src="images/cube-hexagon.png" width=600em class="center-image">
<p>
And it looked great!
</p>
</div>
<button type="button" class="collapsible title subtitle">
<i class="fas fa-angle-right coll-arrow"></i>
<span class="coll-label">Music and Timing</span>
</button>
<div class="collapsible-content" id="coll-music">
<p>
I knew from the start that <b>accurate timing</b> is critical to ensuring the smoothness and feel of rhythm games.
A resulting design feature is that cubes were positioned <b>x-y-time</b> coordinates, rather than <b>x-y-z</b>,
as arriving <i>exactly</i> on the right beat was more important than being at a specific distance.
</p>
<img src="images/cube-axis.jpg" width=150em class="center-image">
<p>
There was another major problem: <b>The game's timing cycle did not run consistently</b>. The timer would wait
until the game finished the current step before <i>starting</i> to count to the next step, and the duration
of each step varied based on computer hardware, screen size, and many other <i>uncontrollable factors</i>.
With the goal of running <b>44,100</b> steps per second, this level of error was unacceptable and would
distort the music to the point that it was unrecognizable.
</p>
<img src="images/beats.png" width=500em class="center-image">
<p>
My first idea was to measure the error and <i>compensate</i> for it - time
each step, and figure out how many beats should occur during it.
</p>
<img src="images/beats-corrected.png" width=500em class="center-image">
<p>
Better, but it still did not line up well. My eventual solution was to <b>flip the script</b> - let the <i>view</i>
control the <i>model</i>, in MVC terms. The song would play uninterrupted on its own thread,
send timing information to the game, and do <i>nothing else</i>, allowing it to run smoothly and stably. The
game itself would handle all the processing of converting coordinates and moving cubes. This way,
even if the framerate dropped or the program slowed, every cube would still arrive <b>right on time</b>.
</p>
</div>
</div>
<div class=" footer">
<div class="footer-links">
<a class="footer-element" target="_blank" href="https://www.linkedin.com/in/andrewhz/">
<i class="fab fa-linkedin"></i>
<span>linkedin</span>
</a>
<a class="footer-element" target="_blank" href="mailto:andrewhzhao@gmail.com">
<i class="fas fa-paper-plane"></i>
<span>email</span>
</a>
<a class="footer-element" target="_blank" href="https://github.com/47hao">
<i class="fab fa-github"></i>
<span>github</span>
</a>
</div>
<span class="footer-credit">site built from scratch</span>
</div>
<script src="collapsible.js"></script>
</body>
</html>