spectroface/index.html at master · DanielRapp/spectroface · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
<html>
  <head>
    <meta charset="utf-8">
    <link rel="stylesheet" type="text/css" href="style.css">
    <link rel="stylesheet" href="http://yui.yahooapis.com/pure/0.5.0/pure-min.css">
  </head>
  <body>

    <div class="content">
      <h1>Recreating the spectrogram face</h1>
      <p>
        I recently stumbled upon the face hidden in the spectrogram of the Aphex Twin track
        called "<a href="https://www.youtube.com/watch?v=M9xMuPWAZW8#t=328">Equation</a>".
        Intrigued, I decided to try my hand at recreating the effect here on the web.
      </p>

      <a href="https://www.youtube.com/watch?v=M9xMuPWAZW8#t=328"><img src="img/aphexBW.png" class="img"></a>

      <h2>Spectrogram</h2>
      <p>
        So what exactly <i>is</i> a spectrogram?
      </p>
      <p>
        A spectrogram is a visual representation of the frequencies which make up
        a sound. Say you whistle a pure "<span data-note="261.6" class="data-note">middle C</span>", then a spectrogram would light
        up right at 261.6 Hz, which is the corresponding
        frequency for that tone. Likewise, the "<span data-note="440" class="data-note">A</span>" note makes the spectrogram turn
        bright white at 440 Hz.
      </p>
      <p>
        If you've ever heard of the <a href="http://jeremykun.com/2012/05/27/the-fourier-transform-a-primer/">fourier</a>
        <a href="http://betterexplained.com/articles/an-interactive-guide-to-the-fourier-transform/">transform</a>;
        a spectrogram is simply the frequency spectrum of the sound at each small moment in time, with the amplitude at each frequency
        visualized as a grayscale value.
      </p>
      <p>
        Click the button to see
        a live spectrogram of what your computer is hearing right now.
        Try whistling.
      </p>
      <div class="live-demo" id="activate-spectrogram">
        <div id="activate">
          <button id="btn-activate" class="pure-button pure-button-primary">Activate spectrogram</button>
        </div>
        <canvas id="spectrogram" class="spectrogram" width="650" height="300"></canvas>
      </div>


      <h2>Constructing the face</h2>
      <p>
        With this in mind, how would we go about converting an image into a sound, which
        can then be interpreted by a spectrogram?
      </p>
      <p>
        The solution is simple. Let
        each row of the image represent a sinusoid (pure tone) at some frequency.
        Say the bottom of the image represents the frequency 5 kHz and the top 15 kHz.
        The grayscale intensity of each pixel determines the amplitude (volume) of each sinusoid.
        So a black pixel on the bottom row means "turn off the 5000 Hz tone", whereas a white pixel means
        "blast a 5000 Hz tone at max volume".
      </p>
      <p>
        Now just scan the image, column by column. Starting with the
        first column, figuring out the intensity
        of the sinuoid for 5000 Hz, then 5001 Hz, then 5002 Hz etc, finally
        outputting them all as a sound (or rather, their sum),
        and then move over to the second column.
      </p>
      <p>
        This will produce the, awful, but pretty darn radical, sound below.
      </p>
      <div class="live-demo-cam">
        <canvas id="spectrogram2" class="spectrogram" width="300" height="300"></canvas>
        <video id="cam1" autoplay muted height="300"></video>
        <div class="audiofy">
          <button id="camToAudio" class="pure-button pure-button-primary">Audiofy yourself</button>
          <button id="vader" class="pure-button pure-button-primary">Audiofy Vader</button>
        </div>
      </div>
      <p>
        <i>
          Note that you'll need to turn on your speakers and run Chrome for this.
          If you're seeing a lot of distortion you may need to lower the volume.
        </i>
      </p>


      <h2>Video</h2>
      <p>
        Let's have a bit of fun. Suppose we just continuously audiofy the video. How would that look/sound?
        Try interacting with the sound by moving around.
      </p>
      <div class="live-demo-video">
        <canvas id="spectrogram3" class="spectrogram" width="300" height="300"></canvas>
        <video id="cam2" autoplay muted height="300"></video>
        <div class="audiofy">
          <button id="videoToAudio" class="pure-button pure-button-primary">Audiofy the video</button>
        </div>
      </div>

      <h2>Take it to the limit</h2>
      <p>
        Something interesting starts to happen if we increase the refresh rate significantly.
      </p>
      <div class="live-demo-video">
        <canvas id="spectrogram4" class="spectrogram" width="300" height="300"></canvas>
        <video id="cam3" autoplay muted height="300"></video>
        <div class="audiofy">
          <button id="instrumentToAudio" class="pure-button pure-button-primary">Start instrument</button>
        </div>
      </div>
      <p>
        It starts acting like an instrument! Notice that if you place your finger over the webcam,
        the sound is muted. You can produce low-pass or high-pass filters by covering only part of the webcam.
      </p>
      <div class="demo">
        <video muted autoplay="autoplay" loop="loop" controls width="400">
          <source src="img/demo.ogv" type="video/ogg">
        </video>
      </div>


      <div id="author">
        Written by <a href="https://github.com/DanielRapp">Daniel Rapp</a>.
        Check out the code for this <a href="https://github.com/DanielRapp/spectroface">on Github</a>.
      </div>
    </div>


    <script src="src/videoToAudio.js"></script>
    <script src="src/spectrogram.js"></script>
    <script src="src/data-note.js"></script>

    <script>
      // Due to a bug in webkit, we have to wait until the page has loaded
      // before we can ask for the mic.
      window.addEventListener('load', function(e) {
        document.getElementById('activate').addEventListener('click', function() {
          var btn = this;

          navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;

          navigator.getUserMedia({ video: true, audio: true }, function(stream) {
            handleMic(stream, drawAudio);
            handleCam(stream);

            // Self destruct!
            btn.parentNode.removeChild(btn);
          }, camError);
        })
      }, false);
    </script>
  </body>
</html>