Recording audio from the device inputs

Configuring the engine for recording audio from the devices input(s):

In order to record audio from the device input (either attached sound module or the internal microphone), you first need to follow the configuration as stated on the Recording and bouncing engine output page. Basically we will need to be able to record the inputs and record to disk.

Also take note of the snippet based recording.

An AudioChannel for audio input

A dedicated AudioChannel is available to process the incoming audio. It is available in C++ as property AudioEngine::inputChannel or through Java via MWEngineInstance.getInputChannel().

The benefit of having an AudioChannel for the input is that this allows you to use all the properties associated with an AudioChannel, such as volume, pan and its ProcessingChain on your incoming audio stream. This means you can apply effects in real time while recording.

Feedback during recording

Also be aware of the benefit of muting the input channel (or lowering its volume). When the audio is recorded from the device's microphone and no headphones are plugged in, this is an open invitation for getting a feedback loop. By having a positive value for the input channels volume, you allow live monitoring of the input as it is being recorded.

Recording incoming audio

To record the incoming audio simply toggle the boolean value to monitor the input channels input (keep in mind the volume setting of the inputs AudioChannel, as described above). The input will be audible regardless of the Sequencer running or not.

In C++

void AudioEngine::recordDeviceInput( bool isRecording );

In Java using the MWEngine class

MWEngine.recordInput( boolean );

Recording incoming audio onto device storage

MWEngine allows you to record the incoming audio from the Android device, while the Sequencer is running. When recording incoming audio from the Android device, the pre mix of the input AudioChannel is written to disk, which means audio is written with the ProcessingChains effects applied, but without the volume (the reason being that it might be more convenient to record audio without live monitoring to prevent feedback from occurring). This means that you can record while other instruments are playing, without recording the output of those instruments.

Also consult the page on Recording and bouncing engine output for further information.

Recording incoming audio in C++

void AudioEngine::setRecordInputToFileState( int maxBuffers, char* outputFileName, bool skipProcessing );
void AudioEngine::unsetRecordInputToFileState();

Given integer maxBuffers describes the maximum amount of buffers that each recorded snippet should hold. String outputFileName describes the desired filename of the .wav-file that will be recorded (NOTE : its directory must be created prior to invoking this method). When the input channels ProcessingChain contains effects, boolean skipProcessing can be used to determine whether to omit the processing prior to writing the input to the file (thus recording the dry "pre-mix" buffer). When set to false, the input plus applied effects ("post-mix") are written to a file.

Recording incoming audio in Java using the MWEngine class

MWEngine.startInputRecording( String outputFileName, boolean skipProcessing );
MWEngine.stopInputRecording();

As with the C++ version, the directory for given outputFileName should exist prior to invoking this method.

Recording both incoming audio and device output onto device storage

Known as "full duplex" recording, where basically the input from the device is recorded simultaneously with the sequenced audio rendered internally by the engine.

Technically you can achieve this by recording the engine output together with an active device input recording, but this method does not account for roundtrip latency.

Understanding roundtrip latency

Roundtrip latency is basically the time it takes between recording a sound and hearing the sound being played back. On most Android devices this is nowadays pretty good with low latency recording paths being available. However, as the Android ecosystem is quite fragmented, certain devices will struggle with providing a low latency audio path. In this case, you will hear a mismatch between the timing of the input channel and the sequenced audio. For instance, consider a karaoke application where a user must sing along to a beat. When the user is timing their singing with the beat, the roundtrip latency would lead to the device only "hearing" the vocal much later (sometimes as large as 400 milliseconds, which makes the performance out of time with the music).

Addressing roundtrip latency

To address roundtrip latency, MWEngine must align the recorded input signal with the recorded sequenced audio (basically "pulling" the recording forward by the latency size).

As mentioned, latency is device specific so should be calculated for each unique device (there is sadly no "magic" constant number we can use). The easiest (and most accurate) way to do this is by using a loopback adapter. However, note that this is a physical hardware device you can't expect each user of your applications users to have in their possession.

To overcome this, you can add a (one-time) setup phase to your application:

The user needs to sing along to a very short clip (maybe a four bar loop of a constant drum pulse where they say a short word to align with the timing of the pulse) and afterwards match their recorded input with the sequenced drum. The setup for that recording being:

• Present user with "start" button, upon click the sequencer starts playback while at the same startInputRecording is invoked (so we are only recording the device input, not the sequenced output of the drum)
• Track the progress for four bars using the Notifier mechanism (also seen in the example Activity). Once four bars have elapsed, you will be stopping both the sequencer and invoking stopInputRecording to stop recording and save it to storage)
• Load the written input recording from storage, and load it into the SampleManager
• Present user with UI where you have a "play" button which at the same time starts the sequencer from the beginning, but also plays a SampleEvent (where the sample is the input recording) at 0 samples offset. So basically the drum is playing again, but the recorded vocal is now playing alongside it.
• The UI should also contain a slider where the user can adjust the offset of the input recording (you want the sliders value to generate a number in milliseconds)
• This value in milliseconds must be converted to an amount in samples (you can use BufferUtility::millisecondsToBuffer for the conversion) and set it as the eventStart for the SampleEvent playing back the recording (note that when its supplied as a start offset you must pass the number as a negative value as we are pulling its playback "forward" (e.g. it starts at an earlier point in time).
• When the user is satisfied with the alignment, they click a "save" button and the value in milliseconds should be stored on their device storage (which you can subsequently read and provide to the startFullDuplexRecording-method when recording begins for a performance).
• This setup screen is not shown again (unless user decides to reconfigure it through a settings menu)

So this becomes a one time setup where all subsequent sessions for the user require no further setup (as the latency will not change for their device). This avoids making this a repetitive task upon each performance (also it will be more honest as people tend to overestimate their timing skills and will have differences in accuracy between performances).

Recording full duplex audio in C++

void AudioEngine::setRecordFullDuplexState( float roundtripLatencyInMs, int maxBuffers, char* outputFile );
void AudioEngine::unsetRecordFullDuplexState();

Given float roundtripLatencyInMs describes the calculated latency in milliseconds. This is used to align the input and sequenced audio streams. The remaining arguments are equal in function to the other recording methods.

Recording full duplex audio in Java using the MWEngine class

MWEngine.startFullDuplexRecording( float roundtripLatencyInMs, String outputFile );
MWEngine.stopFullDuplexRecording();