[Feature request] option to extract audio track from the video

[ekuleshov]

Platforms

Android, iOS, Web, Windows, macOS, Linux

Description

Currently there is an option to mute/remove audio track from the video. It would be really handy to have an option to extract audio track (or audio tracks) from said video.

Why

The audio track can be analyzed to identify specific parts on the video, for things like speech to text transcription or for automatic subtitles.

[hm21]

@ekuleshov I agree with you, that could be useful👍🏻 I can't say exactly when I'll implement it, as I'm currently working on the video clip merging feature. After that, I'll add the background music functionality. Once those are done, I'll have time to work on this feature.

FYI: I've added the "in-progress" label to prevent this thread from auto-closing.

[hm21]

I completely forgot to mention that I implemented that feature. You can extract the audio as shown below:

/// Check if video has audio before extraction (recommended)
final video = EditorVideo.asset('assets/video.mp4');
bool hasAudio = await ProVideoEditor.instance.hasAudioTrack(video);

if (!hasAudio) {
    print('Video has no audio track');
    return;
}

/// Extract with trimming
var config = AudioExtractConfigs(
    video: video,
    format: AudioFormat.aac,
    startTime: Duration(seconds: 10),
    endTime: Duration(seconds: 30),
);

/// Save to file instead of returning as Uint8List
final directory = await getTemporaryDirectory();
String outputPath = '${directory.path}/extracted_audio.mp3';

try {
    await ProVideoEditor.instance.extractAudioToFile(outputPath, config);
} on AudioNoTrackException {
    print('Video has no audio track');
}
/// Alternative read the Uint8List directly like below.
/// Uint8List audioData = await ProVideoEditor.instance.extractAudio(audioConfig);

/// Listen to progress
StreamBuilder<ProgressModel>(
    stream: ProVideoEditor.instance.progressStreamById(config.id),
    builder: (context, snapshot) {
      var progress = snapshot.data?.progress ?? 0;
      return CircularProgressIndicator(value: progress);
    }
)

[ekuleshov]

@hm21 so awesome! thank you
btw, can it extract to wav? That would be most useful for audio analysys

[hm21]

@ekuleshov Unfortunately, that's not supported. The native-code on android/ios didn't support many formats and also depend on the platform, so the only options you can currently choose from are listed below.

/// Audio output formats supported for audio extraction.
///
/// Different platforms may have different support levels for each format.
enum AudioFormat {
  /// MP3 format - widely supported, good compression.
  /// Supported on: Android only (not supported on iOS/macOS)
  mp3,

  /// AAC format - high quality, modern codec.
  /// Supported on: Android, iOS, macOS
  aac,

  /// M4A format - Apple's container for AAC.
  /// Supported on: Android, iOS, macOS
  m4a,

  /// CAF format - Core Audio Format, Apple's flexible container.
  /// Supported on: iOS, macOS (Apple only)
  caf,
}

[ekuleshov]

@ekuleshov Unfortunately, that's not supported. The native-code on android/ios didn't support many formats and also depend on the platform, so the only options you can currently choose from are listed below.

What about raw pcm format? As far as I know it can be converted to on Android and iOS.

Though the wav is basically a wrapper around pcm data with some headers to specify frame rate, number of channels, etc. E.g. https://pub.dev/documentation/wav/latest/wav_file/Wav/write.html

Most speech recognition or audio classification models are taking pcm audio input with 16khz sample rate.

[hm21]

That's a good point. I'll run some tests and get back to you. Btw I'll reopen that feature request so people know there's something in progress.

[hm21]

Version 1.3.0 now also supports the wav format. No idea why but I noticed it was already implemented on the Android side. Looking through my commits, I saw that I later removed it on the Dart side, probably by accident while fixing an issue with another format.

FYI: What I currently don’t support is choosing a specific sample rate, like 16kHz.

[ekuleshov]

That's awesome! Thank you Alex. Appreciate it. Resampling down to 16kHz is easy enough.