GSoC 2024, Libvmaf V0.1 by yigithanyigit · Pull Request #2 · yigithanyigit/FFmpeg

yigithanyigit · 2024-08-21T18:25:49Z

This version directly propagates to AVFrame->metadata . However there are some offset in the frame indexes and doesn't finish at frame index supposed to finish.

For example, Lets say I have an input that 30 fps. If I just pass 2 seconds of that file with -t 2 command, filter should take 60 frames, but in current situation it takes 60 + first vmaf calculated index frames (in my case that was 13. In other words first vmaf calculated and triggered the callback function at 13th frame, 60 + 13 = 73). This problem looks like related to ff_filter_frame function.
I think it should only be called inside the do_vmaf function, but I couldn't find a way that works like that.

Update

I think that sync loss happens because of the framesync. My guess is framesync generating(requesting) frames again when it loss the sync. I am still investigating.

Update 2

Those extra frames actually looks like not re-requested frames. They are actual frames of the input. Program doesn't obeying the command that given (-t 2) and giving extra frames that shouldn't give.

Tested with:
./ffmpeg_g -loglevel info -i images/reference.mp4 -i images/distorted.mp4 -t 2 -lavfi psnr='stats_file=-:stats_version=2' -f null -"
And
./ffmpeg_g -loglevel info -i images/reference.mp4 -i images/distorted.mp4 -t 1 -lavfi libvmaf='n_threads=30:feature=name=psnr|name=ciede:metadata_handler=name=vmaf|name=psnr_y|name=ciede2000' -f null -

Update 3

It works perfectly fine If I don't give -t command. Instead of I trimmed the files and I gave like that. Right now it works like supposed to be.

It also works with complex filtergraphs
./ffmpeg_g -loglevel debug -i images/refer ence.mp4 -i images/distorted.mp4 -filter_complex "[0:v]trim=duration=1, setpts=PTS-STARTPTS[ref];[1:v]trim=duration=1, setpts=PTS-STARTPTS[dist];[ref][dist]libvmaf" -f null -

kylophone · 2024-08-22T04:40:59Z

libavfilter/vf_libvmaf.c

    {"n_subsample", "Set interval for frame subsampling used when computing vmaf.",     OFFSET(n_subsample), AV_OPT_TYPE_INT, {.i64=1}, 1, UINT_MAX, FLAGS},
    {"model",  "Set the model to be used for computing vmaf.",                          OFFSET(model_cfg), AV_OPT_TYPE_STRING, {.str="version=vmaf_v0.6.1"}, 0, 1, FLAGS},
    {"feature",  "Set the feature to be used for computing vmaf.",                      OFFSET(feature_cfg), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 1, FLAGS},
+    {"metadata_handler",  "Set the feature to be propagated as metadata.",              OFFSET(metadata_feature_cfg), AV_OPT_TYPE_STRING, {.str="name=vmaf"}, 0, 1, FLAGS},


I think it would be better to have a bool option to enable metadata writing for all features/models. If you do it in the way you've written, only one feature can be written as metadata. If you do it in the way I've suggested, when the option is set, you can register metadata callbacks for everything.

individual feature names as well as model names should both be available during init.

Actually users can give metadata handlers like this libvmaf='n_threads=30:feature=name=psnr|name=ciede:metadata_handler=name=vmaf|name=psnr_y|name=ciede2000' But obviously this makes hard to count metadata handlers, if someone enters wrong handler it still register that metadata handler.

Ah, I see. I still think a bool is nicer, since you don't have to repeat yourself for each option.

individual feature names as well as model names should both be available during init.

I couldn't find a list of available feature names/models. Can you provide a link for me please?
Thanks

kylophone · 2024-08-22T04:43:52Z

libavfilter/vf_libvmaf.c

+        metadata_cfg->data = s->cb;
+        metadata_cfg->callback = &set_meta;
+
+        err = vmaf_register_metadata_handler(s->vmaf, *metadata_cfg);


Anywhere we're using these new libvmaf metadata apis, we'll want to detect if they are available and guard with macros. See CONFIG_LIBVMAF_CUDA_FILTER for an example. This macro is generated during the configure stage.

The reason for this is compatibility with older libvmaf versions. fyi, these new apis are not part of a libvmaf release yet.

5855c9d
35f34a0

Anywhere we're using these new libvmaf metadata apis, we'll want to detect if they are available and guard with macros. See CONFIG_LIBVMAF_CUDA_FILTER for an example. This macro is generated during the configure stage.

Should those changes need to own functions like libvmaf_cuda or putting guard macros like I did is fine?

kylophone · 2024-08-22T15:53:15Z

libavfilter/vf_libvmaf.c

+    unsigned frame_number;
+    unsigned propagated_handlers_cnt;
+    struct FrameList *next;
+} FrameList;


I wonder if we could use something like av_fifo here instead?

https://ffmpeg.org/doxygen/trunk/group__lavu__fifo.html

Oh yes! Thats what I was looking for!

@kylophone I make some experiments with av_fifo and that might be not a good idea. Vmaf can send frames any order, If we use FIFO then we should make some operations to send them in order to next filter.

kylophone · 2024-08-22T17:07:20Z

libavfilter/vf_libvmaf.c

    {"n_subsample", "Set interval for frame subsampling used when computing vmaf.",     OFFSET(n_subsample), AV_OPT_TYPE_INT, {.i64=1}, 1, UINT_MAX, FLAGS},
    {"model",  "Set the model to be used for computing vmaf.",                          OFFSET(model_cfg), AV_OPT_TYPE_STRING, {.str="version=vmaf_v0.6.1"}, 0, 1, FLAGS},
    {"feature",  "Set the feature to be used for computing vmaf.",                      OFFSET(feature_cfg), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 1, FLAGS},
+    {"metadata_handler",  "Set the feature to be propagated as metadata.",              OFFSET(metadata_feature_cfg), AV_OPT_TYPE_STRING, {.str="name=vmaf"}, 0, 1, FLAGS},


Ah, I see. I still think a bool is nicer, since you don't have to repeat yourself for each option.

kylophone · 2024-08-23T21:19:07Z

libavfilter/vf_libvmaf.c

    int n_subsample;
    char *model_cfg;
+#if CONFIG_LIBVMAF_METADATA_FILTER
    char *feature_cfg;


feature_cfg should not be inside here

kylophone · 2024-08-23T21:20:05Z

libavfilter/vf_libvmaf.c

    {"n_subsample", "Set interval for frame subsampling used when computing vmaf.",     OFFSET(n_subsample), AV_OPT_TYPE_INT, {.i64=1}, 1, UINT_MAX, FLAGS},
    {"model",  "Set the model to be used for computing vmaf.",                          OFFSET(model_cfg), AV_OPT_TYPE_STRING, {.str="version=vmaf_v0.6.1"}, 0, 1, FLAGS},
    {"feature",  "Set the feature to be used for computing vmaf.",                      OFFSET(feature_cfg), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 1, FLAGS},
+    {"metadata_handler",  "Set the feature to be propagated as metadata.",              OFFSET(metadata_feature_cfg), AV_OPT_TYPE_STRING, {.str="name=vmaf"}, 0, 1, FLAGS},


This option needs a guard too

kylophone · 2024-08-23T21:21:23Z

libavfilter/vf_libvmaf.c

 };
 #endif
+
+#if CONFIG_LIBVMAF_METADATA_FILTER


This is unneeded. Not sure if this was intentional, or just a copy/paste error.

Yes this is copy/paste error. Fixed.

kylophone · 2024-08-23T22:14:20Z

libavfilter/vf_libvmaf.c

-#if CONFIG_LIBVMAF_METADATA_FILTER
 const AVFilter ff_vf_libvmaf_metadata = {
-    .name           = "libvmaf_cuda",
+    .name           = "libvmaf_metadata",


We don't need this extra filter at all. The only thing we need is to guard the option in the normal "libvmaf" filter.

I couldn't understand, could you explain more? You want me do something like this; libvmaf_filter_deps="libvmaf libvmaf-metadata"
instead of making
libvmaf_filter_deps="libvmaf"
libvmaf_metadata_filter_deps="libvmaf libvmaf_metadata" in the configure ? If answer is yes, in this way libvmaf filter doesn't work if not it's latest commit.

I couldn't find another way to do this in configuration stage without defining a new filter. If I not define a filter that macro (CONFIG_LIBVMAF_METADATA_FILTER) doesn't generate. I also tried with VMAF_VERSION but since it's string I can't do in macro stage.

That was my mistake, somehow I missed VMAF_VERSION_MAJOR/MINOR/PATCH. Fixed in those commits.

3ad9e15
364b219

yigithanyigit added 7 commits August 8, 2024 13:58

avfilter/vf_libvmaf: Propagatin metadata

9c12800

changes according to new api in libvmaf

d938b52

Remove data_sz according to libvmaf update

1f864c7

Update libvmaf according to support new changes

ec0c3c0

Update vf_libvmaf according to new changes

7b2b247

update vmaf filter according to new API changes (metadata propagation)

6556a11

Remove metastruct

6944fff

yigithanyigit force-pushed the libvmaf-v0.1 branch from 6c4a462 to ab83efc Compare August 22, 2024 00:13

kylophone reviewed Aug 22, 2024

View reviewed changes

yigithanyigit added 3 commits August 23, 2024 18:35

Another Approach

aa7289e

Add macro guard for new api

5855c9d

Add, defined macro guardsto vf_libvmaf

35f34a0

yigithanyigit force-pushed the libvmaf-v0.1 branch from 7702810 to 35f34a0 Compare August 23, 2024 15:35

kylophone reviewed Aug 23, 2024

View reviewed changes

Some bugfix and compilation error fix

a657cf3

yigithanyigit force-pushed the libvmaf-v0.1 branch from 8803754 to a657cf3 Compare August 23, 2024 21:46

kylophone reviewed Aug 23, 2024

View reviewed changes

yigithanyigit added 9 commits August 24, 2024 02:35

remove unrelated change

dfbf384

update guard macros with using vmaf version

3ad9e15

Update vmaf version to 2.0.0 to 3.0.0 in the filter

364b219

Fixed sync issues

e324fc9

Removed unnecessary logs

7a22076

add filters.h

726702d

Fixed build issues and made some optimizations

facfbb3

merge with other branch

f8b66bb

More bugfix

102edf9

Conversation

yigithanyigit commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Update 2

Update 3

Uh oh!

kylophone Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kylophone Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yigithanyigit Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yigithanyigit Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yigithanyigit Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yigithanyigit commented Aug 21, 2024 •

edited

Loading

kylophone Aug 22, 2024 •

edited

Loading

kylophone Aug 22, 2024 •

edited

Loading

yigithanyigit Aug 23, 2024 •

edited

Loading

yigithanyigit Aug 22, 2024 •

edited

Loading

yigithanyigit Aug 23, 2024 •

edited

Loading