RainmeterAPI not working on certain sites.

[RicardoTM05]

I eventually took the bad decision to upgrade both Edge and WebView2 to version 143.0.3650.80 and, as expected, I now have the same drag / context menu issue on YouTube as everybody else, plus I'm being forced to not being able to downgrade either. Yay!

I think the cause might be in the plugin, even if all worked with my old runtime for some reason. To test this, in BangCommand.ini:

• use whatever plugin version and your own newer runtime version
• use an URL like https://www.youtube.com/watch?v=jZriOPN-VmI to go straight to a YouTube video
• use an AllowDualControl of 0 to start clean and without any additional Javascript interference
• have this basic script executed when pressing the "Execute Script" button in that skin:

[TxtExecuteScript]
...
LeftMouseUpAction=[!CommandMeasure "WebView" "ExecuteScript document.body.oncontextmenu = (e) => {if (e.ctrlKey) {console.log('contextmenu'); RainmeterAPI.Bang('[!SkinMenu]');};}; console.log('Script executed!');"]

• save and refresh the skin
• press the "Open DevTools" button and go to the developer tools console
• press the "Execute Script" button to inject the above script, at a time of your choosing after the page was loaded
• clear the console to see better what happens when you do the below
• try to CTRL + RMB for the skin context menu on that YouTube page, and this error should appear:

Let me know if you can reproduce it, know why it happens, or how to fix it if it's a bug (in a way that'll work in older runtimes too).

Also, if possible, there should be a way to forward various syntax errors from the script to the user for debug purposes. Try to use console.log('Script executed!); (i.e. without the ending single quote) in the script and you'll see that it fails silently - which is not necessarily a bad thing, by the way, except the user will be clueless on how to solve the script not working for him in such a case.

P.S. I case you somehow think that fixing this doesn't matter if we move to keyboard hooks for the dual control, you're wrong: it affects custom executing scripts too, as seen above. Reliability, compatiblity and features are key to the plugin, in this order.

Originally posted by @Yincognyto in #9

Apparently, Youtube is avoiding the RainmeterAPI from loading at all. It fails the same way even with a simple RainmeterAPI.Bang('[!Log "Test"]')

I suspect this has to do with iFrames, @NSTechBytes, can you please check this?

[Yincognyto]

Thanks for looking into it.

While iframes do indeed miss mouse events, see a simpler example and try dragging from the left side vs from the right side:
https://www.w3schools.com/html/tryit.asp?filename=tryhtml_iframe_height_width
they don't seem to throw the error I mentioned related to using the RainmeterAPI, which is what I was emphasizing on YouTube.

Also, on YouTube, the mouse events are detected just fine, as you could see, so I imagine that it's not an iframe throwing those errors there. My impression is that avoiding that error will also give a chance for such scripts to work on YouTube too. A simpler example of a site that throws this error could help.

[RicardoTM05]

YouTube is pretty complex by design. And knowing that WebView has a separated method to add HostObjects (which is what adds the RainmeterAPI) to frames is what makes me suspicious of them. Still not sure though. We do need a simpler way to test this.

[NSTechBytes]

There is nothing wrong with the HostObjects code.
This is a limitation of HostObjects.

HostObjects does not work reliably on heavy sites like YouTube.
Iframes, strict CSP, and modern JavaScript structure prevent access.

I recommend changing the AllowDual control logic to use keyboard events,
as mentioned above.

[RicardoTM05]

I failed to implement it using hooks due to WebView being on a different process. Maybe you have better luck on that.

Maybe the answer is around here: Allow input event messages to pass through the browser window

I will still look for ways to fix RainmeterAPI not working on YouTube. Given that it worked for Ying when he was on a older runtime, there’s still the probability to be fixed. Will be asking around.

[Yincognyto]

The 2 pending plugin issues are now: 1) the allow dual control logic; 2) the RainmeterAPI availability. I think both should be pursued:
- dual control like nstechbytes suggested would make it independent of the page particularities, thus hopefully available everywhere
- RainmeterAPI availability on all sites if possible is key to the ExecuteScript command, regardless of changing the dual control logic
The code is not wrong, but can be improved. We're only using the dual control script to test now, not because we want to keep it as the dual control logic. We should try to make RainmeterAPI work on iframes or YouTube if possible, the API is a feature of the plugin.

Here are some of my findings that could possibly help with the host objects availability on problematic sites, maybe we can use that:

• https://github.com/goh-chunlin/WebView2-YouTubeReplayer
• How to use HostObject when Iframe is used in the page. MicrosoftEdge/WebView2Feedback#1409

And the "AI" summary about WebView2 host objects issues on youtube:

Maneuverability (via dual control) and interactivity (via RainmeterAPI) would successfully embed WebView in skins. Afterwards, we can add other features if needed, but we must first have the base working for as many cases / sites as possible, and if possible.

[NSTechBytes]

I believe the issue has already been fixed by RicardoTM, as mentioned here:
#9 (comment)

[RicardoTM05]

I believe the issue has already been fixed by RicardoTM, as mentioned here: #9 (comment)

Yes and no.

I have finally added RainmeterAPI support for top-level iframes which is working fine now. Nested frames are not supported apparently.

Also implemented put_AdditionalAllowedFrameAncestors to allow the user to frame websites, however this feature requires a http-server.

I’ll provide a new testing build in a couple of hours, as I’m not on the PC right now.

The YouTube issue is still there though. I opened an issue on WebView’s GH seeking for help, so maybe they can clarify what the actual issue is.

My current theories are:
Youtube is running inside a nested frame (less probable)
Youtube is blocking scripts some other way.

Ying will be fixing the AllowDualControl script to work correctly inside iFrames in case you don’t find a non-js alternative for it, which we hope you do as it would save us and the users a lot of headaches down the road.

[NSTechBytes]

Sure, there’s no rush. I think AllowDualControl will automatically be fixed once the RainmeterAPI issue is solved, since the problem is that RainmeterAPI isn’t working properly due to iframes.

[RicardoTM05]

Ying and I got to the conclusion that JavaScript’s side AllowaDualControl should be a temporary solution until we get a real solution. The current state of AllowDualControl, even with Yin’s future fix, will not be reliable enough, as there will be websites with elements which will not allow the skin to be dragged.

We can accept that there will be parts of websites, or even complete websites that will unfortunately avoid RainmeterAPI from being injected, but that shouldn’t prohibit the user from dragging the skin or opening the context menu, as those are “basic needs”.

We need to find a way to let the user drag the skin and open the context menu without injecting JavaScript.

I would really appreciate if you could investigate such possibility. Maybe by using hooks or by some other creative means.

[Yincognyto]

I believe the issue has already been fixed by RicardoTM, as mentioned here: #9 (comment)

As Ricardo said, it is only partially fixed. Apart from YT which might be a separate issue than "only" iframes, we now have:

• RainmeterAPI in top-level frames from C++ (advantage: bypasses same origin policy; disadvantage: doesn't cover nested frames)
• RainmeterAPI in possibly all frames from JS (advantage: might cover nested frames; disadvantage: subject to same origin policy)

In other words, both Ricardo from C++, and me from JS, have finally been able to access RainmeterAPI from top-level frames, but each method has its disadvantage. I still need to investigate whether I can recursively look for iframes on the JS side and access RainmeterAPI for each of their document.body objects, so that nested frames are covered too, plus adjusting the skin coordinates when dragging from the JS side, where the iframe coordinates aren't yet accounted for.

To test how it's done from the JS side in a demo example (works for this site only for now, but can be extended), use this URL:

https://www.w3schools.com/html/tryit.asp?filename=tryhtml_iframe_height_width

and this ExecuteScript command:

OnPageLoadFinishAction=[!CommandMeasure WebView """ExecuteScript var f = document.getElementById('iframeResult'), b = (f.contentDocument || f.contentWindow).body; b.onpointerdown = (e) => {console.log('Test'); RainmeterAPI.Bang('[!Log "Test"]');}; document.body.onpointerdown = (e) => {console.log('Test'); RainmeterAPI.Bang('[!Log "Test"]');};"""]

This should print "Test" in both the DevTools console and the Rainmeter log on LMB, in both the left part of the page and the upper right side of it (the right side has nested frames, and I didn't tackle that yet, but the upper right side is a top level frame). From my tests, it works on your 0.0.7 plugin release version too, but obviously the C++ method by Ricardo has the advantage of bypassing the same origin policy, plus it adds the ability to frame websites as a bonus. Because of that, ideally we'd have a blend of the two methods but on the C++ side, if RainmeterAPI can be made available for nested frames as well. If not, I'll try from JS anyway.

[Yincognyto]

This is a limitation of HostObjects.

HostObjects does not work reliably on heavy sites like YouTube. Iframes, strict CSP, and modern JavaScript structure prevent access.

We can accept that there will be parts of websites, or even complete websites that will unfortunately avoid RainmeterAPI from being injected, but that shouldn’t prohibit the user from dragging the skin or opening the context menu, as those are “basic needs”.

Just an update: I found one possible, tricky but very simple solution to the YouTube problem, and this occured, of all things, while tyring to make my dual control script to adjust for the frame coordinates. Basically, I managed to make RainmeterAPI.Bang('[!Log "Test"]') print test to the Rainmeter log, which means anything else pertaining to RainmeterAPI is possible too on YouTube.

This could have positive implications in regard to the injection of scripts into "frames" as well, or at least make things much more obvious if and when they "go wrong" (which they don't actually, it only seems that way, at least from what I noticed on YouTube). I will need to test and adjust things in my script before I can come up with a "final" demo that tackles both the YouTube and "frames" issue, cause I don't like loose ends - just wanted to let you guys know the good news. It would be interesting to find out if this can be used on the C++ side though...