Check if entry displayed in grid container

[dornech]

I wonder if someone has an idea for ma problem. I retrieve data form a (dynamic) list which is updated when scrolling. Unfortunately not all subnodes are displayed (but only 10 of them). I did not find a solution via hidden attribute or something like this

[GCuser99]

Would something like this work?

[dornech]

Now, I want to loop through the entries with Slennium.-
I tried to enter a screenshot lof the DOM but for whatever reason it does not work

[dornech]

After some search I think I found what i was looking for
https://stackoverflow.com/questions/487073/how-to-check-if-element-is-visible-after-scrolling
https://jsfiddle.net/W33YR/411/

[dornech]

I checked the solution and it works as intended - however: only issue was thet variable "top" was not properly initialized, I just changed it to rect_top.

[hanamichi77777]

The issue you are facing is a common challenge in web development known as "Virtual Scrolling" or "DOM Virtualization." Modern UI libraries like MUI (Material UI) only render the elements currently visible in the viewport (plus a small buffer) to maintain high performance. As a result, regardless of the total list size, you are limited to capturing only about 10 elements at a time.

To retrieve the entire dataset, you must implement an "Incremental Scroll & Capture" strategy.

Key Components of the Solution
Deduplication: Use a Scripting.Dictionary object for data storage. This allows you to automatically handle and exclude duplicate items that may be captured during each scroll step.

Smart Waiting: Avoid using unreliable fixed-time waits (Wait commands). Instead, inject a JavaScript hook to precisely detect when both network traffic (Fetch/XHR) and DOM rendering have settled (reached an "Idle" state). This technique is consistent with the approach proposed in #194.

Scroll Loop: Programmatically scroll the container element in increments, wait for the "Idle" state each time, and capture the newly appearing elements.

Below is an implementation example for MUI Grid using SeleniumVBA

' Main scraping process for MUI Data Grid
Sub TestMuiDataGrid_WithHooks()
    Dim driver As WebDriver
    Set driver = New WebDriver
    
    driver.StartChrome
    driver.OpenBrowser
    
    ' Navigate to the MUI Data Grid demo page
    driver.NavigateTo "https://mui.com/x/react-data-grid/"
    
    ' Wait for page and framework initialization
    driver.Wait 4000
    
    ' --- Inject Hooks ---
    InstallNetworkAndDomHooks driver
    
    ' Identify the 2nd scroll container specifically
    Dim scrollContainer As WebElement
    Dim scrollers As WebElements
    Set scrollers = driver.FindElements(By.cssSelector, ".MuiDataGrid-virtualScroller")
    
    ' Ensure at least 2 scrollers exist before proceeding
    If scrollers.Count >= 2 Then
        Set scrollContainer = scrollers.item(2)
    Else
        Debug.Print "Error: Could not find the 2nd scroll container. (Found: " & scrollers.Count & ")"
        driver.Shutdown
        Exit Sub
    End If
    
    ' Dictionary for deduplication
    Dim collectedData As Object
    Set collectedData = CreateObject("Scripting.Dictionary")
    
    Dim previousCount As Long
    Dim currentCount As Long
    Dim noNewItemsCount As Integer
    Dim items As WebElements
    Dim item As WebElement
    Dim itemText As String
    
    Debug.Print "Starting extraction from MUI Data Grid (Target: 30 items)..."
    
    Do
        previousCount = collectedData.Count
        
        ' Retrieve rows ONLY within the 2nd scroll container
        Set items = scrollContainer.FindElements(By.cssSelector, ".MuiDataGrid-row")
        
        For Each item In items
            itemText = item.GetText
            If itemText <> "" And Not collectedData.Exists(itemText) Then
                collectedData.Add itemText, itemText
                
                ' --- STRICT LIMIT CHECK ---
                ' Exit immediately when reaching exactly 30 items
                If collectedData.Count >= 30 Then
                    Debug.Print "Reached the limit of 30 items."
                    GoTo ExitLoop
                End If
                
                If collectedData.Count Mod 10 = 0 Then
                    Debug.Print collectedData.Count & " items collected..."
                End If
            End If
        Next item
        
        ' Scroll the container
        driver.ExecuteScript "arguments[0].scrollTop += arguments[0].clientHeight;", scrollContainer
        
        ' --- Asynchronous Idle Wait ---
        ' Vital for MUI Data Grid to process internal state and render new rows
        WaitForIdleStateAsync driver, 500, 10000
        
        currentCount = collectedData.Count
        
        ' Termination Logic: If no new items are found after multiple scrolls
        If currentCount = previousCount Then
            noNewItemsCount = noNewItemsCount + 1
            If noNewItemsCount >= 3 Then Exit Do
        Else
            noNewItemsCount = 0
        End If
    Loop

ExitLoop:
    ' Output results
    Debug.Print "--- MUI Data Grid Extraction Results (" & collectedData.Count & " items) ---"
    Dim k As Variant
    For Each k In collectedData.keys
        Debug.Print Replace(collectedData(k), vbLf, " | ")
    Next k
    
    driver.Shutdown
    Set driver = Nothing
End Sub

' -------------------------------------------------------------------------
' Inject global hooks to monitor network traffic (Fetch/XHR) and DOM changes
' -------------------------------------------------------------------------
Private Sub InstallNetworkAndDomHooks(oDriver As WebDriver)
    Dim js As String
    js = "if(!window.__niw){(function(){" & vbCrLf
    js = js & "  var s={req:0, lastMut:Date.now(), of:window.fetch, oo:XMLHttpRequest.prototype.open, os:XMLHttpRequest.prototype.send, onchange:null};" & vbCrLf
    js = js & "  window.fetch=function(){s.req++; if(s.onchange)s.onchange(); return s.of.apply(this,arguments).finally(function(){s.req--; if(s.onchange)s.onchange();});};" & vbCrLf
    js = js & "  XMLHttpRequest.prototype.open=function(){return s.oo.apply(this,arguments);};" & vbCrLf
    js = js & "  XMLHttpRequest.prototype.send=function(){s.req++; if(s.onchange)s.onchange(); this.addEventListener('loadend',function(){s.req--; if(s.onchange)s.onchange();}); return s.os.apply(this,arguments);};" & vbCrLf
    js = js & "  var ob=new MutationObserver(function(){s.lastMut=Date.now(); if(s.onchange)s.onchange();});" & vbCrLf
    js = js & "  ob.observe(document,{childList:true,subtree:true,attributes:true});" & vbCrLf
    js = js & "  window.__niw=s;" & vbCrLf
    js = js & "})();}"
    oDriver.ExecuteScript js
End Sub

' -------------------------------------------------------------------------
' Use injected hooks to wait asynchronously until the environment is idle
' -------------------------------------------------------------------------
Private Sub WaitForIdleStateAsync(oDriver As WebDriver, Optional idleTimeout As Long = 500, Optional maxTimeToWait As Long = 30000)
    Dim js As String, ret As Variant
    js = "var idleTimeout=arguments[0], maxTimeout=arguments[1], cb=arguments[2];" & vbCrLf
    js = js & "var s=window.__niw; if(!s){cb('not-armed'); return;}" & vbCrLf
    js = js & "var tm=null, settled=false, safety=setTimeout(function(){done('timeout');}, maxTimeout);" & vbCrLf
    js = js & "function done(v){if(settled)return; settled=true; if(tm)clearTimeout(tm); clearTimeout(safety); cb(v);}" & vbCrLf
    js = js & "function schedule(){" & vbCrLf
    js = js & "  if(tm)clearTimeout(tm);" & vbCrLf
    js = js & "  var timeSinceLastMutation = Date.now() - s.lastMut;" & vbCrLf
    js = js & "  if(s.req===0){" & vbCrLf
    js = js & "    var remainingDelay = Math.max(0, idleTimeout - timeSinceLastMutation);" & vbCrLf
    js = js & "    tm=setTimeout(function(){done('ok');}, remainingDelay);" & vbCrLf
    js = js & "  }" & vbCrLf
    js = js & "}" & vbCrLf
    js = js & "s.onchange=schedule;" & vbCrLf
    js = js & "schedule();"
    
    ret = oDriver.ExecuteScriptAsync(js, idleTimeout, maxTimeToWait)
    If ret <> "ok" Then Err.Raise 404, , "Wait failed or timed out: " & ret
End Sub

[dornech]

Wow, thanks a lot for your effort. I tried with the abovementioned javascript and this works fine as it allows to determine which scroll container entries are visible and which ones not. Duplicate detection I do already with other means; my use case is an automated bank document download and I register al downloades documents in a separte list anyway..