Wakefield Council not pulling data

[nytram-md]

Name of Council

Wakefield Counil

Issue Information

Hi,

I’m using the UK Bin Collection Data HACS integration with Selenium in Home Assistant to get bin collection dates for Wakefield.

Council URL:
https://www.wakefield.gov.uk/where-i-live?uprn=63198222&... (full URL as configured)

Latest Version of both UKBinCollection 163.0 and Home Assistant 2026.2.1

Setup:

Selenium standalone server running in a Debian LXC on Proxmox.

Chromium 144 headless with --no-sandbox and --disable-gpu.

OpenJDK 17 installed.

HA integration pointed to http://:4444.

What works:

Selenium server runs and reports ready (/status endpoint).

Manual Chromium headless launch works but returns an HTML page.

Problem:

When attempting to scrape Wakefield’s site, the output is:

The request is blocked.

All HA entities remain unavailable.

Logs show: No previous data to fall back to.

Verification

• I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
• I have checked my address/postcode/UPRN works on the council's website
• I have provided a detailed explanation of the issue as well as steps to replicate the issue
• I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed

[nytram-md]

Hi

I have managed to get it working without selenium. I have the below in my wakfield_gov_uk.py file and I removed the headerless tick in the configuration for selenium and rebooted Home Assistant and now everything works.

Hope this helps and can be put into production

`from datetime import datetime
from typing import List
import requests
from bs4 import BeautifulSoup
from waste_collection_schedule import Collection

TITLE = "Wakefield Council"
DESCRIPTION = "Source for Wakefield.gov.uk services for Wakefield Council"
URL = "https://wakefield.gov.uk"

TEST_CASES = {
"uprn1": {"uprn": "63024087"},
"uprn2": {"uprn": 63105305},
"uprn3": {"uprn": "63012193"},
}

Maps keywords found in the section heading to icon/alias.

Matched against the lowercased heading, so casing changes on the site don't matter.

TYPES = {
"household": {"icon": "mdi:trash-can", "alias": "Household Waste"},
"general": {"icon": "mdi:trash-can", "alias": "Household Waste"},
"mixed": {"icon": "mdi:recycle", "alias": "Mixed Recycling"},
"recycling": {"icon": "mdi:recycle", "alias": "Mixed Recycling"},
"garden": {"icon": "mdi:leaf", "alias": "Garden Waste"},
}

HOW_TO_GET_ARGUMENTS_DESCRIPTION = {
"en": (
"Enter your UPRN (available from FindMyAddress.co.uk). "
"Alternatively, search your address on the Wakefield site manually and look for 'uprn=' in the URL."
)
}

PARAM_DESCRIPTIONS = {
"en": {
"uprn": "Unique Property Reference Number (UPRN)",
}
}

Real browser headers - the site may check the User-Agent

HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0"
),
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8",
"Accept-Language": "en-GB,en;q=0.5",
}

def _parse_date(text: str):
"""
Parse a date from strings like:
- 'Friday, 6 February 2026'
- 'Next collection - Friday, 20 February 2026'
- 'Last collection - Friday, 6 February 2026'
Returns a date object or None.
"""
text = text.strip()
# Strip a leading label such as "Next collection - "
if " - " in text:
text = text.split(" - ", 1)[1].strip()
# Expect "Weekday, D Month YYYY"
if ", " not in text:
return None
try:
date_part = text.split(", ", 1)[1].strip()
return datetime.strptime(date_part, "%d %B %Y").date()
except (ValueError, IndexError):
return None

def _resolve_type(heading: str):
"""Return (alias, icon) for a section heading, or (None, None) if unrecognised."""
lower = heading.lower()
for keyword, info in TYPES.items():
if keyword in lower:
return info["alias"], info["icon"]
return None, None

class Source:
def init(self, uprn):
self._uprn = uprn

def fetch(self) -> List[Collection]:
    entries = []

    with requests.Session() as sess:
        response = sess.get(
            "https://www.wakefield.gov.uk/where-i-live/",
            params={"uprn": self._uprn, "a": "Your Address"},  # FIX: no set {} wrapper
            headers=HEADERS,
            timeout=30,
        )
        response.raise_for_status()

    soup = BeautifulSoup(response.content, "html.parser")

    # Strategy: find every <strong> tag whose text looks like a bin-type heading,
    # then walk the enclosing container for dates.
    # This is CSS-class-agnostic so site rebuilds won't break it.

    seen_headings = set()

    for strong in soup.find_all("strong"):
        heading = strong.get_text(strip=True)
        alias, icon = _resolve_type(heading)
        if alias is None:
            continue
        # Avoid processing the same heading twice (e.g. duplicate strong tags)
        if alias in seen_headings:
            continue
        seen_headings.add(alias)

        # Walk up to find the nearest block-level container
        container = strong.find_parent(
            lambda tag: tag.name in ("div", "section", "article")
        )
        if container is None:
            container = strong.parent

        collection_dates = set()

        # "Last collection - Friday, 6 February 2026" style text nodes
        for text_node in container.find_all(string=True):
            t = text_node.strip()
            if "collection" in t.lower() and " - " in t:
                date = _parse_date(t)
                if date:
                    collection_dates.add(date)

        # Future collection <li> items: "Friday, 6 March 2026"
        for li in container.find_all("li"):
            date = _parse_date(li.get_text())
            if date:
                collection_dates.add(date)

        for date in sorted(collection_dates):
            entries.append(Collection(date=date, t=alias, icon=icon))

    return entries`