From 031bf47991f3c8c141ed3ca368326353e31c6d83 Mon Sep 17 00:00:00 2001
From: frumza <frumza@gmail.com>
Date: Tue, 12 May 2026 23:52:48 -0400
Subject: [PATCH] linkedin: domain skills for send-connection-request +
 connection-search-scrape

Two new LinkedIn domain skills:

- send-connection-request.md: invite flow with paywall detection,
  rate-limit notes, coordinate-click fallback. Covers the 'Connect
  is an <a> not a <button>' gotcha + the iframe-paywall detection
  pattern.

- connection-search-scrape.md: keyword-filtered People-search via
  connectionOf filter. Anchor-innerText extraction pattern (robust
  to LinkedIn's rotating CSS-module class names), pagination,
  anti-bot guidance, empirical keyword hit-rate table, self-contained
  runnable script.
---
 .../linkedin/connection-search-scrape.md      | 259 ++++++++++++++++++
 .../linkedin/send-connection-request.md       | 150 ++++++++++
 2 files changed, 409 insertions(+)
 create mode 100644 domain-skills/linkedin/connection-search-scrape.md
 create mode 100644 domain-skills/linkedin/send-connection-request.md

diff --git a/domain-skills/linkedin/connection-search-scrape.md b/domain-skills/linkedin/connection-search-scrape.md
new file mode 100644
index 00000000..b39b387c
--- /dev/null
+++ b/domain-skills/linkedin/connection-search-scrape.md
@@ -0,0 +1,259 @@
+# LinkedIn — Scrape someone's connections list via keyword-filtered People search
+
+Mine a target user's network (1st-degree connections, plus optionally 2nd-degree via the asker's graph) by paginating the People-search results with the `connectionOf` filter pre-applied. Keyword sweep beats blanket-scrape — narrows from thousands of connections to ~hundreds of relevant hits, and stays well under LinkedIn's anti-scrape thresholds.
+
+## When to use this
+
+Use this when you have a 1st-degree connection (e.g. a parent, partner, or co-founder) whose own network you want to mine for warm-intro candidates. LinkedIn's People-search lets you filter by "Connections of X" — that returns the intersection of (your 1st+2nd graph) ∩ (X's 1st-degree connections). For a parent or close colleague the overlap is usually broad enough to be useful.
+
+Don't use this for full graph dumps — use the Voyager profile API (`/voyager/api/identity/dash/profiles?q=memberIdentity`) for that. This skill is for **targeted relevance search across someone else's network.**
+
+## The URL pattern
+
+```
+https://www.linkedin.com/search/results/people/
+  ?connectionOf=%5B%22<MEMBER_URN>%22%5D
+  &network=%5B%22F%22%2C%22S%22%5D
+  &keywords=<KEYWORD>
+  &page=<N>
+  &origin=MEMBER_PROFILE_CANNED_SEARCH
+```
+
+- `connectionOf` is a JSON-array-encoded list of `MEMBER_URN` strings (just one URN in practice). Format: `ACoAA...` — get it from the target's `/in/<handle>/` profile page (see "Getting the member URN" below).
+- `network=["F","S"]` = 1st + 2nd degree (relative to the *asker's* graph). Drop `"F"` to exclude direct connections; drop `"S"` to scope to direct only.
+- `keywords=<KW>` is a free-text filter applied across name, headline, current company, past company, education. The same single-keyword caveats apply as anywhere on LinkedIn — narrow keywords beat broad ones.
+- `page=<N>` is 1-indexed. LinkedIn caps People-search at **100 pages × 10 results = 1000 max per query**. Plan keyword sweeps so each keyword returns <1000 hits.
+- `origin=MEMBER_PROFILE_CANNED_SEARCH` is what LinkedIn sets when the user clicks "See all" on a profile's connections — using it identifies the search shape as a legitimate UI flow and reduces friction.
+
+## Getting the member URN
+
+Open the target's profile (`/in/<handle>/`) while logged in, then run:
+
+```python
+js("""(() => {
+    // The URN is embedded in many places. Most reliable:
+    // 1. Look for a 'urn:li:fsd_profile:ACoAA...' in any DOM attribute
+    const m = document.body.innerHTML.match(/urn:li:fsd_profile:([\\w-]+)/);
+    return m ? m[1] : null;
+})()""")
+```
+
+Returns the URN string (`ACoAA...` plus ~30 characters). Use that as `<MEMBER_URN>` in the search URL.
+
+## Result-card extraction — the key DOM pattern
+
+LinkedIn's 2026 People-search DOM uses obfuscated CSS-module class names (e.g. `_83309bd4 _6e63fa0b d343d86c`) that rotate per build. **Don't selector-hunt for cards** — go straight to the profile anchors and use their `innerText` to read the whole card payload.
+
+Each result card has at least one anchor pointing to `/in/<handle>/` whose `innerText` contains the full card payload joined by `\n`:
+
+```
+<Name> • <Degree>
+
+<Headline>
+
+<Location>
+
+Connect
+
+[Past: ...]  | [Summary: ...]  | [Current: ...]   (optional)
+
+<Mutuals>  e.g. "<Mutual Name> and N other mutual connections"
+```
+
+Multiple anchors point at the same profile (image link + name link + headline link); dedupe by `URL.pathname`.
+
+The minimum filter:
+
+```python
+items = js("""(() => {
+    const anchors = Array.from(document.querySelectorAll('a[href*="/in/"]'));
+    const out = [];
+    const seen = new Set();
+    for (const a of anchors) {
+        const u = new URL(a.href);
+        if (!u.pathname.match(/^\\/in\\/[^/]+\\/?$/)) continue;  // strict /in/<handle> only
+        if (!a.innerText.includes('•')) continue;                // skip thumbnail-only links
+        const href = u.pathname.replace(/\\/$/, '');
+        if (seen.has(href)) continue;
+        seen.add(href);
+        out.push({href, text: a.innerText});
+    }
+    return out;
+})()""")
+```
+
+Then parse `text` line-by-line. The first line is `"<Name> • <Degree>"` (split on `•`). The line containing `"mutual connection"` is the mutuals. Lines `"Connect" / "Message" / "Follow"` are buttons — skip. Lines starting `"Past:" / "Summary:" / "Current:"` are auto-generated keyword highlights — useful for relevance scoring. Lines starting `"Select all" / "Add N people to folk"` are list-controls — skip.
+
+## Pagination
+
+Scroll-load won't help — LinkedIn paginates server-side. After landing on `&page=1`, navigate to `&page=2`, etc. Each page returns at most 10 results.
+
+Stopping conditions:
+- A page returns 0 new (deduped) results → keyword exhausted, move on
+- A page returns fewer than ~5 cards → last page of results
+- `page=100` reached → LinkedIn cap; if the keyword is broader than 1000 hits you need a tighter keyword
+
+Sleep ~2s per page-load via `time.sleep(2)` after `wait_for_load`. Sleep ~1s between keywords. LinkedIn does not aggressively rate-limit People-search at this pace from a real logged-in session.
+
+## Anti-bot
+
+**Use a real signed-in Chrome session, not headless.** LinkedIn fingerprints heavily. The `feedback_real_chrome_beats_headless_for_anti_bot` lesson applies — `start_remote_daemon` with a clean profile gets you 1-3 results then a soft-block. The user's actual Chrome with all cookies + extensions + real navigator fingerprint sails through.
+
+No `stealth_patch()` needed when on real-Chrome — only useful in headless/cloud mode.
+
+## Tab management gotcha
+
+`new_tab(url)` creates a new tab but `switch_tab()` is required to make subsequent `js()` / `goto()` calls target it. If the user manually switches tabs between calls, the daemon may follow their focus — list tabs and explicitly switch back:
+
+```python
+tabs = list_tabs(include_chrome=False)
+linkedin_tab = next((t for t in tabs if 'linkedin.com/search' in t.get('url','')), None)
+if linkedin_tab:
+    switch_tab(linkedin_tab['targetId'])
+```
+
+## Keyword strategy
+
+Hit-rate per keyword varies wildly. Sample patterns from a real run against a target with a regional emerging-markets business network (39 keywords, ~380 unique results, ~10 min wall-time):
+
+| Keyword type | Example | Typical hits |
+|---|---|---|
+| **Geographic country** | a target-region country name | 50-80 |
+| **Geographic city** | a target-region city name | 0-20 |
+| **Broad industry** | `engineering`, `trading`, `logistics` | 50-100 (high noise) |
+| **Specific company** | a named target company | 0-3 (high precision) |
+| **Industry term** | `manufacturing`, `procurement`, `mining` | 5-15 |
+| **Niche term** | a specific product or technology | 0 (target may not have these) |
+
+**Geo-keywords yield best per-query precision for finding "people in X country who do Y."** Company-name keywords are highest precision but often return 0 because exact-employer-match is rare in a non-overlapping network. Industry terms are middle — broad enough to surface candidates, narrow enough to filter noise.
+
+## End-to-end runnable script
+
+Self-contained. Replace `CONN_OF` with the target's URN (see "Getting the member URN" above) and `TARGET_LABEL` + `KEYWORDS` for your search. Run via `browser-harness < scrape.py` from a real-Chrome session signed into LinkedIn. Re-runnable + resume-safe — dedupes against prior output, appends only.
+
+```python
+import time, json, urllib.parse
+from pathlib import Path
+
+# === EDIT THESE ===
+CONN_OF = "ACoAA<...REPLACE WITH TARGET URN...>"
+TARGET_LABEL = "target_label"   # used in output filename
+KEYWORDS = [
+    # broad industry
+    "engineering", "trading", "logistics", "manufacturing",
+    # named companies you care about
+    "AcmeCorp", "TargetCo",
+    # geographies
+    "Country1", "City1",
+]
+MAX_PAGES_PER_KEYWORD = 10  # LinkedIn caps at 100; 10 is usually plenty per keyword
+
+# === END EDIT ===
+
+BASE = (f"https://www.linkedin.com/search/results/people/"
+        f"?connectionOf=%5B%22{CONN_OF}%22%5D"
+        f"&network=%5B%22F%22%2C%22S%22%5D"
+        f"&origin=MEMBER_PROFILE_CANNED_SEARCH")
+
+OUT = Path(f"/tmp/{TARGET_LABEL}_connections.jsonl")
+OUT.parent.mkdir(parents=True, exist_ok=True)
+
+seen = set()
+if OUT.exists():
+    for line in OUT.read_text().splitlines():
+        try:
+            r = json.loads(line)
+            seen.add(r.get("href", ""))
+        except Exception:
+            pass
+
+def parse_card(text, href):
+    lines = [l.strip() for l in text.split("\n") if l.strip()]
+    if not lines: return None
+    parts = lines[0].split("•")
+    name = parts[0].strip()
+    degree = parts[1].strip() if len(parts) > 1 else ""
+    headline = lines[1] if len(lines) > 1 else ""
+    location, mutuals, summary = "", "", ""
+    for l in lines[2:]:
+        if l in ("Connect","Message","Follow") or l.startswith("Select all") or l.startswith("Add "):
+            continue
+        if "mutual connection" in l.lower() or " is a mutual" in l.lower():
+            mutuals = l
+        elif l.startswith(("Past:","Summary:","Current:")):
+            summary += (" | " + l) if summary else l
+        elif not location:
+            location = l
+    return {"href": href, "name": name, "degree": degree, "headline": headline,
+            "location": location, "summary": summary, "mutuals": mutuals}
+
+def scrape_current_page():
+    for _ in range(3):
+        js("window.scrollBy(0, 1200)")
+        time.sleep(0.6)
+    js("window.scrollTo(0, document.body.scrollHeight)"); time.sleep(1.2)
+    js("window.scrollTo(0, 0)"); time.sleep(0.4)
+    return js("""(() => {
+        const anchors = Array.from(document.querySelectorAll('a[href*="/in/"]'));
+        const out = [], seen = new Set();
+        for (const a of anchors) {
+            const u = new URL(a.href);
+            if (!u.pathname.match(/^\\/in\\/[^/]+\\/?$/)) continue;
+            if (!a.innerText.includes('•')) continue;
+            const href = u.pathname.replace(/\\/$/, '');
+            if (seen.has(href)) continue;
+            seen.add(href);
+            out.push({href, text: a.innerText});
+        }
+        return out;
+    })()""") or []
+
+for kw in KEYWORDS:
+    url = BASE + "&keywords=" + urllib.parse.quote(kw)
+    goto(url); wait_for_load(timeout=20); time.sleep(2.5)
+    for page in range(1, MAX_PAGES_PER_KEYWORD + 1):
+        if page > 1:
+            goto(url + f"&page={page}"); wait_for_load(timeout=20); time.sleep(2)
+        cards = scrape_current_page()
+        if not cards: break
+        new = 0
+        with OUT.open("a") as f:
+            for c in cards:
+                if c["href"] in seen: continue
+                rec = parse_card(c["text"], c["href"])
+                if not rec: continue
+                rec["keyword"] = kw
+                f.write(json.dumps(rec, ensure_ascii=True) + "\n")
+                seen.add(c["href"]); new += 1
+        print(f"  [{kw}] page {page}: {len(cards)} cards, {new} new")
+        if new == 0 and page > 1: break
+        if len(cards) < 5: break
+    time.sleep(1.0)
+
+print(f"DONE — total unique: {len(seen)}, output: {OUT}")
+```
+
+## Post-processing — scoring for relevance
+
+Raw output is high-recall, low-precision. Run a scoring pass with regex tiers for:
+- **Tier 1 (50pt)** named target companies — exact-match to the specific employers you're hunting
+- **Tier 2 (25pt)** industry vocab — domain-relevant role and function keywords
+- **Tier 3 (20pt)** adjacent-industry vocab — named market participants, related sectors
+- **Geography (15pt)** target-region location strings
+- **Government (10pt)** `minister`, `ambassador`, `consul` when paired with geography (if you're hunting through diplomatic/state channels)
+- **Noise penalty (-15pt)** disqualifying terms relevant to the target's adjacent industries (e.g. for a B2B sourcing search, penalize `tourism`, `hotel`, `fashion`, `media`, `advertising`, `software engineer` — these flood broad networks)
+
+Sort descending. Cut at score ≥ a threshold matched to your tolerance for noise (15 is reasonable when geography alone qualifies).
+
+## What this does NOT give you
+
+- **Email addresses.** Profile anchors don't carry them. Use `voyager_scrape_2026-04-25.py` per-handle for full profile dumps if you need positions/education/etc.
+- **3rd-degree connections of the target.** The `network` parameter is relative to the asker's graph, not the target's. To enumerate the target's full 1st-degree network you'd need to log in as the target.
+- **Hidden profiles.** People who set their connections to private won't appear regardless of the filter.
+
+## Watch out for
+
+- **The URL must use percent-encoded JSON in `connectionOf` and `network`.** `%5B` = `[`, `%5D` = `]`, `%22` = `"`. Bare brackets will silently 400 or redirect to an empty result page.
+- **Profile-card anchor `innerText` includes a trailing `Connect` / `Message` / `Follow` button label.** Filter it out during parsing.
+- **The first profile anchor on a card has the full payload; subsequent same-`/in/`-handle anchors have just the name.** Dedupe by pathname before parsing.
+- **Some profiles render `<Name> \n  • <Degree>` with the bullet on a separate line.** Split on `•` not on `\n` for the first parse.
diff --git a/domain-skills/linkedin/send-connection-request.md b/domain-skills/linkedin/send-connection-request.md
new file mode 100644
index 00000000..66464af8
--- /dev/null
+++ b/domain-skills/linkedin/send-connection-request.md
@@ -0,0 +1,150 @@
+# LinkedIn — Send Connection Request
+
+Send connection requests (with or without note) to a profile at `https://www.linkedin.com/in/<handle>/`.
+
+## Connect button is an `<a>`, not a `<button>`
+
+Critical gotcha: LinkedIn renders the primary Connect action in the profile header as an `<a>` tag styled as a button. Querying `button` only misses it.
+
+```python
+# Works
+link = "document.querySelector('a[aria-label*=\"Invite\"][aria-label*=\"connect\"]')"
+js(f"{link}.click()")
+```
+
+The `aria-label` is `"Invite <FirstName> <LastName> to connect"` (use `*=` substring match to be robust across casing/punctuation).
+
+## Initial modal is in main DOM
+
+After clicking Connect, the first modal — "Add a note to your invitation?" — has two buttons both in the top-level document:
+
+- `button[aria-label="Add a note"]`
+- `button[aria-label="Send without a note"]`
+
+These ARE queryable via JS against the main doc. Poll for `document.querySelector('button[aria-label="Send without a note"]')` to detect when the modal is ready (modal render takes 500–2500ms after click).
+
+## Custom-note dialog is inside an iframe → paywalled on free tier
+
+Clicking "Add a note" navigates to a second dialog rendered inside an iframe at `src="https://www.linkedin.com/preload/"` (747×770). JS scoped to the top document cannot see its buttons. `iframe_target("linkedin.com/preload")` returns `None` in my testing (CDP does not expose this frame through the normal iframe target map).
+
+On a free LinkedIn account, this second dialog is the paywall surface. After a small number of personalized notes per month (observed cap ~5), the iframe renders:
+
+> **"Send unlimited personalized invites with Premium"**
+> "You're out of free custom notes. Bypass the limit with Premium..."
+
+The first "Add a note" modal still appears — the paywall doesn't kick in until you click Add-a-note. So detect the paywall by screenshotting after clicking Add-a-note, or by checking if a textarea is queryable in the top doc afterwards (if not, you've hit the paywall).
+
+## Flow: Send plain connection (no note)
+
+Simplest, always works on free tier. Use this unless you need personalization.
+
+```python
+# 1. Navigate
+goto('https://www.linkedin.com/in/<handle>/')
+time.sleep(5)  # profile header load
+
+# 2. Click Connect (a tag, not button)
+r = js("""(() => {
+  const a = document.querySelector('a[aria-label*="Invite"][aria-label*="connect"]');
+  if (!a) return 'no-connect';
+  a.click();
+  return 'clicked';
+})()""")
+
+# 3. Poll for initial modal (up to 5s)
+for _ in range(10):
+    time.sleep(0.5)
+    if js("!!document.querySelector('button[aria-label=\"Send without a note\"]')"):
+        break
+
+# 4. Click Send without a note
+js("""document.querySelector('button[aria-label="Send without a note"]').click()""")
+time.sleep(2.5)
+
+# 5. Verify — either Pending button present, or Connect link is gone
+pending = js("""({
+  stillInvitable: !!document.querySelector('a[aria-label*="Invite"][aria-label*="connect"]'),
+  pending: !!document.querySelector('button[aria-label*="Pending"]')
+})""")
+```
+
+Signals of success:
+- `stillInvitable: false` (profile header no longer shows Connect)
+- OR `pending: true` (header now shows a Pending button)
+
+## Flow: Send with custom note (free tier, rate-limited)
+
+Use only when you have quota left. Failure mode is the Premium paywall.
+
+```python
+# 1-3: Same as plain-connect up to the initial modal appearing
+
+# 4. Click Add a note
+js("""document.querySelector('button[aria-label="Add a note"]').click()""")
+time.sleep(1.5)
+
+# 5. Check for paywall — if no textarea queryable, you've hit the cap
+ta_exists = js("!!document.querySelector('textarea')")
+if not ta_exists:
+    screenshot('/tmp/paywall.png')
+    # Paywall screenshot will show "Send unlimited personalized invites with Premium"
+    raise RuntimeError("Hit free-tier custom-note quota — no textarea rendered")
+
+# 6. Focus + type
+js("document.querySelector('textarea').focus()")
+type_text(note)  # ≤ 300 chars
+time.sleep(0.8)
+
+# 7. Click Send invitation
+js("""document.querySelector('button[aria-label="Send invitation"], button[aria-label="Send"]').click()""")
+```
+
+## Coordinate-click fallback (when JS can't reach)
+
+If a modal/element is in an iframe or shadow DOM that `js()` can't target, and the element IS visible in a screenshot, use `click(x, y)` with coordinates read from an earlier DOM walk.
+
+Earlier in-doc DOM walk captured the rect:
+
+```python
+# Walk that includes shadow DOMs — catches elements before they migrate to iframes
+r = js("""(() => {
+  function walk(root, out) {
+    root.querySelectorAll('*').forEach(el => {
+      const text = (el.textContent || '').replace(/\\s+/g,' ').trim();
+      if (text === 'Add a note' || text === 'Send without a note') {
+        const rect = el.getBoundingClientRect();
+        if (rect.width > 0 && rect.height > 0) {
+          out.push({
+            tag: el.tagName, text,
+            aria: el.getAttribute('aria-label') || '',
+            rect: {top: Math.round(rect.top), left: Math.round(rect.left), w: Math.round(rect.width), h: Math.round(rect.height)}
+          });
+        }
+      }
+      if (el.shadowRoot) walk(el.shadowRoot, out);
+    });
+  }
+  const results = [];
+  walk(document, results);
+  return results;
+})()""")
+
+# Read rect and click the center
+if r:
+    rect = r[0]['rect']
+    click(rect['left'] + rect['w']//2, rect['top'] + rect['h']//2)
+```
+
+## Rate limit notes
+
+- Custom-note invites: ~5/month on free tier (actual cap varies). Hitting it shows the Premium paywall dialog.
+- Plain "Send without a note" invites: higher cap (hundreds/month, but LinkedIn may throttle if bulk-sent too fast). Space them 2+ seconds apart.
+- LinkedIn also tracks total weekly connection invites separately — ~100/week is the soft cap before "You're approaching the weekly limit" warnings.
+
+## Alternative when Connect isn't shown at all
+
+For very-far-network profiles (4th+ degree, or profiles with "Only mutuals" setting), the Connect link won't appear in the header at all — only Follow and Message (if Open Profile). `a[aria-label*="Invite"][aria-label*="connect"]` returns null. In that case the only options are:
+
+1. Click Follow instead: `button[aria-label^="Follow "]`
+2. Try the sidebar "People similar to X" cards, which may include Connect buttons for 2nd-degree adjacent people
+3. InMail (Premium)