diff --git a/agent-readiness-eval/01-methodology.md b/agent-readiness-eval/01-methodology.md new file mode 100644 index 000000000..093a1a876 --- /dev/null +++ b/agent-readiness-eval/01-methodology.md @@ -0,0 +1,93 @@ +# 01 — Methodology + +## Goal + +Validate that the restructured, task-based SharePoint Embedded (SPE) documentation is +**agent-ready**: a coding agent that a user delegates an SPE question to should be able +to retrieve the answer from the docs alone, answer correctly, and cite the right task +article — without relying on the model's pre-training. + +## Two complementary evaluations + +1. **Structural scoring (deterministic).** A Python scorer grades every task article on + four dimensions (structure, correctness signals, agent-readiness, task framing). + This catches missing metadata, oversized articles, missing navigation, weak titles, + and broken cross-links. See [`02-structural-scoring.md`](02-structural-scoring.md). + +2. **Empirical Q&A (behavioral).** Real questions are answered by **doc-restricted + retrieval agents** that may only read files under `docs/embedded`. The agents + simulate a coding agent doing keyword retrieval: `grep`/`glob` to find a candidate + article, read it, then answer. Outside/world knowledge is prohibited, and the agent + must say "NOT IN DOCS" when the answer genuinely isn't present. See + [`03-qa-round1-21-queries.md`](03-qa-round1-21-queries.md) and + [`04-qa-stress-test-54-queries.md`](04-qa-stress-test-54-queries.md). + +## Why doc-restricted agents + +A general model can answer many SPE questions from pre-training, which would hide gaps +in the docs. Restricting the agent to `docs/embedded` forces every answer to be +*grounded in the documentation*, so a wrong or "NOT IN DOCS" result is a true signal of +a documentation gap, not a model limitation. + +## Agent output contract + +Every Q&A agent returns one structured block per question: + +``` +QID: +ANSWER: +CITED: +FINDABILITY: # how easily keyword search located it +SUFFICIENCY: +GAP: +``` + +`FINDABILITY` and `SUFFICIENCY` separate two distinct failure modes: the content can +exist but be **hard to find** (retrieval problem) or be **findable but incomplete** +(content problem). Both are actionable. + +## Grading scale + +| Score | Meaning | +| --- | --- | +| 3 | Complete + correct, cited the intended task article, easy to find | +| 2 | Correct but partial, hard to find, or cited a legacy/secondary file | +| 1 | Answer missing or materially incomplete (real content gap) | +| 0 | Wrong / hallucinated answer | + +**Negative probes.** Some queries deliberately ask for facts not in the doc set (for +example, SLA, region availability, customer-managed keys). These test *honesty*: an +agent that answers "NOT IN DOCS" scores **3**; one that invents an answer scores **0**. + +## Loop + +1. Build a query bank with persona, question, expected file, and ground truth. +2. Split queries into groups; run one doc-restricted agent per group in parallel. +3. Grade each answer 0–3; record `FINDABILITY`/`SUFFICIENCY`/`GAP`. +4. For every score < 3 that is a real gap, edit the doc to close it. +5. Re-run the structural scorer and link check to confirm no regression. +6. Re-test the fixed queries with a fresh agent. +7. Commit and record results here. + +## Tooling + +- **Retrieval agents:** fast explore agents, scoped by prompt to `docs/embedded` only. +- **Structural scorer:** `score_docs.py` (4 × 25-point rubric). Kept with the session + artifacts; summarized in [`02-structural-scoring.md`](02-structural-scoring.md). +- **Query bank + grades:** stored in a SQLite session database (`qa` for the 21-query + campaign, `qa2` for the 54-query stress test) and exported into the records here. +- **Link integrity:** a small Python pass over every `[..](../x.md)` relative link in + `docs/embedded`, checked against the filesystem. + +## Reproducing + +```powershell +# Structural score (from repo root) +python /score_docs.py + +# Relative-link integrity over docs/embedded +# (walks every ../*.md and ./*.md link and checks it resolves on disk) +``` + +Q&A campaigns are reproduced by re-issuing the prompts in files 03 and 04 to a +retrieval agent restricted to `docs/embedded`, then grading with the scale above. diff --git a/agent-readiness-eval/02-structural-scoring.md b/agent-readiness-eval/02-structural-scoring.md new file mode 100644 index 000000000..3b8cf1301 --- /dev/null +++ b/agent-readiness-eval/02-structural-scoring.md @@ -0,0 +1,72 @@ +# 02 — Structural scoring + +A deterministic Python scorer grades every task article in `docs/embedded` on four +25-point dimensions. It is the fast, repeatable gate that runs before and after every +content change to prevent regressions. + +## Rubric (4 × 25 = 100) + +| Dimension | What it checks (25 pts each) | +| --- | --- | +| **Structure** | Single H1; YAML front matter with `title`/`description`/`ms.date`; heading hierarchy is well-formed; article is right-sized for a context window (not oversized). | +| **Correctness signals** | Has an `**Applies to:**` audience line; uses concrete API/cmdlet/permission names; code fenced with a language; intro paragraph present before the first H2. | +| **Agent-readiness** | Article fits comfortably in a context window; descriptive `description` for retrieval; explicit visible navigation (`## Next steps` and `## Related resources` / `See also`); stable headings usable as anchors. | +| **Task framing** | Verb-first, task-oriented title (e.g., "Create and manage containers"); imperative steps; one primary task per article. Reference and concept pages (`ms.topic` / `task_type: concept`) are exempt from the verb-first rule. | + +### Scorer notes / fixes captured during development + +- The `**Applies to:**` audience line is matched with both bold and plain variants. +- Intro detection must not be defeated by `DOTALL` regex (a bug that was found and fixed). +- Cross-link validity does not require a `../` prefix; same-folder `./x.md` links count. +- Navigation recognizes the variants `Next step`/`Next steps`/`Related`/ + `Related resources`/`See also`. + +## Result + +``` +structure : 25.00 / 25 +correctness : 25.00 / 25 +agent : 25.00 / 25 +task : 25.00 / 25 + TOTAL : 100.00 / 100 + Perfect articles: 43/43 + +=== ARTICLES WITH DEDUCTIONS / ISSUES === + (none) +``` + +The score was re-run after the stress-test fixes (see file 04) and remained +**100.00 / 100, 43/43 perfect**, confirming no regression. + +## Articles graded (43) + +Spread across the task-based information architecture: + +| Section | Count | Examples | +| --- | --- | --- | +| Overview | 3 | `overview.md`, `scenarios-and-use-cases.md`, `whats-new.md` | +| Plan | 7 | `app-tenant-architecture.md`, `choose-app-model.md`, `authentication-permissions.md`, `limits-calling-patterns.md` | +| Build | 16 | `quickstart-vscode.md`, `create-container-type.md`, `manage-files.md`, `open-office-files.md`, `search-containers-files.md`, `agent-experiences.md` | +| Publish | 4 | `prepare-customer-installation.md`, `validate-customer-installation.md` | +| Install & manage (admin) | 10 | `admin-overview.md`, `install-sharepoint-embedded-app.md`, `manage-containers-powershell.md`, `review-audit-events.md` | +| Reference | 6 | `billing-meters.md`, `audit-events.md`, `troubleshooting.md`, `glossary.md` | + +## Link integrity + +A separate pass walks every relative Markdown link (`[..](../x.md)` and `./x.md`) under +`docs/embedded` and resolves it against the filesystem. + +``` +relative md links checked = 297 broken = 0 +``` + +Entry-link integrity was also verified separately: + +- `docs/embedded/overview.md` keeps H1 **"What is SharePoint Embedded?"**, so the + published `/sharepoint/dev/embedded/overview` URL and its `#what-is-sharepoint-embedded` + anchor do not break. +- `docs/index.yml` (the SharePoint-dev landing hub) — both SPE entry cards resolve + (Overview → `/embedded/overview`, quickstart → `/embedded/build/quickstart-vscode`). + A pre-existing broken "Enable SharePoint Embedded" card was repointed to the existing + quickstart. +- `docs/toc.yml` — 46/46 SPE hrefs resolve. diff --git a/agent-readiness-eval/03-qa-round1-21-queries.md b/agent-readiness-eval/03-qa-round1-21-queries.md new file mode 100644 index 000000000..f19a56420 --- /dev/null +++ b/agent-readiness-eval/03-qa-round1-21-queries.md @@ -0,0 +1,70 @@ +# 03 — Q&A campaign: 21 persona queries + +The first empirical campaign tested whether doc-restricted agents could answer realistic +questions from the core SPE personas (new developer, developer, architect, admin, ISV, +compliance, billing, and an undecided "router" user). 21 queries were split into 5 +groups and run by parallel doc-restricted agents, then graded 0–3. + +## Agent prompt (template) + +Each agent received this instruction plus its group's questions: + +> You are evaluating whether a documentation set can answer SharePoint Embedded (SPE) +> questions. You may ONLY read files under `docs/embedded`. Do NOT use outside knowledge. +> Simulate a coding agent doing keyword retrieval: use grep/glob within that folder to +> FIND the answer, then read the matching article. If the docs genuinely do not contain +> the answer, say so honestly — do not invent it. +> +> For EACH question output one block: +> `QID / ANSWER / CITED / FINDABILITY / SUFFICIENCY / GAP`. + +## Queries, expected article, and grades + +| QID | Persona | Question | Intended article | R1 | R2 | +| --- | --- | --- | --- | --- | --- | +| q01 | dev-new | I am brand new to SPE. How do I build my first app? | `build/quickstart-vscode.md` | 3 | 3 | +| q02 | dev | How do I let my app users open and edit Word/Office files from my app? | `build/open-office-files.md` | 3 | 3 | +| q03 | dev | What permission lets my app read content in containers and how do I grant it? | `build/register-application-permissions.md` | 3 | 3 | +| q04 | dev | How do I get notified when a file changes in a container? | `build/respond-to-changes-webhooks.md` | 3 | 3 | +| q05 | dev | How do I create a container type and what is its relationship to my app? | `build/create-container-type.md` | 3 | 3 | +| q06 | dev | How do I search across files in my containers? | `build/search-containers-files.md` | 3 | 3 | +| q07 | dev | How do I archive an inactive container and reactivate it later? | `build/archive-restore-containers.md` | 3 | 3 | +| q08 | dev | How do I add custom metadata columns to containers and query them? | `build/container-metadata.md` | 2 | 3 | +| q09 | architect | Single-tenant vs multitenant: which app model should I choose? | `plan/choose-app-model.md` | 3 | 3 | +| q10 | architect | Who pays for storage, me or my customer? What billing models exist? | `plan/choose-billing-model.md` | 3 | 3 | +| q11 | architect | What size and throughput limits should I design around? | `plan/limits-calling-patterns.md` | 2 | 3 | +| q12 | admin | As a tenant admin, how do I install a vendor SPE app in my tenant? | `admin/install-sharepoint-embedded-app.md` | 3 | 3 | +| q13 | admin | How do I set up billing for an SPE app in my tenant? | `admin/setup-billing-m365-admin-center.md` | 3 | 3 | +| q14 | admin | How do I manage SPE containers with PowerShell? | `admin/manage-containers-powershell.md` | 3 | 3 | +| q15 | admin | What admin role do I need to manage SharePoint Embedded? | `admin/admin-overview.md` | 2 | 3 | +| q16 | compliance | How do I find audit events for SPE container type activity? | `admin/review-audit-events.md` | 3 | 3 | +| q17 | isv | I built an app. How do I prepare it for a customer to install? | `publish/prepare-customer-installation.md` | 3 | 3 | +| q18 | billing | What does SPE charge for? What are the billing meters? | `reference/billing-meters.md` | 3 | 3 | +| q19 | compliance | How do I apply a sensitivity label or block download on a container? | `admin/apply-security-compliance-controls.md` | 3 | 3 | +| q20 | dev | I get access denied / SubscriptionNotRegistered. How do I fix it? | `reference/troubleshooting.md` | 3 | 3 | +| q21 | router | I do not know if I am a developer or admin. Where do I start? | `overview.md` | 2 | 3 | + +## Round 1 result + +- **21/21 answered correctly**, overall quality ≈ **93.7%**. +- Four weaknesses surfaced (all scored 2, "correct but partial / mis-cited"): + +| # | QID(s) | Weakness | Decision | +| --- | --- | --- | --- | +| A | several | Legacy folders (`development/`, `administration/`, `getting-started/`, `compliance/`) compete in keyword search and were sometimes cited instead of the new task article. | **Benign** — answers were still correct; legacy files are intentionally retained as deep-dive sources. Not deleted (out of scope, risky). | +| B | q21 | Router page lacked an explicit developer-vs-admin role identifier. | **Fixed** — added a "Not sure which path?" tip that splits on "do you write code, or manage a tenant?" | +| C | q08 | OWSTEXT custom-property search syntax lived only in the search article; the metadata article didn't cross-link to it. | **Fixed** — added a cross-link from `container-metadata.md` to `search-containers-files.md`. | +| D | q11 | The word "throughput" was absent and the limit-increase process was undocumented. | **Fixed** — added throughput/resource-unit wording and limit-increase guidance to `limits-calling-patterns.md`. | + +## Fixes (committed) + +The B/C/D fixes were committed as `1288ec1c7`: + +- `docs/embedded/overview.md` — dev-vs-admin tip box before the routing table. +- `docs/embedded/build/container-metadata.md` — cross-link to free-text/OWSTEXT search. +- `docs/embedded/plan/limits-calling-patterns.md` — throughput + limit-increase guidance. + +## Round 2 result + +Re-test of the four weak queries (q08, q11, q15, q21) confirmed every gap closed: +**21/21 full score, 100%.** diff --git a/agent-readiness-eval/04-qa-stress-test-54-queries.md b/agent-readiness-eval/04-qa-stress-test-54-queries.md new file mode 100644 index 000000000..4484db27a --- /dev/null +++ b/agent-readiness-eval/04-qa-stress-test-54-queries.md @@ -0,0 +1,648 @@ +# 04 — Stress test: 54 edge-case queries + +The stress test pushed the docs with harder, more specific questions than the persona +campaign: edge cases, exact limits, API nuances, permission distinctions, and **four +deliberate "negative probes"** (facts not in the doc set) to test agent honesty. + +- **54 queries** in 6 groups of 9. +- One doc-restricted retrieval agent per group, run in parallel. +- Graded 0–3 (see [`01-methodology.md`](01-methodology.md) for the scale). + +## Shared agent prompt + +Every group agent received this prompt, with its 9 questions appended: + +``` +You are evaluating whether a documentation set can answer SharePoint Embedded (SPE) +questions. You may ONLY read files under this folder: +C:\...\docs\embedded +Do NOT use any outside/world knowledge. Simulate a coding agent doing keyword +retrieval: use grep/glob within that folder to FIND the answer, then read the matching +article. If the docs genuinely do not contain the answer, say so honestly — do not +invent it. + +For EACH question below, output one block exactly in this format: +QID: +ANSWER: +CITED: +FINDABILITY: +SUFFICIENCY: +GAP: + +Output only the blocks, nothing else. +``` + +--- + +## Group 1 — security / compliance (q22–q30) + +**Questions** + +- q22 Do sensitivity labels applied to a container get inherited by files inside it? +- q23 How do I run eDiscovery against SharePoint Embedded content? +- q24 Can I apply DLP policies to containers? +- q25 How do I set a retention policy on container content? +- q26 What conditional access policy options exist for a container? +- q27 What license do I need to block downloads from a container? +- q28 Which audit fields identify the specific SPE container in an event? +- q29 Does SharePoint Embedded support customer-managed encryption keys? *(negative probe)* +- q30 In which Microsoft 365 geographies/regions is SPE available? *(negative probe)* + +**Raw agent output** + +``` +QID: q22 +ANSWER: NOT IN DOCS +CITED: NONE +FINDABILITY: HARD +SUFFICIENCY: MISSING +GAP: Docs discuss applying sensitivity labels to containers (Set-SPOContainer) but do +not specify whether labels are inherited by files within the container. + +QID: q23 +ANSWER: Yes. Search all SPE containers by selecting "All SharePoint sites" or limit to +specific containers by providing container URLs in the SharePoint sites workload. +CITED: admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q24 +ANSWER: Yes. Apply DLP broadly to all SharePoint sites and SPE containers, or +selectively to specific containers by specifying container URLs. +CITED: admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q25 +ANSWER: Yes, through Microsoft Purview. "All SharePoint sites" applies to all +containers; you can also target specific containers by URL. +CITED: admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md, plan/security-compliance-governance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q26 +ANSWER: Three options: AllowFullAccess, AllowLimitedAccess (web-only), and BlockAccess. +CITED: admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md, plan/security-compliance-governance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q27 +ANSWER: A SharePoint Advanced Management (SAM) license is required to enforce block +download on containers. +CITED: admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q28 +ANSWER: ContainerInstanceId (the specific container instance) and ContainerTypeId. +CITED: admin/review-audit-events.md, admin/apply-security-compliance-controls.md, compliance/security-and-compliance.md +FINDABILITY: EASY +SUFFICIENCY: COMPLETE +GAP: NONE + +QID: q29 +ANSWER: NOT IN DOCS +CITED: NONE +FINDABILITY: NONE +SUFFICIENCY: MISSING +GAP: No documentation on customer-managed encryption keys (BYOK). + +QID: q30 +ANSWER: NOT IN DOCS +CITED: NONE +FINDABILITY: HARD +SUFFICIENCY: MISSING +GAP: Docs mention a region must be specified at container-type creation but do not list +supported M365 geographies. +``` + +**Grading:** q23–q28 → 3. q29, q30 → 3 (honest negative probes). **q22 → 1 (real gap).** + +--- + +## Group 2 — billing / limits / ownership (q31–q39) + +**Questions** + +- q31 How is a single Microsoft Graph API call counted for billing? +- q32 Are downloads through Office clients charged as egress? +- q33 Are admin actions in the SharePoint admin center billed as transactions? +- q34 How many container types can one app own? +- q35 How many container types can a tenant create? +- q36 What is the maximum file size and version count? +- q37 How many owners can a SharePoint Embedded app have in the admin center? +- q38 Can I change the billing type after creating an app in the admin center? +- q39 How long are deleted containers retained before permanent purge? + +**Raw agent output** + +``` +QID: q31 +ANSWER: Each Graph call made explicitly by the app = one transaction. Internal service +calls (eDiscovery, admin actions in SPAC or SPO PowerShell) are not charged as app +transactions. +CITED: reference/billing-meters.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q32 +ANSWER: No. Downloads to the customer's Office Desktop client or WAC are not charged as +egress. Only downloads to other client devices are charged. +CITED: administration/billing/meters.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q33 +ANSWER: No. Admin actions in SPAC are nonchargeable. +CITED: administration/billing/meters.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q34 +ANSWER: One. SPE mandates a 1:1 relationship between owning application and container type. +CITED: development/app-architecture.md, development/limits-calling.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q35 +ANSWER: Up to 25 container types by default ... 5 per developer tenant ... +CITED: getting-started/containertypes.md, development/limits-calling.md +FINDABILITY: EASY SUFFICIENCY: PARTIAL +GAP: "developer tenant" vs "partner tenant" distinction and trial-vs-standard unclear. + +QID: q36 +ANSWER: Max file size 250 GB; max 500 versions per file. +CITED: development/limits-calling.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q37 +ANSWER: Up to three owners. +CITED: administration/consuming-tenant-admin/SPE_app_installation_on_SPAC.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q38 +ANSWER: No. Billing type is permanent and cannot be changed after app creation. +CITED: administration/consuming-tenant-admin/SPE_app_installation_on_SPAC.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q39 +ANSWER: 93 days unless a retention policy changes the outcome; restorable within that window. +CITED: admin/manage-containers-sharepoint-admin-center.md, administration/consuming-tenant-admin/ctapowershell.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE +``` + +**Grading:** q31–q34, q36–q39 → 3. **q35 → 2 (partial — trial vs standard / 5 vs 25 / +developer vs partner tenant distinction unclear).** + +--- + +## Group 3 — auth / permissions / registration (q40–q48) + +**Questions** + +- q40 Difference between `Container.Selected` and `FileStorageContainer.Selected`? +- q41 Is the container type registration API a Microsoft Graph API? +- q42 What token type does the registration API require? +- q43 Which registration call wins if I call it multiple times? +- q44 What permission lets my app manage container membership/permissions? +- q45 Why can't admin consent complete for `Container.Selected`? +- q46 Can I use SPE app-only without any signed-in user? +- q47 What is the owning application relationship to a container type? +- q48 Does my app need a certificate for authentication? + +**Raw agent output** + +``` +QID: q40 +ANSWER: Container.Selected is a SharePoint permission for registration APIs on a +consuming tenant; FileStorageContainer.Selected is a Graph permission for accessing +containers (delegated or app-only). +CITED: build/configure-authentication-authorization.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q41 +ANSWER: No. The registration API is a SharePoint REST API (v2), not a Graph API. +CITED: getting-started/register-api-documentation.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q42 +ANSWER: An app-only token via client credentials grant with a certificate, for the +SharePoint resource. +CITED: getting-started/register-api-documentation.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q43 +ANSWER: The last successful registration call determines the settings used in the +consuming tenant. +CITED: getting-started/register-api-documentation.md, plan/authentication-permissions.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q44 +ANSWER: ManagePermissions (add/remove/update members in container roles). +CITED: getting-started/register-api-documentation.md, development/auth.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q45 +ANSWER: Container.Selected is hidden in the Azure portal, so consent can't complete via +the Enterprise apps pane; use the constructed admin-consent URL. +CITED: reference/troubleshooting.md, development/auth.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q46 +ANSWER: Yes. App-only access works without a signed-in user; needs +FileStorageContainer.Selected application permission with admin consent. +CITED: development/auth.md, plan/authentication-permissions.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q47 +ANSWER: 1:1. Each container type has one owning application with full default access to +its containers and the right to invoke the registration API. +CITED: getting-started/containertypes.md, getting-started/register-api-documentation.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q48 +ANSWER: Yes. The registration API requires a certificate via the client credentials grant. +CITED: getting-started/register-api-documentation.md, build/register-application-permissions.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE +``` + +**Grading:** all q40–q48 → 3. + +--- + +## Group 4 — files / Office / preview / search / webhooks (q49–q57) + +**Questions** + +- q49 How do I upload a large file to a container? +- q50 How do I open an Office file in the desktop client specifically? +- q51 How do I generate a preview/thumbnail link for a file? +- q52 Is container search available in the v1.0 Graph endpoint? +- q53 How long does a webhook subscription last before it expires? +- q54 What content type must my webhook validationToken response use? +- q55 How do I restore a soft-deleted container? +- q56 How do I move data from Azure Blob Storage into SPE? +- q57 How do I search using a custom metadata property? + +**Raw agent output** + +``` +QID: q49 +ANSWER: For larger files use an upload session (LargeFileUploadTask, 320 KB chunks). +CITED: build/manage-files.md, build/migrate-azure-blob-storage.md +FINDABILITY: EASY SUFFICIENCY: PARTIAL +GAP: No explicit size threshold for what counts as a "large" file. + +QID: q50 +ANSWER: Office URI schemes: ms-word:ofe|u|{webUrl}/{file} (and ms-excel, ms-powerpoint). +CITED: build/open-office-files.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q51 +ANSWER: POST .../drives/{driveId}/items/{itemId}/preview; use getUrl from the response. +CITED: build/preview-files.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q52 +ANSWER: No. Search requires /beta/search/query and is preview, delegated-only. For v1.0 +use OData $filter on drive items. +CITED: build/search-containers-files.md +FINDABILITY: HARD SUFFICIENCY: PARTIAL +GAP: Docs show the beta endpoint but don't explicitly state v1.0 lacks search. + +QID: q53 +ANSWER: Max lifetime 4,230 minutes before expiration. +CITED: build/respond-to-changes-webhooks.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q54 +ANSWER: Respond text/plain with the validationToken value as the body. +CITED: build/respond-to-changes-webhooks.md, development/tutorials/using-webhooks.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q55 +ANSWER: Restore-SPODeletedContainer -Identity ; restorable within 93 days; +reactivation instant in first 7 days, up to 24h after. +CITED: administration/consuming-tenant-admin/ctapowershell.md, build/create-manage-containers.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q56 +ANSWER: Download blobs to a stream, create an SPE upload session, upload with +LargeFileUploadTask (320 KB chunks), set conflictBehavior. +CITED: build/migrate-azure-blob-storage.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE + +QID: q57 +ANSWER: Append OWSTEXT to the custom property name in the search query (with +ContainerTypeId); OData $filter for single-container queries. +CITED: build/search-containers-files.md, build/container-metadata.md +FINDABILITY: EASY SUFFICIENCY: PARTIAL +GAP: Less detail on when to use search vs OData filtering. +``` + +**Grading:** q50, q51, q53–q56 → 3. **q49 → 2** (no size threshold). **q52 → 2** +(doesn't state v1.0 lacks search). **q57 → 2** (search-vs-OData guidance thin). + +--- + +## Group 5 — admin / install / onboarding (q58–q66) + +**Questions** + +- q58 Admin consent URL format to install a vendor app? +- q59 Where in the M365 admin center do I configure SPE billing? +- q60 Which PowerShell cmdlet sets a sensitivity label on a container? +- q61 How do I connect PowerShell to manage SPE? +- q62 What happens to users if billing is invalid or SPE is turned off? +- q63 Which permissions must a customer consent to for an ISV app to register and use containers? +- q64 How do I validate that a customer installed my app correctly? +- q65 What does a SharePoint Embedded Administrator role NOT have access to? +- q66 How do I monitor SPE cost in Azure? + +**Raw agent output** + +``` +QID: q58 +ANSWER: https://login.microsoftonline.com//adminconsent?client_id= +CITED: admin/grant-admin-consent-permissions.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q59 +ANSWER: Setup > Files and Content > Automate Content with Microsoft Syntex > Go to +Syntex settings > Apps > SharePoint Embedded. +CITED: admin/setup-billing-m365-admin-center.md +FINDABILITY: EASY SUFFICIENCY: COMPLETE GAP: NONE + +QID: q60 +ANSWER: Set-SPOContainer -Identity -SensitivityLabel