Make detection of binary files more robust

[martenlienen]

The previous heuristic would try to embed non-image files as text, which would fail, for example, for small PDFs. For those files, which contain non-UTF-8 bytes, json-serialize would raise an error, because it would be an invalid UTF-8 string.

The new function has more robust MIME type detection based on Emacs' mailcap and ensures that only files with valid multi-byte content will not be base64 encoded.

Checklist

• I agree to communicate (PR description and comments) with the author myself (not AI-generated).
• I've reviewed all code in PR myself and will vouch for its quality.
• I've read and followed the Contributing guidelines.
• I've filed a feature request/discussion for a new feature.
• I've added tests where applicable.
• I've updated documentation where necessary.
• I've run M-x checkdoc and M-x byte-compile-file.

[xenodium]

Thanks for the PR and making this area more robust.

Had a quick look (apologies under tight time constraints at the moment due to #500).

Left a comment. Mind looking into it?

[xenodium]

Won't small non-image binary files (PDFs, .docx, etc.) that pass the size check will now be embedded as "resource" blocks with base64-encoded content in the text field? Before this PR, json-serialize would error on non-UTF-8 bytes and the condition-case would catch it, falling back to a plain text mention, somewhat accidentally the right behavior.

Maybe resource embed branch in the caller probably needs a (not (map-elt file :base64-p)) guard so binary non-image files fall through to resource_link instead?

[martenlienen]

I worked on this, because I @-mentioned a small PDF file that was then embedded in the message but not base64 encoded. agent-shell then refused any further cooperation, when sending the request failed on json-serialize in acp.

[martenlienen]

This embeds small non-text files as binary resources now as documented here. And I think at least claude would also support embedded PDFs, if these API docs also apply to ACP.

Regarding and agent-shell-text-file-capabilities above, why is this necessary to work with embedded resources? From the naming and the pattern in the image case, I would think that supports-embedded-context would be the right test to check if the agent can deal with embedded context.

[xenodium]

Regarding and agent-shell-text-file-capabilities above, why is this necessary to work with embedded resources?

IIRC, this was somewhat relevant to running agents in a container and preventing access to files outside the container, but this is a moot point since we're crafting a prompt so the user intent is indeed to send the file. We can likely remove it.

[xenodium]

Sorry for the delay. Just saw you made further changes. Thank you!

[xenodium]

Won't we get false positives here for a lot of text cases?

file                           mailcap mime              classified as
/tmp/pr515-demo.el             application/emacs-lisp    BINARY (base64)
/tmp/pr515-demo.sh             application/x-sh          BINARY (base64)
/tmp/pr515-demo.py             nil                       text
/tmp/pr515-demo.json           nil                       text
/tmp/pr515-demo.md             nil                       text

How about we consider peeking at file to determine if binary and use mailcap for mime-type? Maybe something like:

(defun agent-shell--file-binary-p (file-path)
  "Return non-nil if FILE-PATH appears to be binary.
Sniffs for a NUL byte in the first 4 KB — the same fast-path
heuristic used by file(1), git, and grep -I.  Reads at most 4 KB
regardless of file size."
  (with-temp-buffer
    (set-buffer-multibyte nil)
    (insert-file-contents-literally file-path nil 0 4096)
    (goto-char (point-min))
    (search-forward "\0" nil t)))