Feature Request: Allow deeper customization of locals hiding behavior in ExceptionDictTransformer by supporting callbacks for `show_locals`

[sirosen]

I was exploring how the various flags for suppressing / showing / controlling locals behave and had an idea.

Let's suppose I know that there's a certain class of data, trivially identifiable, that my application handles and which I don't want to log. e.g., tokens of some variety.

def is_token(value: str) -> bool:
    return len(value) == 42 and value.startswith(("SPAM:", "EGGS:"))

How can I setup an ExceptionDictTransformer to hide all locals which pass is_token()? I don't think it's easily possible today. But this seems like a valid kind of thing to want to do.

In terms of the current API, I see the options like this:

• suppress is a module-level filter
• locals_hide_dunder and locals_hide_sunder are variable-level filters
• locals_max_string and locals_max_length are variable formatting parameters

Can we phrase these three categories of things as user-controllable hooks or callbacks, where the current behaviors are just "defaults"?

Here's one example of how this might be done, to at least show that it's possible (but results in a bad API):

1. suppress is the module filter, and could be made into suppress: bool | Callable[[str], bool] -- give it a module filename and give back a bool
2. locals_hide_dunder and locals_hide_sunder could be augmented with locals_hide_filter: None | Callable[[bool, bool, <...other args...>], bool] -- where those first two bools are the locals_hide_sunder and locals_hide_dunder values
3. locals_format_repr: None | Callable[int, int, <...other args...>] similarly could take the max string and container lengths

But it shows that it's possible.
Maybe ExceptionDictTransformer could be made to hold composable pieces which would be reusable so that users could build myself such a transformer themselves? Or it could inherit (😬) from a more generic base.

I'm curious if this idea is worth discussing and pursuing. I'd be more than happy to work on it if so.

I didn't see a prior issue for this. Sorry if I missed one. And, of course, thanks so much for structlog. ❤️

[hynek]

let’s pull in @sscherfke since it’s mostly his turf. I think all of this is essentially coming from Rich?

---

Couldn’t you wrap the whole thing and then filter the dict it’s returning?

[sscherfke]

Yes, this code comes essentially from Rich (with permission). I tried to keep it as close to the original as possible in order to make it as easy as possible to "backport" changes from Rich to Structlog. (And I would like to keep it that way if possible :-))

Since all this stuff is quite composable, you might be able to subclass the transformer and use that wit hthe ExceptionRenderer. Would that work for you?

[hynek]

yeah that's what I meant: it would be a bummer to make our lives more complicated by adding a feature that hopefully is possible in a different way.

might be worth talking to the Rich crowd and we just steal it?

[sirosen]

Couldn’t you wrap the whole thing and then filter the dict it’s returning?

That works, but it feels wrong to me because I'd end up paying a cost on every dict being logged, rather than only paying it on tracebacks.

So it would be something like

def filter_event(event_dict):
    if "exception" not in event_dict:
        return event_dict
    return _real_filter(event_dict)  # do the actual stuff

It's 100% workable though.

Since all this stuff is quite composable, you might be able to subclass the transformer and use that wit hthe ExceptionRenderer. Would that work for you?

I'm looking now, and I have to say that this feels right to me. Sorry, I think I over-complicated my needs! 😅
I'm not sure, but I think I was looking at extract() and trying to figure out how to customize that.

Because the renderer gets invoked when there's already an exception in hand, being formatted, the simple subclassing is

class RedactingExceptionRenderer(ExceptionRenderer):
    def __call__(self, exc_info: ExcInfo) -> Any:
        result = super().__call__(exc_info)
        return _filter_func(result)

and that's sufficient to satisfy my basic desire for "the filter function can be expensive, but it only runs when handling exceptions, which should be rare".

---

Is it okay to turn this into a docs issue, and I can test out some worked example and contribute it in docs/recipes.md with a cross-link from docs/exceptions.md?

I think such a doc would also take care of #715.