Request for Entropy-based metric

[max-Hawkins]

After reading about Neurobench and some analogous tools in the 'conventional' computing space, I think it'd be useful to have an entropy based performance metric in NeuroBench. For the idea behind this, here is a 4 page preprint and here is a recent presentation. Summarizing the ideas though, I think such a metric would allow for direct comparisons between neuromorphic and conventional computing systems, accommodate noisy hardware, and account for both structural/connection and activation sparsity in a principled way. @jasonlyik Please let me know what you think. I'm not really a python user, but I can help implement some library in C/C++ if need be for this project.

[jasonlyik]

Hi Max, thanks for your interest in NeuroBench!

The proposal sounds interesting, @ben9809 and I have taken a look at the preprint. Can you tell a little further about how this metric would be implemented in the neural network setting? Is the idea that we will take tensor input and outputs and measure the before/after transformation entropy?

[max-Hawkins]

Thanks for making NeuroBench.

Yeah it'd be something like that. As a simple first-pass, it could be just estimating the entropy of each variable before/after each operation, and summing the estimated entropies across variables to calculate a per-model computation 'cost'. For example, relu(a*x + b) = c could be broken down into the following:

• Entropy of a * x = E_1
• Entropy of (a*x + b) = E_2
• Entropy of c = E_3

Then the total computation cost under an information theory interpretation would be E_1 + E_2 + E_3. We're breaking the application into atomic units of operations, modeling the 'work' done by each operation by the mutual information of the inputs/ouputs pairing, but are estimating this hard-to-calculate value by using the output variable's entropy. In a way, this is a runtime metric that has an upper bound of the bit-width of the output operands. Plug-in estimators of entropy are also reasonably accurate for small (e.g. FP8 or FP4) data types, making them especially useful for AI workloads.

I think an information theoretic performance framework may seem more complicated than necessary (at least within each computing regime individually - like conventional or neuromorphic), but I don't see alternative ways of making hardware comparisons across these boundaries without it. You can always use some application-specific metric like time or energy to solution, but those aren't hardware-centric. I think info theory gives a good, implementation-agnostic, hardware metric.

What do you all think?

[jasonlyik]

This is pretty interesting, I'm curious to see what we'll find for real tasks!

I think a good starting point is the primate_reaching (neural decoding) benchmark: https://github.com/NeuroBench/neurobench/tree/main/examples/primate_reaching

This seems reasonable as a starting point since

• The input data is a binary tensor
• We have small trained neural networks: a 2-layer ANN and 2-layer SNN

I'd be super interested if you can quantify the info theory costs between these two, what do you think?

As for comparing different computing regimes, my reaction is that ideally, we have both real systems available, e.g., an Arduino vs. neuromorphic edge device, and we could directly measure the time and power that both consume while doing the same prediction task. That's the spirit of the NeuroBench systems track.

But, neuromorphic systems are not very mature nor available, so this type of approach is not usually feasible currently. So I think a principled framework comparing at the algorithms level from a theory perspective could be very interesting.

[max-Hawkins]

OK sounds good. I'm going to finish some preliminary toolchain work in Julia first and then will look into python.