Skip to content

Add NetworkStats parser#655

Open
0x4552494b wants to merge 2 commits into
abrignoni:mainfrom
0x4552494b:main
Open

Add NetworkStats parser#655
0x4552494b wants to merge 2 commits into
abrignoni:mainfrom
0x4552494b:main

Conversation

@0x4552494b
Copy link
Copy Markdown

This parser adds support for artifacts from the Android NetworkStats API located at /data/system/netstats/.

@0x4552494b
Copy link
Copy Markdown
Author

Moved artifact icon into the artifacts_v2 header. The linter should pass now.

@stark4n6
Copy link
Copy Markdown
Collaborator

@0x4552494b do you know of any public images that have test data? I haven't been able to find any just yet

@0x4552494b
Copy link
Copy Markdown
Author

Hi @stark4n6,

BinaryHicks Android 11 Image contains some data that can be parsed.

Unfortunately, I'm afraid the script can't comply with the new policy that all new artifacts must be LAVA ready. I can't think of any way to squeeze the data into a 2-dimensional view without omiting relevant info.

Here is a sample of the current output:

screenshot1

@abrignoni
Copy link
Copy Markdown
Owner

abrignoni commented May 26, 2026 via email

@JamesHabben
Copy link
Copy Markdown
Contributor

JamesHabben commented May 26, 2026

Agree with @abrignoni on the split. The identity artifact could even loop through the buckets to sum up things like total transfer value of each identity or count of transfers.

Edit: also it looks like you may be combining sessions for shared user id? If you split those into distinct records, this can allow sorting and filtering in lava, and the timestamps can get the timezone handling too.

@0x4552494b
Copy link
Copy Markdown
Author

Thank you guys for the suggestions. I currently see two problems with the splitting approach.

  1. Network identities don't have a unique identifier. So to reference an identity from the "data buckets view" we'd have to introduce a new index that doesn't exist in the original data.

    and more importantly

  2. In theory the file structure supports entries with multiple NetworkIdentities (the Android sources refer to it as NetworkIdentitySet). An entry could basically look like this:

.
├── NetworkIdentity
├── NetworkIdentity
├── Bucket
│   ├── Netstats
│   ├── Netstats
│   └── Netstats
├── Bucket
│   ├── Netstats
│   └── Netstats
└── ...

I didn't observe this during testing but we can't rule out the possibility that there will ever be an entry with more than one NetworkIdentity. In that case we'd need a way to link Buckets to multiple NetworkIdentities as well as convey that these NetworkIdentities belong together.

@JamesHabben
Copy link
Copy Markdown
Contributor

I think there is enough structure in the source data to support relational output.

The parser is already treating each parsed entry as a grouped structure containing:

  • one or more NetworkIdentity records
  • one or more bucket records

The source format may not expose a native unique ID for the identity set, but the grouping itself appears explicit from parse context.

For relational output, I think we could preserve that relationship by generating a synthetic grouping identifier during parsing. My preference would be to use the file offset of the entry structure rather than an arbitrary incrementing entry index.

For example:

  • entry_offset
    or
  • identity_set_offset

Then both the identity table and bucket table could carry that value as the relational key.

I prefer offset over entry index because:

  • it is derived directly from the source structure
  • it provides forensic provenance/debug traceability
  • it avoids introducing a purely parser-generated ordinal
  • the meaning of the value is transparent to downstream consumers

An incrementing entry ID would also technically work, but offset feels cleaner and more defensible from a forensic/data lineage perspective.

There is still opportunity for aggregation/summarization as well. The offset/set identifier would simply preserve the original grouping relationship while still allowing aggregation by UID, identity fields, durations, totals, etc.

@stark4n6
Copy link
Copy Markdown
Collaborator

@0x4552494b just fyi, tried on Josh's Android 11 image, got some errors for NoneTypes, so might have to add some extra file checks
image

@0x4552494b
Copy link
Copy Markdown
Author

@stark4n6 Weird... I could have sworn this didn't happen when I originally wrote the script.

The problem is in L264, where I’m returning dummy data because otherwise the artifact icon wouldn’t be registered. Once the script is adapted to be LAVA-compatible and return actual data, the issue will resolve automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants