Custom Profile not Recognised

[cmdaltr]

Describe the bug
I have created my own symbol table for Ubuntu 18.04 (5.4.0-42-generic) and imported it using the commands listed below but I get the following error in volatility3 (screenshot also attached):

Volatility 3 Framework 1.0.1
Progress:  100.00		Stacking attempts finished                 
Unsatisfied requirement plugins.PsList.primary: Memory layer for the kernel
Unsatisfied requirement plugins.PsList.vmlinux: Linux kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
	You have the correct symbol file for the requirement
	The symbol file is under the correct directory or zip file
	The symbol file is named appropriately or contains the correct banner


A translation layer requirement was not fulfilled.  Please verify that:
	A file was provided to create this layer (by -f, --single-location or by config)
	The file exists and is readable
	The necessary symbols are present and identified by volatility3
Unable to validate the plugin requirements: ['plugins.PsList.primary', 'plugins.PsList.vmlinux']

Context
Volatility Version: Volatility 3 Framework 1.0.1
Operating System: Ubuntu 20.04 (SIFT Workstation)
Python Version: 3.8.5
Suspected Operating System: Ubuntu 18.04
Command: sudo /usr/local/lib/python3.8/dist-packages/volatility3/vol.py -f /Ubuntu18.04.mem linux.pslist.PsList

To Reproduce
Steps to reproduce the behavior:

wget http://ddebs.ubuntu.com/ubuntu/pool/main/l/linux/linux-image-unsigned-5.4.0-42-generic-dbgsym_5.4.0-42.46_amd64.ddeb
sudo dpkg -x linux-image-unsigned-5.4.0-42-generic-dbgsym_5.4.0-42.46_amd64.ddeb /tmp/
sudo ./dwarf2json linux --elf /tmp/usr/lib/debug/boot/vmlinux-5.4.0-42-generic > /usr/local/lib/python3.8/dist-packages/volatility3/volatility3/symbols/linux/vmlinux-5.4.0-42-generic.json
sudo /usr/local/lib/python3.8/dist-packages/volatility3/vol.py -f /Ubuntu18.04.mem linux.pslist.PsList
sudo chmod 755 -R /usr/local/lib/python3.8/dist-packages/volatility3/volatility3/symbols/linux/

Expected behavior
I expect volatility3 to be able to read the custom symbol table I have created and as a result read the memory image I have captured

[ikelos]

Hiya, could you please check that the symbol file you made is listed in the output of the isfinfo plugin, and if so and there's a banner there, that the banner matches the output of the banners plugin run against the image? That will help us diagnose why volatility isn't happy with it... 5:)

[cmdaltr]

The symbol file I made does not show in the output for isfinfo
I'm guessing that means it is not valid? Based on the instructions above - what have I missed?
FYI this is the banners output for the file:

Offset	Banner

0x10b6001a0	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)
0x10c18dd94	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)
0x10c6c8468	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)
0x12a96655f	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)
0x13fec98d0	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)

[ikelos]

If the symbol file isn't even listed in the output for isfinfo then probably it means the file hasn't ended up in the right place, although it certainly looks like it's in the right place from your commands? It may also be that the JSON file doesn't contain a banner (you can check the JSON to see if there's a constant_data field under the banner symbol, which should be a base64 encoding of the kernel banner). If that isn't there, then volatility might think the file isn't valid and so not bother listing it. You can also clear out your cache (using --clear-cache) and then run volatility with -vvvvv to see if it has any problems with it?

[cmdaltr]

The JSON file contains a banner:

      "address": 18446744071595622816,
      "constant_data": "TGludXggdmVyc2lvbiA1LjQuMC00Mi1nZW5lcmljIChidWlsZGRAbGd3MDEtYW1kNjQtMDM4KSAoZ2NjIHZlcnNpb24gOS4zLjAgKFVidW50dSA5LjMuMC0xMHVidW50dTIpKSAjNDYtVWJ1bnR1IFNNUCBGcmkgSnVsIDEwIDAwOjI0OjAyIFVUQyAyMDIwIChVYnVudHUgNS40LjAtNDIuNDYtZ2VuZXJpYyA1LjQuNDQpCgA="
    },
    "linux_proc_banner": {

[cmdaltr]

And this is the output from using -vvvv flag:

Volatility 3 Framework 1.0.1
INFO     volatility3.cli: Volatility plugins path: ['/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/plugins', '/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/symbols', '/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a linux category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.primary
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.vmlinux
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
INFO     volatility3.framework.automagic: Running automagic: LinuxBannerCache
INFO     volatility3.framework.automagic.symbol_cache: Building linux caches...
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Stacked LimeLayer using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LinuxIntelStacker
DEBUG    volatility3.framework.automagic.linux: No suitable linux banner could be matched
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: LimeLayer
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: FileLayer
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['LimeLayer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: LinuxSymbolFinder
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.primary
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.vmlinux

Unsatisfied requirement plugins.PsList.primary: Memory layer for the kernel
Unsatisfied requirement plugins.PsList.vmlinux: Linux kernel symbols

A symbol table requirement was not fulfilled.  Please verify that:
	You have the correct symbol file for the requirement
	The symbol file is under the correct directory or zip file
	The symbol file is named appropriately or contains the correct banner


A translation layer requirement was not fulfilled.  Please verify that:
	A file was provided to create this layer (by -f, --single-location or by config)
	The file exists and is readable
	The necessary symbols are present and identified by volatility3
Unable to validate the plugin requirements: ['plugins.PsList.primary', 'plugins.PsList.vmlinux']

[ikelos]

Hmmm, no issues with the linux caches, and the constant_data looks to be present. It's very strange that isfinfo isn't listing it? Could you please try:

1. Put the JSON file in /some/directory/somewhere/linux/file.json (the linux bit is important)
2. Run vol.py -vvvvvv -s /some/directory/somewhere isfinfo 2>&1 > output.txt
3. Attach the output.txt file to this bug.

We need to figure out why it's not got that file listed, and isn't classing it as a file it can't process in the verbose logging. 5:S

[cmdaltr]

output.txt

[ikelos]

Hmmmm, so it looks like it was finding the mac.zip file in /home/sansforensics/, which suggests that you could create a linux directory at /home/sansforensics/linux and put the JSON file in there. It clearly found a number of other JSON files, including ones that aren't the right format, which means if it didn't list the JSON file in there, it didn't consider it at all. What surprises me about that is that the earlier output listed where it would read symbols from:

INFO     volatility3.cli: Volatility symbols path: ['/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/symbols', '/usr/local/lib/python3.8/dist-packages/volatility3/volatility3/framework/symbols']

but the isfinfo output was giving a lot more files/directories than just those. I don't know whether sansforensics alters the installation in any way, but it might be worth checking with a copy from git?

[cmdaltr]

I don't think sansforensics does alter anything because I have pulled the latest version of vol3 from GitHub already - vol3 does not come preinstalled.

I have tried the same steps as above on a clean Ubuntu 18.04 host and I am still getting errors. Perhaps I have created the profile incorrectly?
Do you have a Ubuntu 18.04 symbol JSON file you could share?

[ikelos]

Then I'm really not sure what's going on? It's clearly crawling all the JSON files it can find (and under a bunch of directories I wouldn't have expected it to)? The output.txt didn't have the logging output (which the 2>&1 should have piped into the same place). It would be interesting to see the first few lines of that, because they specify which directories are searched under the for the JSON file. The code where it walks all the JSON files is here.

The symbol files need to follow the JSON schema specified by their version type in the metadata. I don't have one matching your kernel exactly, but here's an older one that should show you what it should look like. Hopefully you can compare the contents and check that the file is right (my guess is the file just isn't even being looked at for some reasons, but it's not yet clear why)...
ubuntu:5.4.0-26-generic64.zip

[cmdaltr]

Thank you - this is the JSON I have created - please let me know of any errors
vmlinux-5.4.0-42-generic.json.zip

[ikelos]

That's very interesting. The schema validator says the contents of the file is fine, but dumping it in the right place and running isfinfo isn't showing it for me either... 5:S I'll have a dig around and see if I can figure out what's up. Thanks for sticking with it! 5:)

[ikelos]

Ok, that was my bad, it wasn't showing up because the file wasn't actually there. 5:P When it's there I get the following:

file:///home/personal/workspace/volatility3/volatility3/symbols/linux/vmlinux-5.4.0-42-generic.json	Unknown	18	10088	150900	1724	-	Linux version 5.4.0-42-generic (buildd@lgw01-amd64-038) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 (Ubuntu 5.4.0-42.46-generic 5.4.44)

So it looks like it should be fine, and again, it suggests the symbol path isn't being read correctly, or it doesn't have permission to read that file for some reason?