New lightgrep #486
New lightgrep #486
Conversation
…rmance regression. The timings below are from the following command: ./src/bulk_extractor -F ../lightgrep/pytest/keys/shuf10.txt -Z -o ~/be_timed_output_without_thread_local_`printf %04d $i` -E scan_lightgrep ~/ev/terry-2009-12-11-002.E01 Thread_local? Clocktime (Min.) Clocktime (Max.) Clocktime (Average) Scan Lightgrep Time (Min.) Scan Lightgrep Time (Max.) Scan Lightgrep Time (Average) FALSE 162.965479 168.628229 164.2545712 494.810946 528.368114 504.1799554 TRUE 163.681386 173.587754 167.233617 499.815450 532.324335 516.4901762 This reverts commit 0ca43ec.
Update lightgrep scanner for bulk_extractor 2.0
|
Ah, with the build failure, I don't think we tried running |
|
The most significant thing that's changed is we now have a full release of lightgrep v1.5, available here: https://github.com/strozfriedberg/lightgrep/releases/tag/1.5.0 We can change @simsong I would appreciate it if you could look at the diff for scan_lightgrep.cpp, as the typical scanner setup and usage is what I'm most unsure of. |
|
Yay!----Sent from my phone.On Feb 11, 2025, at 9:55 PM, Jon Stewart ***@***.***> wrote:
Ah, with the build failure, I don't think we tried running make distdir-am, probably just plain old make. Nice to have automated tests!
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Sure. I will review in the next day or so.Generally, I don't review draft PRs. I missed when you tagged me. And the top of the PR said it wasn’t ready for review…----Sent from my phone.On Feb 11, 2025, at 10:05 PM, Jon Stewart ***@***.***> wrote:
The most significant thing that's changed is we now have a full release of lightgrep v1.5, available here: https://github.com/strozfriedberg/lightgrep/releases/tag/1.5.0
We can change Makefile.am to fix the distdir-am target and whatever comes afterwards. We'll also adjust the CONFIGURE_ scripts to pull lightgrep v1.5.0 from the above URL. With the CONFIGURE_ scripts, is there some way to add commonality to all of them?
@simsong I would appreciate it if you could look at the diff for scan_lightgrep.cpp, as the typical scanner setup and usage is what I'm most unsure of.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
| const char* contentsCStr = contents.c_str(); | ||
| // Add all the patterns from the files in one fell swoop | ||
| if (lg_add_pattern_list(Fsm, PatternInfo, contentsCStr, itr->c_str(), DefaultEncodingsCStrings, 2, &opts, &err) < 0) { | ||
| if (lg_add_pattern_list(Fsm, contentsCStr, itr.c_str(), DefaultEncodingsCStrings, NumDefaultEncodings, &opts, &err) < 0) { |
There was a problem hiding this comment.
lg_add_pattern_list() seems able to return -1 without populating the error chain, so this function can return false without printing a more descriptive error message.
lg_add_pattern_list() needs more unit testing in lightgrep...
|
@jonstewart - I've converted this to a draft until the crashing issue is resolved. |
|
I believe I have a fix for lightgrep as a PR at strozfriedberg/lightgrep#11. It may need a little more work for cleanup—unsurprising that a stupid oversight bug is hanging out in fiddly code. Once the PR is merged, I'll tag a v1.5.1 release of lightgrep for this PR to target. @simsong can you provide some guidance about the various build scripts? do you have them all automated with github actions? In particular, what's needed to make a Windows build? |
|
I'm not sure the status of the windows build. But I'll update it to incorporate lightgrep and build with that soon.
… On Mar 20, 2025, at 3:37 PM, Jon Stewart ***@***.***> wrote:
jonstewart
left a comment
(simsong/bulk_extractor#486)
I believe I have a fix for lightgrep as a PR at strozfriedberg/lightgrep#11. It may need a little more work for cleanup—unsurprising that a stupid oversight bug is hanging out in fiddly code. Once the PR is merged, I'll tag a v1.5.1 release of lightgrep for this PR to target.
@simsong can you provide some guidance about the various build scripts? do you have them all automated with github actions? In particular, what's needed to make a Windows build?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.
<strozfriedberg/lightgrep#11>
<https://github.com/simsong> <#486 (comment)> <https://github.com/notifications/unsubscribe-auth/AAMFHLHXIHUOUZZMEOJROT32VLAC7AVCNFSM6AAAAABW6QHWRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBQGQ4DOMRRGM>
jonstewart
left a comment
(simsong/bulk_extractor#486)
<#486 (comment)>
I believe I have a fix for lightgrep as a PR at strozfriedberg/lightgrep#11 <strozfriedberg/lightgrep#11>. It may need a little more work for cleanup—unsurprising that a stupid oversight bug is hanging out in fiddly code. Once the PR is merged, I'll tag a v1.5.1 release of lightgrep for this PR to target.
@simsong <https://github.com/simsong> can you provide some guidance about the various build scripts? do you have them all automated with github actions? In particular, what's needed to make a Windows build?
—
Reply to this email directly, view it on GitHub <#486 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMFHLHXIHUOUZZMEOJROT32VLAC7AVCNFSM6AAAAABW6QHWRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBQGQ4DOMRRGM>.
You are receiving this because you were mentioned.
|
Now let's see this run in GitHub Actions