Service is unable to load huge Password File

[ForumSchlampe]

System
Windows Server 2012 R2 most recent patch level

Reproduce:
Install OpenPasswordFilter + Service as described
Use as password list weakpass 2.0 ( https://weakpass.com/wordlist/1859 )
Start the Service

Problem
A huge amount of event entries with id 101
First logged Errortext:
"Died trying to ingest line number 25643819"
At this point the service was at about 900MB
It counted up very slowly but the error was persistent for each new line.

The password in the above line is nothing special
KDUAS

[brockrob]

That's a 28GB wordlist. Even if you can get that loaded, I would not expect it to be fast.

If you really want to (and you actually have domain controllers with that much RAM), you need to start worrying about https://www.codeproject.com/Articles/483475/Memory-Limits-in-a-NET-Process

Now that we're querying the haveibeenpwned API, there's probably not much value in using such a huge and exhaustive wordlist anyway.

[ForumSchlampe]

I know, similar to the one haveibeenpwned list without API

Do we realy touch those limits with the service, doesnt we run the service in "User-mode virtual address space for each 64-bit process" which is pretty huge on x64 systems?

While our Domain Controllers cannot reach the Internet, we depend on local available wordlists. Amount of memory wont be a problem, we try to avoid to use a sql database on a 3rd system so anything for a working solution runs on the domain controllers.

[brockrob]

The password match list goes into a hashset, and that has limitations: https://blog.mischel.com/2008/04/09/hashset-limitations/

You can probably make some significant headway by pruning your wordlist. There's no point in having entries that are shorter than your domain's minimum password length setting, and match checking is case insensitive so you can lowercase and deduplicate the entire list too.

[ForumSchlampe]

Already trying what is left after pruning the wordlist...but takes time

While i am not very familiar with programming, if we hash the list in advance, does this limit still apply?

[ForumSchlampe]

Did a fork with an collegue to fix this (mainly it is now a 64bit only project) so the limits will not apply, big files are no problem

Pruning the wordlist resulted in a 16gb file, loading this was a mess (takes up to ~90min and consumes 100gb ram) but while running it is fast, very fast!

[brockrob]

That's a lot of RAM for a DC!

I'd take a look at the dll if it's taking 90 minutes for the service to come up. It's configured to fail open, meaning allow any password changes if the service is unreachable.

[ForumSchlampe]

So we`ve done some improvements in our fork to handle this, if the service doesnt respond the response is negative/false

It is now possible to load very huge password files (if theres enough memory), optionally use mssql/mysql database (not entire sure if mysql implementation is the best but it works) and haveibeenpwned api is now configurable.