File descriptor exhaustion

[sjcappella]

Long running fuzzing sessions will eventually exhaust available file descriptors due to state syncs seeming to maintain a connection even after the client has retrieved the state. Haven't pin-pointed the actual offending line of of code yet. Will probably look into it more.

[richo]

Fascinating (And sorry for not seeing these issues, evidently they got lost in my inbox).

I'll have a look today- how long was longrunning? I've successfully left roving going for O(weeks) and not had issues. In any case, I have some hunches about where this is, I'll see if I can repro today.

[richo]

uh, to be clear is this the server or the client leaking the fd's?

[sjcappella]

Ah, sorry I wasn't more specific. I blew that setup away a while ago. If I remember correctly, it was the client leaking the FDs. I had about 64 instances running across 3 boxes and eventually they all ran out. I can try and set this up this weekend and see if I can reproduce. But I just remember that after having it run for a while, eventually there was a file based resource exhaustion that prevented synching.

[richo]

I can pretty easily try to repro, I just wanted to make sure I knew what I was reproing.

I'll update this with more info

[richo]

Were you running with a really low limit on fd's by any chance?

With a really low limit (50) I've been able to repro what I believe is it (The tar step fails, I haven't dug but I bet it's it). However what's interesting is that with a higher limit (1000) it doesn't crash.

My theory is that whatever I'm leaking eventually gets freed by GC, but that with a sufficiently low limit, the GC pass never happens. The patch I just pushed doesn't help.

[sjcappella]

I was running stock Ubuntu 15.04 x64 without any changes to max fd settings. I did do some reading into how Go handles fd's when I first encountered the issue and after researching Go's fd and GC, saw that Go will use GC to mange them, so I wasn't sure if that was the issue or not. Your guess is much more accurate than mine. I haven't had the chance to seriously look at Go.