usertrap: disable syscall patching when ptraced

[ret2libc]

Workaround the issue in #12266 .

It's not clear to me yet why the issue appears, but it happens intermittently when ptracing a program. I think also #11649 is related. This patch is similar to --systrap-disable-syscall-patching but instead of disabling syscall patching globally, we just disable it for some tasks.

What do you think?

[konstantin-s-bogom]

IIRC the issue with debuggers and systrap is that they enable single-stepping, which is completely incompatible with the "syshandler" routine used to handle patched syscalls. When the process is inside this routine and subsequently receives a SIGTRAP for every instruction, the signal handler proceeds to blow away the register state of the user process.

So this patch is insufficient, because if the workload enters some piece of code where patches have already been applied before patching got disabled, this will result in exactly the same error.

I've described what I think the proper solution is in #11649 (we need to roll back the patches in addition to disabling them).

All that said I don't see too much harm with merging this (maybe with a better comment and a TODO to implement the rest). This partial fix may be enough for a good chunk of ptraced workloads. @avagin WDYT?

[ret2libc]

You are right, I guess it will not work when you ptrace-attach to an already existing process that has been patched, but it will work better for programs that are ptraced as soon as cloned, right? I will improve the comment!

[ret2libc]

I've rewritten the comment and added the TODO. Let me know what else I can do!

[konstantin-s-bogom]

Sorry for not replying for a while, I was out. Thanks for fixing the comment! I think you should also add a loud warning log in case State.nextTrap > 0, which indicates that this subprocess already had some patched syscalls; something like LIKELY ERROR: Attached tracer to process with patched syscalls! Systrap is not fully compatible with ptrace/debuggers, program may die unexpectedly soon!

[ret2libc]

@konstantin-s-bogom interestingly, I hit the the warning even by just running gdb -q --batch -ex "run" --args /bin/true. A few syscalls have already been patched even there.

[avagin]

@konstantin-s-bogom interestingly, I hit the the warning even by just running gdb -q --batch -ex "run" --args /bin/true. A few syscalls have already been patched even there.

Because gdb attaches to a child process before it execs the target binary...

[ret2libc]

Thanks!