This article does a nice job of explaining why pthread cancellation is hopeless.
> If we could know that no signal handler is ran between the flag check and the syscall, then we’d be safe.
If you're willing to write assembly, you can accomplish this without rseq. I got it working many years ago on a bunch of platforms. [1] It's similar to what they did in this article: define a "critical region" between the initial flag check and the actual syscall. If the signal happens here, ensure the instruction pointer gets adjusted in such a way that the syscall is bypassed and EINTR returned immediately. But it doesn't need any special kernel support that's Linux-only and didn't exist at the time, just async signal handlers.
(rseq is a very cool facility, btw, just not necessary for this.)
No you can't since the compiler will likely inline the syscall (or vsyscall) in your functions. So there's no way to know the instruction pointer is in the right section. The only way is to pay for no-inline cost and have a wrapper that's calling the syscall, so it's a huge cost to pay for a very rare feature (cancelling a thread abruptly is a no-no in most coding conventions).
I'm afraid that every part of what you just wrote was wrong.
> No you can't since the compiler will likely inline the syscall (or vsyscall) in your functions.
Do you mean the SYSCALL instruction? The standard practice is to make syscalls through glibc's wrappers. The compiler can't inline stuff across a shared library boundary because it doesn't know what version of the shared library will be requested at runtime. Using alternate non-inlineable wrappers (with some extra EINTR magic) does not newly impose the cost of out-of-lined functions.
It'd be possible to allow this instruction to be inlined into your binary's code (rather than using glibc shared library calls), but basically no one does, because this cost is insignificant compared to the context switch.
In general, inlining can be a big performance win, but mostly not because of the actual cost of the function call itself. It's more that sometimes huge optimizations are possible when the caller and callee are considered together. But these syscall wrappers don't have a lot of expense for the compiler to optimize away.
Do you mean the actual syscall (kernel code)? This is a different binary across a protection boundary; even more reason it can't be inlined.
vsyscall (or its modern equivalent, vDSO) is not relevant here. That's only for certain calls such as `gettimeofday` that do not block and so never return EINTR and (in pthread cancellation terms) are not "cancellation points". There is just no reason to do this for them. And again, the compiler can't inline it, because it doesn't know what code the kernel will supply at runtime.
> The only way is to pay for no-inline cost and have a wrapper that's calling the syscall, so it's a huge cost to pay for a very rare feature (cancelling a thread abruptly is a no-no in most coding conventions).
It's an insignificant cost that you're already paying.
The article is proposing a much safer alternative to cancelling a thread abruptly: using altered syscall wrappers that ensure EINTR is returned if a signal arrives before (even immediately before) entering kernel space. That's the same thing my sigsafe library does.
> If we could know that no signal handler is ran between the flag check and the syscall, then we’d be safe.
If you're willing to write assembly, you can accomplish this without rseq. I got it working many years ago on a bunch of platforms. [1] It's similar to what they did in this article: define a "critical region" between the initial flag check and the actual syscall. If the signal happens here, ensure the instruction pointer gets adjusted in such a way that the syscall is bypassed and EINTR returned immediately. But it doesn't need any special kernel support that's Linux-only and didn't exist at the time, just async signal handlers.
(rseq is a very cool facility, btw, just not necessary for this.)
[1] Here's the Linux/x86_64 syscall wrapper: https://github.com/scottlamb/sigsafe/blob/master/src/x86_64-... and the signal handler: https://github.com/scottlamb/sigsafe/blob/master/src/x86_64-...