If something is so hard to implement that everyone who tries gets it wrong (to a first approximation), then maybe the concept is bad. Or, at least, the concept isn't fully baked and is missing something critical.
Most implementations of toasts-with-actions that I've seen don't have the problem OP described. I more often find myself manually dismissing them than wishing they'd have stuck around longer.
