The point of a timing attack is that some operations take different amounts of time depending on the plaintext and/or key. Depending on the attack in question and your access to the system, this potentially lets you draw conclusions about what was encrypted by observing how quickly the system responds to various requests.
If the custom algorithm is an outer layer, it only processes data that has already been encrypted by another presumably strong algorithm. Even if there's a timing attack, breaking the outer layer can't help you unless you can also break the well-studied inner layer. If the custom algorithm sees the actual plaintext directly, the timing attack can let you straight to the original message, no matter how strong any of your outer layers are.
Indeed, if the home-made RSA algorithm is the outer layer, and someone manages to break it, then we're back to the status quo. And the status quo is unbreakable encryption, right?
On the other hand, if the home-made RSA algorithm is part of the inner layer, how is it different than any of the other poorly designed code that we use in the user facing side of encryption? If an API call goes through 100,000 lines of business logic, and the result gets encrypted and sent as an API response, that's okay, it happens billions of times per day. In terms of security, what does it matter if a few of those 100,000 lines are a home-made RSA implementation?
If the custom algorithm is an outer layer, it only processes data that has already been encrypted by another presumably strong algorithm. Even if there's a timing attack, breaking the outer layer can't help you unless you can also break the well-studied inner layer. If the custom algorithm sees the actual plaintext directly, the timing attack can let you straight to the original message, no matter how strong any of your outer layers are.