I'm sorry to be the one to tell you, but Signal 100% stores your contacts. They keep your name, your photo, your phone number, and a list of every person you've contacted using Signal. That data is permanently stored on their servers.
> "This should be updated for the recent changes where contacts are uploaded to Signal’s servers and stored permanently along with Groups V2 and other data, protected by a 4-digit minimum PIN and Intel SGX – there have been concerns 5 raised 2 in these forums, particularly if one of your contacts chooses a brute-forceable PIN which in the context of an Intel SGX vulnerability 1 could leak a lot of contact data if hacked, even if you choose a strong password."
See the two links sited in that comment for more information on why it isn't actually stored in "secure" way.
See: https://community.signalusers.org/t/can-signal-please-update...
> "This should be updated for the recent changes where contacts are uploaded to Signal’s servers and stored permanently along with Groups V2 and other data, protected by a 4-digit minimum PIN and Intel SGX – there have been concerns 5 raised 2 in these forums, particularly if one of your contacts chooses a brute-forceable PIN which in the context of an Intel SGX vulnerability 1 could leak a lot of contact data if hacked, even if you choose a strong password."
See the two links sited in that comment for more information on why it isn't actually stored in "secure" way.