To charge him with defrauding investors requires a whole different group of people to get involved.

Additionally, those people need enough evidence to have a chance of conviction. "He refused to answer questions about it" is not actually evidence.