I used to live in one too (not yours). It's weird how often people would tell me that this was infeasible and impossible to do it at all.

I also almost bought one that was in a converted school. I'm sure some people could come up with reasons for why that is impossible too.

Zoning is the only absolute impediment I can think of and that is a self imposed one.