Forcing investors to "be somewhere they don't want to be" is a barrier to attracting people, the Immigration Minister says.