People who live near new traffic restrictions in a city have said they feel "betrayed" and "ignored" by their local council.