The towns combined provide the military force... I think the Empire would have run them down eventually, but they would have lost too many resources to make it worth it since they were still expanding elsewhere and needed troops.