I was wondering the same thing about Negan. There's Hilltop and the Kingdom and the fishing village within a relatively small area. I wondered why he gave up so easily on such a small sampling of teritory before agreeing to go back to jail.
I can’t say I was impressed. The Whisperers lacked any mystery or menace, with one being taken hostage so soon. I agree that, oddly enough, the Negan parts of the story ended up being the most intriguing. But would he really want to come back to his cell?