Well, like you know, in Scrum, just have a Scrum of Scrums, that’ll work out.
I actually giggled on my morning walk because it was the same visceral reaction I had reading the first time I came across the concept. I have pretty much the same reaction to how well a lot of Agile techniques will scale. They all depend on communication and collocation so a huge project team seems antithetical. My general rule is to run screaming from really large projects, or to see if they can be done with a smaller elite team.
Luckily this is all pretty much theoretical in my present world since our largest projects have at most 8-10 developers, and most of them are more along the lines of 2-3 developers per project.