Baseball. While most will discredit it because there's a ton of stop-and-go in the sport. There's almost nothing harder than hitting a 90-100 miles per hour fastball, then trying to hit that same ball breaking 6-12 inches off/on up/down on the plate going 60-70 miles per hour.
It takes years of scouting reports,
And most of all natural super-human hand-eye coordination.
...when I lived in Cincinnati I met the pitcher Jose Rijo, which before his injury was an awesome pitcher. I probably was like 13 years old or so and he was doing a demonstration at a charity of how a fastball looks coming at you. And he was maybe throwing 80% of what he was capable of. They would have you stand behind a chain-linked fence (similar to the fences pitchers stand behind in batting practice) and I the ball flew by me. Even to this day there's no damn way I'm hitting that.
I started playing union this past year for my school and it is pretty easy to play/pickup. I'm sure most people who play will tell you that. Is it a tough sport? Sure, but not difficult to play. Obviously any of these sports are tough to play at the highest level, but rugby is far easier than baseball, boxing, golf, hockey, etc.