I didn't see this particular question asked like this before, but I didn't get much chance to read back through all the forums, so forgive me if I've already been answered.
OK, so let's first start off by talking about smaller blades. Medium sized daggers and small sized short swords. A halfling using a small short sword does 1D4 damage, crits on 19-20 for X2 damage and can use weapon finesse. If that same halfling uses a medium dagger, it does all the same stuff, but at a -2 to hit (right?). The difference is that the weapon is sized wrong for the halfling. Maybe the handle is too thick, the pommel too heavy, whatever. The point is, that halfling can get a small sized short sword, and be just fine.
Now on to the large sized bastard sword: 2D8 damage, crits on 19-20 for X2 damage, but needs a feat to use, and can only be used with two hands. And oh yeah, -2 to attack rolls because it isn't sized for a medium sized creature. So why not just resize the handle (or whatever) to make it easier to use for a medium sized creature; just like they do for the small short sword/medium dagger. The only change would be losing the -2 to hit (unless you're a large sized creature).
The game logic and world logic both allow for it, so why not?
|The Black Bard|
Some quick real world math, spoilered for space sake:
Medium is 4-8 feet, our average knight is 6, right in the middle, perfect! A 12' creature would be the average of large (8-16 feet tall range), so we can double the sword dimensions. Nice and simple!
Our large bastard sword is, on average, 102 inches long, or 8.5 feet. It would not be able to stand vertically in the average house without going into the ceiling. The grip makes up only 16 inches of that. And while the equipment chart says to only double the weight of large items, realistically it should be cubed, as all 3 of its dimensions are doubling. So the 4 lb. sword should weigh technically close to 64 lbs. Even cutting that in half (which leaves us with so little material the blade is now structurally compromised) is still a 32 pound blade. Thats approximately the weight of 4 gallons of milk stretched across an 8' pole.
A truly strong character CAN use such a blade and make it work: they take the -2 to hit, and because they are ridiculously strong (say, 8 points higher than the average human, so 18, yes, that is ridiculously strong. Str 30 is INSANELY strong, literally against common sense) they still end up with an attack bonus of +3 at level 1. While common folk have a +0, likely not with such an intimidating weapon.
Ultimately, leverage comes into play. This is represented by the carrying capacity and object break DC adjustments for large and smaller creatures. A pixie can never get the stuck lid off a pickle jar, and a storm giant can never fail (won't even break the jar unless he tries, by the rules). Leverage, aka the -2 for wrong size, can't be overcome.
Regarding game rules, a medium dagger and a small short sword are not the same thing, even if they are statistically identical. If they were, the rules would say (treat small short swords as daggers). In older editions, the game did just that, for simplicity.
The bastard sword is a 2 handed martial weapon, that with a feat, can be used as a 1 handed exotic weapon. This is a case where they are actually the same thing. So your comparison can be viewed as a sort of "apples and oranges".
Game logic can be argued to not support it (and I suppose can be argued to, but I find it a stretch). World logic is actually against it, if the world involves anything resembling real world basic physics like gravity and inertia to create leverage issues.
tldr: You can already fight with a large weapon, and physics imposes a limiter on it. Outside of rewriting physics (via magic, likely), that limiter is going to stay.
Source: A lot of time at Renaissance Faires and SCA wars, holding, swinging, fighting with, and forging such blades. Blah blah, personal experience is meaningless, blah blah, weapon cords, etc.
Only kinda related to you question, but...
A medium dagger is actually considered a one handed weapon for a small sized creature. Thus it can be used two handed for 1.5 STR, and it actually CAN'T be used with weapon finesse. These are actually pretty meaningful differences that have come up in a game I was in.
I agree with everything The Black Bard said.