There is more than 1 type of banner. There is at least one magical banner (Advanced Player's Guide):
A lord's banner is a cloth flag or standard, typically at least 2 feet wide and 4 feet long (though some are up to twice that size), meant to be carried and displayed on a lance, polearm, frame, or staff. It has no effect when not mounted properly or when laying on the ground. It normally depicts the insignia or heraldic symbol of a particular noble.
And a nonmagical banner from Ultimate Equipment:
This is a banner, flag, or pennant. You tie it to a pole, lance, or polearm. Most are woven, dyed, or painted with a pattern or symbol, such as a knight's crest or a country's flag. A simple banner with one field color and a simple insignia such as a weapon or shield costs 1 gp. A banner with two to four field colors and a complex insignia such as a lion or dragon costs 5 gp. A detailed banner with four or more field colors and a very complex insignia, such as a heraldic device with eight or more sections, costs 20 gp.
A flag (Inner Sea World Guide), and the Flagbearer feat (same source), do require it to be held in hand.
A flag is a colorful banner that bears the heraldry or symbol of a nation or organization. A character with the Flagbearer feat who bears a flag can grant additional combat bonuses to nearby allies. Carrying a flag in combat requires a free hand.
The banner for the cavalier class just needs to be displayed, with no listed limitations on how.