Average size is fairly meaningless. Average according to where and when? How many people would be a normal garrison, and how many a minimum? Is it a motte-and-bailey castle made largely of wood and earth, a standalone stone keep, or a full-size stone concentric castle? I know you said they won't have the layout, but these are basic questions necessary to answer you.
Who they need to bring depends a bit on how much supply they can expect from outside. It doesn't sound like the area is settled enough to supply food - if this is true they'll need wagonloads of food frequently. If they're bringing a bunch of farmers to settle the savannah then they'll still need those wagonloads until the first crops come in.
If I'm wrong and locals can and will supply food then it's a question of how much else they might supply. The party might need their own wheelwright for wagons or they might not for example.
A blacksmith, a stonemason, a carpenter, a stablemaster and someone to run the place when they're off dealing with adventurer-type problems are about the minimum specialists they'll need IMO. Numbers of soldiers, cooks & servants, staff for some or all of the above will depend on the needs of the castle.
For other proffesionals I would include a judge, lumberjacks, miners, a few shopkeepers, clerics/healers, an alchemist, an engineer, some scouts with survival skills, and a proffesional entertainer. This is even if the party can do some of these things as they will probably be occupied at times.
Also I want to point out that if the players start out lv9 and have a divine and an arcane caster the party is self sufficient. They would have access to spells to make/do anything they would want people to