I've GMed/DMed for many years, but I'm running my first open-world game where PCs can literally explore any place on a map. I have created a huge, custom random-encounter chart, maps, city stats, etc.
I'm finding that despite their excitement to play such a game, they tend to loose focus or interest without "railroading" them or pushing a plot line in their direction. Actually, I presented them with a plotline last weekend at our usual game. They decided to explore a new town, thus not following the plotline, then things went downhill from there as they lost focus and interest.
I could use some constructive advice about how best to run/plan/whatever such a game.
I recommend taking a look at Kingmaker and Ultimate Campaign; both of these would be useful in your endeavor. Kingmaker is a sandbox type campaign, focused on exploration (at least at the beginning), but with story that keeps things moving.
What you need is to have some reason for your people to be out there doing stuff. A charter for a government (as done in Kingmaker) or merchant group, or some other entity, would be a good example. This will motivate your people to move around the hex map. In certain hexes, place planned encounters that add flavor or plot points about the campaign world, and don't forget to throw in random encounters.
You're running into reality. Players think they want to create their own plot, but they rarely can.
What you have to do is come up with a plot, but it needs to be a big one. Don't tie it down to any specific places on the map. Come up with a conflict and group of villains. Get the party to accept a job/mission that would bring them into the conflict even if peripherally.
Then let them go on the map. Let them find info and clues on the plot as they wander around from town to town. Let them dig in to anything they see, just keep feeding them info about the plot. A good method is to let rolls that are made to gather info (for anything they're looking for) get some info on the plot. Let the info be more/better the higher the roll.
Then start throwing in encounters where your vilains are either involved in what the party is taking on that week, or just make some of the random encounters be with the villains. Just work it in when you feel it's appropriate.
The building volume of info they have on your villains, combined with an increasing number of encounters will get the party wanting to take out your villains.
As you give them info keep it vauge, and don't tie yourself down. Let the parties wanderings, and their interpretations "lead" them to your bad guys. If you try to get the party to figure out exactly where you placed the lair, they are not going to. If you let their conclusions be where you place the lair it will work every time.
Basically feed them info. Let them determine what it all means. Place your bad guys based on the parties conclusions. In the end the party will have written the story, and fought the bad guys they were looking for. They'll feel that they were clever enough to make all the right choices, and never realize they were simply on a very mobile and flexible railroad.
Make your plot more adaptive, then. Here's an example.
When visiting a town in a world full of magic and monsters, surely there's some quests they can pick up. Maybe they overhear a rumor that something isn't right at the tanner's, the mayor's, or the general store. Strange noises have been coming from the graveyard. Little plot hooks that give them a couple encounters in a given location. Goblins have been raiding the town every few weeks, and the militia isn't enough to stop them. A merchant needs protection traveling to the next town.
The next 3 or so of these small events should have a tie-in to a larger event. The bandit leader, the necromancer, and the goblin chief all wear the same signet ring or amulet or whatever. The third one will also carry a message on him from Count Midboss with instructions on when to attack the town.
Well, if the PCs eventually investigate this Count Midboss character, they discover he's been terrorizing various townships through front organizations (the necromancer, the bandits, and the goblins, plus others) to make the towns more reliant on <something he provides them>, until eventually he will claim these towns as his own, expanding his lands without needing an actual army. If the party doesn't pursue this hook, just keep dropping hints that he's the cause of the misfortune plaguing the land. Perhaps the local baron eventually (after town 5 or something) comes to the same conclusion and offers a reward for them to discretely stop Count Midboss's plan, since he's trying to steal away the baron's towns.
Then you add another layer of intrigue once they've dealt with the Count (who may or may not have been a vampire). It turns out the Count was working for a shadow organization that serves a neighboring country. Once he had these towns under his control, he was planning on letting the other country's armies march through his lands and attack this country's capitol directly. This would plunge the land into a huge war, and the shadow organization would use the energy from all the killing to summon a terrible fiend from the great beyond, because that's how crazy cultists do.
Once the cult is dealt with, it turns out the cult leader was a drow masquerading as a human. He told the cult that they were trying to take over the land, but he just wanted to destroy everything on the surface because drow hate surface dwellers.
So what we have here is a naturally developing plotline that can advance or be ignored as the party wants. If they choose to ignore the plot hooks, then just keep dropping the hooks for a while and they will probably eventually take one of them. And you can do this at any of the levels. You don't have to railroad them into a plot like this. If they're having fun just exploring, then let them explore. But hint that Count Midboss's town is pretty cool and they should check it out. Hint that the capitol city where the cultists hang out is even cooler. And hasn't everyone wanted to explore the underworld? No? Then cool, we can keep exploring little farming communities and stuff.
Everything the others have been saying is excellent and right on the money. I have one more thing to add though.
Keep in mind the value of the NPC. A good NPC adds a very human element to a game. You want people to take the bait, and nothing can do that like having some sort of investment in the events at hand. You don't have to make the NPCs fighters - in fact, keeping them low level even if they are warriors (in the sense that they do battle, not the NPC class) is a good idea. It's not about wowing the PCs with the NPC's power, it's about endearing someone to a cause by making it personal. My group is playing "City of Seven Spears" (which isn't open world but still) and, after being attacked by an evil boggard, decided they hated the boggards and wanted nothing to do with them. However, a few weeks later they ran into a particular, lower level oracle boggard who they immediately loved. All but one member of the group decided they were pro-boggard now, and put a lot of their resources into making an alliance work. Note: You can go too far with this by making the journey ABOUT the NPC. You don't want that. You want an NPC to be a sort of bridge that allows the PCs to become the main players in a certain scenario themselves.
Actually I have a second thing: Figure out your players' stories, and find the sorts of plots that make them tick. Some people will jump for joy at the idea of treasure, but when I tried that with my group they shrugged it off. My group in particular really doesn't care unless they're influencing other people in some way. They're more likely to help a village in trouble than they are to go spelunking for the mythic sword. Some characters will love other things than those two: they might have a particular aversion to a particular sort of monster, or they might be looking for a very particular artifact, or they might just want to live a normal life (in which case the danger best come to them). Consider what sort of plots make your characters tick, and write your campaign around those ideas.