The below changes describe a method to get away from "cast and forget" type casting. I'm not claiming there aren't holes or issues and only present it as a rough draft of the idea.
The basic premise is this: a caster can cast any number of spells per day that they know and have components to cast. However, while casting itself is unlimited, casting a spell of higher than the normal level a caster can cast requires a certain cool down period before another spell of similar level or higher may be cast.
Any caster may technically cast any spell on their list so long as they know it or have access to it either via a spellbook, scroll, etc and have components to cast it.
The caveat to this is that all casting times, cool downs and potential ramifications are dependent on the relationship between the caster level of the caster and the level of the spell. A caster who can normally cast a level 4 spell, can cast a level 4 spell as a standard action as normal so long as the normal casting time is such. However, a level 1 caster casting a level 4 spell would take 1 hour if the spell normally takes a standard action, for example. These timescales for casting adjust along the scale of swift action, move action, standard action, full-round action, 1 minute, 1 hour, 1 day and 1 week and adjust by +1 per the level of the spell above the level the caster can normally cast. Any casting times taking longer than 1 hour require an uninterrupted time with which to cast, the caster may not take breaks for any reason and must make concentration checks for any interruptions and the caster must make any relevant saves for forgoing food and water during the time they spend casting a spell. Forced march rules may be used if it is thematically appropriate to do so.
Higher level casters may cast spells more quickly and work the timescales in the opposite direction, but this requires a caster be able to cast spells of a level 3 higher than the spell in question. For example, a caster able to cast 9th level spells may cast spells rated at levels 4-6 as a move action and spells with level 0-3 as a swift action. This may allow casters to cast more than 1 spell per turn but no spell may be cast by a caster of any level in any time increment shorter than a swift action through either metamagic feats, class abilities, traits, etc except where the normal casting time of a spell is indicated to be such, such as Feather Fall, which is an immediate action.
They should also make some sort of save for channeling more energy than their level normally allows. This should be in the form of either a will or fortitude save depending on either the level of the spell or its effects. Concentration checks work normally per RAW.
Regardless if a caster fails or succeeds a save to cast a higher level spell, as long as they have all the required components, makes any relevant concentration checks or other saves normally encountered in casting other than those below, the spell effect works normally even if casting the spell kills the caster. The only caveat to this is that if the caster dies as the result of casting a spell too high powered for them and the target of the spell has a spell resist rating, the target is assumed to automatically be immune to the spell effects as per the normal rules of spell resistance as if the caster failed their caster level check.
Casters should also face potential drawbacks in the form of fortitude or will saves for casting spells higher than their normal level allows. Spells that affect the target's mind should require a will save of DC 10 + (5/level of spell over normal the caster can cast) caster level check. Failing this save drives the caster mad in some way and the caster gains a condition as deemed appropriate by the GM for a number of rounds equal to the difference between the spells a caster may normally cast and the spell attempted. For example, a level 1 caster casting a 9th level spell would be driven mad for 8 rounds if he failed his will save.
If a spell inflicts damage, the caster must make a fortitude save using the above rules except the caster takes on the amount of damage of the spell in question as per the spell's description and only for 1 round unless the spell effect lasts multiple rounds and ends when the spells effects normally would end.
Spells targeting the body (polymorph, teleport, etc) are also a fortitude save and similarly act upon the caster instead should they fail their save.
Spell resistance works normally. Failing any of the above saves negates any affect on the intended target. Reflex saves are not included as the effect works directly and immediately upon the caster and is impossible to diminish, negate or otherwise avoid these effects even if you have a spell resistance rating, damage reduction or abilities such as Evasion.
All spell components cost at least 1 GP and are therefore "expensive." Components must be purchased and spent on a per casting basis. If a GP amount of a components would be unwieldy in size (e.g. if 1 GP of cloth would equal multiple bolts) then increase the quality of the item such that its quantity in GP may be carried easily enough on ones person so as to be able to cast the spell as many times as normal as if these rules were not in effect. If a quality of material is not available within an area then spells dependent upon that material may not be cast even if the caster has the eschew materials feat.
Lastly, casting a spell higher than one's normal level of spells requires a cool down equal to half the time required to cast the spell before another spell of the same level or higher may be cast regardless if the casting was successful or not. If a level 1 caster casts a level 6 spell, they may not cast another level 6 or higher spell for 3 days, 12 hours. These cool downs stack. Casting a level 7 spell followed by a level 6 spell requires a week before another level 6 or higher spell may be cast. As long as a caster is under such a cool down, they are considered fatigued.
Instead of cool down periods wouldn't it make more sense (and be easier to track) to have a tiered spell point system?
So you have a pool for level 1, 2 and 3 spells, you run out of 3rd spell points and you can convert over 2nd level points (at a harsh ratio) to fill that pool. Each time a spell pool is empty (per spell level) you gain a fatigue level. Again, stressing harsh conversion ratios where it would tactically make more sense to keep your 1st level pool intact and not dumping all the points to replenish higher level pools (unless you're really desperate).
I like cool down mechanics but I think points are easier to keep track of, especially when dealing with different levels/cool down times, assigning X amount of points (per tier/day) would be easier to manage imo.
One other boon to points is that since it's point dependant (and you have pools per level) the recovery time/ability to cast will not be fixed i.e. a week for anyone who casts a 7th level spelled followed by a 6th level spell. Spell point recovery based on level would be dependent upon level, feats, rest, environment, etc, and not just the spell level being cast having a fixed cool down timer. Once you gain enough points back and you can cast the spell again. Other variables (ex: fatigue and casting checks) can be instituted to manage balance/abuse issues.