Rushing the main plot does accelerate the process, of course. Not to mention that Witcher 3 is a very long game and, if you are thorough with it, you will gain ability points once every three hours or so. And while one ability point might not seem that much of an investment, don’t forget that you also need to spend an entire ability slot on this and those are much more precious. So, as long as you stock up on alcohest (which isn’t even that expensive – just 25 crowns a pop, not to mention you’ll find plenty of it in the random loot), you can still rest to regen quite effectively.
Sure, meditation ceases to replenish your vitality there, yet it still recharges your Swallow potion. Sun and Stars: seems like a nice early game option for the last two difficulties but, in truth, it isn’t that great.