A lot is personal preference. You want just enough swingweight to feel the clubhead throughout the swing, but not too much where it's cumbersome to swing. I usually start around D0 and work from there.
The best thing to do is to try a few clubs and determine which one feels best (in weight) for your swing and the results that come from it and you have a good comfortable swingweight.
Don't go off of feel only because what feels good in your hands might not swing properly for you.
SW is going to effect flex. The higher the sw goes the more flexible the shaft becomes. Tape 1 dime at a time to the back of your clubhead. This equals 1 sw point. If you like it add 2 grams. If two dimes feels good add 4 grams of lead tape to back center of club.
My personal preference is progressive swingweights.
Swingweight is supposed to be a relative measure of how much force is required to swing a club. Most clubs are made to conform to a very narrow swingweight range. While it is necessary to make the clubheads progressively lighter for the longer clubs, I think they should not be made light enough to keep the swingweights in a narrow range. Centrifugal force and momentum are extremely important in the classic golf swing. The amount of centrifugal force and momentum created depends on the total weight of a club and on clubhead speed.
In my opinion, the swingweight for the pitching wedge in a man’s set should be at least D2 for regular flex shafts and at least D4 for stiff shafts. For women’s clubs, it should be at least C6. For each ½ inch longer club, the swingweight should go up by ½ point. So if your pitching wedge is 36 inches and D2, your 44 inch driver should be E0 (8 points higher). Would you then need stiffer shafts in your metalwoods? Maybe, but probably not. 8 points is not a huge increase in swingweight. The difference you would notice would be that the timing and extension with your longer clubs would be easier, and they would feel a bit more like your short irons. With the increased centrifugal force, your clubhead speed would probably not be reduced at all.
Swingweight is all about feel, there is not a right or wrong swingweight, it's just a number. Olazabel has the reputation of being able to tell the difference between 1 or 2 SW points. Average golfers (non-tour players) like us generally cannot tell the difference between a few SW points. You'll get the guy that says he can of course. Put 2 clubs out, ask which has a higher SW, he's got a 50% chance of being right. Put 30 clubs out, 29 all the same SW, 1 is 2 SW points higher, watch what happens. If you want a swingweighted set, the real issue would be consistency, within a SW point of each other, excluding wedges of course.
BG1 has brought up a good point. He is referring to MOI matching. When I MOI match a set, it's not "etched in stone", but each club generally gets 1 to 1.5 SW points heavier as I go. If you can build a set that requires the SAME amount of force to move each club, that's MOI matching, every club feels the same, generally equates to consistency of feel.
BG1: You say that 8 sw points isn't that much, I can't agree with that. I can sure tell the difference in my D6 irons, compared to my old set that are D2. And I happen to have a LW that is E4, and it has a totally different feel then the D6 SW. I can't see too many golfers playing with a 45 inch driver that's a E1, as you mention. I just got two new drivers, and they are both D0, even with the length is 46 inches. Granted they do feel rather light, but I don't think I'd like them at E1 or E2, as you state. Who plays a driver with a E1 swing weight that you know of?
Back to the original post, if you are quick from the top you may want to look into counterweighting, which would in fact lower the swingweight. Total clubweight plays a bigger role than swingweight does in helping with tempo. At least this has been my experience.
When I was in my 20s, I used E4 for woods and D9 for irons. At 35 I was down to D8 woods and D5 irons. Now at 65 my woods are D5 and the irons are D2. I've heard that in his 30s Jack Nicklaus swung a C10 driver. I guess it's what you feel comfortable with rather than any figures off a chart.