So cheking some damage table son the wiki, i came to the conclusion that it's better to leave all your swords at 4* (the damage differences are of no more than 20 points) and upgrade them later, but first get a krogmo trinket. With the bonus trinket, you'd get even more damage on all weapons at once instead of needing to upgrade them one by one. This may make an impact on some upgrade plans.
Is this true? (to non Skolver + BTS users of course)
I don't exactly know, but from what I've seen, each level of damage bonus (Low, Medium, High, etc.) adds 5%-6% to your damage. Using this fact and some of the damage tables on the wiki, maybe you can figure it out.
Another thing to consider is that weapons are weakened, when you use them in shallower strata than they're designed for (e.g. 5-star in Stratum 4). If your goal is to do jelly runs, for example, then 5-star weapons are little better than 4-star weapons anyway. Another thing to consider is that some items become significantly different when they reach 5-star; for example, the DA charge attack gets three projectiles.