The following proposal is based on the concept that the devs don't have infinite time to dedicate to this system, so our suggestions should be easy to implement.
We have some quite different opinions on this topic, but I guess that we all agree on the point that the minimum penalty is too high and that we would like a bit more flexibility in "Evilness".
The risk vs reward is present currently, but it is very binary. By being red you risk a lot for a lot of reward. Many would probably like the option to be evil but with less reward and less risk.
There are also a few players which would want pvp kills to be less punishing and less rewarding, tipically by implementing partial looting. Personally when I hear talks of partial looting in a full loot game, I die a little inside, but in this particular case I think that we can find a match.
In particular, I suggest the following:
Allow an action from a Red player to rummage through the belongings of a knocked down player. This doesn't open a loot window, but just rewards the Red player with half the gold in that player's bag. Most importantly, this happens without the player getting killed, just knocked down. The Red player loses karma when doing this, but a lot less than performing a murder.
This small change actually has a lot of effects, because we are now distinguishing between the archetypes of "bandit" and "murderer". You can be a red player which goes around assaulting and robbing as a style of play, without fully murdering, but still looking for a good fight. Maybe stealing some carts while at it. Such a red player when he is finally taken down will have a low penalty, since he was less oppressive on the other players.
You can obviously still be a murderer with the same current system, which is fine. You choose the most evil route and want the highest rewards, so when you get killed, you suffer harsh consequences.
Being assaulted by a bandit means losing half your gold and part of your health bar. You keep your equip, half your gold and your loot. It is far less crippling and I think that it meets the needs of the pve/pvp crowd.
In addition to this, the action of knocking down a player should give a karma penalty which depends on how purple was the health bar of that player. The weaker he was, the more "evil" it is to attack them, since it is more crippling for them. Attacking someone at full health bar and robbing them of some gold is a lesser evil, since the player will then just keep doing what he was already doing. Instead if the player was at 30% purple health bar before the assault, it means that you attacked a weak target (less risk for you) and that he now probably has to go back (more crippling), which means that you lose more karma.
Now, we can talk about a less punishing system for the not so evil players. The current jail system is fine for any karma lower than -3k. At -3k it should correctly start from 24h, since you most likely murdered someone or robbed a lot of people (Any act of murdering shall immediately set you to at least -3k, because you are now a murderer).
If you are not evil enough when caught, the jail system is not good.
If we implement something like 500 gold and 2 hours, hunting them will not be worthwile for the Sheriff, which means that we will miss the sheriff/bandit player interaction. As such, the gold penalty imposed should be still be quite high even at low evil levels, but you are not sent to jail, you can play normally. Your bounty is posted outside the jail and you can pay it by yourself or others can do it, like the current system. You have 1 week to pay it. If you don't or if you ever set yourself evil again in that time, your karma becomes immediately -3k minus your previous karma, which means that if you are caught again there are at least 24 hours of jail waiting for you.
So, all in all this system introduces with relatively few changes (mostly numerical changes and re using what is already there) a way for players to play the non consensual pvp part of the game without suffering the very harsh consequences of the current jail system, while being less oppressive on the victims. The old system is still in place for those players that wish to be very evil and cause big troubles for the victims.