I finally connected why this bothers me.
Inherited slavery is WRONG! Because the debt that landed the father in slavery should not be transferred to the son. That assumes that the father was justly enslaved (not as far fetched as it first appears, think about it).
Blacks from Africa, for the most part, were enslaved for losing some local war. Enslavement was part of losing these things. Often the slave's children were raised as part of the conquering tribe, thus not slaves. When they got here though, their kids became slaves, and their kids, and their kids ad nauseum. That's how chattel slavery works.
If chattel slavery is wrong because the children who have done nothing to deserve being enslaved then blaming me for the oppression of blacks when I have never owned a slave is also wrong; regardless of whether gramps owned thousands. And while "freeing the slaves" was not why the Union government fought the war, it WAS the recruiting slogan that got many to join and fight. The debt to the slaves was paid in blood, by white men mostly. I guess if you don't consider that enough we can put you back into chains because I certainly don't wanna pay as much as the people who freed your ancestor paid.
It's even more galling to me that the descendants of slaves, who have never been slaves themselves, demand that I, who have never owned a slave, pay for the crimes of my forebears against their forebears. Never mind that my forebears didn't get here until 1873 on my Dad's side and 1924 on my Mom's. Wasn't there, didn't do it are positive defenses!
I have some of the same issues with the fixes being as bad as the original wrong with the wealth that was stolen from WW2 Jews. The people who own it now are very unlikely to be the person who stole it. The heirs of the original owner are demanding that the current possessor fork it over regardless of how it came into their possession. I am sorry the thieves got away with your stuff, but they paid in blood for the theft (remember that whole WW2 thing?). Want us to put Nazi Germany back so you can go through their courts? Any takers? Anyone?
The war crimes trials in Israel are another thing that bugs me. Isreal did not exist until 1947. That nation trying nazis for crimes committed in 1939-1945 seems ex post factum to me. Prior to 1947 it was impossible to break Israeli law because Israel did not exist. The war criminals also did not break Nazi German laws when they committed their "crimes". I do not think they should have gone unpunished for their deeds, just that the wrong government entity was doing the trial.