I am by no means an expert but when it comes to composting, it all breaks down eventually.
I thought it was more worm farms that you didn't want those things in, But I too put all that stuff into my compost, the only things that are definite no no's are diseased or insect infested waste.
I stick everything in also including meat scraps. The worms devour it all in short time.
I have geckos that live under the lid to get every tiny fly type insect that hatches from the compost.
Actually it doesn't hurt if you have a community of sawflies living in there because they break down the larger pieces for the worms.
If you were to put a very large number of citrus skins in there every day, that would certainly be to the detriment of the worms or microbiome. But a few each week is nothing that any self-respecting compost worm can't handle!
And another reason for using them is if you have some slatters, they will go live in the upside-down citrus skin so you can simply take the whole skin out complete with its slatters and burn or drown in the weed tea bucket. But remember, they can walk on water due to surface tension.
I do add citrus peel to my biochar along with some compost into my biochar to breakdown/inoculate it over the winter. I am not sure if it works but I am hoping the acid in the peel will offset the alkaline nature of my biochar.
If you are really concerned about citrus peel in the compost - what you can do is to dry it out (on the window sill will do it nicely) then boil it in your jug or kettle to get rid of the lime scale that builds up (leave it overnight sitting in the water to get a better result) and then after the oil has been boiled out of the peel - then you can add it to the compost or worm farm quite happily.
I have two compost areas, one with introduced worms and one that is just straight compost and any native worms that happen to find their way in. I try to limit citrus peels, meat, and onions in the worm compost, but I dump everything into the other area.