I stick everything in also including meat scraps. The worms devour it all in short time.
I have geckos that live under the lid to get every tiny fly type insect that hatches from the compost.
Actually it doesn't hurt if you have a community of sawflies living in there because they break down the larger pieces for the worms.
If you were to put a very large number of citrus skins in there every day, that would certainly be to the detriment of the worms or microbiome. But a few each week is nothing that any self-respecting compost worm can't handle!
And another reason for using them is if you have some slatters, they will go live in the upside-down citrus skin so you can simply take the whole skin out complete with its slatters and burn or drown in the weed tea bucket. But remember, they can walk on water due to surface tension.