Sounds like a R4C situation.

Perhaps a miscellaneous case file at the USDC is in order, so you can record your refusal for cause there.

I'm sure there is a lot "they can do", because "they" are probably under the impression that "they" own you via the many carefully crafted hidden adhesion contracts, such as BC, DL, SSN, or the mere fact of your existence in that particular space and time.

Without knowing any more details, such as what agency is trying to contract with you and for what, it's impossible to tell what a prudent course of action might be, except to say:
do a cost-benefit analysis before you get into a fight, know who your allies are, and most importantly, know who's your Daddy?