Figured it out

So the solution involved "correctly" overriding both Measure and Arrange. I have to make sure I call Measure() and Arrange() on all of the children in the Adorner to make sure that I report everything correctly. Note that DesiredSize isn't valid unless you call Measure() on an object.