When you get the coordinates of your Mouse on click, grab the label control that exists in that position. Then you just have to offset your panel based on the labels position instead of the mouse position.
Here is a thread on how to achieve that:
WPF Get Elements under mouse[
^]