The location of the point which was

*originally*at the bottom right is (w, h) in local coordinates. The local axes become x'=(cos a, sin a) and y'=(-sin a, cos a) so the location of that point becomes (w cos a - h sin a, w sin a + h cos a), with (w,h) = the original size of the rectangle and a = the angle through which you have rotated it (anticlockwise).

You can do the same thing for the other three points, whose local coordinates relative to the top left (you state that this is the origin) are (0,0), (w,0) and (0,h), meaning that their rotated locations are:

TL = (0, 0)

TR = (w cos a, w sin a)

BL = (-h sin a, h cos a)

You can then find out which is the most 'bottom right' by finding which has the largest dot product with the direction vector to specify 'bottom-rightness' which is (1, 1)* – which is simply the sum of coordinates, but you should understand why.

*: Actually the standard direction vector is norm(1,1) = (0.707, 0.707), but as we are comparing values to each other, the rescaling doesn't matter in this case.