Somehow one of your numbers is infinite.
Let iz be ipBo.z2bcxS.
I'd imagine the subexpression (xStep - iz) is yielding infinity because either:
(i) xStep is 0 and so is iz. Or
(ii xStep is equal to iz.
Either way, the results of rotatedX, rotatedY and rotatedXS are now infinity (any operation on infinity typically yields infinity or NaN).
So when you attempt to (implictly) convert it to a decimal it fails, because the decimal type cannot represent infinite numbers of NaN.
Hope this helps.
See my comment above (accidentally posted in RyanDev's response) for more suggestions. With consideration It seems likely that the issue is writing a double to a decimal - I don't think the DataRow will, by default, attempt any conversions. The simplest solution to this may be to write using:
dr["RotatedX^2"] = (decimal)rotatedXS;
I'd still make some of the other changes I suggested - using type-safe data access methods can help catch such errors earlier.