My assumptions:
- You are interesting in a 2D Scenario
- You have a set of N points (only intersting in case N > 3)
A General solution for this 2D is:
Determine the convex hull for your set of Points. How to do that you can find e.g.
https://en.wikipedia.org/wiki/Quickhull[
^]
Every Point which is not in the result of the convex hull is inside. Points of the convex hull are "at the border".
You can that extend also for 3D Scenarios.
I hope it helps.
Regards