Rubber band Zooming functionality

There are many ways to implement zoom functionalty in a graphic system.
Suppose your final display is in a texture and that is displayed to the screen. For unzoomed display, usually vertex buffer created with texture coordinate as follows, maps entire texture(X coordinate from 0 to 1 and Y coordinate from 0 to 1)

// Format x,y,z, TU, TV
GLFloat fVertexBuffer[]
{ -1.0f,1.0f, 0.0f,    0.0f, 1.0f, // Left Top  corner
 -1.0f,-1.0f, 0.0f,   0.0f, 0.0f), // Left Bottom
 1.0f , -1.0f, 0.0f,  1.0f, 0.0f, // Right bottom
 1.0f, 1.0f,  0.0f,   1.0f, 1.0f // Right top
};

To get a zoomed image of the final texture, you can map small area of the texture to screen.

// Format x,y,z, TU, TV
GLFloat fVertexBufferZoom[]
{ -1.0f,1.0f, 0.0f,    0.25f, 0.75f, // Left Top  corner
 -1.0f,-1.0f, 0.0f,   0.25f, 0.25f), // Left Bottom
 1.0f , -1.0f, 0.0f,  0.75f, 0.25f, // Right bottom
 1.0f, 1.0f,  0.0f,   0.75f, 0.75f // Right top
};

Another suggestion is to change the projection based on your zoom area. If you are using orthographic projection, you can simply change the projection area based on the zoom area selected by the rectangle.

For more details on texture mapping and projection, refer any opengl or graphics reference documents.

Here is a OpenGL tutor, can be used to get more idea about opengl projection and texture mapping conceps.
http://user.xmission.com/~nate/tutors.html