first of all you have to retrive datatable in form of json or xml an client side using ajax.
after that use this..
window.open('data:application/vnd.ms-excel,' + your data(in json or xml format));
Another way create html table as per your datatable formate (hidden).
window.open('data:application/vnd.ms-excel,' + $('#yourDivData').html());