Pretty simple:
This just creates the list of files to keep:
List<FileInfo> keepers = Directory.GetFiles(@"D:\Temp", "*.jpg").Select(f => new FileInfo(f)).ToList();
Then just:
string[] files = Directory.GetFiles(@"D:\Temp");
var deleteList = files.Except(keepers.Select(fi => fi.FullName));
foreach (string s in deleteList)
{
File.Delete(s);
}