Neglecting user presentation issues which you have done zero of lets look at the code you have given us
scanf("%d",&a[i]);
while(a[i]<=10||a[i]>=100)
scanf("%d",&a[i]);
Will accept the first entry whether or not it is valid, then check for validity on the next line.
while(a[i]<=10||a[i]>=100)
scanf("%d",&a[i]);
Those lines are correct and all that is needed as it does the required check.
Then all you need to do is check if the number already exists .. you just got entry a[i].
The valid prior entries are in array a[0]... a[i-1]
So all you need to do is check if a[i] matches any entry in a[0] .. a[i-1] ... if it does don't accept it.
You go off into lala land with j and k loops .. HINT you need only 1 loop which goes from a[0] to a[i-1] read sentence above.
Hint you can simply undo the increment of i, if you need to dump the last value.
Final hint you need only two loops and 1 if statement besides your check statement above to do the job efficiently.
Finish the code then look at presentation and improvements and removing unnecessary stuff.
SOLUTION AS YOU GOT STUCK:
int a[5];
int i;
for (i = 0; i < 5; i++){
while ( (scanf("%d", &a[i]) != 1) || (a[i] <= 10) || (a[i] >= 100) ) {};
int j;
for (j = 0; j < i; j++){
if ((a[j]) == (a[i])){
i--;
break;
}
}
}