I want to ask a question about C programming

Neglecting user presentation issues which you have done zero of lets look at the code you have given us

scanf("%d",&a[i]);
while(a[i]<=10||a[i]>=100)
  scanf("%d",&a[i]);

Will accept the first entry whether or not it is valid, then check for validity on the next line.

while(a[i]<=10||a[i]>=100)
  scanf("%d",&a[i]);

Those lines are correct and all that is needed as it does the required check.

Then all you need to do is check if the number already exists .. you just got entry a[i].
The valid prior entries are in array a[0]... a[i-1]

So all you need to do is check if a[i] matches any entry in a[0] .. a[i-1] ... if it does don't accept it.

You go off into lala land with j and k loops .. HINT you need only 1 loop which goes from a[0] to a[i-1] read sentence above.
Hint you can simply undo the increment of i, if you need to dump the last value.
Final hint you need only two loops and 1 if statement besides your check statement above to do the job efficiently.

Finish the code then look at presentation and improvements and removing unnecessary stuff.

SOLUTION AS YOU GOT STUCK:

int a[5];
int i;
// For each of the required 5 numbers
for (i = 0; i < 5; i++){
    // The check .. Must be first a valid from Scanf (you forgot that) ...and then must be not less that 10 .. not greater than 100
    // READ Scanf SPECS ... On success, the function returns the number of items filled ... SO YOU WANT 1.
    // Blind loop until all three criteria met ,, scanf returns 1 and number between 10-100
    while ( (scanf("%d", &a[i]) != 1) || (a[i] <= 10) || (a[i] >= 100) ) {};

    // Check previous entries since we have a valid one
    // If you look i = 0 (first number) will not enter loop as j can not be less than i
    int j;
    for (j = 0; j < i; j++){
        // look for a match a[i] == a[j]
        if ((a[j]) == (a[i])){
            // You have a match so the current number is invalid IT EXISTS ALREADY
            // So decrement i so the increment is undone
            i--;
            // then break out of the j loop
            break;
        }
    }
}