"But are my fears justified?"
No. Not even slightly!
Memory fragmentation is not caused by how you access the memory, it's caused by allocating and freeing chunks of memory in such a way that lots of smaller, useless holes appear that can't be reused as a "bigger chunk" because they aren't contiguous.
There is a good description here:
Developer Library » Memory Fragmentation, your worst nightmare[
^] - and it has pictures, without which it's really, really difficult to explain what is happening!
But accessing your pointers via pointer arithmetic doesn't have anything to do with fragmentation, as it doesn't cause any new allocation of memory or return any memory to the pool - those have to be explicit in C via malloc and free