About compiler design

It is a big question you're posting. In general what you want is an engine that is capable of translating a finite state machine in code that is executed and then traced. You're better off looking for some library that provides such environment.

There are a lot of tools that can greatly help you on that.
Have a look at classic flex[^] and bison[^] or to 'modern' boost's spirit[^].
If you're not stuck with C/C++, then a look at (java-based) ANTLR[^] is worthy.
Finally, if you want to have even some fun, check out Lua[^] and its powerful LPeg library[^].
:)

This is a really huge thing that you are attempting but it is doable. How else do we have all the compilers that we use? Even their creators had to start somewhere.

Start writing a simple arithmetic expression parser. Try it by yourselves first before looking at a working code.

Then I suggest you read and understand a simple working code like that calculator example given in Strourstrup's The C++ Programming Language. That is what every compiler/interpreter must do.

You may then move on to something a bit more useful like an interpreter. There are scores of material to help you with that.

Compiler must be attempted only after these steps. It is more than just parsing text and taking actions. First, you must be very VERY good with the language for which you want to build the compiler(you can't implement proper rules without knowing them all properly). Then you must have understanding of the low level representation of high level code. There are more such advanced stuff you will encounter, all of which will appear more clear as you progress.

Here's another parser builder: Gold Parser Builder[^]. It's free; you give it a syntax document and it generates state machines that identify the tokens (keywords, constants, symbols, etc.) in the input and invokes your code that implements the rules of the grammar.