I think you will need to set the
Condition property on each service/custom action/folder depending on how your setup is organized. The
Condition will look at a Custom Property and determine based on the value of the property whether or not to install. The checkboxes on your custom dialog would set the value of the Custom Property.
This should get you started:
Customizing Windows Installer Packages [
^]
Note: You may need to use Orca (or some other tool for editing MSI databases) to create your Custom Property.