Ajax.BeginForm has got two events for you. OnSuccess and OnFailure. The value for these are call back methods .(event handlers). So what you do in your controller for OnSuccess is wrong. OnSuccess keyword has nothing to do with server side controllers.
What you need to do is create two javascript functions one for OnSuccess and one for OnFailure. The OnFailure function will be called for any exceptions that occur during your JQuery call. OnSuccess will be called if there is no exception occurred.
In your scenario, return a flag to OnSuccess event handler and check the value of flag and accordingly hide or show your modal.
These examples will give you an idea
Using Ajax.BeginForm() with ASP.NET MVC[
^]
Ajax modal forms in ASP.Net MVC |[
^]