For some reason the default controller factory throws an HttpException when a controller is not found:

protected internal virtual IController GetControllerInstance(RequestContext requestContext, Type controllerType) {
    if (controllerType == null) {
        throw new HttpException(404,
            String.Format(
                CultureInfo.CurrentUICulture,
                MvcResources.DefaultControllerFactory_NoControllerFound,
                requestContext.HttpContext.Request.Path));
    }
    // ...
}

Not sure why this was done this way. Maybe it's just my ignorance but it seems that it should directly return a 404 not an HttpException. Now if we're using a custom controller factory we can modify this behavior. Here is the ExtensibleControllerFactory (Mentioned earlier) modified to deal with not finding a controller:

public class ExtensibleControllerFactory : DefaultControllerFactory
{
    private readonly Func<Type, System.Web.Mvc.Controller> _factory;
    private readonly ControllerActionInvoker _actionInvoker;
    private readonly IController _notFoundController;

    public ExtensibleControllerFactory(
        Func<Type, System.Web.Mvc.Controller> factory) :
        this(factory, null, new NotFoundController()) { }

    public ExtensibleControllerFactory(
        Func<Type, System.Web.Mvc.Controller> factory, 
        ControllerActionInvoker actionInvoker) : 
        this(factory, actionInvoker, new NotFoundController()) { }

    public ExtensibleControllerFactory(
        Func<Type, System.Web.Mvc.Controller> factory,
        ControllerActionInvoker actionInvoker,
        IController notFoundController)
    {
        _factory = factory;
        _actionInvoker = actionInvoker;
        _notFoundController = notFoundController;
    }

    protected override IController GetControllerInstance(
        RequestContext requestContext,
        Type controllerType)
    {
        if (controllerType == null) return _notFoundController;
        var controller = _factory(controllerType);
        if (_actionInvoker != null) controller.ActionInvoker = _actionInvoker;
        return controller;
    }

    private class NotFoundController : IController
    {
        public void Execute(RequestContext requestContext)
        { requestContext.HttpContext.Response.StatusCode = 404; }
    }
}

Here if the controller cannot be found a special controller is returned that handles that situation. This implementation defaults to using one that simply returns a 404 but you can pass in a custom one if desired.

Our site receives literally hundreds of 404's daily (Usually from hackers fishing, IE http://www.oursite.com/admin/login.php). Generating an exception for these is completely unnecessary (for us anyways) and we just want a 404 to be directly returned. This approach enables you to do that.