Evidently the aspnet_compiler.exe does not care about .refresh files (From my tests and what I read on the internets). It would be nice if it did as it would reduce the steps needed to build a WSP (Web Site Project) on a build server. I ended up just manually copying the assemblies into the bin folder before the build as follows in this nant script:

<mkiisdir dirpath="..\BabelFish.ServiceHost.Web" vdirname="BabelFish.ServiceHost.Web" />

<copy todir="..\BabelFish.ServiceHost.Web\bin">
fileset basedir="..\BabelFish.Services\bin">
include name="*.dll" />

<exec program="C:\WINDOWS\Microsoft.NET\Framework\${framework.version}\aspnet_compiler.exe" useruntimeengine="true">
arg value="-p" />
arg value="..\BabelFish.ServiceHost.Web" />
arg value="-v" />
arg value="BabelFish.ServiceHost.Web" />

<deliisdir vdirname="BabelFish.ServiceHost.Web" />