One of the things I really wanted with our REST API is per method basic authentication. So lets say we have a library with a books resource. We want everyone to be able to read this resource but we want people to authenticate when modifying this resource. As far as I could tell there is no way to do this OOB with WCF REST without resorting to breaking your resource up into two parts, read and write and hosting the secure one in a separate IIS app with basic auth enabled. The only way I could figure out how to do this properly is implement basic auth in an operation invoker. This way you could define authentication on a per method basis as follows (With the BasicAuthenticationInvoker) without breaking the resource up:

[ServiceContract]
public interface IBookService
{
    [WebGet(UriTemplate = "/{isbn}")]
    [OperationContract]
    Book GetBook(string isbn);
            
    [WebInvoke(UriTemplate = "/{isbn}", Method=Verbs.Delete)]
    [OperationContract]
    [BasicAuthenticationInvoker]
    void DeleteBook(string isbn);
}

It's pretty simple to do this and doesn't require integration with IIS. First define a class that will act as a behavior attribute and an operation invoker:

public class BasicAuthenticationInvoker : Attribute, IOperationBehavior, IOperationInvoker 
{
}

Next, implement the operation behavior. We will store the original invoker to call if the user successfully authenticates, our invoker will basically act as a proxy. We can ignore the other three implemented methods.

#region Private Fields

    private IOperationInvoker _invoker;

#endregion

#region IOperationBehavior Members

    public void ApplyDispatchBehavior(OperationDescription operationDescription, 
        DispatchOperation dispatchOperation)
    {
        _invoker = dispatchOperation.Invoker;
        dispatchOperation.Invoker = this;
    }

    public void ApplyClientBehavior(OperationDescription operationDescription, 
        ClientOperation clientOperation) { }
    public void AddBindingParameters(OperationDescription operationDescription, 
        BindingParameterCollection bindingParameters) { }
    public void Validate(OperationDescription operationDescription) { }

#endregion

Now implement the operation invoker. First we call our private authenticate method, if this is successful we'll call the invoker, otherwise we return nothing.

#region IOperationInvoker Members

    public object Invoke(object instance, object[] inputs, out object[] outputs)
    {
        if (Authenticate("New York Public Library"))
            return _invoker.Invoke(instance, inputs, out outputs);
        else
        {
            outputs = null;
            return null;
        }
    }

    public object[] AllocateInputs() { return _invoker.AllocateInputs(); } 

    public IAsyncResult InvokeBegin(object instance, object[] inputs, 
        AsyncCallback callback, object state)
    { throw new NotSupportedException(); }

    public object InvokeEnd(object instance, out object[] outputs, IAsyncResult result)
    { throw new NotSupportedException(); }

    public bool IsSynchronous { get { return true; } }

#endregion

The private authentication methods are as follows. The authenticate method checks the username and password and if successful returns true, otherwise it sets the authenticate header and the status code to unauthorized.

private bool Authenticate(string realm)
{
    string[] credentials = 
        GetCredentials(WebOperationContext.Current.IncomingRequest.Headers);

    if (credentials != null && 
        credentials[0] == "tony" && 
        credentials[1] == "clifton") return true;

    WebOperationContext.Current.OutgoingResponse.Headers["WWW-Authenticate"] = 
        string.Format("Basic realm=\"{0}\"", realm);
    WebOperationContext.Current.OutgoingResponse.StatusCode = 
        HttpStatusCode.Unauthorized;
    return false;
}

private string[] GetCredentials(WebHeaderCollection headers)
{
    string credentials = WebOperationContext.Current.IncomingRequest.
        Headers["Authorization"];
    if (credentials != null) credentials = credentials.Trim();

    if (!string.IsNullOrEmpty(credentials))
    {
        try
        {
            string[] credentialParts = credentials.Split(new char[] { ' ' });
            if (credentialParts.Length == 2 && 
                credentialParts[0].Equals("basic", 
                StringComparison.OrdinalIgnoreCase))
            {
                credentials = ASCIIEncoding.ASCII.GetString(
                    Convert.FromBase64String(credentialParts[1]));
                credentialParts = credentials.Split(new char[] { ':' });
                if (credentialParts.Length == 2) return credentialParts;
            }
        }
        catch { }
    }

    return null;
}

This same approach could also be used to examine parameters as part of the authentication process (If you are using tokens or the like).