C＃HttpClient.SendAsync始终返回404，但URL在浏览器中有效

我正在开发一个C＃控制台应用程序，用于测试URL是否有效。它适用于大多数URL。但是我们发现在某些情况下，应用程序始终会从目标站点获得404响应，但URL实际上在浏览器中有效。当我在DHC（Dev HTTP Client）等工具中尝试使用这些URL时，它们也可以使用。

I am developing an C# console application for testing whether a URL is valid or not. It works well for most of URLs. But we found that there are some cases the application always got 404 response from target site but the URLs actually work in the browser. And those URLs also works when I tried them in the tools such as DHC (Dev HTTP Client).

在一开始，我虽然可能是未添加正确标题的原因。但是，在尝试使用Fiddler编写具有相同标头的http请求之后，它就可以在Fiddler中使用。

In the beginning, I though that this could be the reason of not adding right headers. But after tried using Fiddler to compose a http request with same headers, it works in Fiddler.

那么我的代码有什么问题？ .NET HttpClient中是否存在任何错误？

So what's wrong with my code? Is there any bug in .NET HttpClient?

以下是我的测试应用程序的简化代码：

Here are the simplified code of my test application:

class Program
{
    static void Main(string[] args)
    {
        var urlTester = new UrlTester("http://www.hffa.it/short-master-programs/fashion-photography");

        Console.WriteLine("Test is started");

        Task.WhenAll(urlTester.RunTestAsync());

        Console.WriteLine("Test is stoped");
        Console.ReadKey();
    }


    public class UrlTester
    {
        private HttpClient _httpClient;
        private string _url;

        public UrlTester(string url)
        {
            _httpClient = new HttpClient 
            { 
                Timeout = TimeSpan.FromMinutes(1)
            };

            // Add headers
            _httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36");
            _httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate,sdch");
            _httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
            _httpClient.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4");

            _url = url;
        }

        public async Task RunTestAsync()
        {
            var httpRequestMsg = new HttpRequestMessage(HttpMethod.Get, _url);

            try
            {
                using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
                {
                    Console.WriteLine("Response: {0}", response.StatusCode);
                }
            }
            catch (HttpRequestException e) 
            {
                Console.WriteLine(e.InnerException.Message);
            }
        }
    }

}