HttpClient 代理使用 proxy   https://www.cnblogs.com/yxnyd/p/9801396.html HttpClient(二)HttpClient使用Ip代理与处理连接超时

HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时

https://www.cnblogs.com/yxnyd/p/9801396.html

HttpClient(二)HttpClient使用Ip代理与处理连接超时

 

前言

  其实前面写的那一点点东西都是轻轻点水,其实HttpClient还有很多强大的功能:

  (1)实现了所有 HTTP 的方法(GET,POST,PUT,HEAD 等)
  (2)支持自动转向
  (3)支持 HTTPS 协议
  (4)支持代理服务器

一、HttpClient使用代理IP

1.1、前言

  在爬取网页的时候,有的目标站点有反爬虫机制,对于频繁访问站点以及规则性访问站点的行为,会采集屏蔽IP措施。
  这时候,代理IP就派上用场了。可以使用代理IP,屏蔽一个就换一个IP。
  关于代理IP的话 也分几种 透明代理、匿名代理、混淆代理、高匿代理,一般使用高匿代理。    

1.2、几种代理IP

  1)透明代理(Transparent Proxy)

    REMOTE_ADDR = Proxy IP
    HTTP_VIA = Proxy IP
    HTTP_X_FORWARDED_FOR = Your IP
    透明代理虽然可以直接“隐藏”你的IP地址,但是还是可以从HTTP_X_FORWARDED_FOR来查到你是谁。

  2)匿名代理(Anonymous Proxy)

    REMOTE_ADDR = proxy IP
    HTTP_VIA = proxy IP
    HTTP_X_FORWARDED_FOR = proxy IP
    匿名代理比透明代理进步了一点:别人只能知道你用了代理,无法知道你是谁。
    还有一种比纯匿名代理更先进一点的:混淆代理

  3)混淆代理(Distorting Proxies)

    REMOTE_ADDR = Proxy IP
    HTTP_VIA = Proxy IP
    HTTP_X_FORWARDED_FOR = Random IP address
    如上,与匿名代理相同,如果使用了混淆代理,别人还是能知道你在用代理,但是会得到一个假的IP地址,伪装的更逼真。

  4)高匿代理(Elite proxy或High Anonymity Proxy)

    REMOTE_ADDR = Proxy IP
    HTTP_VIA = not determined
    HTTP_X_FORWARDED_FOR = not determined
    可以看出来,高匿代理让别人根本无法发现你是在用代理,所以是最好的选择。
    一般我们搞爬虫 用的都是 高匿的代理IP;
    那代理IP 从哪里搞呢 很简单 百度一下,你就知道 一大堆代理IP站点。 一般都会给出一些免费的,但是花点钱搞收费接口更加方便。

1.3、实例来使用代理Ip

  使用 RequestConfig.custom().setProxy(proxy).build() 来设置代理IP  

HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
package com.jxlg.study.httpclient;

import com.sun.org.apache.regexp.internal.RE;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.IOException;

public class UseProxy {
    public static void main(String[] args) throws IOException {
        //创建httpClient实例
        CloseableHttpClient httpClient = HttpClients.createDefault();
        //创建httpGet实例
        HttpGet httpGet = new HttpGet("http://www.tuicool.com");
        //设置代理IP,设置连接超时时间 、 设置 请求读取数据的超时时间 、 设置从connect Manager获取Connection超时时间、
        HttpHost proxy = new HttpHost("58.60.255.82",8118);
        RequestConfig requestConfig = RequestConfig.custom()
                .setProxy(proxy)
                .setConnectTimeout(10000)
                .setSocketTimeout(10000)
                .setConnectionRequestTimeout(3000)
                .build();
        httpGet.setConfig(requestConfig);
        //设置请求头消息
        httpGet.setHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36");
        CloseableHttpResponse response = httpClient.execute(httpGet);

        if (response != null){
            HttpEntity entity = response.getEntity();  //获取返回实体
            if (entity != null){
                System.out.println("网页内容为:"+ EntityUtils.toString(entity,"utf-8"));
            }
        }
        if (response != null){
            response.close();
        }
        if (httpClient != null){
            httpClient.close();
        }
    }
}
HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时

1.4、实际开发中怎么去获取代理ip

  我们可以使用HttpClient来 爬取 http://www.xicidaili.com/ 上最新的20条的高匿代理IP,来保存到 链表中,当一个IP被屏蔽之后获取连接超时时,

  就接着取出 链表中的一个IP,以此类推,可以判断当链表中的数量小于5的时候,就重新爬取 代理IP 来保存到链表中。

1.5、HttpClient连接超时及读取超时

  httpClient在执行具体http请求时候 有一个连接的时间和读取内容的时间;

  1)HttpClient连接时间

    所谓连接的时候 是HttpClient发送请求的地方开始到连接上目标url主机地址的时间,理论上是距离越短越快,

    线路越通畅越快,但是由于路由复杂交错,往往连接上的时间都不固定,运气不好连不上,HttpClient的默认连接时间,据我测试,

    默认是1分钟,假如超过1分钟 过一会继续尝试连接,这样会有一个问题 假如遇到一个url老是连不上,会影响其他线程的线程进去,说难听点,

    就是蹲着茅坑不拉屎。所以我们有必要进行特殊设置,比如设置10秒钟 假如10秒钟没有连接上 我们就报错,这样我们就可以进行业务上的处理,

    比如我们业务上控制 过会再连接试试看。并且这个特殊url写到log4j日志里去。方便管理员查看。

  2)HttpClient读取时间

    所谓读取的时间 是HttpClient已经连接到了目标服务器,然后进行内容数据的获取,一般情况 读取数据都是很快速的,

    但是假如读取的数据量大,或者是目标服务器本身的问题(比如读取数据库速度慢,并发量大等等..)也会影响读取时间。

    同上,我们还是需要来特殊设置下,比如设置10秒钟 假如10秒钟还没读取完,就报错,同上,我们可以业务上处理。     

  比如我们这里给个地址 http://central.maven.org/maven2/,这个是国外地址 连接时间比较长的,而且读取的内容多 。很容易出现连接超时和读取超时。  

  我们如何用代码实现呢?

  HttpClient给我们提供了一个RequestConfig类 专门用于配置参数比如连接时间,读取时间以及前面讲解的代理IP等。

  例子:

HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
 
package com.jxlg.study.httpclient;

import org.apache.http.HttpEntity;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.IOException;

public class TimeSetting {
    public static void main(String[] args) throws IOException {
        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpGet httpGet = new HttpGet("http://central.maven.org/maven2/");
        RequestConfig config = RequestConfig.custom()
                .setConnectTimeout(5000)
                .setSocketTimeout(5000)
                .build();
        httpGet.setConfig(config);
        httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36");
        CloseableHttpResponse response = httpClient.execute(httpGet);
        if (response != null){
            HttpEntity entity = response.getEntity();
            System.out.println("网页内容为:"+ EntityUtils.toString(entity,"UTF-8"));
        }
        if (response != null){
            response.close();
        }
        if (httpClient != null){
            httpClient.close();
        }
    }
}
HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
 

由于项目中有用到HttpClient异步发送大量http请求,所以做已记录

思路:使用HttpClient连接池,多线程

 
public class HttpAsyncClient {
    private static int socketTimeout = 500;// 设置等待数据超时时间0.5秒钟 根据业务调整

    private static int connectTimeout = 2000;// 连接超时

    private static int poolSize = 100;// 连接池最大连接数

    private static int maxPerRoute = 100;// 每个主机的并发最多只有1500

    private static int connectionRequestTimeout = 3000; //从连接池中后去连接的timeout时间

    // http代理相关参数
    private String host = "";
    private int port = 0;
    private String username = "";
    private String password = "";

    // 异步httpclient
    private CloseableHttpAsyncClient asyncHttpClient;

    // 异步加代理的httpclient
    private CloseableHttpAsyncClient proxyAsyncHttpClient;

    public HttpAsyncClient() {
        try {
            this.asyncHttpClient = createAsyncClient(false);
            this.proxyAsyncHttpClient = createAsyncClient(true);
        } catch (Exception e) {
            e.printStackTrace();
        }

    }

    public CloseableHttpAsyncClient createAsyncClient(boolean proxy)
            throws KeyManagementException, UnrecoverableKeyException,
            NoSuchAlgorithmException, KeyStoreException,
            MalformedChallengeException, IOReactorException {

        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectionRequestTimeout(connectionRequestTimeout)
                .setConnectTimeout(connectTimeout)
                .setSocketTimeout(socketTimeout).build();

        SSLContext sslcontext = SSLContexts.createDefault();

        UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(
                username, password);

        CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, credentials);

        // 设置协议http和https对应的处理socket链接工厂的对象
        Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder
                .<SchemeIOSessionStrategy> create()
                .register("http", NoopIOSessionStrategy.INSTANCE)
                .register("https", new SSLIOSessionStrategy(sslcontext))
                .build();

        // 配置io线程
        IOReactorConfig ioReactorConfig = IOReactorConfig.custom().setSoKeepAlive(false).setTcpNoDelay(true)
                .setIoThreadCount(Runtime.getRuntime().availableProcessors())
                .build();
        // 设置连接池大小
        ConnectingIOReactor ioReactor;
        ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
        PoolingNHttpClientConnectionManager conMgr = new PoolingNHttpClientConnectionManager(
                ioReactor, null, sessionStrategyRegistry, null);

        if (poolSize > 0) {
            conMgr.setMaxTotal(poolSize);
        }

        if (maxPerRoute > 0) {
            conMgr.setDefaultMaxPerRoute(maxPerRoute);
        } else {
            conMgr.setDefaultMaxPerRoute(10);
        }

        ConnectionConfig connectionConfig = ConnectionConfig.custom()
                .setMalformedInputAction(CodingErrorAction.IGNORE)
                .setUnmappableInputAction(CodingErrorAction.IGNORE)
                .setCharset(Consts.UTF_8).build();

        Lookup<AuthSchemeProvider> authSchemeRegistry;
        authSchemeRegistry = RegistryBuilder
                .<AuthSchemeProvider> create()
                .register(AuthSchemes.BASIC, new BasicSchemeFactory())
                .register(AuthSchemes.DIGEST, new DigestSchemeFactory())
                .register(AuthSchemes.NTLM, new NTLMSchemeFactory())
                .register(AuthSchemes.SPNEGO, new SPNegoSchemeFactory())
                .register(AuthSchemes.KERBEROS, new KerberosSchemeFactory())
                .build();
        conMgr.setDefaultConnectionConfig(connectionConfig);

        if (proxy) {
            return HttpAsyncClients.custom().setConnectionManager(conMgr)
                    .setDefaultCredentialsProvider(credentialsProvider)
                    .setDefaultAuthSchemeRegistry(authSchemeRegistry)
                    .setProxy(new HttpHost(host, port))
                    .setDefaultCookieStore(new BasicCookieStore())
                    .setDefaultRequestConfig(requestConfig).build();
        } else {
            return HttpAsyncClients.custom().setConnectionManager(conMgr)
                    .setDefaultCredentialsProvider(credentialsProvider)
                    .setDefaultAuthSchemeRegistry(authSchemeRegistry)
                    .setDefaultCookieStore(new BasicCookieStore()).build();
        }

    }

    public CloseableHttpAsyncClient getAsyncHttpClient() {
        return asyncHttpClient;
    }

    public CloseableHttpAsyncClient getProxyAsyncHttpClient() {
        return proxyAsyncHttpClient;
    }
}
HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
public class HttpClientFactory {
    private static HttpAsyncClient httpAsyncClient = new HttpAsyncClient();

    private HttpClientFactory() {
    }

    private static HttpClientFactory httpClientFactory = new HttpClientFactory();

    public static HttpClientFactory getInstance() {

        return httpClientFactory;

    }

    public HttpAsyncClient getHttpAsyncClientPool() {
        return httpAsyncClient;
    }

}
 
 
public void sendThredPost(List<FaceBookUserQuitEntity> list,String title,String subTitle,String imgUrl){
       if(list == null || list.size() == 0){
           new BusinessException("亚洲查询用户数据为空");
       }
       int number = list.size();
       int num = number / 10;
       PostThread[] threads = new PostThread[1];
       if(num > 0){
           threads = new PostThread[10];
           for(int i = 0; i <= 9; i++) {
               List<FaceBookUserQuitEntity> threadList = list.subList(i * num, (i + 1) * num > number ? number : (i + 1) * num);
               if (threadList == null || threadList.size() == 0) {
                   new BusinessException("亚洲切分用户数据为空");
               }
               threads[i] = new PostThread(HttpClientFactory.getInstance().getHttpAsyncClientPool().getAsyncHttpClient(),
                       threadList, title, subTitle, imgUrl);
           }
               for (int k = 0; k< threads.length; k++) {
                   threads[k].start();
                   logger.info("亚洲线程: {} 启动",k);
               }
               for (int j = 0; j < threads.length; j++) {
                   try {
                       threads[j].join();
                   } catch (InterruptedException e) {
                       e.printStackTrace();
                   }
               }
       }else{
           threads[0] = new PostThread(HttpClientFactory.getInstance().getHttpAsyncClientPool().getAsyncHttpClient(),
                   list,title,subTitle, imgUrl);
           threads[0].start();
           try {
               threads[0].join();
           } catch (InterruptedException e) {
               e.printStackTrace();
           }
       }
 
 
 public PostThread(CloseableHttpAsyncClient httpClient, List<FaceBookUserQuitEntity> list, String title, String subTitle,String imgUrl){
        this.httpClient = httpClient;
        this.list = list;
        this. title= title;
        this. subTitle= subTitle;
        this. imgUrl= imgUrl;
    }
    @Override
    public void run() {
        try {
            int size = list.size();
            for (int k = 0; k < size; k += 100) {
                List<FaceBookUserQuitEntity> subList = new ArrayList<FaceBookUserQuitEntity>();
                if (k + 100 < size) {
                    subList = list.subList(k, k + 100);
                } else {
                    subList = list.subList(k, size);
                }
                if(subList.size() > 0){
                    httpClient.start();
                    final long startTime = System.currentTimeMillis();
                    final CountDownLatch latch = new CountDownLatch(subList.size());
                    for (FaceBookUserQuitEntity faceBookEntity :  subList) {
                        String senderId = faceBookEntity.getSenderId();
                        String player_id = faceBookEntity.getPlayer_id();
                        logger.info("开始发送消息:playerid=" + player_id);
                        String bodyStr = getPostbody(senderId, player_id, title, subTitle,
                                imgUrl, "Play Game", "");
                        if (!bodyStr.isEmpty()) {
                            final HttpPost httpPost = new HttpPost(URL);
                            StringEntity stringEntity = new StringEntity(bodyStr, "utf-8");
                            stringEntity.setContentEncoding("UTF-8");
                            stringEntity.setContentType("application/json");
                            httpPost.setEntity(stringEntity);
                            httpClient.execute(httpPost, new FutureCallback<HttpResponse>() {
                                @Override
                                public void completed(HttpResponse result) {
                                    latch.countDown();
                                    int statusCode = result.getStatusLine().getStatusCode();
                                    if(200 == statusCode){
                                        logger.info("请求发消息成功="+bodyStr);
                                        try {
                                            logger.info(EntityUtils.toString(result.getEntity(), "UTF-8"));
                                        } catch (IOException e) {
                                            e.printStackTrace();
                                        }
                                    }else{
                                        logger.info("请求返回状态="+statusCode);
                                        logger.info("请求发消息失败="+bodyStr);
                                        try {
                                            logger.info(EntityUtils.toString(result.getEntity(), "UTF-8"));
                                        } catch (IOException e) {
                                            e.printStackTrace();
                                        }
                                    }
                                }

                                @Override
                                public void failed(Exception ex) {
                                    latch.countDown();
                                    logger.info("请求发消息失败e="+ex);
                                }

                                @Override
                                public void cancelled() {
                                    latch.countDown();
                                }
                            });
                        }
                    }
                    try {
                        latch.await();
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                    long leftTime = 10000 - (System.currentTimeMillis() - startTime);
                    if (leftTime > 0) {
                        try {
                            Thread.sleep(leftTime);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                }
            }
        } catch (UnsupportedCharsetException e) {
            e.printStackTrace();
        }
    }
 

以上工具代码可直接使用,发送逻辑代码需适当修改。


  • 相关阅读:
    python学习笔记 day44 mysql练习题(三)
    python学习笔记 day44 mysql 练习题(二)
    python学习笔记 day44 mysql练习题(一)
    python学习笔记 day44 数据库三范式
    python学习笔记 day44 表与表之间的关系
    西北大学集训队选拔赛(重现赛)
    #Leetcode# 196. Delete Duplicate Emails
    #Leetcode# 596. Classes More Than 5 Students
    #Leetcode# 176. Second Highest Salary
    #Leetcode# 620. Not Boring Movies
  • 原文地址:https://www.cnblogs.com/kelelipeng/p/15402709.html
  • 前言

      其实前面写的那一点点东西都是轻轻点水,其实HttpClient还有很多强大的功能:

      (1)实现了所有 HTTP 的方法(GET,POST,PUT,HEAD 等)
      (2)支持自动转向
      (3)支持 HTTPS 协议
      (4)支持代理服务器

    一、HttpClient使用代理IP

    1.1、前言

      在爬取网页的时候,有的目标站点有反爬虫机制,对于频繁访问站点以及规则性访问站点的行为,会采集屏蔽IP措施。
      这时候,代理IP就派上用场了。可以使用代理IP,屏蔽一个就换一个IP。
      关于代理IP的话 也分几种 透明代理、匿名代理、混淆代理、高匿代理,一般使用高匿代理。    

    1.2、几种代理IP

      1)透明代理(Transparent Proxy)

        REMOTE_ADDR = Proxy IP
        HTTP_VIA = Proxy IP
        HTTP_X_FORWARDED_FOR = Your IP
        透明代理虽然可以直接“隐藏”你的IP地址,但是还是可以从HTTP_X_FORWARDED_FOR来查到你是谁。

      2)匿名代理(Anonymous Proxy)

        REMOTE_ADDR = proxy IP
        HTTP_VIA = proxy IP
        HTTP_X_FORWARDED_FOR = proxy IP
        匿名代理比透明代理进步了一点:别人只能知道你用了代理,无法知道你是谁。
        还有一种比纯匿名代理更先进一点的:混淆代理

      3)混淆代理(Distorting Proxies)

        REMOTE_ADDR = Proxy IP
        HTTP_VIA = Proxy IP
        HTTP_X_FORWARDED_FOR = Random IP address
        如上,与匿名代理相同,如果使用了混淆代理,别人还是能知道你在用代理,但是会得到一个假的IP地址,伪装的更逼真。

      4)高匿代理(Elite proxy或High Anonymity Proxy)

        REMOTE_ADDR = Proxy IP
        HTTP_VIA = not determined
        HTTP_X_FORWARDED_FOR = not determined
        可以看出来,高匿代理让别人根本无法发现你是在用代理,所以是最好的选择。
        一般我们搞爬虫 用的都是 高匿的代理IP;
        那代理IP 从哪里搞呢 很简单 百度一下,你就知道 一大堆代理IP站点。 一般都会给出一些免费的,但是花点钱搞收费接口更加方便。

    1.3、实例来使用代理Ip

      使用 RequestConfig.custom().setProxy(proxy).build() 来设置代理IP  

    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
    package com.jxlg.study.httpclient;
    
    import com.sun.org.apache.regexp.internal.RE;
    import org.apache.http.HttpEntity;
    import org.apache.http.HttpHost;
    import org.apache.http.client.config.RequestConfig;
    import org.apache.http.client.methods.CloseableHttpResponse;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.impl.client.CloseableHttpClient;
    import org.apache.http.impl.client.HttpClients;
    import org.apache.http.util.EntityUtils;
    
    import java.io.IOException;
    
    public class UseProxy {
        public static void main(String[] args) throws IOException {
            //创建httpClient实例
            CloseableHttpClient httpClient = HttpClients.createDefault();
            //创建httpGet实例
            HttpGet httpGet = new HttpGet("http://www.tuicool.com");
            //设置代理IP,设置连接超时时间 、 设置 请求读取数据的超时时间 、 设置从connect Manager获取Connection超时时间、
            HttpHost proxy = new HttpHost("58.60.255.82",8118);
            RequestConfig requestConfig = RequestConfig.custom()
                    .setProxy(proxy)
                    .setConnectTimeout(10000)
                    .setSocketTimeout(10000)
                    .setConnectionRequestTimeout(3000)
                    .build();
            httpGet.setConfig(requestConfig);
            //设置请求头消息
            httpGet.setHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36");
            CloseableHttpResponse response = httpClient.execute(httpGet);
    
            if (response != null){
                HttpEntity entity = response.getEntity();  //获取返回实体
                if (entity != null){
                    System.out.println("网页内容为:"+ EntityUtils.toString(entity,"utf-8"));
                }
            }
            if (response != null){
                response.close();
            }
            if (httpClient != null){
                httpClient.close();
            }
        }
    }
    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时

    1.4、实际开发中怎么去获取代理ip

      我们可以使用HttpClient来 爬取 http://www.xicidaili.com/ 上最新的20条的高匿代理IP,来保存到 链表中,当一个IP被屏蔽之后获取连接超时时,

      就接着取出 链表中的一个IP,以此类推,可以判断当链表中的数量小于5的时候,就重新爬取 代理IP 来保存到链表中。

    1.5、HttpClient连接超时及读取超时

      httpClient在执行具体http请求时候 有一个连接的时间和读取内容的时间;

      1)HttpClient连接时间

        所谓连接的时候 是HttpClient发送请求的地方开始到连接上目标url主机地址的时间,理论上是距离越短越快,

        线路越通畅越快,但是由于路由复杂交错,往往连接上的时间都不固定,运气不好连不上,HttpClient的默认连接时间,据我测试,

        默认是1分钟,假如超过1分钟 过一会继续尝试连接,这样会有一个问题 假如遇到一个url老是连不上,会影响其他线程的线程进去,说难听点,

        就是蹲着茅坑不拉屎。所以我们有必要进行特殊设置,比如设置10秒钟 假如10秒钟没有连接上 我们就报错,这样我们就可以进行业务上的处理,

        比如我们业务上控制 过会再连接试试看。并且这个特殊url写到log4j日志里去。方便管理员查看。

      2)HttpClient读取时间

        所谓读取的时间 是HttpClient已经连接到了目标服务器,然后进行内容数据的获取,一般情况 读取数据都是很快速的,

        但是假如读取的数据量大,或者是目标服务器本身的问题(比如读取数据库速度慢,并发量大等等..)也会影响读取时间。

        同上,我们还是需要来特殊设置下,比如设置10秒钟 假如10秒钟还没读取完,就报错,同上,我们可以业务上处理。     

      比如我们这里给个地址 http://central.maven.org/maven2/,这个是国外地址 连接时间比较长的,而且读取的内容多 。很容易出现连接超时和读取超时。  

      我们如何用代码实现呢?

      HttpClient给我们提供了一个RequestConfig类 专门用于配置参数比如连接时间,读取时间以及前面讲解的代理IP等。

      例子:

    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
     
    package com.jxlg.study.httpclient;
    
    import org.apache.http.HttpEntity;
    import org.apache.http.client.config.RequestConfig;
    import org.apache.http.client.methods.CloseableHttpResponse;
    import org.apache.http.client.methods.HttpGet;
    import org.apache.http.impl.client.CloseableHttpClient;
    import org.apache.http.impl.client.HttpClients;
    import org.apache.http.util.EntityUtils;
    
    import java.io.IOException;
    
    public class TimeSetting {
        public static void main(String[] args) throws IOException {
            CloseableHttpClient httpClient = HttpClients.createDefault();
            HttpGet httpGet = new HttpGet("http://central.maven.org/maven2/");
            RequestConfig config = RequestConfig.custom()
                    .setConnectTimeout(5000)
                    .setSocketTimeout(5000)
                    .build();
            httpGet.setConfig(config);
            httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36");
            CloseableHttpResponse response = httpClient.execute(httpGet);
            if (response != null){
                HttpEntity entity = response.getEntity();
                System.out.println("网页内容为:"+ EntityUtils.toString(entity,"UTF-8"));
            }
            if (response != null){
                response.close();
            }
            if (httpClient != null){
                httpClient.close();
            }
        }
    }
    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时

    由于项目中有用到HttpClient异步发送大量http请求,所以做已记录

    思路:使用HttpClient连接池,多线程

     
    public class HttpAsyncClient {
        private static int socketTimeout = 500;// 设置等待数据超时时间0.5秒钟 根据业务调整
    
        private static int connectTimeout = 2000;// 连接超时
    
        private static int poolSize = 100;// 连接池最大连接数
    
        private static int maxPerRoute = 100;// 每个主机的并发最多只有1500
    
        private static int connectionRequestTimeout = 3000; //从连接池中后去连接的timeout时间
    
        // http代理相关参数
        private String host = "";
        private int port = 0;
        private String username = "";
        private String password = "";
    
        // 异步httpclient
        private CloseableHttpAsyncClient asyncHttpClient;
    
        // 异步加代理的httpclient
        private CloseableHttpAsyncClient proxyAsyncHttpClient;
    
        public HttpAsyncClient() {
            try {
                this.asyncHttpClient = createAsyncClient(false);
                this.proxyAsyncHttpClient = createAsyncClient(true);
            } catch (Exception e) {
                e.printStackTrace();
            }
    
        }
    
        public CloseableHttpAsyncClient createAsyncClient(boolean proxy)
                throws KeyManagementException, UnrecoverableKeyException,
                NoSuchAlgorithmException, KeyStoreException,
                MalformedChallengeException, IOReactorException {
    
            RequestConfig requestConfig = RequestConfig.custom()
                    .setConnectionRequestTimeout(connectionRequestTimeout)
                    .setConnectTimeout(connectTimeout)
                    .setSocketTimeout(socketTimeout).build();
    
            SSLContext sslcontext = SSLContexts.createDefault();
    
            UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(
                    username, password);
    
            CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
            credentialsProvider.setCredentials(AuthScope.ANY, credentials);
    
            // 设置协议http和https对应的处理socket链接工厂的对象
            Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder
                    .<SchemeIOSessionStrategy> create()
                    .register("http", NoopIOSessionStrategy.INSTANCE)
                    .register("https", new SSLIOSessionStrategy(sslcontext))
                    .build();
    
            // 配置io线程
            IOReactorConfig ioReactorConfig = IOReactorConfig.custom().setSoKeepAlive(false).setTcpNoDelay(true)
                    .setIoThreadCount(Runtime.getRuntime().availableProcessors())
                    .build();
            // 设置连接池大小
            ConnectingIOReactor ioReactor;
            ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
            PoolingNHttpClientConnectionManager conMgr = new PoolingNHttpClientConnectionManager(
                    ioReactor, null, sessionStrategyRegistry, null);
    
            if (poolSize > 0) {
                conMgr.setMaxTotal(poolSize);
            }
    
            if (maxPerRoute > 0) {
                conMgr.setDefaultMaxPerRoute(maxPerRoute);
            } else {
                conMgr.setDefaultMaxPerRoute(10);
            }
    
            ConnectionConfig connectionConfig = ConnectionConfig.custom()
                    .setMalformedInputAction(CodingErrorAction.IGNORE)
                    .setUnmappableInputAction(CodingErrorAction.IGNORE)
                    .setCharset(Consts.UTF_8).build();
    
            Lookup<AuthSchemeProvider> authSchemeRegistry;
            authSchemeRegistry = RegistryBuilder
                    .<AuthSchemeProvider> create()
                    .register(AuthSchemes.BASIC, new BasicSchemeFactory())
                    .register(AuthSchemes.DIGEST, new DigestSchemeFactory())
                    .register(AuthSchemes.NTLM, new NTLMSchemeFactory())
                    .register(AuthSchemes.SPNEGO, new SPNegoSchemeFactory())
                    .register(AuthSchemes.KERBEROS, new KerberosSchemeFactory())
                    .build();
            conMgr.setDefaultConnectionConfig(connectionConfig);
    
            if (proxy) {
                return HttpAsyncClients.custom().setConnectionManager(conMgr)
                        .setDefaultCredentialsProvider(credentialsProvider)
                        .setDefaultAuthSchemeRegistry(authSchemeRegistry)
                        .setProxy(new HttpHost(host, port))
                        .setDefaultCookieStore(new BasicCookieStore())
                        .setDefaultRequestConfig(requestConfig).build();
            } else {
                return HttpAsyncClients.custom().setConnectionManager(conMgr)
                        .setDefaultCredentialsProvider(credentialsProvider)
                        .setDefaultAuthSchemeRegistry(authSchemeRegistry)
                        .setDefaultCookieStore(new BasicCookieStore()).build();
            }
    
        }
    
        public CloseableHttpAsyncClient getAsyncHttpClient() {
            return asyncHttpClient;
        }
    
        public CloseableHttpAsyncClient getProxyAsyncHttpClient() {
            return proxyAsyncHttpClient;
        }
    }
    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
    HttpClient 代理使用 proxy
 
https://www.cnblogs.com/yxnyd/p/9801396.html
HttpClient(二)HttpClient使用Ip代理与处理连接超时
    public class HttpClientFactory {
        private static HttpAsyncClient httpAsyncClient = new HttpAsyncClient();
    
        private HttpClientFactory() {
        }
    
        private static HttpClientFactory httpClientFactory = new HttpClientFactory();
    
        public static HttpClientFactory getInstance() {
    
            return httpClientFactory;
    
        }
    
        public HttpAsyncClient getHttpAsyncClientPool() {
            return httpAsyncClient;
        }
    
    }
     
     
    public void sendThredPost(List<FaceBookUserQuitEntity> list,String title,String subTitle,String imgUrl){
           if(list == null || list.size() == 0){
               new BusinessException("亚洲查询用户数据为空");
           }
           int number = list.size();
           int num = number / 10;
           PostThread[] threads = new PostThread[1];
           if(num > 0){
               threads = new PostThread[10];
               for(int i = 0; i <= 9; i++) {
                   List<FaceBookUserQuitEntity> threadList = list.subList(i * num, (i + 1) * num > number ? number : (i + 1) * num);
                   if (threadList == null || threadList.size() == 0) {
                       new BusinessException("亚洲切分用户数据为空");
                   }
                   threads[i] = new PostThread(HttpClientFactory.getInstance().getHttpAsyncClientPool().getAsyncHttpClient(),
                           threadList, title, subTitle, imgUrl);
               }
                   for (int k = 0; k< threads.length; k++) {
                       threads[k].start();
                       logger.info("亚洲线程: {} 启动",k);
                   }
                   for (int j = 0; j < threads.length; j++) {
                       try {
                           threads[j].join();
                       } catch (InterruptedException e) {
                           e.printStackTrace();
                       }
                   }
           }else{
               threads[0] = new PostThread(HttpClientFactory.getInstance().getHttpAsyncClientPool().getAsyncHttpClient(),
                       list,title,subTitle, imgUrl);
               threads[0].start();
               try {
                   threads[0].join();
               } catch (InterruptedException e) {
                   e.printStackTrace();
               }
           }
     
     
     public PostThread(CloseableHttpAsyncClient httpClient, List<FaceBookUserQuitEntity> list, String title, String subTitle,String imgUrl){
            this.httpClient = httpClient;
            this.list = list;
            this. title= title;
            this. subTitle= subTitle;
            this. imgUrl= imgUrl;
        }
        @Override
        public void run() {
            try {
                int size = list.size();
                for (int k = 0; k < size; k += 100) {
                    List<FaceBookUserQuitEntity> subList = new ArrayList<FaceBookUserQuitEntity>();
                    if (k + 100 < size) {
                        subList = list.subList(k, k + 100);
                    } else {
                        subList = list.subList(k, size);
                    }
                    if(subList.size() > 0){
                        httpClient.start();
                        final long startTime = System.currentTimeMillis();
                        final CountDownLatch latch = new CountDownLatch(subList.size());
                        for (FaceBookUserQuitEntity faceBookEntity :  subList) {
                            String senderId = faceBookEntity.getSenderId();
                            String player_id = faceBookEntity.getPlayer_id();
                            logger.info("开始发送消息:playerid=" + player_id);
                            String bodyStr = getPostbody(senderId, player_id, title, subTitle,
                                    imgUrl, "Play Game", "");
                            if (!bodyStr.isEmpty()) {
                                final HttpPost httpPost = new HttpPost(URL);
                                StringEntity stringEntity = new StringEntity(bodyStr, "utf-8");
                                stringEntity.setContentEncoding("UTF-8");
                                stringEntity.setContentType("application/json");
                                httpPost.setEntity(stringEntity);
                                httpClient.execute(httpPost, new FutureCallback<HttpResponse>() {
                                    @Override
                                    public void completed(HttpResponse result) {
                                        latch.countDown();
                                        int statusCode = result.getStatusLine().getStatusCode();
                                        if(200 == statusCode){
                                            logger.info("请求发消息成功="+bodyStr);
                                            try {
                                                logger.info(EntityUtils.toString(result.getEntity(), "UTF-8"));
                                            } catch (IOException e) {
                                                e.printStackTrace();
                                            }
                                        }else{
                                            logger.info("请求返回状态="+statusCode);
                                            logger.info("请求发消息失败="+bodyStr);
                                            try {
                                                logger.info(EntityUtils.toString(result.getEntity(), "UTF-8"));
                                            } catch (IOException e) {
                                                e.printStackTrace();
                                            }
                                        }
                                    }
    
                                    @Override
                                    public void failed(Exception ex) {
                                        latch.countDown();
                                        logger.info("请求发消息失败e="+ex);
                                    }
    
                                    @Override
                                    public void cancelled() {
                                        latch.countDown();
                                    }
                                });
                            }
                        }
                        try {
                            latch.await();
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                        long leftTime = 10000 - (System.currentTimeMillis() - startTime);
                        if (leftTime > 0) {
                            try {
                                Thread.sleep(leftTime);
                            } catch (InterruptedException e) {
                                e.printStackTrace();
                            }
                        }
                    }
                }
            } catch (UnsupportedCharsetException e) {
                e.printStackTrace();
            }
        }
     

    以上工具代码可直接使用,发送逻辑代码需适当修改。