通常我们在写爬虫时候,为了规避目标网站限制通常采用代理方式调用,普遍的做法是通过定时任务去一些免费代理网站获取代理服务IP+Port,远程调用时候通过循环可用的代理去抓取目标网站内容;
OpenFeign配置代理服务器的步骤总结如下:
- OpenFeign使用方式:见之前文章@FeignClient中的fallbackFactory,fallback不起作用的正确使用示例
- application.yml添加配置,采用okhttp方式远程调用
feign:
httpclient:
enabled: false
okhttp:
enabled: true
- pom.xml配置增加
<dependency>
<groupId>io.github.openfeign</groupId>
<artifactId>feign-okhttp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
- Feign自动化配置增加如下代码
@Bean
public OkHttpClientFactory okHttpClientFactory(OkHttpClient.Builder builder) {
return new ProxyOkHttpClientFactory(builder);
}
static class ProxyOkHttpClientFactory extends DefaultOkHttpClientFactory {
public ProxyOkHttpClientFactory(OkHttpClient.Builder builder) {
super(builder);
//配置IP、端口
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("222.162.185.176", 4524));
builder.proxy(proxy);
/*
builder.proxyAuthenticator(new Authenticator() {
@Override
public Request authenticate(Route route, Response response) throws IOException {
//设置代理服务器账号密码
String credential = Credentials.basic("admin", "admin");
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
});*/
//如果要配置限制域则加上下面
/*List<Proxy> proxyList = new ArrayList<>(1);
proxyList.add(proxy);
builder.proxySelector(new ProxySelector() {
//限制域
Set<String> domainList;
@Override
public List<Proxy> select(URI uri) {
if (uri == null || !domainList.contains(uri.getHost())) {
return Collections.singletonList(Proxy.NO_PROXY);
}
return proxyList;
}
@Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
}
});*/
}
}
- 将自己的代理服务器ip+port替换即可;关于如何寻找高质量免费的代理有时间单独更新一篇博文总结一下;
评论区