前言
生产环境上的几个由dubbo发布的微服务,支付,订单系统模块,当测试反馈或者用户反馈相应慢的时候,排查到底是在哪个节点服务出问题时,这可就麻烦了,这里牵涉到整个链路跟踪。当时也用过傻办法,每个方法出入口都有对应的消耗时间然后痛苦的去查看日志排查。这里记录下使用dubbo整合zipkin来跟踪整个服务链路消耗时间。
准备安装文件
以下为大家准备了安装文件,也可以自己到网上下载
下载地址:curl -sSL https://zipkin.io/quickstart.sh | bash -s
准备文件 | 下载地址 |
---|---|
zipkin.jar | 链接: https://pan.baidu.com/s/1dlUjcLjemTcm7jnc0HDiwQ 提取码: typ4 |
Zipkin安装启动
由于是一个jar文件,直接下载就可以通过java -jar方式允许(前提先安装java环境)
启动
1
2[root@master zipkin]# java -jar zipkin.jar
# 看到如下信息就代码启动成功了访问控制台
以上为zipkin的安装启动,还是比较简单的。不过按照这种方式安装的 Zipkin Server 使用的存储类型是 inMemory 的。当服务器停机之后,所有收集到的 trace 信息会丢失,不适用于生产系统。如果在生产系统中使用,需要配置另外的存储类型。Zipkin 支持 MySql、Cassandra、和 ElasticSearch。推荐使用 Cassandra 和 ElasticSearch。后续上生产后,在记录下对应的配置,这里也仅仅是先把整个流程跑通。接下来就是需要对需要收集的项目进行简单改造,增加配置信息。
Dubbo项目ZipKin配置
增加pom配置依赖(全局依赖)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60<brave.version>4.19.2</brave.version>
<zipkin-reporter.version>2.1.3</zipkin-reporter.version>
<zipkin.version>2.8.1</zipkin.version>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter</artifactId>
<version>2.6.0</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-sender-okhttp3</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-context-log4j2</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-instrumentation-http</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-instrumentation-http-tests</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-instrumentation-servlet</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.brave</groupId>
<artifactId>brave-tests</artifactId>
<version>4.19.2</version>
</dependency>
<dependency>
<groupId>io.zipkin.zipkin2</groupId>
<artifactId>zipkin</artifactId>
<version>${zipkin.version}</version>
</dependency>
<dependency>
<groupId>io.zipkin.java</groupId>
<artifactId>zipkin</artifactId>
<version>${zipkin.version}</version>
</dependency>
<dependency>
<groupId>io.zipkin.java</groupId>
<artifactId>zipkin-junit</artifactId>
<version>${zipkin.version}</version>
</dependency>增加zipkin公共模块config,以及对应的filter代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110package com.smarttrip.pay.common.zipkin;
import brave.Tracing;
import brave.servlet.TracingFilter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import zipkin2.Span;
import zipkin2.reporter.AsyncReporter;
import zipkin2.reporter.Reporter;
import zipkin2.reporter.Sender;
import zipkin2.reporter.okhttp3.OkHttpSender;
import javax.servlet.Filter;
/**
* @program: 配置文件
* @description:
* @author: lishijia
* @create: 2019-01-23 10:31
**/
public class TracingConfig {
/**
* 配置zipkin服务地址
*/
"${zipkin.tracing.endpoint:endpoint}") (
private String zipkinEndPoint;
"${zipkin.tracing.local-service-name:local-service-name}") (
private String localServiceName;
/**
* 配置sender
* @return
*/
public Sender sender(){
OkHttpSender sender = OkHttpSender
.newBuilder()
.endpoint(zipkinEndPoint)
.build();
return sender;
}
/**
* 配置reporter
* @param sender
* @return
*/
public Reporter<Span> reporter(Sender sender){
return AsyncReporter
.builder(sender)
.build();
}
/**
* 配置dubbo-consumer tracing
* @param reporter
* @return
*/
public Tracing tracing(Reporter reporter){
return Tracing.newBuilder()
.localServiceName(localServiceName)
.spanReporter(reporter)
.build();
}
/**
* 配置http tracing
* @param reporter
* @return
*/
public Tracing tracing2(Reporter reporter){
return Tracing.newBuilder()
.localServiceName(localServiceName + "_http")
.spanReporter(reporter)
.build();
}
/**
* 配置servlet filter
* @param tracing2
* @return
*/
public Filter filter(Tracing tracing2){
return TracingFilter.create(tracing2);
}
/**
* 注册filter
* @param filter
* @return
*/
public FilterRegistrationBean filterRegistration(Filter filter) {
FilterRegistrationBean registration = new FilterRegistrationBean();
registration.setFilter(filter);
registration.addUrlPatterns("/*");
registration.setName("zipkin-filter");
registration.setOrder(1);
return registration;
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113package com.smarttrip.pay.common.zipkin;
import brave.Span;
import brave.Tracer;
import brave.propagation.Propagation;
import com.alibaba.dubbo.remoting.exchange.ResponseCallback;
import com.alibaba.dubbo.rpc.*;
import com.alibaba.dubbo.rpc.protocol.dubbo.FutureAdapter;
import com.alibaba.dubbo.rpc.support.RpcUtils;
import com.alibaba.fastjson.JSON;
import zipkin2.Endpoint;
import java.net.InetSocketAddress;
import java.util.Map;
import java.util.concurrent.Future;
/**
* @program:
* @description:
* @author: lishijia
* @create: 2019-01-23 10:31
**/
public class ZipkinHelper {
static final Propagation.Setter<Map<String, String>, String> SETTER =
new Propagation.Setter<Map<String, String>, String>() {
public void put(Map<String, String> carrier, String key, String value) {
carrier.put(key, value);
}
public String toString() {
return JSON.toJSONString(this);
}
};
static final Propagation.Getter<Map<String, String>, String> GETTER =
new Propagation.Getter<Map<String, String>, String>() {
public String get(Map<String, String> carrier, String key) {
return carrier.get(key);
}
public String toString() {
return JSON.toJSONString(this);
}
};
static void buildSpan(Span span, Span.Kind kind, InetSocketAddress remoteAddress, String service, String method){
if (!span.isNoop()) {
span.kind(kind).start();
span.kind(kind);
span.name(service + "/" + method);
Endpoint.Builder remoteEndpoint = Endpoint.newBuilder().port(remoteAddress.getPort());
if (!remoteEndpoint.parseIp(remoteAddress.getAddress())) {
remoteEndpoint.parseIp(remoteAddress.getHostName());
}
span.remoteEndpoint(remoteEndpoint.build());
}
}
static Result spanTracing(Span span, Tracer tracer, Invoker<?> invoker, Invocation invocation, RpcContext rpcContext){
boolean isOneway = false;
boolean deferFinish = false;
try (Tracer.SpanInScope scope = tracer.withSpanInScope(span)) {
Result result = invoker.invoke(invocation);
if (result.hasException()) {
onError(result.getException(), span);
}
isOneway = RpcUtils.isOneway(invoker.getUrl(), invocation);
Future<Object> future = rpcContext.getFuture(); // the case on async client invocation
if (future instanceof FutureAdapter) {
deferFinish = true;
((FutureAdapter) future).getFuture().setCallback(new ZipkinHelper.FinishSpanCallback(span));
}
return result;
} catch (Exception e) {
onError(e, span);
throw e;
} finally {
if (isOneway) {
span.flush();
} else if (!deferFinish) {
span.finish();
}
}
}
static void onError(Throwable error, Span span) {
span.error(error);
if (error instanceof RpcException) {
span.tag("dubbo.error_code", Integer.toString(((RpcException) error).getCode()));
}
}
static final class FinishSpanCallback implements ResponseCallback {
final Span span;
FinishSpanCallback(Span span) {
this.span = span;
}
public void done(Object response) {
span.finish();
}
public void caught(Throwable exception) {
onError(exception, span);
span.finish();
}
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60package com.smarttrip.pay.common.zipkin;
/**
* @program: DubboZipkinProviderFilter是dubbo服务端过滤器,用于拦截zipkin上下文消息
* @description:
* @author: lishijia
* @create: 2019-01-2310:33
**/
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.TraceContext;
import brave.propagation.TraceContextOrSamplingFlags;
import com.alibaba.dubbo.common.Constants;
import com.alibaba.dubbo.common.extension.Activate;
import com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory;
import com.alibaba.dubbo.rpc.*;
import com.alibaba.dubbo.rpc.support.RpcUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
(group = Constants.PROVIDER)
public class DubboZipkinProviderFilter implements Filter {
private static final Logger log = LoggerFactory.getLogger(DubboZipkinProviderFilter.class);
private SpringExtensionFactory springExtensionFactory = new SpringExtensionFactory();
private Tracer tracer;
// tracing上下文消息提取
private TraceContext.Extractor<Map<String, String>> extractor;
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
log.info("dubbo zipkin provider filter......");
Tracing tracing = springExtensionFactory.getExtension(Tracing.class, "tracing");
tracer = tracing.tracer();
if (null == tracer){
return invoker.invoke(invocation);
}
if (null == extractor){
extractor = tracing.propagation().extractor(ZipkinHelper.GETTER);
}
TraceContextOrSamplingFlags extracted = extractor.extract(invocation.getAttachments());
Span span = extracted.context() != null
? tracer.joinSpan(extracted.context())
: tracer.nextSpan(extracted);
RpcContext rpcContext = RpcContext.getContext();
ZipkinHelper.buildSpan(span, Span.Kind.SERVER, rpcContext.getRemoteAddress(), invoker.getInterface().getSimpleName(),
RpcUtils.getMethodName(invocation));
return ZipkinHelper.spanTracing(span, tracer, invoker, invocation, rpcContext);
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57package com.smarttrip.pay.common.zipkin;
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.TraceContext;
import com.alibaba.dubbo.common.Constants;
import com.alibaba.dubbo.common.extension.Activate;
import com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory;
import com.alibaba.dubbo.rpc.*;
import com.alibaba.dubbo.rpc.support.RpcUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
/**
* @program: DubboZipkinConsumerFilter是dubbo消费端过滤器,用于拦截zipkin上下文消息
* @description:
* @author: lishijia
* @create: 2019-01-23 10:32
**/
(group = Constants.CONSUMER)
public class DubboZipkinConsumerFilter implements Filter {
private static final Logger log = LoggerFactory.getLogger(DubboZipkinConsumerFilter.class);
private SpringExtensionFactory springExtensionFactory = new SpringExtensionFactory();
private Tracer tracer;
// tracing上下文消息注入
private TraceContext.Injector<Map<String, String>> injector;
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
log.info("dubbo zipkin consumer filter......");
Tracing tracing = springExtensionFactory.getExtension(Tracing.class, "tracing");
tracer = tracing.tracer();
if (tracer == null){
return invoker.invoke(invocation);
}
if (null == injector){
injector = tracing.propagation().injector(ZipkinHelper.SETTER);
}
RpcContext rpcContext = RpcContext.getContext();
Span span = tracer.nextSpan();
injector.inject(span.context(), invocation.getAttachments());
ZipkinHelper.buildSpan(span, Span.Kind.CONSUMER, rpcContext.getRemoteAddress(), invoker.getInterface().getSimpleName(),
RpcUtils.getMethodName(invocation));
return ZipkinHelper.spanTracing(span, tracer, invoker, invocation, rpcContext);
}
}配置文件
以下配置文件在每个服务端,消费端都需要添加
1
2
3
4
5# 启用自动配置
# 在resources目录新建META-INF目录:
# 新建文件:spring.factories 内容如下:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.smarttrip.pay.common.zipkin.TracingConfig1
2
3
4
5# 在provider,consumer启动工程下增加如下配置过滤器:
# 在`META-INF`添加`dubbo`文件夹,
# 新建文件:`com.alibaba.dubbo.rpc.Filter` 内容如下:
zipkinConsumerDubboFilter=com.smarttrip.pay.common.zipkin.DubboZipkinConsumerFilter
zipkinProviderDubboFilter=com.smarttrip.pay.common.zipkin.DubboZipkinProviderFilter配置每个节点的名称以及zipkin上报地址
1
2
3
4zipkin:
tracing:
local-service-name: zipkin-provider
endpoint: http://192.168.226.100:9411/api/v2/spans
针对项目的改造增加的代码、配置基本上就完成了
通过调用服务验证ZipKin
通过如下图可以看到每个链路总消耗时间,
查看单个链路在每个服务上锁消耗的时间,然后可以再去排查对应的问题所在
总结
通过Zipkin跟踪整条链路消耗时间情况来快速定位问题,尤其是整个项目是在微服务化的情况下运行。那么这种工具就是必备的了。自己也可以针对Zipkin下载源码来打包,可以二次开发,增加对应的预警功能等。后续上生产后,会把存储改造为物理存储,即Cassandra、和 ElasticSearch这种nosql数据库。