bRPC – Builtin Services

Docs: builtin services

Thu, 12 Aug 2021 00:00:00 +0000

Builtin Services

Builtin services expose internal status of servers in different pespectives, making development and debugging over brpc more efficient. brpc serves builting services via HTTP, which can be easily accessed through curl and web browsers. Servers respond plain text or html according to User-Agent in the request header, or you may append ?console=1examplerpc_view for proxying.

Following 2 screenshots show accesses to builtin services from a web browser and a terminal respectively. Note that the logo is the codename inside Baidu, and being modified to brpc in opensourced version.

From a web browser

From a terminal

Security Mode

To avoid potential attacks and information leaks, builtin services musthere for more details.

Main services:

/status: displays brief status of all services.

/vars: lists user-customizable counters on miscellaneous metrics.

/connections: lists all connections and their stats.

/flags: lists all gflags, some of them are modifiable at run-time.

/rpcz: traces all RPCs.

cpu profiler: analyzes CPU hotspots.

heap profiler: shows how memory are allocated.

contention profiler: analyzes lock contentions.

Other services

/version shows version of the server. Call Server::set_version() to specify version of the server, or brpc would generate a default version like brpc_server_<service-name1>_<service-name2> ...

/health shows whether this server is alive or not.

/protobufs shows scheme of all protobuf messages inside the server.

/vlogVLOG that can be enabled(not working with glog).

/dir: browses all files on the server, convenient but too dangerous, disabled by default.

/threads: displays information of all threads of the process, hurting performance significantly when being turned on, disabled by default.

Docs: status

Thu, 12 Aug 2021 00:00:00 +0000

/status/vars, but stats are grouped differently.

Meanings of the fields above:

non_service_error: number of errors raised outside processing code of the service. When a valid service is obtained, the subsequent error is regarded as service_error, otherwise it is regarded as non_service_error (such as request parsing failed, service name does not exist, request concurrency exceeding limit, etc.). As a contrast, failing to access back-end servers during the processing is an error of the service, not a non_service_error. Even if the response written out successfully stands for failure, the error is counted into the service rather than non_service_error.
connection_count: number of connections to the server from clients, not including number of outward connections which are displayed at /vars/rpc_channel_connection_count.
example.EchoService: Full name of the service, including the package name defined in proto.
Echo (EchoRequest) returns (EchoResponse): Signature of the method. A service can have multiple methods. Click links on request/response to see schemes of the protobuf messages.
count: Number of requests that are succesfully processed.
error: Number of requests that are failed to process.
latency: average latency in recent 60s/60m/24h/30d from right to left-bvar_dump_interval) on plain texts.
latency_percentiles-bvar_dump_interval). Curves with historical values are shown on html.
latency_cdfCDF, only available on html.
max_latency: max latency in recent 60s/60m/24h/30d from right to left-bvar_dump_interval) on plain texts.
qps: QPS(Queries Per Second) in recent 60s/60m/24h/30d from right to left-bvar_dump_interval) on plain texts.
processing: (renamed to concurrency in master) Number of requests being processed by the method. If this counter can’t hit zero when the traffic to the service becomes zero, the server probably has bugs, such as forgetting to call done->Run() or stuck on some processing steps.

brpc::Describable.

class MyService : public XXXService, public brpc::Describable {
public:
 ...
 void Describe(std::ostream& os, const brpc::DescribeOptions& options) const {
 os << "my_status: blahblah";
 }
};

For example:

Docs: vars

Thu, 12 Aug 2021 00:00:00 +0000

bvar/vars/vars/VARNAMEbvar to know how to add bvars for your program. brpc extensively use bvar to expose internal status. If you are looking for an utility to collect and display metrics of your application, consider bvar in the first place. bvar definitely can’t replace all counters, essentially it moves contentions occurred during write to read: which needs to combine all data written by all threads and becomes much slower than an ordinary read. If read and write on the counter are both frequent or decisions need to be made based on latest values, you should not use bvar.

Query methods

/vars : List all exposed bvars

/vars/NAME：List the bvar whose name is NAME

/vars/NAME1,NAME2,NAME3：List bvars whose names are either NAME1, NAME2 or NAME3.

/vars/foo*,b$r: List bvars whose names match given wildcard patterns. Note that $ matches a single character instead of ? which is a reserved character in URL.

Following animation shows how to find bvars with wildcard patterns. You can copy and paste the URL to others who will see same bvars that you see. (values may change)

There’s a search box in the upper-left corner on /vars page, in which you can type part of the names to locate bvars. Different patterns are separated by , : or space.

/vars is accessible from terminal as well:

$ curl brpc.baidu.com:8765/vars/bthread*
bthread_creation_count : 125134
bthread_creation_latency : 3
bthread_creation_latency_50 : 3
bthread_creation_latency_90 : 5
bthread_creation_latency_99 : 7
bthread_creation_latency_999 : 12
bthread_creation_latency_9999 : 12
bthread_creation_latency_cdf : "click to view"
bthread_creation_latency_percentiles : "[3,5,7,12]"
bthread_creation_max_latency : 7
bthread_creation_qps : 100
bthread_group_status : "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "
bthread_num_workers : 24
bthread_worker_usage : 1.01056

View historical trends

Clicking on most of the numerical bvars shows historical trends. Each clickable bvar records values in recent 60 seconds, 60 minutes, 24 hours and 30 days, which are 174 numbers in total. 1000 clickable bvars take roughly 1M memory.

Calculate and view percentiles

x-ile (short for x-th percentile) is the value ranked at N * x%-th position amongst a group of ordered values. E.g. If there’re 1000 values inside a time window, sort them in ascending order first. The 500-th value(1000 * 50%) in the ordered list is 50-ile(a.k.a median), the 990-th(1000 * 99%) value is 99-ile, the 999-th value is 99.9-ile. Percentiles give more information on how latencies distribute than mean values, and being helpful for analyzing behavior of the system more accurately. Industrial-grade services often require SLA to be not less than 99.97% (the requirement for 2nd-level services inside Baidu, >=99.99% for 1st-level services), even if a system has good average latencies, a bad long-tail area may still break SLA. Percentiles do help analyzing the long-tail area.

Percentiles can be plotted as a CDF or percentiles-over-time curve.

Following diagram plots percentiles as CDF, where the X-axis is the ratio(ranked-position/total-number) and the Y-axis is the corresponding percentile. E.g. The Y value corresponding to X=50% is 50-ile. If a system requires that “99.9% requests need to be processed within Y milliseconds”, you should check the Y at 99.9%.

CDF ? When a Y=y is chosen, the corresponding X means “percentage of values <= y”. Since values are sampled randomly (and uniformly), the X can be viewed as “probability of values <= y”, or P(values <= y), which is just the definition of CDF.

PDF. If we divide the Y-axis of the CDF into many small-range segments, calculate the difference between X values of both ends of each segment, and use the difference as new value for X-axis, a PDF curve would be plotted, just like a normal distribution rotated 90 degrees clockwise. However density of the median is often much higher than others in a PDF and probably make long-tail area very flat and hard to read. As a result, systems prefer showing distributions in CDF rather than PDF.

Here’re 2 simple rules to check if a CDF curve is good or not:

The flatter the better. A horizontal line is an ideal CDF curve which means that there’re no waitings, congestions or pauses, very unlikely in practice.
The area between 99% and 100% should be as small as possible: right-side of 99% is the long-tail area, which has a significant impact on SLA.

A CDF with slowly ascending curve and small long-tail area is great in practice.

Following diagram plots percentiles over time and has four curves. The X-axis is time and Y-axis from top to bottom are 99.9% 99% 90% 50% percentiles respectively, plotted in lighter and lighter colors (from orange to yellow).

Hovering mouse over the curves shows corresponding values at the time. The tooltip in above diagram means “The 99% percentile of latency before 39 seconds is 330 microseconds”. The diagram does not include the 99.99-ile curve which is usually significantly higher than others, making others hard to read. You may click bvars ended with “_latency_9999” to read the 99.99-ile curve separately. This diagram shows how percentiles change over time, which is helpful to analyze performance regressions of systems.

brpc calculates latency distributions of services automatically, which do not need users to add manually. The metrics are as follows:

bvar::LatencyRecorderbvar-c++ for details):

#include <bvar/bvar.h>

...
bvar::LatencyRecorder g_latency_recorder("client"); // expose this recorder
...
void foo() {
 ...
 g_latency_recorder << my_latency;
 ...
}

If the application already starts a brpc server, values like client_latency, client_latency_cdf can be viewed from /vars as follows. Clicking them to see (dynamically-updated) curves:

Non brpc server

here.

Docs: connections

Thu, 12 Aug 2021 00:00:00 +0000

connections服务可以查看所有的连接。一个典型的页面如下：

server_socket_count: 5

CreatedTime	RemoteSide	SSL	Protocol	fd	BytesIn/s	In/s	BytesOut/s	Out/s	BytesIn/m	In/m	BytesOut/m	Out/m	SocketId
2015/09/21-21:32:09.630840	172.22.38.217:51379	No	http	19	1300	1	269	1	68844	53	115860	53	257
2015/09/21-21:32:09.630857	172.22.38.217:51380	No	http	20	1308	1	5766	1	68884	53	129978	53	258
2015/09/21-21:32:09.630880	172.22.38.217:51381	No	http	21	1292	1	1447	1	67672	52	143414	52	259
2015/09/21-21:32:01.324587	127.0.0.1:55385	No	baidu_std	15	1480	20	880	20	88020	1192	52260	1192	512
2015/09/21-21:32:01.325969	127.0.0.1:55387	No	baidu_std	17	4016	40	1554	40	238879	2384	92660	2384	1024

channel_socket_count: 1

CreatedTime	RemoteSide	SSL	Protocol	fd	BytesIn/s	In/s	BytesOut/s	Out/s	BytesIn/m	In/m	BytesOut/m	Out/m	SocketId
2015/09/21-21:32:01.325870	127.0.0.1:8765	No	baidu_std	16	1554	40	4016	40	92660	2384	238879	2384	0

channel_short_socket_count: 0

上述信息分为三段：

第一段是server接受(accept)的连接。
第二段是server与下游的单连接（使用brpc::Channel建立），fd为-1的是虚拟连接，对应第三段中所有相同RemoteSide的连接。
第三段是server与下游的短连接或连接池(pooled connections)，这些连接从属于第二段中的相同RemoteSide的虚拟连接。

表格标题的含义：

RemoteSide : 远端的ip和端口。
SSL：是否使用SSL加密，若为Yes的话，一般是HTTPS连接。
Protocol : 使用的协议，可能为baidu_std hulu_pbrpc sofa_pbrpc memcache http public_pbrpc nova_pbrpc nshead_server等。
fd : file descriptor（文件描述符），可能为-1。
BytesIn/s : 上一秒读入的字节数
In/s : 上一秒读入的消息数（消息是对request和response的统称）
BytesOut/s : 上一秒写出的字节数
Out/s : 上一秒写出的消息数
BytesIn/m: 上一分钟读入的字节数
In/m: 上一分钟读入的消息数
BytesOut/m: 上一分钟写出的字节数
Out/m: 上一分钟写出的消息数
SocketId ：内部id，用于debug，用户不用关心。

典型截图分别如下所示：

单连接：

连接池：

短连接：

Docs: flags

Thu, 12 Aug 2021 00:00:00 +0000

flags服务了解每个flag的具体功能。如果你的程序还没有使用gflags，我们建议你使用，原因如下：

命令行和文件均可传入，前者方便做测试，后者适合线上运维。放在文件中的gflags可以reload。而configure只支持从文件读取配置。
你可以在浏览器中查看brpc服务器中所有gflags，并对其动态修改（如果允许的话）。configure不可能做到这点。
gflags分散在和其作用紧密关联的文件中，更好管理。而使用configure需要聚集到一个庞大的读取函数中。

Usage of gflags

gflags一般定义在需要它的源文件中。#include <gflags/gflags.h>后在全局scope加入DEFINE_<type>(<name>, <default-value>, <description>); 比如：

#include <gflags/gflags.h>
...
DEFINE_bool(hex_log_id, false, "Show log_id in hexadecimal");
DEFINE_int32(health_check_interval, 3, "seconds between consecutive health-checkings");

一般在main函数开头用ParseCommandLineFlags处理程序参数：

#include <gflags/gflags.h>
...
int main(int argc, char* argv[]) {
 google::ParseCommandLineFlags(&argc, &argv, true/*表示把识别的参数从argc/argv中删除*/);
 ...
}

如果要从conf/gflags.conf中加载gflags，则可以加上参数-flagfile=conf/gflags.conf。如果希望默认（什么参数都不加）就从文件中读取，则可以在程序中直接给flagfile赋值，一般这么写

google::SetCommandLineOption("flagfile", "conf/gflags.conf");

程序启动时会检查conf/gflags.conf是否存在，如果不存在则会报错：

$ ./my_program
conf/gflags.conf: No such file or directory

官方文档。

flagfile

在命令行中参数和值之间可不加等号，而在flagfile中一定要加。比如./myapp -param 7是ok的，但在./myapp -flagfile=./gflags.conf对应的gflags.conf中一定要写成 -param=7 或 –param=7，否则就不正确且不会报错。

在命令行中字符串可用单引号或双引号包围，而在flagfile中不能加。比如./myapp -name="tom"或./myapp -name='tom'都是ok的，但在./myapp -flagfile=./gflags.conf对应的gflags.conf中一定要写成 -name=tom 或 –name=tom，如果写成-name=“tom"的话，引号也会作为值的一部分。配置文件中的值可以有空格，比如gflags.conf中写成-name=value with spaces是ok的，参数name的值就是value with spaces，而在命令行中要用引号括起来。

flagfile中参数可由单横线(如-foo)或双横线(如–foo)打头，但不能以三横线或更多横线打头，否则的话是无效参数且不会报错!

flagfile中以#开头的行被认为是注释。开头的空格和空白行都会被忽略。

flagfile中可以使用--flagfile包含另一个flagfile。

Change gflag on-the-fly

flags服务可以查看服务器进程中所有的gflags。修改过的flags会以红色高亮。“修改过”指的是修改这一行为，即使再改回默认值，仍然会显示为红色。

/flags：列出所有的gflags

/flags/NAME：查询名字为NAME的gflag

/flags/NAME1,NAME2,NAME3：查询名字为NAME1或NAME2或NAME3的gflag

/flags/foo*,b$r：查询名字与某一统配符匹配的gflag，注意用$代替?匹配单个字符，因为?在url中有特殊含义。

访问/flags/NAME?setvalue=VALUE即可动态修改一个gflag的值，validator会被调用。

为了防止误修改，需要动态修改的gflag必须有validator，显示此类gflag名字时有(R)后缀。

修改成功后会显示如下信息：

尝试修改不允许修改的gflag会显示如下错误信息：

设置一个不允许的值会显示如下错误（flag值不会变化）：

r31658之后支持可视化地修改，在浏览器上访问时将看到(R)下多了下划线：

点击后在一个独立页面可视化地修改对应的flag：

填入true后确定：

返回/flags可以看到对应的flag已经被修改了：

关于重载gflags，重点关注：

避免在一段代码中多次调用同一个gflag，应把该gflag的值保存下来并调用该值。因为gflag的值随时可能变化，而产生意想不到的结果。
使用google::GetCommandLineOption()访问string类型的gflag，直接访问是线程不安全的。
处理逻辑和副作用应放到validator里去。比如修改FLAGS_foo后得更新另一处的值，如果只是写在程序初始化的地方，而不是validator里，那么重载时这段逻辑就运行不到了。

如果你确认某个gflag不需要额外的线程同步和处理逻辑就可以重载，那么可以用如下方式为其注册一个总是返回true的validator：

DEFINE_bool(hex_log_id, false, "Show log_id in hexadecimal");
BRPC_VALIDATE_GFLAG(hex_log_id, brpc::PassValidate/*always true*/);

这个flag是单纯的开关，修改后不需要更新其他数据（没有处理逻辑），代码中前面看到true后面看到false也不会产生什么后果（不需要线程同步），所以我们让其默认可重载。

对于int32和int64类型，有一个判断是否为正数的常用validator：

DEFINE_int32(health_check_interval, 3, "seconds between consecutive health-checkings");
BRPC_VALIDATE_GFLAG(health_check_interval, brpc::PositiveInteger);

以上操作都可以在命令行中进行：

$ curl brpc.baidu.com:8765/flags/health_check_interval
Name | Value | Description | Defined At
---------------------------------------
health_check_interval (R) | 3 | seconds between consecutive health-checkings | src/brpc/socket_map.cpp

1.0.251.32399后增加了-immutable_flags，打开后所有的gflags将不能被动态修改。当一个服务对某个gflag值比较敏感且不希望在线上被误改，可打开这个开关。打开这个开关的同时也意味着你无法动态修改线上的配置，每次修改都要重启程序，对于还在调试阶段或待收敛阶段的程序不建议打开。

Docs: rpcz

Thu, 12 Aug 2021 00:00:00 +0000

dapper一个长期运行的例子。

关于开销：我们的实现完全规避了线程竞争，开销极小，在qps 30万的测试场景中，观察不到明显的性能变化，对大部分应用而言应该是“free”的。即使采集了几千万条请求，rpcz也不会增加很多内存，一般在50兆以内。rpcz会占用一些磁盘空间（就像日志一样），如果设定为存一个小时的数据，一般在几百兆左右。

开关方法

-enable_rpcz选项会在启动后开启。

Name	Value	Description	Defined At
enable_rpcz (R)	true (default:false)	Turn on rpcz	src/baidu/rpc/builtin/rpcz_service.cpp
rpcz_hex_log_id (R)	false	Show log_id in hexadecimal	src/baidu/rpc/builtin/rpcz_service.cpp
rpcz_database_dir	./rpc_data/rpcz	For storing requests/contexts collected by rpcz.	src/baidu/rpc/span.cpp
rpcz_keep_span_db	false	Don’t remove DB of rpcz at program’s exit	src/baidu/rpc/span.cpp
rpcz_keep_span_seconds (R)	3600	Keep spans for at most so many seconds	src/baidu/rpc/span.cpp

若启动时未加-enable_rpcz，则可在启动后访问SERVER_URL/rpcz/enable动态开启rpcz，访问SERVER_URL/rpcz/disable则关闭，这两个链接等价于访问SERVER_URL/flags/enable_rpcz?setvalue=true和SERVER_URL/flags/enable_rpcz?setvalue=false。在r31010之后，rpc在html版本中增加了一个按钮可视化地开启和关闭。

这里。

数据展现

/rpcz展现的数据分为两层。

第一层

看到最新请求的概况，点击链接进入第二层。

第二层

看到某系列(trace)或某个请求(span)的详细信息。一般通过点击链接进入，也可以把trace=和span=作为query-string拼出链接

内容说明：

时间分为了绝对时间（如2015/01/21-20:20:30.817392，小数点后精确到微秒）和前一个时间的差值（如. 19，代表19微秒)。
trace=ID有点像“session id”，对应一个系统中完成一次对外服务牵涉到的所有服务，即上下游server都共用一个trace-id。span=ID对应一个server或client中一个请求的处理过程。trace-id和span-id在概率上唯一。
第一层页面中的request=和response=后的是数据包的字节数，包括附件但不包括协议meta。第二层中request和response的字节数一般在括号里，比如"Responded(13)“中的13。
点击链接可能会访问其他server上的rpcz，点浏览器后退一般会返回到之前的页面位置。
I’m the last call, I’m about to …都是用户的annotation。

Annotation

TRACEPRINTF打印内容到事件流中，比如：

TRACEPRINTF("Hello rpcz %d", 123);

这条annotation会按其发生时间插入到对应请求的rpcz中。从这个角度看，rpcz是请求级的日志。如果你用TRACEPRINTF打印了沿路的上下文，便可看到请求在每个阶段停留的时间，牵涉到的数据集和参数。这是个很有用的功能。

Docs: cpu profiler

Thu, 12 Aug 2021 00:00:00 +0000

brpc可以分析程序中的热点函数。

开启方法

链接libtcmalloc_and_profiler.a
1. crash.可能由于tcmalloc不及时归还内存，越界访问不会crash。
2. 如果tcmalloc使用frame pointer而不是libunwind回溯栈，请确保在CXXFLAGS或CFLAGS中加上-fno-omit-frame-pointer，否则函数间的调用关系会丢失，最后产生的图片中都是彼此独立的函数方框。
定义宏BRPC_ENABLE_CPU_PROFILER, 一般加入编译参数-DBRPC_ENABLE_CPU_PROFILER。
这里。

注意要关闭Server端的认证，否则可能会看到这个：

$ tools/pprof --text localhost:9002/pprof/profile
Use of uninitialized value in substitution (s///) at tools/pprof line 2703.
http://localhost:9002/profile/symbol doesn't exist

server端可能会有这样的日志：

FATAL: 12-26 10:01:25: * 0 [src/brpc/policy/giano_authenticator.cpp:65][4294969345] Giano fails to verify credentical, 70003
WARNING: 12-26 10:01:25: * 0 [src/brpc/input_messenger.cpp:132][4294969345] Authentication failed, remote side(127.0.0.1:22989) of sockfd=5, close it

查看方法

通过builtin service的 /hotspots/cpu 页面查看
通过pprof 工具查看，如 tools/pprof –text localhost:9002/pprof/profile

控制采样频率

启动前设置环境变量：export CPUPROFILE_FREQUENCY=xxx

默认值为: 100

控制采样时间

url加上?seconds=秒数，如/hotspots/cpu?seconds=5

图示

下图是一次运行cpu profiler后的结果：

左上角是总体信息，包括时间，程序名，总采样数等等。
View框中可以选择查看之前运行过的profile结果，Diff框中可选择查看和之前的结果的变化量，重启后清空。
代表函数调用的方框中的字段从上到下依次为：函数名，这个函数本身（除去所有子函数）占的采样数和比例，这个函数及调用的所有子函数累计的采样数和比例。采样数越大框越大。
方框之间连线上的数字表示被采样到的上层函数对下层函数的调用数，数字越大线越粗。

热点分析一般开始于找到最大的框最粗的线考察其来源及去向。

cpu profiler的原理是在定期被调用的SIGPROF handler中采样所在线程的栈，由于handler（在linux 2.6后）会被随机地摆放于活跃线程的栈上运行，cpu profiler在运行一段时间后能以很大的概率采集到所有活跃线程中的活跃函数，最后根据栈代表的函数调用关系汇总为调用图，并把地址转换成符号，这就是我们看到的结果图了。采集频率由环境变量CPUPROFILE_FREQUENCY控制，默认100，即每秒钟100次或每10ms一次。在实践中cpu profiler对原程序的影响不明显。

pprof或gperftools中的pprof进行profiling。

比如pprof --text localhost:9002 --seconds=5的意思是统计运行在本机9002端口的server的cpu情况，时长5秒。一次运行的例子如下：

$ tools/pprof --text 0.0.0.0:9002 --seconds=5
Gathering CPU profile from http://0.0.0.0:9002/pprof/profile?seconds=5 for 5 seconds to
/home/gejun/pprof/echo_server.1419501210.0.0.0.0
Be patient...
Wrote profile to /home/gejun/pprof/echo_server.1419501210.0.0.0.0
Removing funlockfile from all stack traces.
Total: 2946 samples
1161 39.4% 39.4% 1161 39.4% syscall
248 8.4% 47.8% 248 8.4% bthread::TaskControl::steal_task
227 7.7% 55.5% 227 7.7% writev
87 3.0% 58.5% 88 3.0% ::cpp_alloc
74 2.5% 61.0% 74 2.5% __read_nocancel
46 1.6% 62.6% 48 1.6% tc_delete
42 1.4% 64.0% 42 1.4% brpc::Socket::Address
41 1.4% 65.4% 41 1.4% epoll_wait
35 1.2% 66.6% 35 1.2% memcpy
33 1.1% 67.7% 33 1.1% __pthread_getspecific
33 1.1% 68.8% 33 1.1% brpc::Socket::Write
33 1.1% 69.9% 33 1.1% epoll_ctl
28 1.0% 70.9% 42 1.4% brpc::policy::ProcessRpcRequest
27 0.9% 71.8% 27 0.9% butil::IOBuf::_push_back_ref
27 0.9% 72.7% 27 0.9% bthread::TaskGroup::ending_sched

省略–text进入交互模式，如下图所示：

$ tools/pprof localhost:9002 --seconds=5
Gathering CPU profile from http://0.0.0.0:9002/pprof/profile?seconds=5 for 5 seconds to
/home/gejun/pprof/echo_server.1419501236.0.0.0.0
Be patient...
Wrote profile to /home/gejun/pprof/echo_server.1419501236.0.0.0.0
Removing funlockfile from all stack traces.
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 2954 samples
1099 37.2% 37.2% 1099 37.2% syscall
253 8.6% 45.8% 253 8.6% bthread::TaskControl::steal_task
240 8.1% 53.9% 240 8.1% writev
90 3.0% 56.9% 90 3.0% ::cpp_alloc
67 2.3% 59.2% 67 2.3% __read_nocancel
47 1.6% 60.8% 47 1.6% butil::IOBuf::_push_back_ref
42 1.4% 62.2% 56 1.9% brpc::policy::ProcessRpcRequest
41 1.4% 63.6% 41 1.4% epoll_wait
38 1.3% 64.9% 38 1.3% epoll_ctl
37 1.3% 66.1% 37 1.3% memcpy
35 1.2% 67.3% 35 1.2% brpc::Socket::Address

MacOS的额外配置

在MacOS下，gperftools中的perl pprof脚本无法将函数地址转变成函数名，解决办法是：

standalone pprof，并把下载的pprof二进制文件路径写入环境变量GOOGLE_PPROF_BINARY_PATH中
安装llvm-symbolizer（将函数符号转化为函数名），直接用brew安装即可：brew install llvm

火焰图

FlameGraph工具，将环境变量FLAMEGRAPH_PL_PATH正确设置到本地的/path/to/flamegraph.pl后启动server即可。

Docs: heap profiler

Thu, 12 Aug 2021 00:00:00 +0000

brpc可以分析内存是被哪些函数占据的。heap profiler的原理是每分配满一些内存就采样调用处的栈，“一些”由环境变量TCMALLOC_SAMPLE_PARAMETER控制，默认524288，即512K字节。根据栈表现出的函数调用关系汇总为我们看到的结果图。在实践中heap profiler对原程序的影响不明显。

开启方法

链接libtcmalloc_and_profiler.a
1. 如果tcmalloc使用frame pointer而不是libunwind回溯栈，请确保在CXXFLAGS或CFLAGS中加上-fno-omit-frame-pointer，否则函数间的调用关系会丢失，最后产生的图片中都是彼此独立的函数方框。

在shell中export TCMALLOC_SAMPLE_PARAMETER=524288官方文档建议设置为524288。这个变量也可在运行前临时设置，如TCMALLOC_SAMPLE_PARAMETER=524288 ./server。如果没有这个环境变量，可能会看到这样的结果：

$ tools/pprof --text localhost:9002/pprof/heap
Fetching /pprof/heap profile from http://localhost:9002/pprof/heap to
/home/gejun/pprof/echo_server.1419559063.localhost.pprof.heap
Wrote profile to /home/gejun/pprof/echo_server.1419559063.localhost.pprof.heap
/home/gejun/pprof/echo_server.1419559063.localhost.pprof.heap: header size >= 2**16

这里。

注意要关闭Server端的认证，否则可能会看到这个：

$ tools/pprof --text localhost:9002/pprof/heap
Use of uninitialized value in substitution (s///) at tools/pprof line 2703.
http://localhost:9002/pprof/symbol doesn't exist

server端可能会有这样的日志：

FATAL: 12-26 10:01:25: * 0 [src/brpc/policy/giano_authenticator.cpp:65][4294969345] Giano fails to verify credentical, 70003
WARNING: 12-26 10:01:25: * 0 [src/brpc/input_messenger.cpp:132][4294969345] Authentication failed, remote side(127.0.0.1:22989) of sockfd=5, close it

图示

左上角是当前程序通过malloc分配的内存总量，顺着箭头上的数字可以看到内存来自哪些函数。

点击左上角的text选择框可以查看文本格式的结果，有时候这种按分配量排序的形式更方便。

左上角的两个选择框作用分别是：

–max_profiles_kept调整。
Diff：和选择的profile做对比。表示什么都不选。如果你选择了之前的某个profile，那么将看到View框中的profile相比Diff框中profile的变化量。

下图演示了勾选Diff和Text的效果。

在Linux下，你也可以使用pprof脚本（tools/pprof）在命令行中查看文本格式结果：

$ tools/pprof --text db-rpc-dev00.db01:8765/pprof/heap
Fetching /pprof/heap profile from http://db-rpc-dev00.db01:8765/pprof/heap to
/home/gejun/pprof/play_server.1453216025.db-rpc-dev00.db01.pprof.heap
Wrote profile to /home/gejun/pprof/play_server.1453216025.db-rpc-dev00.db01.pprof.heap
Adjusting heap profiles for 1-in-524288 sampling rate
Heap version 2
Total: 38.9 MB
35.8 92.0% 92.0% 35.8 92.0% ::cpp_alloc
2.1 5.4% 97.4% 2.1 5.4% butil::FlatMap
0.5 1.3% 98.7% 0.5 1.3% butil::IOBuf::append
0.5 1.3% 100.0% 0.5 1.3% butil::IOBufAsZeroCopyOutputStream::Next
0.0 0.0% 100.0% 0.6 1.5% MallocExtension::GetHeapSample
0.0 0.0% 100.0% 0.5 1.3% ProfileHandler::Init
0.0 0.0% 100.0% 0.5 1.3% ProfileHandlerRegisterCallback
0.0 0.0% 100.0% 0.5 1.3% __do_global_ctors_aux
0.0 0.0% 100.0% 1.6 4.2% _end
0.0 0.0% 100.0% 0.5 1.3% _init
0.0 0.0% 100.0% 0.6 1.5% brpc::CloseIdleConnections
0.0 0.0% 100.0% 1.1 2.9% brpc::GlobalUpdate
0.0 0.0% 100.0% 0.6 1.5% brpc::PProfService::heap
0.0 0.0% 100.0% 1.9 4.9% brpc::Socket::Create
0.0 0.0% 100.0% 2.9 7.4% brpc::Socket::Write
0.0 0.0% 100.0% 3.8 9.7% brpc::Span::CreateServerSpan
0.0 0.0% 100.0% 1.4 3.5% brpc::SpanQueue::Push
0.0 0.0% 100.0% 1.9 4.8% butil::ObjectPool
0.0 0.0% 100.0% 0.8 2.0% butil::ResourcePool
0.0 0.0% 100.0% 1.0 2.6% butil::iobuf::tls_block
0.0 0.0% 100.0% 1.0 2.6% bthread::TimerThread::Bucket::schedule
0.0 0.0% 100.0% 1.6 4.1% bthread::get_stack
0.0 0.0% 100.0% 4.2 10.8% bthread_id_create
0.0 0.0% 100.0% 1.1 2.9% bvar::Variable::describe_series_exposed
0.0 0.0% 100.0% 1.0 2.6% bvar::detail::AgentGroup
0.0 0.0% 100.0% 0.5 1.3% bvar::detail::Percentile::operator
0.0 0.0% 100.0% 0.5 1.3% bvar::detail::PercentileSamples
0.0 0.0% 100.0% 0.5 1.3% bvar::detail::Sampler::schedule
0.0 0.0% 100.0% 6.5 16.8% leveldb::Arena::AllocateNewBlock
0.0 0.0% 100.0% 0.5 1.3% leveldb::VersionSet::LogAndApply
0.0 0.0% 100.0% 4.2 10.8% pthread_mutex_unlock
0.0 0.0% 100.0% 0.5 1.3% pthread_once
0.0 0.0% 100.0% 0.5 1.3% std::_Rb_tree
0.0 0.0% 100.0% 1.5 3.9% std::basic_string
0.0 0.0% 100.0% 3.5 9.0% std::string::_Rep::_S_create

brpc还提供一个类似的growth profiler分析内存的分配去向（不考虑释放）。

MacOS的额外配置

standalone pprof，并把下载的pprof二进制文件路径写入环境变量GOOGLE_PPROF_BINARY_PATH中
安装llvm-symbolizer（将函数符号转化为函数名），直接用brew安装即可：brew install llvm

Docs: contention profiler

Thu, 12 Aug 2021 00:00:00 +0000

brpc可以分析花在等待锁上的时间及发生等待的函数。

开启方法

这里。

图示

当很多线程争抢同一把锁时，一些线程无法立刻获得锁，而必须睡眠直到某个线程退出临界区。这个争抢过程我们称之为contention。在多核机器上，当多个线程需要操作同一个资源却被一把锁挡住时，便无法充分发挥多个核心的并发能力。现代OS通过提供比锁更底层的同步原语，使得无竞争锁完全不需要系统调用，只是一两条wait-free，耗时10-20ns的原子操作，非常快。而锁一旦发生竞争，一些线程就要陷入睡眠，再次醒来触发了OS的调度代码，代价至少为3-5us。所以让锁尽量无竞争，让所有线程“一起飞”是需要高性能的server的永恒话题。

cpu profiler中。cpu profiler可以抓到特别频繁的锁（以至于花费了很多cpu），但耗时真正巨大的临界区往往不是那么频繁，而无法被cpu profiler发现。**contention profiler和cpu profiler好似互补关系，前者分析等待时间（被动），后者分析忙碌时间。**还有一类由用户基于condition或sleep发起的主动等待时间，无需分析。

目前contention profiler支持pthread_mutex_t（非递归）和bthread_mutex_t，开启后每秒最多采集1000个竞争锁，这个数字由参数-bvar_collector_expected_per_second控制（同时影响rpc_dump）。

Name	Value	Description	Defined At
bvar_collector_expected_per_second	1000	Expected number of samples to be collected per second	bvar/collector.cpp

如果一秒内竞争锁的次数Ｎ超过了1000，那么每把锁会有1000/N的概率被采集。在我们的各类测试场景中（qps在10万-60万不等）没有观察到被采集程序的性能有明显变化。

我们通过实际例子来看下如何使用contention profiler，点击“contention”按钮（more左侧）后就会开启默认10秒的分析过程。下图是libraft中的一个示例程序的锁状况，这个程序是3个节点复制组的leader，qps在10-12万左右。左上角的Total seconds: 2.449是采集时间内（10秒）在锁上花费的所有等待时间。注意是“等待”，无竞争的锁不会被采集也不会出现在下图中。顺着箭头往下走能看到每份时间来自哪些函数。

上图有点大，让我们放大一个局部看看。下图红框中的0.768是这个局部中最大的数字，它代表raft::LogManager::get_entry在等待涉及到bvar::detail::UniqueLockBase的函数上共等待了0.768秒（10秒内）。我们如果觉得这个时间不符合预期，就可以去排查代码。

点击上方的count选择框，可以查看锁的竞争次数。选择后左上角变为了Total samples: 439026，代表采集时间内总共的锁竞争次数（估算）。图中箭头上的数字也相应地变为了次数，而不是时间。对比同一份结果的时间和次数，可以更深入地理解竞争状况。