protobuf使用说明书

protobuf使用说明书

本说明旨在指导和帮助用户如何简单快速的使用protobuf,提供了java和python两种使用示例。如果对其实现算法及其他细节感兴趣请参考protobuf官方文档,但是需要翻墙访问,不过在我天朝程序员面前,天空飘来五个字儿:这都不叫事儿!

一、环境准备

1、下载protoc编译器压缩包,解压后将bin目录配置到环境变量。

2、安装protobuf依赖库:

  • python:pip install protobuf 或者下载源码进行手动安装
  • java:配置maven(gradle)依赖,或者下载jar包手动添加外部依赖。pom依赖示例如下:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>vdcoding</groupId>
  <artifactId>com.vdcoding</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <dependencies>
  	<dependency>
  	    <groupId>com.google.protobuf</groupId>
  	    <artifactId>protobuf-java</artifactId>
  	    <version>3.5.1</version>
  	</dependency>
	<dependency>
	    <groupId>com.google.protobuf</groupId>
	    <artifactId>protobuf-java-util</artifactId>
	    <version>3.5.1</version>
	</dependency>
  </dependencies>
</project>

二、准备proto文件,样例文件protobuf_demo.proto如下:

//声明proto语法,还支持proto2,但是最好使用proto3,做了很多简化和优化,具体区别这里不做说明
syntax = "proto3";
/*
声明编译后的代码的包名称,用于防止命名空间冲突
python是依赖本地系统目录层级,用不到该选项,但最好声明
*/
package com.vdcoding;

/*
声明在java中引用的包名称,如果不声明则使用文件开头定义package的名称
如果proto文件被编译为非java代码,则该选项不会有任何影响
*/
option java_package = "com.vdcoding";

/*声明在java中引用的类名称,如果不声明则使用文件名称经过驼峰式命名后作为类名称
编译后会根据声明的包和类名称生成完整层级的包目录
*/
option java_outer_classname = "ProtoDemo";

/*
选择编译模式,只对C++和java有效果,有三个可选值:
1、SPEED: 默认值,编译出的代码高度优化过,包括全部可用接口,而且运行速度嘎嘎的
编译后的代码行数2200+
2、CODE_SIZE: 编译出的代码相较SPEED体积略小,也就是没有经过特殊优化,所以速度略慢,
但是包括全部可用接口,如果proto文件内容巨大,而且对速度没有严苛的要求,可以选择该值
编译后的代码行数:1700+
3、LITE_RUNTIME: 编译器会依赖一个瘦身版的库libprotobuf-lite,所以编译出的代码
很轻量,并且只提供部分主要接口,但是速度和SPEED一样。适合运行于手机平台上。
编译后的代码行数:1600+
*/
option optimize_for = SPEED;


message OnlineUser{
    enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }
    message UserInfo{
        uint64 uid = 1;
        string usre_name = 2;
        PhoneType type = 3;
        uint32 phone_num = 4;
        uint32 cli_type = 5;
    }
    uint32 rsp_code = 1;
    string product = 2;
    uint64 query_time = 3;
    //repeated相当于java中的ArrayList,python中的list
    repeated uint32 random = 4;
    repeated UserInfo user_list = 5;
}

必要的说明:

1、字段限制:可以在字段前添加required或optional标记,例如:required uint32 uid=1,用来标记某个字段在序列化时是否必传,如果不添加该标记则默认为optional,例子protobuf_demo.proto中的所有字段都是可选的,也建议这样做,伸缩性好。

2、默认值处理,即在序列化与反序列化时不给相应字段传值的默认值处理

  • string类型默认为空字符串
  • bytes类型默认为空字节串
  • bool类型默认为false
  • 数字类型默认为0
  • enum类型默认为枚举类型中的第一个定义的值,而且必须为0
  • repeated默认值为空数组

关于protobuf支持的字段类型,这里给出C++、Java、python所对应的类型,可参考如下列表:

.proto Type Notes C++ Type Java Type Python Type
double double double float
float float float float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long
uint32 Uses variable-length encoding. uint32 int int/long
uint64 Uses variable-length encoding. uint64 long int/long
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 228. uint32 int int
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 256. uint64 long int/long
sfixed32 Always four bytes. int32 int int
sfixed64 Always eight bytes. int64 long int/long
bool bool boolean bool
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode
bytes May contain any arbitrary sequence of bytes. string ByteString str

三、编译protobuf文件,将其转换为目标语言的代码文件

第一步已经配置好protoc编译器,通过命令行进入proto文件所在的目录,然后在命令行输入如下命令:

1、java版本:

protoc --java_out=./ protobuf_demo.proto

编译后会在当前目录下按照定义的package和classname生成包和类文件ProtoDemo.java

2、python版本:

python: protoc --python_out=./ protobuf_demo.proto

编译后在当前目录下生成protobuf_demo_pb2.py文件

关于protoc支持的其他命令行选项可输入protoc –help查看。

四、利用生成的代码文件进行实际应用开发,这一部分通过一个网络IO的例子来演示如何利用protobuf提供的API进行序列化与反序列化操作。

1、首先定义一个sockerserver来接收序列化的数据并将数据原路返回,这部分server端的代码利用python定义

#echo_server.py

from socketserver import BaseRequestHandler
from socketserver import ThreadingTCPServer


class EchoServer(BaseRequestHandler):
    def handle(self):
        print("Client connected:", self.client_address)
        data = self.request.recv(4096)
        print("Received pb data:", data)
        self.request.send(data)

server = ThreadingTCPServer(("127.0.0.1", 6000), EchoServer)
server.serve_forever()

2、定义client端,用于向echo_server发送及接收数据,client端的代码中需要引用第三步编译生成的代码文件及第一步安装的protobuf库中所提供的各种API。

Python版本:

#protodemo.py

import time
from socket import socket, AF_INET, SOCK_STREAM
from threading import Thread

from google.protobuf.json_format import MessageToDict, ParseDict
from protobuf_demo_pb2 import OnlineUser


class Client:
    def __init__(self, ip, port):
        self.soc = socket(AF_INET, SOCK_STREAM)
        self.soc.connect((ip, port))
        self.listener = Thread(target=self.on_message)
        self.listener.setDaemon(True)
        self.listener.start()

    def on_message(self):
        while self.soc:
            data = self.soc.recv(4096)
            if data:
                print("Received pb data:", data)
                #实际应用时,这里应该先解析到单个数据包的大小,再截取对应长度的数据做解析
                user.ParseFromString(data)
                #将反序列化后的user转换为字典对象,protobuf还提供了MessageToJson用来转换为json对象
                user_dict = MessageToDict(user, 
                                    including_default_value_fields=True, 
                                    preserving_proto_field_name=True)
                print("Parse pb to dict:", user_dict)

    def send_data(self, data):
        self.soc.send(data)
        print("Send pb data:", data)
    
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, tb):
        self.soc.close()

user1 = {
    "uid": 123456,
    "usre_name": 'tester',
    "type": 2,
    "phone_num": "119",
    "cli_type": 5
}
demo_struct = {
    "rsp_code": 0,
    "product": None,
    "query_time": int(time.time()),
    # "random": [888, 999],
    "user_list": [user1]
}

#实例化一个protobuf message对象
user = OnlineUser()
#将字典对象的数据赋值给message对象
ParseDict(demo_struct, user)
#message对象可以直接通过属性访问
print("User list:", user.user_list)
#序列化为pb二进制数据用于IO操作
ser_user = user.SerializeToString()

with Client('127.0.0.1', 6000) as client:
    client.send_data(ser_user)
    time.sleep(1)

启动echo_server,然后运行protodemo.py会得到如下输出:

User list: [uid: 123456
usre_name: "tester"
type: WORK
phone_num: 119
cli_type: 5
]
Send pb data: b'\x18\xec\xbb\x93\xd5\x05*\x12\x08\xc0\xc4\x07\x12\x06tester\x18\x02 w(\x05'
Received pb data: b'\x18\xec\xbb\x93\xd5\x05*\x12\x08\xc0\xc4\x07\x12\x06tester\x18\x02 w(\x05'
Parse pb to dict: {'query_time': '1520754156', 'user_list': [{'uid': '123456', 'usre_name': 'tester', 'type': 'WORK', 'phone_num': 119, 'cli_type': 5}], 'rsp_code': 0, 'product': '', 'random': []}

Java版本TestProto.java:

package com.vdcoding;

import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.nio.channels.SocketChannel;

import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat.Printer;
import com.google.protobuf.util.JsonFormat;

import com.vdcoding.ProtoDemo.OnlineUser;;

public class TestProto {
	private Socket socket;
	private SocketChannel channel = null;
	public TestProto(String ip, int port) throws IOException {
		channel = SocketChannel.open();
		socket = channel.socket();
		socket.connect(new InetSocketAddress(ip, port), 5000);
		System.out.println("Connect to server:" + socket.getRemoteSocketAddress());
	}
	
	public void sendData(Message message){
		try {
			OutputStream os = socket.getOutputStream();
			//将protobuf message写入输出流
			message.writeTo(os);
			os.flush();
			System.out.println("Send data:" + message);
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
	
	public void messageListener(){
		Printer printer = JsonFormat.printer();
		while(socket.isConnected()){
			try {
				OnlineUser user = OnlineUser.parseFrom(socket.getInputStream());
				if(user.getRspCode()!=0){
					System.out.println("Received data:" + user);
					System.out.println("Convert user to json:" + printer.includingDefaultValueFields().print(user));
				}
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}
	public static void main(String[] args){
		OnlineUser.UserInfo.Builder userInfo = 
				OnlineUser.UserInfo.newBuilder()
				.setUid(123123123)
				.setUsreName("Paditon")
				.setCliType(16)
				.setPhoneNum(110)
				.setType(OnlineUser.PhoneType.MOBILE);
		OnlineUser.Builder builder = OnlineUser.newBuilder();
		builder.setRspCode(12);	//整型不设置值默认为0
		builder.setProduct("hello world");
		builder.setQueryTime(System.currentTimeMillis());
		builder.addRandom(999);
		builder.addRandom(888);
		builder.addUserList(userInfo);
		Message onlineuser = builder.build();
		try {
			TestProto tp = new TestProto("127.0.0.1", 6000);
			tp.sendData(onlineuser);
			Thread t = new Thread(new Runnable() {
				@Override
				public void run() {
					tp.messageListener();
				}
			});
			t.setDaemon(true);
			t.start();
			Thread.sleep(1000);
		} catch (InterruptedException e) {
				e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

输出差不多这里就不贴出来了。通过一个简单的例子来展示了下protobuf api的使用,当然其api是很多的,具体的可参考API官方文档。

说点什么

avatar
  订阅  
提醒