1# Protocol Specification
2
3The ttrpc protocol is client/server protocol to support multiple request streams
4over a single connection with lightweight framing. The client represents the
5process which initiated the underlying connection and the server is the process
6which accepted the connection. The protocol is currently defined as
7asymmetrical, with clients sending requests and servers sending responses. Both
8clients and servers are able to send stream data. The roles are also used in
9determining the stream identifiers, with client initiated streams using odd
10number identifiers and server initiated using even number. The protocol may be
11extended in the future to support server initiated streams, that is not
12supported in the latest version.
13
14## Purpose
15
16The ttrpc protocol is designed to be lightweight and optimized for low latency
17and reliable connections between processes on the same host. The protocol does
18not include features for handling unreliable connections such as handshakes,
19resets, pings, or flow control. The protocol is designed to make low-overhead
20implementations as simple as possible. It is not intended as a suitable
21replacement for HTTP2/3 over the network.
22
23## Message Frame
24
25Each Message Frame consists of a 10-byte message header followed
26by message data. The data length and stream ID are both big-endian
274-byte unsigned integers. The message type is an unsigned 1-byte
28integer. The flags are also an unsigned 1-byte integer and
29use is defined by the message type.
30
31 +---------------------------------------------------------------+
32 | Data Length (32) |
33 +---------------------------------------------------------------+
34 | Stream ID (32) |
35 +---------------+-----------------------------------------------+
36 | Msg Type (8) |
37 +---------------+
38 | Flags (8) |
39 +---------------+-----------------------------------------------+
40 | Data (*) |
41 +---------------------------------------------------------------+
42
43The Data Length field represents the number of bytes in the Data field. The
44total frame size will always be Data Length + 10 bytes. The maximum data length
45is 4MB and any larger size should be rejected. Due to the maximum data size
46being less than 16MB, the first frame byte should always be zero. This first
47byte should be considered reserved for future use.
48
49The Stream ID must be odd for client initiated streams and even for server
50initiated streams. Server initiated streams are not currently supported.
51
52## Mesage Types
53
54| Message Type | Name | Description |
55|--------------|----------|----------------------------------|
56| 0x01 | Request | Initiates stream |
57| 0x02 | Response | Final stream data and terminates |
58| 0x03 | Data | Stream data |
59
60### Request
61
62The request message is used to initiate stream and send along request data for
63properly routing and handling the stream. The stream may indicate unary without
64any inbound or outbound stream data with only a response is expected on the
65stream. The request may also indicate the stream is still open for more data and
66no response is expected until data is finished. If the remote indicates the
67stream is closed, the request may be considered non-unary but without anymore
68stream data sent. In the case of `remote closed`, the remote still expects to
69receive a response or stream data. For compatibility with non streaming clients,
70a request with empty flags indicates a unary request.
71
72#### Request Flags
73
74| Flag | Name | Description |
75|------|-----------------|--------------------------------------------------|
76| 0x01 | `remote closed` | Non-unary, but no more data expected from remote |
77| 0x02 | `remote open` | Non-unary, remote is still sending data |
78
79### Response
80
81The response message is used to end a stream with data, an empty response, or
82an error. A response message is the only expected message after a unary request.
83A non-unary request does not require a response message if the server is sending
84back stream data. A non-unary stream may return a single response message but no
85other stream data may follow.
86
87#### Response Flags
88
89No response flags are defined at this time, flags should be empty.
90
91### Data
92
93The data message is used to send data on an already initialized stream. Either
94client or server may send data. A data message is not allowed on a unary stream.
95A data message should not be sent after indicating `remote closed` to the peer.
96The last data message on a stream must set the `remote closed` flag.
97
98The `no data` flag is used to indicate that the data message does not include
99any data. This is normally used with the `remote closed` flag to indicate the
100stream is now closed without transmitting any data. Since ttrpc normally
101transmits a single object per message, a zero length data message may be
102interpreted as an empty object. For example, transmitting the number zero as a
103protobuf message ends up with a data length of zero, but the message is still
104considered data and should be processed.
105
106#### Data Flags
107
108| Flag | Name | Description |
109|------|-----------------|-----------------------------------|
110| 0x01 | `remote closed` | No more data expected from remote |
111| 0x04 | `no data` | This message does not have data |
112
113## Streaming
114
115All ttrpc requests use streams to transfer data. Unary streams will only have
116two messages sent per stream, a request from a client and a response from the
117server. Non-unary streams, however, may send any numbers of messages from the
118client and the server. This makes stream management more complicated than unary
119streams since both client and server need to track additional state. To keep
120this management as simple as possible, ttrpc minimizes the number of states and
121uses two flags instead of control frames. Each stream has two states while a
122stream is still alive: `local closed` and `remote closed`. Each peer considers
123local and remote from their own perspective and sets flags from the other peer's
124perspective. For example, if a client sends a data frame with the
125`remote closed` flag, that is indicating that the client is now `local closed`
126and the server will be `remote closed`. A unary operation does not need to send
127these flags since each received message always indicates `remote closed`. Once a
128peer is both `local closed` and `remote closed`, the stream is considered
129finished and may be cleaned up.
130
131Due to the asymmetric nature of the current protocol, a client should
132always be in the `local closed` state before `remote closed` and a server should
133always be in the `remote closed` state before `local closed`. This happens
134because the client is always initiating requests and a client always expects a
135final response back from a server to indicate the initiated request has been
136fulfilled. This may mean server sends a final empty response to finish a stream
137even after it has already completed sending data before the client.
138
139### Unary State Diagram
140
141 +--------+ +--------+
142 | Client | | Server |
143 +---+----+ +----+---+
144 | +---------+ |
145 local >---------------+ Request +--------------------> remote
146 closed | +---------+ | closed
147 | |
148 | +----------+ |
149 finished <--------------+ Response +--------------------< finished
150 | +----------+ |
151 | |
152
153### Non-Unary State Diagrams
154
155RC: `remote closed` flag
156RO: `remote open` flag
157
158 +--------+ +--------+
159 | Client | | Server |
160 +---+----+ +----+---+
161 | +--------------+ |
162 >-------------+ Request [RO] +----------------->
163 | +--------------+ |
164 | |
165 | +------+ |
166 >-----------------+ Data +--------------------->
167 | +------+ |
168 | |
169 | +-----------+ |
170 local >---------------+ Data [RC] +------------------> remote
171 closed | +-----------+ | closed
172 | |
173 | +----------+ |
174 finished <--------------+ Response +--------------------< finished
175 | +----------+ |
176 | |
177
178 +--------+ +--------+
179 | Client | | Server |
180 +---+----+ +----+---+
181 | +--------------+ |
182 local >-------------+ Request [RC] +-----------------> remote
183 closed | +--------------+ | closed
184 | |
185 | +------+ |
186 <-----------------+ Data +---------------------<
187 | +------+ |
188 | |
189 | +-----------+ |
190 finished <---------------+ Data [RC] +------------------< finished
191 | +-----------+ |
192 | |
193
194 +--------+ +--------+
195 | Client | | Server |
196 +---+----+ +----+---+
197 | +--------------+ |
198 >-------------+ Request [RO] +----------------->
199 | +--------------+ |
200 | |
201 | +------+ |
202 >-----------------+ Data +--------------------->
203 | +------+ |
204 | |
205 | +------+ |
206 <-----------------+ Data +---------------------<
207 | +------+ |
208 | |
209 | +------+ |
210 >-----------------+ Data +--------------------->
211 | +------+ |
212 | |
213 | +-----------+ |
214 local >---------------+ Data [RC] +------------------> remote
215 closed | +-----------+ | closed
216 | |
217 | +------+ |
218 <-----------------+ Data +---------------------<
219 | +------+ |
220 | |
221 | +-----------+ |
222 finished <---------------+ Data [RC] +------------------< finished
223 | +-----------+ |
224 | |
225
226## RPC
227
228While this protocol is defined primarily to support Remote Procedure Calls, the
229protocol does not define the request and response types beyond the messages
230defined in the protocol. The implementation provides a default protobuf
231definition of request and response which may be used for cross language rpc.
232All implementations should at least define a request type which support
233routing by procedure name and a response type which supports call status.
234
235## Version History
236
237| Version | Features |
238|---------|---------------------|
239| 1.0 | Unary requests only |
240| 1.2 | Streaming support |
View as plain text