Application data can be sent using streams and also datagrams if the datagram option is negotiated. Picoquic provides two APIs for sending data on stream, and also two APIs for sending datagram frames: queuing APIs and “just in time” APIs. Receiving is always “just in time”, using the “callback” API.
QUIC streams come in four classes:
Bidirectional streams enable data in both directions, while unidirectional streams enable sending data from the endpoint that create them to its peer.
Streams are identified by a 64 bit stream ID. The type of stream is encoded in the 2 least significant bits of the stream ID. Streams numbers should be opened in sequence in the specified class.
Before using any of the stream APIs, the application must either learn the
stream ID of a remote stream when receiving a callback related to that stream,
or select a local bidirectional or unidirectional stream ID, using the
picoquic_get_next_local_stream_id API:
/* Obtain the next available stream ID in the local category */
uint64_t picoquic_get_next_local_stream_id(picoquic_cnx_t* cnx, int is_unidir);
That API should be used when creating a local stream.
Streams in picoquic are created implicitly, either when the application starts using them or when the first data for the stream are received from the peer.
Picoquic provides a “queuing” API to add data to a stream, and an alternative “just in time” API. When sending “just in time”, picoquic does not build an internal data queue. Instead, the application is called when the stack is ready to send a stream frame for the specified stream, gets access to the packet that is being formatted, and writes the application data directly into that packet just before the packet is sent on the network.
Using the just in time API is a bit more complex than just queuing data, but avoiding copying data in queues reduces memory and CPU consumption. For example, when serving an image on a web page, the HTTP stack can compose stream data frames directly from the image file, instead of loading the image in memory and copying it in the stream queue.
For “real time” applications, the just in time API also has the advantage of sending the most up to date data. Suppose for example an application that provides the time over the network. With the “just in time” API, it can read the clock immediately before sending the packet, providing more accurate time to the receiver.
Applications can “add” data to a stream by calling either picoquic_add_to_stream
or picoquic_add_to_stream_with_ctx.
int picoquic_add_to_stream(picoquic_cnx_t* cnx,
uint64_t stream_id, const uint8_t* data, size_t length, int set_fin);
int picoquic_add_to_stream_with_ctx(picoquic_cnx_t * cnx,
uint64_t stream_id, const uint8_t * data, size_t length, int set_fin,
void * app_stream_ctx);
These two functions are identical, except for the app_stream_ctx parameter,
which is actually not used in the sending process but in only in the receive
callbacks.
When using these API, the data to be sent is copied and added to an internal queue per stream. It will be sent on the network as soon as flow control, congestion control and scheduling priorities permit.
With the just in time API, the application:
marks a stream as active using the API “picoquic_mark_active_stream”.
receives the callback picohttp_callback_provide_data when the stack is ready
to send data on the stream.
services the callback by reserving a buffer using the API “picoquic_provide_stream_data_buffer”, and copying the application data in that buffer.
The APIs are documented in “picoquic.h”:
/* Mark stream as active, or not.
* If a stream is active, it will be polled for data when the transport
* is ready to send. The polling will only start after all currently
* queued data has been sent.
*/
int picoquic_mark_active_stream(picoquic_cnx_t* cnx,
uint64_t stream_id, int is_active, void* v_stream_ctx);
/* If a stream is marked active, the application will receive a callback with
* event type "picoquic_callback_prepare_to_send" when the transport is ready to
* send data on a stream. The "length" argument in the call back indicates the
* largest amount of data that can be sent, and the "bytes" argument points
* to an opaque context structure. In order to prepare data, the application
* needs to call "picoquic_provide_stream_data_buffer" with that context
* pointer, with the number of bytes that it wants to write, with an indication
* of whether or not the fin of the stream was reached, and also an indication
* of whether or not the stream is still active. The function
* returns the pointer to a memory address where to write the byte -- or
* a NULL pointer in case of error. The application then copies the specified
* number of bytes at the provided address, and provide a return code 0 from
* the callback in case of success, or non zero in case of error.
*/
uint8_t* picoquic_provide_stream_data_buffer(void* context, size_t nb_bytes, int is_fin, int is_still_active);
Nobody is perfect. An application may “think” that it is ready to send data
and set the “active” flag using picoquic_mark_active_stream, only to
receive a callback and find out that it does not have anything more to
send.
The application can in that case state that it has nothing to send by
calling picoquic_provide_stream_data_buffer and setting the number
of bytes to zero, e.g.:
/* Not sending here! */
(void)picoquic_provide_stream_data_buffer(context, 0, 0, 0);
In this example, the fin flag is set to 0, and the still active flag set to 0.
The application might call the API “picoquic_mark_active_stream” later,
and mark the stream as active again.
The application could combine nb_bytes_ = 0 (no data to send now) and fin = 1,
if finished sending that stream. The stack will then send a stream data frame with
no content but the FIN bit set, marking the end of the stream.
The application could also set is_still_active=1, or call picoquic_mark_active_stream
from within the call back, which is equivalent. But that’s a bit of a gamble.
The stack will try to fill the current packet with some other content,
but will immediately repeat the callback when ready to send another packet.
That kind of “hyper active polling” may not be the best for performance, except in the
case when the application message does not fit in the length of the buffer, maybe
because some other data was already written in the outgoing packet. The next callback
should provide a larger buffer.
An application may use the “queuing” and “just in time” API on the same stream.
For example, the HTTP implementation uses the queuing API to write HTTP frame
headers to the stream, and then uses the “just in time” API to write the actual
page data. The rule is simple: picoquic will always write all the queued data
before issuing the callback picohttp_callback_provide_data.
In multipath environments, it is sometimes desirable to “pin” a QUIC stream to a specific path. Applications that want to pin path to streams must first monitor the state of paths, which is provided by callbacks such as:
/* A new path is available, or a suspended path is available again */
picoquic_callback_path_available,
/* An available path is suspended */
picoquic_callback_path_suspended,
/* An existing path has been deleted */
picoquic_callback_path_deleted,
/* Some path quality parameters have changed */
picoquic_callback_path_quality_changed
In these API, each path is identified by a 64 bit unique_path_id. This identifier
can be used to set the “affinity” between a stream and a path:
int picoquic_set_stream_path_affinity(picoquic_cnx_t* cnx,
uint64_t stream_id, uint64_t unique_path_id);
When the affinity is set, picoquic will wait until the path is ready
to send queued data for that stream, or to issue the callback
picoquic_callback_prepare_to_send for that stream.
When stream data are received, the contents are queued temporarily until
it can be delivered in order. At that point, the application will
receive a callback picoquic_callback_stream_data or, if this is
the last data for a stream, picoquic_callback_stream_data.
The callback will indicate the stream ID. The stream_ctx argument
will be set to the value associated with the stream in the last call
to one of:
int picoquic_set_app_stream_ctx(picoquic_cnx_t* cnx,
uint64_t stream_id, void* app_stream_ctx);
int picoquic_mark_active_stream(picoquic_cnx_t* cnx,
uint64_t stream_id, int is_active, void* v_stream_ctx);
int picoquic_add_to_stream_with_ctx(picoquic_cnx_t * cnx, uint64_t stream_id,
const uint8_t * data, size_t length, int set_fin, void * app_stream_ctx);
If an application discards a stream context, it should be careful to
remove the memory association to this context in the stack. It can do
that by setting the stream_ctx argument to NULL in one of the previous
API, or by calling:
/* Remove association between stream and context */
void picoquic_unlink_app_stream_ctx(picoquic_cnx_t* cnx, uint64_t stream_id);
When an application has finished sending data on a stream, it should set the
FIN bit in its last call to picoquic_add_to_stream_with_ctx or to
picoquic_provide_stream_data_buffer.
When the peer has set the FIN bit and the last stream frame has been received
in order, the application receives the callback picoquic_callback_stream_fin,
which may also deliver the last data bytes for the stream.
Application can also abruptly close the stream using the RESET mechanism, by calling the API:
int picoquic_reset_stream(picoquic_cnx_t* cnx,
uint64_t stream_id, uint64_t local_stream_error);
If a stream is reset by the peer, the application will receive the callback
picoquic_callback_stream_reset.
The RESET mechanism only works for the sending direction. If a stream is
bidirectional, the application may well continue to receive data from
the peer until the peer’s side of the stream is closed or reset. An
application that receives the callback picoquic_callback_stream_reset
may continue to send data on the stream until it decides to close
or reset it.
If an application resets a stream, Picoquic will immediately stop sending on that stream. This includes not just new data, but also previous data that would need to be repeated because of packet loss.
Picoquic considers that a bidirectional stream is closed if it is closed or reset in both directions – the application has sent the FIN of the stream or reset the stream, and received the FIN or reset callback. All resource associated with the stream are freed after that. For unidirectional streams, picoquic considers a local stream closed if the application has sent the FIN of the stream, and a remote stream close if the application has received the FIN callback.
An application may send QUIC datagrams if the datagram extension has been negotiated during the QUIC handshake. Like for streams, datagrams can be either queued or sent “just in time”. Just like for streams, sending datagrams “just in time” saves CPU and memory, and allows real time applications to fill datagram with up to date data.
In addition, the queuing API limits the size of the datagrams to 1200
bytes (PICOQUIC_DATAGRAM_QUEUE_MAX_LENGTH), so they can
fit in the minimum packet length supported by QUIC. With the
just in time API, datagram sizes are only limited by the packet size
and also by the maximum size negotiated during the handshake.
Datagrams can be queued using the API:
int picoquic_queue_datagram_frame(picoquic_cnx_t* cnx, size_t length, const uint8_t* bytes);
Trying to queue datagram larger than PICOQUIC_DATAGRAM_QUEUE_MAX_LENGTH will result
in an error PICOQUIC_ERROR_DATAGRAM_TOO_LONG.
With the just in time API, the application:
indicates its readiness to send datagrams marks a stream as active using
the API picoquic_set_datagram_ready.
receives the callback picoquic_callback_prepare_datagram when the stack is ready
to send datagrams in a packet.
services the callback by reserving a buffer using the
API picoquic_provide_datagram_buffer or picoquic_provide_datagram_buffer_ex,
and copying the application data in that buffer.
The APIs are defined in picoquic.h:
int picoquic_mark_datagram_ready(picoquic_cnx_t* cnx, int is_ready);
uint8_t* picoquic_provide_datagram_buffer(void* context, size_t length);
uint8_t* picoquic_provide_datagram_buffer_ex(void* context, size_t length,
picoquic_datagram_active_enum is_active);
The old API picoquic_provide_datagram_buffer does not have the
is_active argument, which makes the “reneging” and “multipath” scenarios
harder to handle. It was kept so as to not require changes in old
applications, but new applications should preferably use the “extended’
API.
There may be circumstances when an application indicates that it is ready to send datagrams, but cannot send data when it receives the callback.
If the application marked the context ready by mistake, it should use
the extended API picoquic_provide_datagram_buffer_ex to signal
that it has no data to send. This is a way of saying “oops”. The
stack will stop polling for datagrams, until there is a new call to
picoquic_mark_datagram_ready.
If the application does have data to send but the available length indicated in the callback is too small, it should set the “length” argument to 0, and the “is_active” argument to picoquic_datagram_active_any_path. The stack will try to immediately reissue the callback in the next packet, hopefully with more space available.
In a multipath environment, it is sometimes useful to specify on which path datagrams should be sent. For example, if sending realtime Voice over IP, the application may want to send all the voice datagrams on the same path, to minimize delay jitter. It may also want to choose a path with a low latency and low packet loss.
In a multipath connection, the API picoquic_mark_datagram_ready signals
application readiness to send datagrams on any available path. The application can
use the API picoquic_mark_datagram_ready_path to request sending on a specific path:
int picoquic_mark_datagram_ready_path(picoquic_cnx_t* cnx,
uint64_t unique_path_id, int is_path_ready);
If the generic API is used, the application will receive a callback
picoquic_callback_prepare_datagram each time a datagram can be sent
on any outgoing packet. If it uses the path specific API, it will only
receive the callback when a datagram can be sent on one of the
paths marked “ready”.
The application using multipath should use the “extended” variant of the datagram buffer API:
uint8_t* picoquic_provide_datagram_buffer_ex(void* context, size_t length,
picoquic_datagram_active_enum is_active);
In that variant, the is_active enum provides options to describe whether
the application wants to continue sending datagrams on this path, or on
all paths, as stated in picoquic.h:
/*
* In multipath environments, the application can use the API
* `picoquic_mark_datagram_ready_path` to signal that is is ready to send
* datagrams on a specific path. The picoquic_provide_datagram_path_ex
* API allows the application to mark 4 different level of activity:
*
* - picoquic_datagram_not_active: not active on this path or any other.
* - picoquic_datagram_active_any_path: active, but not specifically on this path.
* - picoquic_datagram_active_this_path_only: ready to send datagrams on this
* path, but not on other paths unless they were specifically marked.
* - picoquic_datagram_active_this_path_and_others: has traffic ready to
* send on this path, and some different traffic ready for any other path.
*/
typedef enum {
picoquic_datagram_not_active = 0,
picoquic_datagram_active_any_path = 1,
picoquic_datagram_active_this_path_only = 2,
picoquic_datagram_active_this_path_and_others = 3
} picoquic_datagram_active_enum;
The QUIC specifications make no guarantee that datagram frames will be received in order, or at all. Picoquic allows applications some degree of control with three callbacks:
/* Ack for packet carrying datagram-frame received from peer */
picoquic_callback_datagram_acked,
/* Packet carrying datagram-frame probably lost */
picoquic_callback_datagram_lost,
/* Packet carrying datagram-frame was not really lost */
picoquic_callback_datagram_spurious,
The bytes and length arguments point to the datagram frame that
was sent. The application is freed to undertake any corrective action,
but it should be aware that this is a “best effort” API, based on
the acknowledgements of the carrying packets. Some implementations
of QUIC may well acknowledge these packets before the data is
actually processed by the application. In particular, some Web Transport
stacks are known to place the datagrams in a queue for delivery to
the application, and to delete them if the queue overflows.
The application will receive the callback picoquic_callback_datagram when a
datagram is received.