Design doc for xseg signaling

This document describes the current state of signaling mechanism on xseg and proposes a new designe to decouple the signal types from the peer types.

Current State

Peers use the xseg requests to pass data between them and the xseg signaling mechanisms to signal one another. The peers are separated in the domain they operate. Currently there are two domains, kernel and user domain. Each domain has certain peer types designed for that domain, each with its own peer ops. The signaling mechanism that will be used to signal a peer, is tight to the peer operations and in general with the peer type.

In more detail, each port where the peer binds to, gets a signal descriptor that will be used to signal the peer. This signal descriptor is provided by the peer type definition and points to the peer instance data. These data will be given to the peer-type provided functions, to signal the peer.

This approach has worked so far, but has a major limitation. It tightly couples the peer type with the signaling mechanism. This does not allow having multiple signaling mechanisms for the same peer type.

Proposed changes

This design attempts to introduce separate signal types and to decouple them from the peer types. This comprises of the following:

1. Introduce signal types to the xseg mechanism.
2. Modify xseg to make use of the new signal types, instead of relying to the peer-type signaling.

Signal types

Signal types are an xseg way to describe a signaling mechanism. They are used to wrap the fundamental signaling mechanisms provided by the OS and the expose a consistent API to signal peers, independently from the peer type or domain.

A peer can use a signal descriptor to inform other interested parties, how to signal them. The signal descriptor already holds instance specific data that are used to signal the peer. We extend the signal descriptor to hold the signal type that should be used. That way a signal descriptor describes to all interested parties, how a peer should prepare itself to accept signals or which action and on which data should be taken to notify a peer.

Each signal type is specified for use with a peer in certain domain, with hooks to support receiving signals from the other domains. The signal types are domain-wise interoperable. That means, that a peer of a domain, can use all the available signal types specified for that domain. A peer can use only compatible signal descriptors, i.e, signal descriptors from the same domain as the peer.

Each peer must allocate and use at least one signal descriptor, to be able to receive signals. It must provide each port it binds to, with a signal descriptor that will be used to notify the peer, when the port is signaled.

Signal type propagation

Each signal type has a unique name. To avoid the expensive string comparison each time we want to get the signal ops from the signal descriptor, each signal type is given a unique integer id. To support dynamic signal type loading, it’s impractical to statically assign integer ids. So they should be dynamically decided and propagated consistently among the peers. The same thing must happen for the memory location which pools signal-type specific signal descriptor data.

This can be done using the same mechanism that is currently used for peer type propagation.

Decoupling the signal types from the peer types have the following consequences:

1. Peer types do not need to be synced among the different peers using the shared data, and should be replaced by the signal types.
2. The peer data is not needed any more, and can be replaced with xobj_handlers that return signal descriptor structs of the specified type.

Reference counting

Each signal descriptor can be access by possible all peers on the segment. This means that the peer that allocates the signal descriptor, cannot safely free it, unless it knows for sure that nobody else is using it. To address this, each signal descriptor will be reference counted and will be freed only when the references drop to zero.

More over, each peer cannot safely quit a signal type. If there are active signal descriptors which have not yet been released, it is not safe to clean up local signal initializations. So each peer type counts in a peer private manner, the active signal descriptors. This can be achieved by counting the initializations and cleanups of the signal descriptors.

Implementation details

Structs

• Signal descriptors

  A generic signal descriptor struct is introduced to hold the signal type and the instance specific data of the peer. Each signal type has its own data, so the signal descriptor points to memory location in the segment where these data reside.

  /* Generic signal descriptor struct.
   * Contains the type of the signal descriptor and a pointer to the actual struct
   */
  struct signal_desc {
      uint64_t type, /* type of signal_desc */
      xptr sd /* pointer to the signal-type specific descriptor struct */
  }
  

• Signal types

  A signal type must define its name and provide function on how to perform certain operations.

  /*
   * Signal descriptor operations
   */
  struct xseg_signal_type {
      {
          int (*init_signal_desc)(struct xseg *xseg, void *sd),
          void (*quit_signal_desc)(struct xseg *xseg, void *sd),
          int (*local_signal_init)(struct xseg *xseg),
          void (*local_signal_quit(struct xseg *xseg),
          int (*remote_signal_init)(void),
          void (*remote_signal_quit)(void),
          int (*prepare_wait)(struct xseg *xseg, void *sd),
          int (*cancel_wait)(struct xseg *xseg, void *sd),
          int (*wait_signal)(struct xseg *xseg, void *sd, uint32_t usec_timeout),
          int (*signal)(struct xseg *xseg, void *sd),
      },
      "signal_type_name"
  }
  

  These operations are:

  • Initialization/cleanup of an individual signal descriptor.

    int (*init_signal_desc)(struct xseg *xseg, void *sd)
    void (*quit_signal_desc)(struct xseg *xseg, void *sd)
    

  • Initialization/cleanup needed to be able to accept signals of this type.

    int (*local_signal_init)(struct xseg *xseg)
    void (*local_signal_quit(struct xseg *xseg)
    

  • Initialization/cleanup needed to be able to send signals of this type.

    void (*local_signal_quit(struct xseg *xseg),
    int (*remote_signal_init)(void),
    

• Ports

  Xseg ports must me extended to hold a pointer to the signal descriptor of the peer that uses the port.

  struct xseg_port {
      ...
      xptr signal_desc /* pointer to struct signal_desc */
      ...
  }
  

API Calls

Generic library wide structs:

static struct xseg_signal_type *__signal_types[XSEG_NR_SIGNAL_TYPES];
static unsigned int __nr_signals;

New calls:

• xseg_get_signal_desc(struct xseg *xseg, char *type)
  

  • Description: Allocate and initialize a new signal descriptor of this type.

  • Calls:

    __find_signal_type()
    __alloc_signal_desc(xseg, signal_type_id)
    xobj_get()
    signal_type->init_signal_desc()
    

• xseg_put_signal_desc(struct xseg *xseg, struct signal_desc *sd)
  

  • Description: Put a signal descriptor. Cleanup and free a previously allocated signal descriptor, if the caller is the last user

  • Calls:

    if xobj_put() == 0
            signal_type->quit_signal_desc()
            __free_signal_desc(xseg, sd)
    

  • Notes: Putting a signal descriptor means that it cannot be used again. It does not mean that the peer will not receive any more signals of this signal type. Also, since the signal descriptors are reference counted (see other design doc), the cleanup and the deallocation are not guaranteed when this calls completes.

Modified calls:

• xseg_prepare_wait(struct xseg *xseg, void *sd);
  

  • Description: Prepare a signal descriptor for waiting on it.

  • Calls:

    signal_desc_type->prepare_wait()
    

• xseg_cancel_wait(struct xseg *xseg, void *sd)
  

  • Description: Cancels waiting on a signal descrptor.

  • Calls:

    signal_desc_type->cancel_wait()
    

• xseg_wait_signal(struct xseg *xseg, void *sd, uint32_t usec_timeout)
  

  • Description: Waits on a signal descriptor for a maximum usec_timeout usecs.

  • Calls:

    signal_desc_type->wait_signal()
    

• xseg_local_signal_init(struct xseg *xseg, char *signal_type);
  

  • Description: Initializations needed to be able to accept signals from this signal descriptor type.

  • Calls:

    signal_desc_type->local_signal_init(struct xseg *xseg);
    

• xseg_local_signal_quit(struct xseg *xseg, char *signal_type);
  

  • Description: Cleanup after initializations.

  • Calls:

    signal_desc_type->local_signal_quit(struct xseg *xseg);
    

• xseg_bind_port(struct xseg *xseg, xport pornto, void *sd);
  

  • Description: Bind to the specified portno, and attach the given signal_descriptor to it.
  • Calls:

Typical usage scenario

1. Initialize signal receiving for a signal type.
2. Get a new signal descriptor for the signal type (sd).
3. Prepare waiting on the signal descriptor.
4. Accept new request.
5. Cancel waiting on the signal descriptor.
6. Process request.
7. Prepare waiting on the signal descriptor.
8. Wait on the signal descriptor.

...

9. Put the signal descriptor.
10. Quit the signal type.